fix(seo): trailing-slash URLs, duplicate consolidation and sitemap lastmod (#1153)

* fix(seo): use trailing slashes in canonical and sitemap URLs

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(seo): consolidate duplicate versioned and markdown pages

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* feat(seo): add git-based lastmod to sitemap

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>

Author: mazzma12

crowdsec-docs/docs/appsec/configuration_creation_testing.md

@@ -72,7 +72,7 @@ inband_rules:
 
 AppSec configuration files declare **which rules to load** in **in-band** *(blocking)* and/or **out-of-band** *(non-blocking)* mode, define how matches are handled (for example, default remediation), and let you tweak processing via hooks like `on_load`, `pre_eval`, `post_eval`, and `on_match`.
 
-For the full list of keys, see [Configuration Syntax](configuration).
+For the full list of keys, see [Configuration Syntax](configuration.md).
 
 :::info
 When loading multiple AppSec configs, _hooks_ and _appsec rules_ are appended, and for conflicting options (for example, `default_remediation`), the last one takes precedence.

crowdsec-docs/docs/getting_started/install_drupal_plugin.mdx

@@ -8,7 +8,7 @@ The CrowdSec Drupal plugin provides real-time protection against malicious actor
 
 ## Overview
 
-The Drupal plugin leverages the [PHP Soft Agent](install_php_softagent) to connect your site to CrowdSec's collaborative security ecosystem. This integration allows your Drupal site to:
+The Drupal plugin leverages the [PHP Soft Agent](install_php_softagent.md) to connect your site to CrowdSec's collaborative security ecosystem. This integration allows your Drupal site to:
 
 - **Detect threats**: Automatically identify brute force attacks and scanning attempts
 - **Share intelligence**: Contribute attack data to the global CrowdSec network

crowdsec-docs/docs/getting_started/sdk_intro.mdx

@@ -15,7 +15,7 @@ Why Integrate the SDK:
 
 ## Supported SDKs
 
-* [Python SDK](install_pyagent)
-* [PHP SDK](install_php_softagent)
+* [Python SDK](install_pyagent.md)
+* [PHP SDK](install_php_softagent.md)
 
 Whether you're building a WAF, SIEM, or a custom security tool, the CrowdSec SDKs make it easy to contribute to and benefit from a collaborative defense network.

crowdsec-docs/docs/intro.mdx

@@ -50,7 +50,7 @@ In addition to the core "detect and react" mechanism, CrowdSec is committed to s
 Under the hood, the Security Engine has various components:
 
 -   The [Log Processor](/log_processor/intro.mdx) handles detection. It analyzes logs from [various data sources](/log_processor/data_sources/introduction.md) and [HTTP requests](/appsec/intro.md) from compatible web servers.
--   The [Appsec](appsec/intro) feature is part of the Log Processor. It filters HTTP requests from compatible web servers.
+-   The [Appsec](appsec/intro.md) feature is part of the Log Processor. It filters HTTP requests from compatible web servers.
 -   The [Local API](local_api/intro.md) acts as a middleman:
     -   Between the [Log Processors](/log_processor/intro.mdx) and the [Remediation Components](/u/bouncers/intro) which are in charge of enforcing decisions.
     -   And with the [Central API](/central_api/intro.md) to share alerts and receive blocklists.
@@ -65,7 +65,7 @@ This architecture supports simple standalone setups and more distributed deploym
 -   Centralized logs (rsyslog, Loki, ...): [Run CrowdSec next to your log pipeline](/u/user_guides/log_centralization), not on production workloads.
 -   Kubernetes: See [our Helm chart](/u/getting_started/installation/kubernetes).
 -   Containers: Use the [Docker data source](/log_processor/data_sources/docker.md).
--   WAF only: Start with the [AppSec quickstart](appsec/intro).
+-   WAF only: Start with the [AppSec quickstart](appsec/intro.md).
 
 Distributed architecture example:
 

crowdsec-docs/docs/log_processor/scenarios/format.md

@@ -575,7 +575,7 @@ It is useful when you want to have further scenarios that will rely on past-over
 cache_size: 5
 ```
 
-By default, a bucket holds [capacity](format#capacity) events "in memory".
+By default, a bucket holds [capacity](#capacity) events "in memory".
 However, for a number of cases, you don't want this, as it might lead to excessive memory consumption.
 
 By setting `cache_size` to a positive integer, we can control the maximum in-memory cache size of the bucket, without changing its capacity and such. It is useful when buckets are likely to stay alive for a long time or ingest a lot of events to avoid storing a lot of events in memory.

crowdsec-docs/docusaurus.config.ts

@@ -1,3 +1,4 @@
+import { execSync } from "node:child_process";
 import type * as Preset from "@docusaurus/preset-classic";
 import type { NavbarItem } from "@docusaurus/theme-common";
 import type { Config } from "@docusaurus/types";
@@ -227,10 +228,6 @@ const redirects = [
 		to: "/u/console/ip_reputation/api_keys",
 	},
 	{ from: "/u/cti_api/ip_report", to: "/u/console/ip_reputation/ip_report" },
-	{
-		from: "/u/cti_api/search_queries",
-		to: "/u/console/ip_reputation/search_ui",
-	},
 	{
 		from: "/u/cti_api/advanced_search",
 		to: "/u/console/ip_reputation/search_ui_advanced",
@@ -264,6 +261,19 @@ function redirectsGlobalDataPlugin() {
 	};
 }
 
+// A shallow clone dates every file to the last commit, which would give every sitemap URL the same <lastmod>.
+function hasFullGitHistory(): boolean {
+	try {
+		return (
+			execSync("git rev-parse --is-shallow-repository", { stdio: ["ignore", "pipe", "ignore"] })
+				.toString()
+				.trim() === "false"
+		);
+	} catch {
+		return false;
+	}
+}
+
 const config: Config = {
 	future: {
 		v4: {
@@ -275,6 +285,8 @@ const config: Config = {
 	tagline: "CrowdSec - Real-time & crowdsourced protection against aggressive IPs",
 	url: "https://docs.crowdsec.net",
 	baseUrl: "/",
+	// Amplify serves `page/index.html` at `page/` and redirects `page` there, so canonical and sitemap URLs must end with a slash.
+	trailingSlash: true,
 	onBrokenLinks: "warn",
 	onBrokenMarkdownLinks: "warn",
 	favicon: "img/crowdsec_no_txt.png",
@@ -335,7 +347,7 @@ const config: Config = {
 			// or nobody who dismissed the previous banner will ever see the new one.
 			id: "banner_botdetection",
 			content:
-				'<a href="/docs/next/appsec/bot_detection/intro">Stop scrapers and headless browsers — discover CrowdSec Bot Detection</a>',
+				'<a href="/docs/next/appsec/bot_detection/intro/">Stop scrapers and headless browsers — discover CrowdSec Bot Detection</a>',
 			backgroundColor: "#F8AB13",
 			textColor: "#131132",
 			isCloseable: true,
@@ -386,6 +398,10 @@ const config: Config = {
 							banner: "none",
 							path: "/",
 						},
+						// Near-duplicate of the newer versions: indexing it only splits ranking signals.
+						"v1.7": {
+							noIndex: true,
+						},
 						// EOL: kept online for existing links, but out of the index.
 						// plugin-sitemap drops noindex routes too, so it also leaves sitemap.xml.
 						"v1.6": {
@@ -407,6 +423,16 @@ const config: Config = {
 				theme: {
 					customCss: "./src/css/custom.css",
 				},
+				sitemap: {
+					lastmod: hasFullGitHistory() ? "datetime" : null,
+					changefreq: null,
+					priority: null,
+					// Docs of the version served at /docs/ declare their /docs/next/ copy as canonical (src/theme/DocRoot/Layout).
+					createSitemapItems: async ({ defaultCreateSitemapItems, ...params }) =>
+						(await defaultCreateSitemapItems(params)).filter(
+							(item) => !/^\/docs\/(?!next\/|v\d)/.test(new URL(item.url).pathname)
+						),
+				},
 			} satisfies Preset.Options,
 		],
 	],

crowdsec-docs/src/theme/DocRoot/Layout/index.tsx

@@ -5,11 +5,13 @@
  * This override injects a custom SecondaryNavbar above that content, which adds:
  *   - A breadcrumb trail (Home > Section > [current page path])
  *   - A version dropdown (when multiple doc versions exist)
+ * On older versions it also points the canonical URL at the current version's copy of the page.
  *
  * Docusaurus picks this file automatically because it lives at
  * src/theme/DocRoot/Layout/index.tsx, shadowing the original in node_modules.
  */
 
+import Head from "@docusaurus/Head";
 import Link from "@docusaurus/Link";
 import type { PropSidebarBreadcrumbsItem } from "@docusaurus/plugin-content-docs";
 import {
@@ -21,6 +23,8 @@ import {
 	useVersions,
 } from "@docusaurus/plugin-content-docs/client";
 import { useHistorySelector } from "@docusaurus/theme-common";
+import useDocusaurusContext from "@docusaurus/useDocusaurusContext";
+import { applyTrailingSlash } from "@docusaurus/utils-common";
 import { SECTION_MAP } from "@site/src/sectionMap";
 import BackToTopButton from "@theme/BackToTopButton";
 import type { Props } from "@theme/DocRoot/Layout";
@@ -32,6 +36,21 @@ import React, { type ReactNode, useState } from "react";
 
 import styles from "./styles.module.css";
 
+// Older versions duplicate the current docs; without this, each copy competes for the same searches.
+function CanonicalToCurrentDoc({ pluginId }: { pluginId: string }): ReactNode {
+	const { siteConfig } = useDocusaurusContext();
+	const { activeVersion, alternateDocVersions } = useActiveDocContext(pluginId);
+	const currentDoc = alternateDocVersions.current;
+	if (!activeVersion || activeVersion.name === "current" || !currentDoc) return null;
+
+	const path = applyTrailingSlash(currentDoc.path, { trailingSlash: siteConfig.trailingSlash, baseUrl: siteConfig.baseUrl });
+	return (
+		<Head>
+			<link rel="canonical" href={`${siteConfig.url}${path}`} />
+		</Head>
+	);
+}
+
 function VersionDropdown({ pluginId }: { pluginId: string }): ReactNode {
 	const versions = useVersions(pluginId);
 	const activeDocContext = useActiveDocContext(pluginId);
@@ -133,10 +152,12 @@ function SecondaryNavbar(): ReactNode {
 
 export default function DocRootLayout({ children }: Props): ReactNode {
 	const sidebar = useDocsSidebar();
+	const activePlugin = useActivePlugin();
 	const [hiddenSidebarContainer, setHiddenSidebarContainer] = useState(false);
 
 	return (
 		<div className={styles.docsWrapper}>
+			<CanonicalToCurrentDoc pluginId={activePlugin?.pluginId ?? "default"} />
 			<BackToTopButton />
 			<SecondaryNavbar />
 			<div className={styles.docRoot}>

crowdsec-docs/src/theme/NotFound/Content/index.tsx

@@ -15,8 +15,10 @@ export default function NotFoundContent({ className }: Props): ReactNode {
 
 	useEffect(() => {
 		if (!pluginData?.redirects) return;
-		const path = window.location.pathname.replace(siteConfig.baseUrl.replace(/\/$/, ""), "") || "/";
-		const match = pluginData.redirects.find((r) => r.from === path);
+		// Amplify appends a trailing slash before serving the 404 page, while most `from` paths have none.
+		const withoutTrailingSlash = (p: string) => p.replace(/(.)\/$/, "$1");
+		const path = withoutTrailingSlash(window.location.pathname.replace(siteConfig.baseUrl.replace(/\/$/, ""), "") || "/");
+		const match = pluginData.redirects.find((r) => withoutTrailingSlash(r.from) === path);
 		if (match) {
 			window.location.replace(match.to);
 		}

crowdsec-docs/unversioned/cti_api/search_queries.md

@@ -58,10 +58,10 @@ It is possible to search any field available in the JSON document below:
 
 We provide all the possible values for each of the following fields:
 
--   [`behaviors`](taxonomy/behaviors)
--   [`classifications.classifications`](taxonomy/classifications)
--   [`classifications.false_positives`](taxonomy/false_positives)
--   [`attack_details`](taxonomy/scenarios)
+-   [`behaviors`](taxonomy/behaviors.mdx)
+-   [`classifications.classifications`](taxonomy/classifications.mdx)
+-   [`classifications.false_positives`](taxonomy/false_positives.mdx)
+-   [`attack_details`](taxonomy/scenarios.mdx)
 
 To perform a search query, specify the field from the JSON document you wish to search followed by a colon `:` and the value you are looking for between double quotes.
 

crowdsec-docs/unversioned/tracker_api/api_authentication.mdx

@@ -10,7 +10,7 @@ import TabItem from "@theme/TabItem";
 
 To use the Live Exploit Tracker API, you need an **API key**. Contact the CrowdSec team to obtain yours if you haven't already.
 
-The same API key works for both the [web interface](./web_interface) and the REST API.
+The same API key works for both the [web interface](./web_interface.mdx) and the REST API.
 
 ## Base URL
 
@@ -33,7 +33,7 @@ curl -X 'GET' \
 
 ## SDKs
 
-CrowdSec provides official SDKs for convenient API access with typed models, authentication handling, and a better developer experience. See the [SDKs & Libraries](./api_sdks) page for installation instructions, examples, and the full list of available SDKs.
+CrowdSec provides official SDKs for convenient API access with typed models, authentication handling, and a better developer experience. See the [SDKs & Libraries](./api_sdks.mdx) page for installation instructions, examples, and the full list of available SDKs.
 
 **Quick start with the Python SDK:**
 
@@ -66,9 +66,9 @@ The API returns standard HTTP status codes:
 
 ## Next Steps
 
-- [Explore CVEs](./api_cves) — List, search, and get detailed intelligence
-- [Fingerprint Rules](./api_fingerprints) — Monitor product-level probing activity
-- [Integrations & Blocklists](./api_integrations) — Create firewall integrations and subscribe to CVEs
-- [Browse by Vendor, Product, or Tag](./api_lookups) — Explore the coverage landscape
-- [SDKs & Libraries](./api_sdks) — Official SDKs for Python and more
+- [Explore CVEs](./api_cves.mdx) — List, search, and get detailed intelligence
+- [Fingerprint Rules](./api_fingerprints.mdx) — Monitor product-level probing activity
+- [Integrations & Blocklists](./api_integrations.mdx) — Create firewall integrations and subscribe to CVEs
+- [Browse by Vendor, Product, or Tag](./api_lookups.mdx) — Explore the coverage landscape
+- [SDKs & Libraries](./api_sdks.mdx) — Official SDKs for Python and more
 - [API Reference](https://admin.api.crowdsec.net/v1/docs#tag/Cves) — Full interactive API documentation