Collect PostgreSQL deadlock reports from the server log (#2661)

.github/workflows/build.yml

@@ -725,8 +725,8 @@ jobs:
       #
       # The values are the product's own derivation from the live hypertable count
       # (TimescaleSupport.HypertableCount = the 62-collector catalog + collection_log = 63):
-      #   timescaledb.max_background_workers = HypertableCount + 2       = 70
-      #   max_worker_processes = 3 + (HypertableCount + 2) + 8           = 81
+      #   timescaledb.max_background_workers = HypertableCount + 2       = 71
+      #   max_worker_processes = 3 + (HypertableCount + 2) + 8           = 82
       # Hard-coded here because a workflow cannot call into the product — so
       # CiClusterWorkerSizingTests parses THIS FILE and fails the build if either number stops
       # matching the formula as collectors are added, and CiClusterWorkerSizingLiveTests asserts
@@ -744,8 +744,8 @@ jobs:
           Add-Content -Path "$dataDir\postgresql.conf" -Value "shared_preload_libraries = 'timescaledb'"
           Add-Content -Path "$dataDir\postgresql.conf" -Value "port = 5541"
           Add-Content -Path "$dataDir\postgresql.conf" -Value "listen_addresses = '127.0.0.1'"
-          Add-Content -Path "$dataDir\postgresql.conf" -Value "timescaledb.max_background_workers = 70"
-          Add-Content -Path "$dataDir\postgresql.conf" -Value "max_worker_processes = 81"
+          Add-Content -Path "$dataDir\postgresql.conf" -Value "timescaledb.max_background_workers = 71"
+          Add-Content -Path "$dataDir\postgresql.conf" -Value "max_worker_processes = 82"
           & "$bin\pg_ctl.exe" -D $dataDir -l $logFile -w start
           if ($LASTEXITCODE -ne 0) { if (Test-Path $logFile) { Get-Content $logFile -Tail 50 }; throw "pg_ctl start failed ($LASTEXITCODE)" }
           & "$bin\createdb.exe" -h 127.0.0.1 -p 5541 -U darling darling

.github/workflows/nightly.yml

@@ -369,8 +369,8 @@ jobs:
       # customer runs and made scheduler-racing failures luck-of-the-slot instead of reproducible.
       # Values are the product's own derivation from the live hypertable count
       # (TimescaleSupport.HypertableCount = the 62-collector catalog + collection_log = 63):
-      #   timescaledb.max_background_workers = HypertableCount + 2  = 70
-      #   max_worker_processes = 3 + (HypertableCount + 2) + 8      = 81
+      #   timescaledb.max_background_workers = HypertableCount + 2  = 71
+      #   max_worker_processes = 3 + (HypertableCount + 2) + 8      = 82
       # Kept honest by CiClusterWorkerSizingTests (parses this file against the formula) and
       # CiClusterWorkerSizingLiveTests (asserts the running cluster serves them). Must configure
       # the cluster identically to build.yml's darling-pg job: that guard parses the appended
@@ -391,8 +391,8 @@ jobs:
           Add-Content -Path "$dataDir\postgresql.conf" -Value "shared_preload_libraries = 'timescaledb'"
           Add-Content -Path "$dataDir\postgresql.conf" -Value "port = 5541"
           Add-Content -Path "$dataDir\postgresql.conf" -Value "listen_addresses = '127.0.0.1'"
-          Add-Content -Path "$dataDir\postgresql.conf" -Value "timescaledb.max_background_workers = 70"
-          Add-Content -Path "$dataDir\postgresql.conf" -Value "max_worker_processes = 81"
+          Add-Content -Path "$dataDir\postgresql.conf" -Value "timescaledb.max_background_workers = 71"
+          Add-Content -Path "$dataDir\postgresql.conf" -Value "max_worker_processes = 82"
           & "$bin\pg_ctl.exe" -D $dataDir -l $logFile -w start
           if ($LASTEXITCODE -ne 0) { if (Test-Path $logFile) { Get-Content $logFile -Tail 50 }; throw "pg_ctl start failed ($LASTEXITCODE)" }
           & "$bin\createdb.exe" -h 127.0.0.1 -p 5541 -U darling darling

CHANGELOG.md

@@ -8,6 +8,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ## [Unreleased]
 
 ### Added
+- **PostgreSQL deadlocks, not just the count** ([#2661]) - we collected `pg_stat_database.deadlocks`, a number that goes up, and nothing else. PostgreSQL writes a complete report to its server log: the wait graph, every participant, and each one's full statement text. `get_pg_deadlocks` lists them with the victim, the lock modes and the resources; `get_pg_deadlock_detail` returns the graph as the server wrote it. In one respect this beats the SQL Server graph, which names the victim's statement and often leaves the other side as a handle. It needs **nothing configured on the target** - unlike plan capture there is no setting that suppresses a deadlock report, so it works on a managed fleet today; the only precondition is being able to read the log, which plan capture already established. The collector re-reads an overlapping tail on purpose so a report cut in half at one edge is whole in the next, and every row carries a hash of its graph so the same report is reported once, with a sighting count, rather than once per cycle.
 - **PostgreSQL configuration, and what changed in it** ([#2658]) - `pg_settings` was never collected, so neither "what is `work_mem` set to on this server" nor "what changed last Tuesday" had an answer, and the second is the kind that cannot be recovered later at any price: a configuration history nobody recorded is not sitting on the server waiting to be read. `get_pg_server_config` reports the settings somebody actually chose, non-default first, with where each value came from and whether changing it needs a restart or a reload. `get_pg_server_config_changes` reports value changes between snapshots, old beside new. Both name `pending_restart` loudly - the state where `postgresql.conf` has been edited and reloaded but the running server is still on the old value, so the file and the server disagree with no symptom until a restart months later changes behaviour during someone else's incident.
 - **Test an index from the predicate grid** ([#2612]) - right-click a row in PostgreSQL predicate statistics and ask whether the planner would actually use an index on that column. The command shipped with no caller: the only way to reach it was hand-writing a row into the command queue, which is how it was tested and is not a feature. It hangs off that grid and nowhere else, which is the shape it was scoped to - on demand only, never scheduled, driven from a row somebody is already looking at. The confirmation says what it costs the server before it runs (nothing executed, no index built, session reset), and a predicate whose estimate error is already large is flagged BEFORE the round trip, because an index does not fix a plan built on a wrong row count.
 - **Azure SQL DB now reports every database's size, not just the connected one** ([#2643], raised from the field) - `sys.database_files` is database-scoped, so a Viewer pointed at `master` showed `master`'s two files and nothing else, which is correct and reads exactly like a broken collector. `sys.resource_stats` is a master-only view carrying `storage_in_megabytes` per database, so from a `master` connection the siblings now appear too - as one row each, labelled `(whole database)` with a NULL `file_id`, because that view has no per-file breakdown and a fabricated file name would make the grid look complete and be wrong. The sibling read runs through `sp_executesql`: the view does not exist in a user database and name resolution happens at parse time, so a guarded UNION still fails with 208 everywhere else. Verified against a live Azure SQL Database from both a `master` and a user-database connection.
@@ -3015,3 +3016,4 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 [#2655]: https://github.com/erikdarlingdata/PerformanceMonitor/issues/2655
 [#2658]: https://github.com/erikdarlingdata/PerformanceMonitor/issues/2658
 [#2659]: https://github.com/erikdarlingdata/PerformanceMonitor/issues/2659
+[#2661]: https://github.com/erikdarlingdata/PerformanceMonitor/issues/2661

Darling/Darling.Tests/PgSchemaGeneratorTests.cs

@@ -51,7 +51,7 @@ Availability Group collectors (#991) = 38, plus plan_correction (#1952 automatic
            create tables and one store can hold both engines' data, so splitting it per engine would
            fragment DDL generation. Dispatch is gated separately, by engine, in
            CollectorCatalog.AppliesTo(definition, target). */
-        Assert.Equal(67, CollectorCatalog.All.Count);
+        Assert.Equal(68, CollectorCatalog.All.Count);
 
         /* Uniqueness is asserted AGAINST THE COUNT rather than against a second literal. The literals here
            had drifted to 45 while the real figure tracked the count, so the test that exists to catch a
@@ -637,6 +637,7 @@ public void EveryPostgresRung_IsIdenticalToTheGeneratedSchema()
             (98, PgPredicateStatsCollector.Instance),
             (99, PgPlanCaptureCollector.Instance),
             (102, PgServerConfigCollector.Instance),
+            (103, PgDeadlocksCollector.Instance),
         };
 
         /* Every PostgreSQL collector must appear above. One added without a rung listed here would

Darling/Darling.Tests/ServerPageTabsTests.cs

@@ -83,6 +83,8 @@ public sealed class ServerPageTabsTests
         ["get_pg_lock_stats"] = "pg_lock_stats",
         ["get_pg_write_stats"] = "pg_write_stats",
         ["get_pg_server_config"] = "pg_server_config",
+        ["get_pg_deadlocks"] = "pg_deadlocks",
+        ["get_pg_deadlock_detail"] = "pg_deadlocks",
         ["get_pg_server_config_changes"] = "pg_server_config",
         ["get_pg_replication_stats"] = "pg_replication_stats",
             ["get_pg_top_queries"] = "pg_statement_stats",

Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs

@@ -1148,6 +1148,8 @@ private static CatalogRead R(string category, string description, params Catalog
             ["get_pg_write_stats"] = R(CatData, "Checkpoint and WAL write activity across the window: timed versus requested checkpoints, buffers written by whom, and WAL volume.", PServer(), PHours(24), PAsOf()),
             ["get_pg_server_config"] = R(CatData, "The PostgreSQL server's configuration from pg_settings, non-default first, saying where each value came from and whether changing it needs a restart. Reports pending_restart, where the file and the running server disagree.", PServer(), PLimit(100), PBool("include_defaults", false)),
             ["get_pg_server_config_changes"] = R(CatData, "PostgreSQL configuration parameters whose value CHANGED in the window, old beside new. Nothing else can reconstruct this after the fact.", PServer(), PHours(168), PLimit(100), PAsOf()),
+            ["get_pg_deadlocks"] = R(CatData, "PostgreSQL deadlocks reported in the window, with the victim, the lock modes and resources, and the victim's statement. Needs nothing configured on the target.", PServer(), PHours(24), PLimit(25), PAsOf()),
+            ["get_pg_deadlock_detail"] = R(CatData, "PostgreSQL deadlock graphs in full: the whole wait graph and every participant's statement, as the server wrote it. Newest first, or one by deadlock_hash.", PServer(), PText("deadlock_hash"), PLimit(5)),
             ["get_pg_replication_stats"] = R(CatData, "Health of CONNECTED replicas from pg_stat_replication, with the worst lag in the window beside the latest. Counterpart of get_pg_replication_slots.", PServer(), PHours(24), PLimit(25), PAsOf()),
             ["get_pg_blocking"] = R(CatData, "PostgreSQL blocking chains that were sampled, with the root blocker attributed. A sample, not an event log.", PServer(), PHours(24), PLimit(50), PAsOf()),
             ["get_pg_database_stats"] = R(CatData, "PostgreSQL per-database temp-file spills, cache hit ratio, deadlocks and commit/rollback split, differenced across the window.", PServer(), PHours(24), PLimit(20), PAsOf()),
@@ -1626,6 +1628,8 @@ cannot parse exactly rather than silently matching nothing. */
             ["get_pg_write_stats"] = (c, pg, an) => DarlingMcpPgServerStateTools.GetPgWriteStats(pg, Server(c), Hours(c, 24), as_of: AsOf(c)),
             ["get_pg_server_config"] = (c, pg, an) => DarlingMcpPgServerStateTools.GetPgServerConfig(pg, Server(c), Rows(c, "limit", 100), QueryBool(c, "include_defaults", false)),
             ["get_pg_server_config_changes"] = (c, pg, an) => DarlingMcpPgServerStateTools.GetPgServerConfigChanges(pg, Server(c), Hours(c, 168), Rows(c, "limit", 100), as_of: AsOf(c)),
+            ["get_pg_deadlocks"] = (c, pg, an) => DarlingMcpPgDeadlockTools.GetPgDeadlocks(pg, Server(c), Hours(c, 24), Rows(c, "limit", 25), as_of: AsOf(c)),
+            ["get_pg_deadlock_detail"] = (c, pg, an) => DarlingMcpPgDeadlockTools.GetPgDeadlockDetail(pg, Server(c), Str(c, "deadlock_hash"), Rows(c, "limit", 5)),
             ["get_pg_replication_stats"] = (c, pg, an) => DarlingMcpPgReplicationStatsTools.GetPgReplicationStats(pg, Server(c), Hours(c, 24), Rows(c, "limit", 25), as_of: AsOf(c)),
             ["get_pg_blocking"] = (c, pg, an) => DarlingMcpPgBlockingTools.GetPgBlocking(pg, Server(c), Hours(c, 24), Rows(c, "limit", 50), as_of: AsOf(c)),
             ["get_pg_database_stats"] = (c, pg, an) => DarlingMcpPgDatabaseTools.GetPgDatabaseStats(pg, Server(c), Hours(c, 24), Rows(c, "limit", 20), as_of: AsOf(c)),

Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs

@@ -4820,6 +4820,7 @@ await DarlingObservability.LogCollectionAsync(
         ["pg_statement_stats"] = (r, s, ct) => r.RunAsync(PgStatementStatsCollector.Instance, s, ct),
         ["pg_wraparound_stats"] = (r, s, ct) => r.RunAsync(PgWraparoundStatsCollector.Instance, s, ct),
         ["pg_server_config"] = (r, s, ct) => r.RunAsync(PgServerConfigCollector.Instance, s, ct),
+        ["pg_deadlocks"] = (r, s, ct) => r.RunAsync(PgDeadlocksCollector.Instance, s, ct),
         ["pg_xmin_horizon"] = (r, s, ct) => r.RunAsync(PgXminHorizonCollector.Instance, s, ct),
         ["pg_replication_slots"] = (r, s, ct) => r.RunAsync(PgReplicationSlotsCollector.Instance, s, ct),
         ["pg_autovacuum_stats"] = (r, s, ct) => r.RunAsync(PgAutovacuumStatsCollector.Instance, s, ct),

Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpHostService.cs

@@ -653,6 +653,9 @@ McpToolTypeRegistrationTests now derives the check by reflection instead of trus
                 .WithGeminiCompatibleTools<DarlingMcpPgPredicateTools>()
                 .WithGeminiCompatibleTools<DarlingMcpPgReplicationStatsTools>()
                 .WithGeminiCompatibleTools<DarlingMcpPgWaitSamplingTools>()
+                /* get_pg_deadlocks / get_pg_deadlock_detail (#2661) - the reports themselves, out of the
+                   server log, rather than pg_stat_database's count. */
+                .WithGeminiCompatibleTools<DarlingMcpPgDeadlockTools>()
                 .WithGeminiCompatibleTools<DarlingMcpMemoryGrantTools>()
                 .WithGeminiCompatibleTools<DarlingMcpPlanCacheSchedulerTools>()
                 .WithGeminiCompatibleTools<DarlingMcpJobTools>()

Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpInstructions.cs

@@ -62,7 +62,7 @@ public static string Build(DarlingPeerDirectory.Snapshot peers)
 
         ## Tool Reference
 
-        This server exposes 128 tools. 86 are the same names Performance Monitor Lite exposes, spanning diagnostic analysis, plan analysis, data reads at core and diagnostic depth, resource contention + jobs, trends, system-health parse-on-read, alerts + health overview, and the Default Trace. The remaining 42 are unique to Darling: twenty-five are the PostgreSQL reads (Aurora/PostgreSQL targets only Darling's central store can hold), eight are the Custom Views tools (seven manage the saved views — the one view-authoring write surface — and `describe_custom_view_catalog` returns the read-only compose vocabulary those authoring tools draw from), three are alert-tuning write tools (`update_alert_settings` tunes the alert engine's thresholds; `create_mute_rule` / `delete_mute_rule` manage the mute rules) that write only the shared alert configuration in the monitoring store, two are server-onboarding write tools (`add_servers` bulk-adds monitored servers; `remove_server` removes one) that add or remove rows in the monitoring store's monitored-server registry, `get_fleet_overview` and `get_ag_health` are the two cross-server reads only a central store can answer, `get_store_metrics` reads the monitoring store's OWN hourly size/compression/growth series for capacity forecasting, and `get_blocking` is Darling's name for the blocked-process-report read that Lite exposes as `get_blocked_process_reports` — a naming difference, not a capability gap. Every data-read tool reads the data the collectors already captured into the store — a stored read, never a live query against the monitored server.
+        This server exposes 130 tools. 86 are the same names Performance Monitor Lite exposes, spanning diagnostic analysis, plan analysis, data reads at core and diagnostic depth, resource contention + jobs, trends, system-health parse-on-read, alerts + health overview, and the Default Trace. The remaining 44 are unique to Darling: twenty-seven are the PostgreSQL reads (Aurora/PostgreSQL targets only Darling's central store can hold), eight are the Custom Views tools (seven manage the saved views — the one view-authoring write surface — and `describe_custom_view_catalog` returns the read-only compose vocabulary those authoring tools draw from), three are alert-tuning write tools (`update_alert_settings` tunes the alert engine's thresholds; `create_mute_rule` / `delete_mute_rule` manage the mute rules) that write only the shared alert configuration in the monitoring store, two are server-onboarding write tools (`add_servers` bulk-adds monitored servers; `remove_server` removes one) that add or remove rows in the monitoring store's monitored-server registry, `get_fleet_overview` and `get_ag_health` are the two cross-server reads only a central store can answer, `get_store_metrics` reads the monitoring store's OWN hourly size/compression/growth series for capacity forecasting, and `get_blocking` is Darling's name for the blocked-process-report read that Lite exposes as `get_blocked_process_reports` — a naming difference, not a capability gap. Every data-read tool reads the data the collectors already captured into the store — a stored read, never a live query against the monitored server.
 
         ### Reading an empty result