feat(canopy): N2: blob store backups and paired restore

.github/workflows/ci.yml

@@ -198,6 +198,11 @@ jobs:
       - name: WMI VSS end-to-end
         run: cargo test -p bestool --lib wmi_shadow_roundtrip -- --ignored --nocapture
 
+      # A follower reading its source out of the leader's shadow: the data is
+      # rewritten after the freeze, so serving live bytes fails the test.
+      - name: Shared volume capture end-to-end
+        run: cargo test -p bestool --lib a_shared_capture_reads_the_bytes_from_when_it_froze -- --ignored --nocapture
+
   clippy:
     name: Clippy
     runs-on: ubuntu-latest

.workhorse/specs/canopy/backup.md

@@ -11,8 +11,8 @@ Canopy owns scheduling, retention, maintenance, inspection, and alerting; the de
 ## Backup definitions
 
 A backup is configured by a TOML definition file in the backups directory — `/etc/bestool/backups/*.toml` on Unix, a per-platform data directory on Windows — one definition per file (so configuration management can drop in a single file per backup).
-A definition carries a `type` (the Canopy-facing label), optional `[tags]` (extra kopia tags), optional ordered `[[pre]]` and `[[post]]` command hooks, and exactly one method table — `[simple]` or `[postgresql]` — selecting a built-in method.
-A definition with no method table, or with more than one, is a load error.
+A definition carries a `type` (the Canopy-facing label), an optional `after` (the type it follows, see "Follower backups"), optional `[tags]` (extra kopia tags), optional ordered `[[pre]]`, `[[post]]`, `[[pre_restore]]` and `[[post_restore]]` command hooks, and exactly one method table — `[simple]` or `[postgresql]` — selecting a built-in method.
+A definition with no method table, or with more than one, is a load error, as is a definition naming itself in `after`.
 The `type` is the only identity that matters to Canopy; the filename is informational.
 
 Backups are generic: a definition names a method and a target, and `type` is just a label.
@@ -22,6 +22,7 @@ A `tamanu-postgres` backup is a definition that selects the `postgresql` method;
 
 ```toml
 type = "tamanu-postgres"          # required — the Canopy backup-type label
+after = "other-type"              # optional — run after that type's backups, restore with it
 
 [tags]                            # optional — extra kopia tags (string to string)
 component = "database"
@@ -31,6 +32,12 @@ command = ["/usr/bin/systemctl", "stop", "example"]
 
 [[post]]                          # optional, ordered — run after cleanup
 command = ["/usr/bin/systemctl", "start", "example"]
+
+[[pre_restore]]                   # optional, ordered — run before this def's restore
+command = ["/usr/bin/systemctl", "stop", "example"]
+
+[[post_restore]]                  # optional, ordered — run after this def's restore
+command = ["/usr/bin/systemctl", "start", "example"]
 ```
 
 A hook is a table with a `command` array, run argv-style (no shell).
@@ -43,7 +50,8 @@ There must be exactly one method table.
 
 ```toml
 [simple]                          # snapshot a path as-is
-path = "/var/lib/example"         # required
+path = "/var/lib/example"         # exactly one of path / path_command
+# path_command = ["bestool", "tamanu", "blob-root"]   # resolve the path by command instead
 ```
 
 #### PostgreSQL
@@ -60,7 +68,10 @@ socket = "/var/run/postgresql"    # optional — override the unix socket direct
 
 ## Methods
 
-The `simple` method hands kopia a configured path verbatim; it contributes no extra tags and needs no preparation or cleanup.
+The `simple` method hands kopia a path verbatim; it contributes no extra tags and needs no preparation or cleanup.
+The path is either fixed (`path`) or resolved on every run by an argv-style command (`path_command`) whose output must be a single line naming an absolute path; a failed or malformed resolution fails the run.
+Run-time resolution serves a source whose location lives outside the definition and can move under it (the Tamanu blob store root is a database-backed setting an administrator can change, printed by `bestool tamanu blob-root`), so the capture follows the live location instead of a hardcoded path silently going stale.
+At restore, the same resolution names the destination, so a store restored after its database lands where the freshly restored database expects it.
 
 The `postgresql` method takes a crash-consistent physical copy of a postgres cluster, described under "The postgresql method" below.
 
@@ -102,6 +113,31 @@ A run:
    Any run that started kopia reports (success or failure); a run that exited idle at step 3 reports nothing.
    A failed report is logged and surfaced as a non-zero exit, but is not retried — Canopy's repository inspection is the backstop for a lost report.
 
+## Follower backups
+
+A definition may declare `after = "<type>"`, making it a follower of that type.
+When a run of the followed type completes a backup successfully, the driver then runs each of its followers, sequentially in type order, before returning; following is transitive, and each type in a chain runs at most once however the definitions are arranged.
+A run that failed, was skipped because its type was already running, or exited dormant runs no followers.
+A follower is otherwise an ordinary definition: it registers as a capability, may be scheduled by Canopy or run manually on its own, and reports its runs like any other type.
+
+Following is for a capture that must be a superset of what another capture references.
+The Tamanu blob store definition follows the database definition, so every blob the database capture references is already stored when the store capture begins; blobs are immutable and never removed while referenced, so the store capture can only hold more, never less, than the database capture needs.
+A follower run triggered on its own (by schedule or by hand) is still safe on these terms, being a superset for every earlier capture of the followed type; what only the chain provides is a store capture promptly after each database capture.
+
+### Sharing a capture
+
+Where a leader's capture freezes more than its own source, a follower whose source is inside that frozen tree is read out of the leader's capture instead of live.
+A Windows VSS shadow copy freezes an entire volume, so every source on it is inside; btrfs and thin-LVM freeze the subvolume or logical volume the leader's data sits on, which holds whatever else is under the same mount; a streamed base backup copies the cluster directory alone and holds nothing besides.
+Path prefix alone does not decide it on the Linux backends: a btrfs snapshot does not descend into a nested subvolume, nor an LVM one into a filesystem mounted inside it, and a source there reads as an empty directory, so sharing turns on being on the same subvolume or mount.
+VSS is the only backend that shares a capture today, matching the volume as the path spells it, case-insensitively; a source the capture never froze is read live as before.
+
+This is one snapshot where there were two, and it is also the only way the pair describes a single instant: a follower reading live captures the store as it is minutes later, after its leader's data froze.
+Such a follower therefore reports its leader's freeze instant as its own, being what its data actually describes.
+
+The capture is released once the whole chain has drained, rather than at the end of the run that took it, since that is the point after which nothing can still be reading from it.
+Releasing it is best-effort: every run that used it has finished and reported by then, so a teardown failure is a leak to warn about rather than an outcome to fail.
+A run told to hold its capture keeps it as a rollback point instead, and shares nothing.
+
 ## The moment the data froze
 
 A run reports the instant it froze the data it backs up — the point in time the backup represents — which is distinct from when the upload finished and from when Canopy received the report.
@@ -146,6 +182,7 @@ kopia's snapshot source host is set to the server id, so a backup's source is at
 The source path is stable across runs for a given backup type, so kopia's snapshot history, deduplication, and retention attribute to one source.
 
 Every snapshot is tagged with the device id, the run id, and the backup type, plus any tags the definition or the method contribute; the canopy-owned tags take precedence so a definition cannot override them.
+Every snapshot also carries its backup type as its description, so a repository listing identifies each snapshot without its tags being read.
 
 ## Local cache
 
@@ -217,10 +254,27 @@ A restore can equally take its source from a capture held on the device, describ
 Selection is by id across the whole repository — not scoped to the server issuing the restore — so a replacement host can restore a backup taken by the server it succeeds.
 It restores the snapshot into a staging area on the same filesystem as the target so the final move is atomic, then hands off to the method.
 
+Restoring a type also restores its followers, so a cycle like database-and-blob-store comes back as a consistent pair.
+A follower's snapshot is selected rather than named: the earliest snapshot of the follower's type, from the same source host as the chosen snapshot, taken at or after it.
+At-or-after is the safety rule: a later follower snapshot is a superset of what the restored data references, an earlier one may not be, and is never selected; when none exists at or after, the restore refuses.
+The whole cycle is planned up front, before any data is touched, and each follower restore is then a full restore of its own, with its own credentials, run id, and report, in chain order, so a follower whose target is resolved by `path_command` resolves it against data its leader has just restored.
+`--no-followers` restores the named type alone; restoring a follower's type explicitly by snapshot id remains the operator's manual path around a refusal.
+`--target` redirects only the named type's destination while followers would still restore over their live paths, so combining it with planned followers is refused; pass `--no-followers` alongside it.
+Follower snapshots are recognised by the backup type they carry, as a tag or as their description.
+
 The `postgresql` method's restore is a full automated swap: it stops the cluster, moves the existing data directory aside (kept, not deleted), moves the restored tree into place with the right ownership and permissions, starts the cluster via plain crash recovery, and verifies it accepts connections.
 A WAL reset is only attempted as a logged last resort if the cluster will not start.
 The `simple` method's restore lays the files back at its path or a given target.
 
+### Restore hooks
+
+A definition's `[[pre_restore]]` and `[[post_restore]]` hooks bracket its own restore, in order, argv-style, and a failing one on either side fails the restore run.
+They exist for data a service holds open: the method can lay files back, but it has no way to know which process has to let go of them first or be restarted afterwards to see them.
+Both run around the method, not around the download — a staging failure never stops a service.
+Each definition's hooks are its own, so in a restored cycle a follower's hooks bracket the follower's data landing, not its leader's.
+A failed `[[pre_restore]]` hook stops the restore before anything is laid down, so the existing data is untouched.
+A failed `[[post_restore]]` hook leaves the restored data in place — it is a report that the service did not come back, not a reason to undo the restore.
+
 Restore refuses to overwrite existing data by default.
 To proceed an operator passes an explicit confirmation flag (for non-interactive use) or answers an interactive double confirmation; with neither, over occupied data, it refuses.
 Migrations, configuration sync, and version upgrades are left to the operator.

crates/bestool/Cargo.toml

@@ -122,6 +122,7 @@ tamanu = [ # enable all tamanu subcommands
 	"tamanu-artifacts",
 	"tamanu-backup",
 	"tamanu-backup-configs",
+	"tamanu-blob-root",
 	"tamanu-config",
 	"tamanu-url",
 	"tamanu-doctor",
@@ -142,6 +143,9 @@ alertd-tamanu = ["alertd", "tamanu-config", "dep:bestool-postgres", "bestool-tam
 tamanu-artifacts = ["__tamanu", "dep:comfy-table", "dep:detect-targets", "dep:target-tuples"]
 tamanu-backup = ["__tamanu", "file", "tamanu-config", "dep:bestool-psql", "dep:algae-cli", "dep:duct"]
 tamanu-backup-configs = ["__tamanu", "tamanu-backup", "dep:walkdir", "dep:zip"]
+# Print the blob store root, resolved from Tamanu's `blobStorage.root` setting
+# in the database, for blob store backup defs to use as their `path_command`.
+tamanu-blob-root = ["__tamanu", "tamanu-config", "dep:bestool-postgres"]
 tamanu-config = ["__tamanu"]
 tamanu-doctor = [
 	"__tamanu",

crates/bestool/USAGE.md

@@ -72,6 +72,7 @@ This document contains the help content for the `bestool` command-line program.
 * [`bestool tamanu artifacts`↴](#bestool-tamanu-artifacts)
 * [`bestool tamanu backup`↴](#bestool-tamanu-backup)
 * [`bestool tamanu backup-configs`↴](#bestool-tamanu-backup-configs)
+* [`bestool tamanu blob-root`↴](#bestool-tamanu-blob-root)
 * [`bestool tamanu config`↴](#bestool-tamanu-config)
 * [`bestool tamanu db-url`↴](#bestool-tamanu-db-url)
 * [`bestool tamanu doctor`↴](#bestool-tamanu-doctor)
@@ -507,6 +508,9 @@ Restore a backup from Canopy's repository
    Takes a hold id, as shown by `bestool canopy hold list`.
 * `--target <PATH>` — Override the destination (the simple method's path); postgresql always targets its configured cluster
 * `--clobber-existing-data-yes-i-am-sure` — Proceed even if the destination already contains data (non-interactive)
+* `--no-followers` — Restore only the named type, skipping the defs that follow it.
+
+   By default, restoring a type also restores each def that declares `after` on it, from the earliest snapshot of that def's type taken at or after the one being restored, never an earlier one, which could lack content the restored data references.
 * `--config <DIR>` — Override the registration directory
 * `--backups-dir <DIR>` — Override the backups definition directory
 
@@ -1528,6 +1532,7 @@ Alias: t
 * `artifacts` — List available artifacts for a Tamanu version
 * `backup` — Backup a local Tamanu database to a single file
 * `backup-configs` — Backup local Tamanu-related config files to a zip archive
+* `blob-root` — Print the Tamanu blob store root
 * `config` — Find and print the current Tamanu config
 * `db-url` — Generate a DATABASE_URL connection string
 * `doctor` — Gather server info + healthchecks for a Tamanu install
@@ -1904,6 +1909,22 @@ If `--key` or `--key-file` is provided, the backup file will be encrypted. Note
 
 
 
+## `bestool tamanu blob-root`
+
+Print the Tamanu blob store root.
+
+The root is Tamanu's `blobStorage.root` setting (database-backed and editable in the admin panel, so no config file carries it), resolved against the server package directory when not absolute. A blob store backup def names this command as its `path_command`, so every capture and restore follows the live setting instead of a hardcoded path going stale.
+
+**Usage:** `bestool tamanu blob-root [OPTIONS]`
+
+###### **Options:**
+
+* `-p`, `--package <PACKAGE>` — Package to read the setting for (central-server or facility-server).
+
+   Detected from the config and database when not given.
+
+
+
 ## `bestool tamanu config`
 
 Find and print the current Tamanu config.

crates/bestool/src/actions/alertd.rs

@@ -266,6 +266,9 @@ fn backup_runner() -> bestool_alertd::BackupRunner {
 			BackupEvent::Started { run_id } => json!({"event": "started", "runId": run_id}),
 			BackupEvent::Phase(phase) => json!({"event": "phase", "phase": phase}),
 			BackupEvent::Progress(status) => json!({"event": "progress", "status": status}),
+			BackupEvent::Follower { backup_type } => {
+				json!({"event": "follower", "type": backup_type})
+			}
 			BackupEvent::Done {
 				snapshot_id,
 				bytes_uploaded,