perf(sync): encode only what changed, and detect when the compressors move

.claude/rules/project-stage.md

@@ -25,6 +25,16 @@ The compatibility contract is the one the release machinery already encodes —
   fields (`dir`, `build`, `presync`, `metadata`) live in users' committed project
   files. Renaming or removing one breaks those projects — treat the recipe schema like
   any other released API.
+- **Changing asset preparation is expensive, not unsafe.** The compressor settings
+  (gzip level, brotli quality/window), the compressor *implementations* (a `brotli`
+  or `flate2` bump, or a different `flate2` backend selected by feature
+  unification), and `MAX_CHUNK_SIZE` all change the bytes a sync stores. Nothing
+  needs freezing to stay correct: `asset-prep`'s compression canary detects a change
+  and makes the next sync re-prepare and re-upload every asset, and the
+  [`docs/verifying-contents.md`](../../docs/verifying-contents.md) contract is
+  already version-scoped. But each change costs every deployed canister one full
+  re-upload and invalidates previously-published state hashes, so make it
+  deliberately — ideally on a series bump — rather than as a drive-by `cargo update`.
 
 Two things do **not** change:
 

.claude/rules/testing.md

@@ -7,10 +7,12 @@ Tests are organized around these components. Each runs independently.
 - **Location**: inline `#[cfg(test)]` modules in each [`crates/asset-prep/src/`](../../crates/asset-prep/src/)`*.rs` file
 - **Run**: `cargo test -p asset-prep`
 
-`asset-prep` is the native, local-only crate that turns a `dist/` directory into prepared assets + redirect rules + the canonical `state-hash::Manifest` (shared by `sync-core` and the `state-hash-cli` verifier). It owns directory scanning, MIME detection, encoding selection, `_headers`/`_redirects` parsing, html-handling synthesis, the 404 convention, content chunking/hashing, and the `dist → Manifest` builder.
+`asset-prep` is the native, local-only crate that turns a `dist/` directory into prepared assets + redirect rules + the canonical `state-hash::Manifest` (shared by `sync-core` and the `state-hash-cli` verifier). It owns directory scanning, MIME detection, encoding selection, `_headers`/`_redirects` parsing, html-handling synthesis, the 404 convention, content chunking/hashing, the compression canary, and the `dist → Manifest` builder.
 
 **Add tests here when** you change how files are discovered, how MIME/encodings are chosen, how `_headers`/`_redirects` parse, how html-handling/404 rules are synthesised, or how content is chunked/hashed into a manifest.
 
+**Golden vectors live here** ([`content.rs`](../../crates/asset-prep/src/content.rs), [`canary.rs`](../../crates/asset-prep/src/canary.rs)): the compressors' output bytes and the canary fingerprint are pinned, because nothing in semver protects them and a drift silently costs every deployed canister a full re-upload. A failure there is a signal, not a nuisance — read the guidance above the vectors before updating them.
+
 ## State hash (`state-hash`)
 
 - **Location**: inline `#[cfg(test)]` in [`crates/state-hash/src/lib.rs`](../../crates/state-hash/src/lib.rs)
@@ -40,7 +42,7 @@ Tests are organized around these components. Each runs independently.
 
 ## End-to-end (`e2e`)
 
-- **Location**: [`crates/e2e/`](../../crates/e2e/) — split across focused test files (e.g. `sync.rs`, `redirects.rs`, `etag.rs`, `streaming.rs`, `protection.rs`, `recipe.rs`)
+- **Location**: [`crates/e2e/`](../../crates/e2e/) — split across focused test files (e.g. `sync.rs`, `redirects.rs`, `etag.rs`, `streaming.rs`, `protection.rs`, `recipe.rs`, `state_hash.rs`)
 - **Run**: `cargo test -p e2e`
 
 E2E tests verify that the canister and plugin work correctly together through the `icp` CLI against a live local replica — deploy, re-sync, content update/deletion, serving, certification, redirects, headers, streaming/range, ETag, env cookie, upgrade persistence, access protection, and recipe resolution.

.github/workflows/ci.yml

@@ -77,10 +77,15 @@ jobs:
           exit 1
         env:
           ICP_CLI_GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+      # `--locked` everywhere: the committed Cargo.lock is what pins the
+      # compressor builds whose output bytes end up in deployed canisters and in
+      # `state_hash` (see the golden vectors in `asset-prep`). Without it a stale
+      # lock is silently updated in CI, so a run could test different bytes than
+      # a developer did. With it, that fails the build instead.
       - name: Run tests
         run: | # nextest doesn't run doctests, so those stay on `cargo test --doc`
-          cargo nextest run --all-targets --no-fail-fast
-          cargo test --doc
+          cargo nextest run --locked --all-targets --no-fail-fast
+          cargo test --locked --doc
 
   fmt:
     name: cargo fmt
@@ -110,7 +115,7 @@ jobs:
         uses: actions-rust-lang/setup-rust-toolchain@150fca883cd4034361b621bd4e6a9d34e5143606 # v1.15.4
       - name: Run clippy
         run: |
-          cargo clippy --tests --benches --keep-going -- -D warnings
+          cargo clippy --locked --tests --benches --keep-going -- -D warnings
 
   benchmark:
     name: canbench

Cargo.toml

@@ -20,16 +20,22 @@ ic-http-certification = "3.0.3"
 ic-response-verification = "3.0.3"
 ic-representation-independent-hash = "3.0.3"
 
+# Compressors. Their output bytes are part of what a deployed canister stores and
+# what `state_hash` covers, and semver doesn't protect them: a compatible bump may
+# re-tune an encoder, and flate2's bytes also depend on which backend feature
+# unification selects. Left unpinned deliberately — a version pin can't see a
+# backend swap, and `asset-prep`'s golden vectors fail on any drift either way.
+brotli = "8.0.2"
+flate2 = "1.0"
+
 # Others
 anyhow = "1.0.103"
 assert_cmd = "2"
 base64 = "0.22.1"
-brotli = "8.0.2"
 canbench-rs = "0.6.0"
 candid = "0.10.18"
 candid_parser = "0.4.0"
 ciborium = "0.2.2"
-flate2 = "1.0"
 futures = "0.3"
 # Pinned to the versions icp-cli uses so recipe-gen's tests render recipes the
 # same way the real CLI does.

Makefile

@@ -40,13 +40,16 @@ wasm: canister plugin
 # Recipes are phony on purpose: cargo already does incremental freshness
 # tracking, so we always invoke it (cheap when nothing changed) and let it decide
 # whether a rebuild is needed, then copy the result into dist/.
+#
+# `--locked` because the shipped wasms embed the compressors whose output bytes
+# every deployed canister stores; a silently-updated lock would change them.
 canister:
-	cargo build -p canister --target $(CANISTER_TARGET) --profile $(CANISTER_PROFILE)
+	cargo build --locked -p canister --target $(CANISTER_TARGET) --profile $(CANISTER_PROFILE)
 	@mkdir -p $(DIST)
 	cp $(CANISTER_OUT) $(DIST)/canister.wasm
 
 plugin:
-	cargo build -p sync-plugin --target $(PLUGIN_TARGET) --profile $(PLUGIN_PROFILE)
+	cargo build --locked -p sync-plugin --target $(PLUGIN_TARGET) --profile $(PLUGIN_PROFILE)
 	@mkdir -p $(DIST)
 	cp $(PLUGIN_OUT) $(DIST)/plugin.wasm
 

certified-assets.did

@@ -141,6 +141,13 @@ type SetRedirectRulesArguments = record {
   rules: vec RedirectRule;
 };
 
+// Record the client's compression fingerprint over the assets this sync wrote.
+// Emitted only after a sync prepared every asset from scratch, so the stored
+// value describes the whole asset set. See `preparation_canary`.
+type SetPreparationCanaryArguments = record {
+  canary: blob;
+};
+
 // A single mutation applied during a sync.
 type Operation = variant {
   CreateAsset: CreateAssetArguments;
@@ -149,6 +156,7 @@ type Operation = variant {
   DeleteAsset: DeleteAssetArguments;
   SetAssetHeaders: SetAssetHeadersArguments;
   SetRedirectRules: SetRedirectRulesArguments;
+  SetPreparationCanary: SetPreparationCanaryArguments;
 };
 
 // ───────── Sync: session ─────────
@@ -254,8 +262,19 @@ service: {
   // mapping is positional per call, uploads may be issued concurrently.
   upload_chunks: (UploadChunksArguments) -> (vec ChunkId);
 
+  // The compression fingerprint recorded by the client that last prepared every
+  // asset (32 bytes), or 32 zero bytes if none has. A syncing client compares it
+  // against its own to decide whether the stored compressed encodings are the
+  // ones its compressors would produce — if not, it re-prepares everything
+  // rather than trusting an unchanged uncompressed hash. Opaque to the canister.
+  preparation_canary: () -> (blob) query;
+
   // Apply a group of operations. Perform all successfully, or reject.
-  execute_operations: (ExecuteOperationsArguments) -> ();
+  // The call flagged `is_final` finalizes the sync and returns the canonical
+  // state hash recomputed over the now-final state (the same value `state_hash`
+  // then reports); every other call returns `null`. Returning it here saves a
+  // finishing sync a second round trip.
+  execute_operations: (ExecuteOperationsArguments) -> (opt blob);
 
   // ───────── Authorization ─────────
   // A set of extra principals authorized to sync assets. Canister controllers

crates/asset-prep/src/canary.rs

@@ -0,0 +1,190 @@
+//! A fingerprint of *how this build compresses*, used to make lazy encoding safe.
+//!
+//! `sync-core` skips encoding an asset whose uncompressed hash the canister
+//! already holds, inferring that the stored compressed encodings are the ones
+//! this build would produce. That inference is only sound while the compressors
+//! behave identically — and nothing guarantees they will. Compressed output is
+//! not part of a crate's API surface, so a semver-compatible `brotli` bump may
+//! legally re-tune its heuristics; `flate2`'s DEFLATE bytes depend on which
+//! backend feature unification selected; and neither is fixed by a
+//! specification (RFC 7932 and RFC 1951 define *decoders*).
+//!
+//! So instead of freezing those dependencies by convention and hoping nobody
+//! bumps them, the sync records what its compressors actually did. The canary is
+//! the hash of a frozen input compressed by *this* build; the canister stores the
+//! value written alongside its assets. A mismatch means "the compressors moved",
+//! and the sync falls back to re-preparing everything eagerly — which is exactly
+//! the pre-lazy-encoding behaviour, so the canister re-converges on its own.
+//!
+//! **This is detection, not proof.** Compressor behaviour is a function over all
+//! inputs and the canary samples it at one point, so a change that alters real
+//! bundles but not this input would slip through. [`INPUT`] is therefore built
+//! from deliberately varied material — long repeats, prose, structured text and
+//! incompressible noise — to exercise as many encoder decision paths as a few
+//! kilobytes can. What it catches reliably is what actually happens in practice:
+//! a version bump or a backend swap, both of which change essentially every
+//! output. A miss leaves the sync exactly where it would be with no canary at
+//! all, so this can only reduce the exposure, never add to it.
+//!
+//! [`INPUT`] is frozen: changing it changes every canister's canary and forces
+//! one full re-upload each. `input_is_frozen` guards it in CI.
+
+use sha2::{Digest, Sha256};
+use wire_types::Encoding;
+
+use crate::content::Content;
+use crate::prepare::MAX_CHUNK_SIZE;
+
+/// Size of the canary input. Big enough to cross the encoders' internal block
+/// boundaries and exercise several match strategies, small enough that
+/// compressing it costs single-digit milliseconds even at brotli q11.
+const INPUT_LEN: usize = 6 * 1024;
+
+/// The frozen canary input: four regions with deliberately different characters,
+/// so an encoder change that only fires on one kind of data still shows up.
+///
+/// 1. a long repeat (long-match / distance-cache paths),
+/// 2. prose-like text (static dictionary, literal entropy coding),
+/// 3. structured source-like text (mixed punctuation, short repeats),
+/// 4. high-entropy bytes (the no-match path, literal coding).
+fn input() -> Vec<u8> {
+    let mut out = Vec::with_capacity(INPUT_LEN);
+    let quarter = INPUT_LEN / 4;
+
+    while out.len() < quarter {
+        out.extend_from_slice(b"aaaaaaaaaaaaaaaabbbbbbbbbbbbbbbb");
+    }
+    out.truncate(quarter);
+
+    while out.len() < 2 * quarter {
+        out.extend_from_slice(
+            b"the quick brown fox jumps over the lazy dog while the sun sets slowly. ",
+        );
+    }
+    out.truncate(2 * quarter);
+
+    while out.len() < 3 * quarter {
+        out.extend_from_slice(
+            br#"{"key":"value","n":12345,"nested":{"a":[1,2,3],"b":null},"s":"text"}"#,
+        );
+    }
+    out.truncate(3 * quarter);
+
+    // A fixed xorshift64* stream: deterministic, no dependencies, and not
+    // meaningfully compressible. The constants are frozen along with the rest.
+    let mut state: u64 = 0x2545_F491_4F6C_DD1D;
+    while out.len() < INPUT_LEN {
+        state ^= state >> 12;
+        state ^= state << 25;
+        state ^= state >> 27;
+        out.extend_from_slice(&state.wrapping_mul(0x2545_F491_4F6C_DD1D).to_le_bytes());
+    }
+    out.truncate(INPUT_LEN);
+
+    out
+}
+
+/// This build's compression fingerprint: the hash of the frozen input under
+/// every compressor, folded together with the preparation constants that shape
+/// the stored form.
+///
+/// Deliberately routed through [`Content::encode`] — the same call real assets
+/// take — so any change to how we compress is reflected here, not just changes
+/// in the underlying crates. `MAX_CHUNK_SIZE` is folded in because it decides
+/// the chunk layout a skipped asset's manifest entry depends on.
+pub fn fingerprint() -> Result<[u8; 32], String> {
+    let content = Content {
+        data: input(),
+        media_type: mime::APPLICATION_OCTET_STREAM,
+    };
+
+    let mut hasher = Sha256::new();
+    hasher.update(b"certified-assets preparation canary v1");
+    hasher.update((MAX_CHUNK_SIZE as u64).to_le_bytes());
+    for encoding in [Encoding::Gzip, Encoding::Brotli] {
+        let encoded = content.encode(encoding)?;
+        hasher.update(encoding.label().as_bytes());
+        hasher.update((encoded.data.len() as u64).to_le_bytes());
+        hasher.update(Sha256::digest(&encoded.data));
+    }
+    Ok(hasher.finalize().into())
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    /// The canary input is frozen: changing it invalidates every canister's
+    /// stored canary and forces a full re-upload on each. If this fails because
+    /// you meant to change it, bump the domain-separation string in
+    /// `fingerprint` too, and expect the re-uploads.
+    #[test]
+    fn input_is_frozen() {
+        let digest: [u8; 32] = Sha256::digest(input()).into();
+        assert_eq!(digest, INPUT_DIGEST, "canary input drifted");
+    }
+
+    /// SHA-256 of [`input`]. A golden value, not a derived one.
+    const INPUT_DIGEST: [u8; 32] = [
+        113, 10, 158, 48, 203, 255, 77, 203, 177, 240, 217, 133, 13, 80, 34, 205, 230, 236, 159,
+        173, 213, 2, 80, 98, 23, 34, 183, 131, 37, 107, 232, 145,
+    ];
+
+    #[test]
+    fn input_has_the_expected_shape() {
+        let input = input();
+        assert_eq!(input.len(), INPUT_LEN);
+        // The four regions must actually differ, or the varied-material argument
+        // in this module's docs is false.
+        let quarter = INPUT_LEN / 4;
+        let regions: Vec<&[u8]> = input.chunks(quarter).collect();
+        for (i, a) in regions.iter().enumerate() {
+            for b in regions.iter().skip(i + 1) {
+                assert_ne!(a, b, "canary regions must differ");
+            }
+        }
+    }
+
+    /// The whole point: the fingerprint is stable for a fixed build, so an
+    /// unchanged toolchain never triggers a spurious re-upload.
+    #[test]
+    fn fingerprint_is_deterministic() {
+        assert_eq!(fingerprint().unwrap(), fingerprint().unwrap());
+    }
+
+    /// The fingerprint value itself, pinned — the CI half of the pair. A drift
+    /// here means the next deploy against any existing canister re-prepares and
+    /// re-uploads every asset, and every published state hash stops matching.
+    /// That is a legitimate thing to do, but not by accident: see the guidance
+    /// on the golden vectors in `content.rs`, which say *which* compressor moved.
+    #[test]
+    fn fingerprint_is_pinned() {
+        assert_eq!(
+            fingerprint().unwrap(),
+            [
+                245, 226, 54, 136, 144, 3, 34, 211, 233, 170, 231, 7, 236, 41, 240, 14, 77, 42,
+                176, 104, 40, 40, 127, 55, 244, 101, 113, 73, 212, 101, 239, 247,
+            ],
+            "compression fingerprint drifted"
+        );
+    }
+
+    /// ...and it is actually sensitive to compressor output, not just to the
+    /// constants folded in around it.
+    #[test]
+    fn fingerprint_tracks_compressed_bytes() {
+        let content = Content {
+            data: input(),
+            media_type: mime::APPLICATION_OCTET_STREAM,
+        };
+        let brotli = content.encode(Encoding::Brotli).unwrap();
+        let mut hasher = Sha256::new();
+        hasher.update(Sha256::digest(&brotli.data));
+        let independent: [u8; 32] = hasher.finalize().into();
+        // Not equal (different domain), but if brotli output were ignored the
+        // fingerprint could not depend on it at all — this asserts the encode
+        // path runs and produces bytes to hash.
+        assert!(!brotli.data.is_empty());
+        assert_ne!(fingerprint().unwrap(), independent);
+    }
+}