[BUG] Server store lost a node block (suspected during mass repack): every push fails verification with 'Deserialize node block failed: Address not found', and clone of any revision fails — client still holds the block but cannot re-upload it

[JKConnor]

Summary

A localhost loreserver store silently lost (or corrupt-wrote) at least one node block. From that point on:

• lore push of the main branch fails every time with:

  [Error] pushing branch to remote: code: 'Internal error', message: "Failed to collect new
  fragments for verification: Deserialize node block failed: Address not found:
  cdcb669db3961400be0a6393b2c3c584d1927286a270e6b6f94dfbd7e099b019-00000000000000000000000000000000"
  

  (0.8.4 adds the source ref: lore-revision\src\branch\push.rs:892)
• A fresh lore clone of the repository fails at every revision with
  [Error] Failed to deserialize revision state node block after Cloned 62/63 files (53.24 MiB/281.36 MiB) — the server can no longer serve one complete revision of anything.
• Local commits keep working; the local repository passes lore repository verify ("Verified repository state integrity").

The nasty part: the client still has the missing block, but there is no way to give it back to the server. Push's fragment collection only queries the addresses that are new in the pushed revisions, so the lost block is never re-offered, and verification fails server-side forever.

Environment

• Client: lore 0.8.3+201, server: loreserver 0.8.3 at time of corruption. Reproduced identically after upgrading both to 0.8.4+283 / loreserver 0.8.4.
• Windows 11 Home 10.0.26200, client and server on the same machine (lore://127.0.0.1:41337/...).
• Single-user repo, 96 revisions, mixed text + binary content, server immutable store ≈ 9.7 GB across the 256 shard dirs, flush_delay_seconds = 10 in both [immutable_store.local] and [mutable_store.local].

Timeline / suspected trigger

• Heavy autonomous push activity through the night (multiple large commits).
• At 02:45 local, every one of the 256 immutable-store shard pack files was rewritten within a ~2-minute window (file mtimes 02:45:25–02:47:24). We did not run gc manually — is there internal scheduled/threshold compaction in 0.8.3's server? This mass rewrite is our prime suspect for dropping the block.
• Pushes kept succeeding all day afterwards (they only verified recently-changed subtrees).
• The first push whose verification had to walk the older unchanged subtree containing the lost node failed with the error above, and has failed on every retry, across server restarts and the 0.8.4 upgrade.
• Possibly relevant: 0.8.4's changelog fixes a use-after-free in write_fragmented when the chunker future is cancelled — if that could produce a corrupt-written or dropped block under 0.8.3, it would fit this picture.

Diagnostics collected

lore repository store immutable query <address> for the failing address:

Address cdcb669d…-00000000000000000000000000000000 (local)
Status: Stored (metadata and payload)
Payload: 31584 bytes (cached)
Content: 49280 bytes
Flags: 0x150008
Address cdcb669d…-00000000000000000000000000000000 (remote)
Status: Not found

Push with --debug shows the client querying only the 2 new fragment addresses ("Queried 2 fragments, 0 missing", "Push 0/0 fragment(s)") before the server-side verification error — confirming the lost block is never re-offered.

Curiosity observed while probing: pushing a newly created branch pointing at the same head "succeeds" (updates the remote branch pointer, transfers 0 fragments, and prints updating current local latest to 0000000000000000000000000000000000000000000000000000000000000000) even though the store is in this state — so branch-create pushes appear to skip the verification that main's update runs. A subsequent clone of that branch fails the same way, so the pointer now leads to an unservable revision.

We have preserved a full copy of the damaged server store directory (9.7 GB) and can run further diagnostics against it on request (we can't share repository content, but can share store-level metadata, sizes, error output, and debug logs).

Asks

1. Recovery path: a supported way to heal a server store when a client still holds the missing fragment — e.g. lore push --repair (re-offer the full closure of the pushed revision), or a low-level "upload this address" command. Right now the only options appear to be rebuilding the repo and abandoning history.
2. fsck for stores: a server-store verification command (the client has lore repository verify; the server side appears to have nothing), ideally runnable offline against a store directory.
3. Root cause: if 0.8.3's server does scheduled/threshold compaction, please check it for a window where an indexed block's payload can be dropped (see the 02:45 mass-repack timeline above). Possibly related structural error report: [Error] Failed to find subnode: Node 36368 has parent 0 but was reached as a child of 11036 #90.