You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The reference cost ledger added for #343 (cost_entry table, CostLedger protocol, /_reference/experiments/{E}/cost) is not covered by checkpoint export/import. A checkpoint round-trip therefore silently loses every recorded spend row: the wire state (tasks / ideas / variants / events / submissions / workers / groups) round-trips, cost does not.
Concretely, after POST /v0/experiments/{E}/checkpoint → teardown → POST /v0/checkpoints/import, GET /_reference/experiments/{E}/cost on the receiver returns an empty ledger even though the source experiment had rows. Same failure shape as the empty-repo.bundle bug (#294/#131): structurally valid archive, silently incomplete content.
Why it was deferred
The checkpoint archive layout is normative — spec/v0/10-checkpoints.md §3 pins the directory tree and §5's manifest files object names each logical component, with checkpoint_format_version at "1" and an importer MUST-reject on an unrecognized version. Adding a cost.jsonl component to the bundle is a normative format change (new manifest field + format-version handling + conformance implications), which the #343 milestone-1 PR was explicitly scoped away from (its balloon-guard: implement within current spec, propose spec surgery rather than doing it).
Options
Bump the checkpoint format version and add cost as a first-class component (files.cost). Cleanest, but it is spec surgery: §5 prose, the manifest schema, an importer that accepts v1 archives without the component, and a conformance thought-pass on §12 cross-reference validation. Pairs naturally with whatever revision gives cost a normative home (the spec-change plan proposed on Cost instrumentation: persist per-role token/$ so runs can report what they cost #343) — if cost becomes a Variant field or a first-class record, it round-trips through the existing components and this issue dissolves.
Reference-only sidecar next to the archive. An extra file the reference exporter writes outside the normative tree (or a separate /_reference/ export endpoint). No spec change, but two artifacts to move instead of one, and the sidecar is easy to lose — which is the failure mode this issue is about.
Accept the loss and document it. Cost is derived bookkeeping, not protocol state; an operator who checkpoints for disaster recovery arguably cares about resumability, not accounting history. Cheapest, and wrong for the R2/R3 use case: the whole point of Cost instrumentation: persist per-role token/$ so runs can report what they cost #343 is being able to say what a completed run cost, and a long run that survives a checkpoint restore would report only its post-restore spend.
Recommendation: option 1, bundled with #343's normative-home decision rather than as standalone work.
Detection
Until this is fixed the gap should at least be visible: smoke-checkpoint.sh passed while #294 existed because it only asserted structural validity. If the deferral persists, add an explicit assertion (or a logged warning at export time when the ledger is non-empty) so "checkpoint drops cost" is a stated property rather than a surprise.
Problem
The reference cost ledger added for #343 (
cost_entrytable,CostLedgerprotocol,/_reference/experiments/{E}/cost) is not covered by checkpoint export/import. A checkpoint round-trip therefore silently loses every recorded spend row: the wire state (tasks / ideas / variants / events / submissions / workers / groups) round-trips, cost does not.Concretely, after
POST /v0/experiments/{E}/checkpoint→ teardown →POST /v0/checkpoints/import,GET /_reference/experiments/{E}/coston the receiver returns an empty ledger even though the source experiment had rows. Same failure shape as the empty-repo.bundlebug (#294/#131): structurally valid archive, silently incomplete content.Why it was deferred
The checkpoint archive layout is normative —
spec/v0/10-checkpoints.md§3 pins the directory tree and §5's manifestfilesobject names each logical component, withcheckpoint_format_versionat"1"and an importer MUST-reject on an unrecognized version. Adding acost.jsonlcomponent to the bundle is a normative format change (new manifest field + format-version handling + conformance implications), which the #343 milestone-1 PR was explicitly scoped away from (its balloon-guard: implement within current spec, propose spec surgery rather than doing it).Options
files.cost). Cleanest, but it is spec surgery: §5 prose, the manifest schema, an importer that accepts v1 archives without the component, and a conformance thought-pass on §12 cross-reference validation. Pairs naturally with whatever revision gives cost a normative home (the spec-change plan proposed on Cost instrumentation: persist per-role token/$ so runs can report what they cost #343) — if cost becomes aVariantfield or a first-class record, it round-trips through the existing components and this issue dissolves./_reference/export endpoint). No spec change, but two artifacts to move instead of one, and the sidecar is easy to lose — which is the failure mode this issue is about.Recommendation: option 1, bundled with #343's normative-home decision rather than as standalone work.
Detection
Until this is fixed the gap should at least be visible:
smoke-checkpoint.shpassed while #294 existed because it only asserted structural validity. If the deferral persists, add an explicit assertion (or a logged warning at export time when the ledger is non-empty) so "checkpoint drops cost" is a stated property rather than a surprise.Context
reference/packages/eden-storage/src/eden_storage/cost.py