Skip to content

Cost ledger rows are not carried in checkpoint export/import (#343 deferral) #344

Description

@ealt

Problem

The reference cost ledger added for #343 (cost_entry table, CostLedger protocol, /_reference/experiments/{E}/cost) is not covered by checkpoint export/import. A checkpoint round-trip therefore silently loses every recorded spend row: the wire state (tasks / ideas / variants / events / submissions / workers / groups) round-trips, cost does not.

Concretely, after POST /v0/experiments/{E}/checkpoint → teardown → POST /v0/checkpoints/import, GET /_reference/experiments/{E}/cost on the receiver returns an empty ledger even though the source experiment had rows. Same failure shape as the empty-repo.bundle bug (#294/#131): structurally valid archive, silently incomplete content.

Why it was deferred

The checkpoint archive layout is normativespec/v0/10-checkpoints.md §3 pins the directory tree and §5's manifest files object names each logical component, with checkpoint_format_version at "1" and an importer MUST-reject on an unrecognized version. Adding a cost.jsonl component to the bundle is a normative format change (new manifest field + format-version handling + conformance implications), which the #343 milestone-1 PR was explicitly scoped away from (its balloon-guard: implement within current spec, propose spec surgery rather than doing it).

Options

  1. Bump the checkpoint format version and add cost as a first-class component (files.cost). Cleanest, but it is spec surgery: §5 prose, the manifest schema, an importer that accepts v1 archives without the component, and a conformance thought-pass on §12 cross-reference validation. Pairs naturally with whatever revision gives cost a normative home (the spec-change plan proposed on Cost instrumentation: persist per-role token/$ so runs can report what they cost #343) — if cost becomes a Variant field or a first-class record, it round-trips through the existing components and this issue dissolves.
  2. Reference-only sidecar next to the archive. An extra file the reference exporter writes outside the normative tree (or a separate /_reference/ export endpoint). No spec change, but two artifacts to move instead of one, and the sidecar is easy to lose — which is the failure mode this issue is about.
  3. Accept the loss and document it. Cost is derived bookkeeping, not protocol state; an operator who checkpoints for disaster recovery arguably cares about resumability, not accounting history. Cheapest, and wrong for the R2/R3 use case: the whole point of Cost instrumentation: persist per-role token/$ so runs can report what they cost #343 is being able to say what a completed run cost, and a long run that survives a checkpoint restore would report only its post-restore spend.

Recommendation: option 1, bundled with #343's normative-home decision rather than as standalone work.

Detection

Until this is fixed the gap should at least be visible: smoke-checkpoint.sh passed while #294 existed because it only asserted structural validity. If the deferral persists, add an explicit assertion (or a logged warning at export time when the ledger is non-empty) so "checkpoint drops cost" is a stated property rather than a surprise.

Context

Metadata

Metadata

Assignees

No one assigned

    Labels

    cluster:durabilitySubstrate persistence, recovery, checkpointsenhancementNew feature or requestpriority:2-plannedPlanned roadmap work; well-scoped, not urgenttriage:readyIssue triage disposition

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions