[feat][RFC] Dual-mode checkpoint storage with LangGraph DeltaChannel

[Vanzeren]

Before you start

• I searched existing issues and this is not a duplicate.

Problem / motivation

Summary

Adopt LangGraph 1.2's DeltaChannel for the messages channel as an opt-in
checkpoint storage mode (checkpoint_channel_mode: full | delta, default
full), cutting checkpoint storage and write-side serialization from O(N²) to
O(N) in conversation turns. The feature ships with a dual-mode safety
framework (process-frozen mode, per-checkpoint metadata markers, fail-closed
compatibility gate), a single materialized-access choke point for all
checkpoint consumers, and mechanism-preserving rewrites of run rollback and
context compaction.

Motivation

Default checkpointing serializes the full message list into every checkpoint,
so a thread's storage grows quadratically with turns. Upstream's reference
numbers for a 200-turn Deep Agents run: 5.3 GB → 129 MB (41×) with
DeltaChannel. DeerFlow research/coding threads hit the same growth pattern —
they routinely run hundreds of turns with large tool outputs.

Delta mode stores only a sentinel per checkpoint plus the per-step writes;
reads reconstruct state by replaying ancestor writes through the reducer
(snapshot_frequency=1000 bounds replay). All checkpointer backends
(memory/sqlite/postgres) serve both modes unchanged — the semantics live in
the compiled graph's channel table, not the saver.

Proposed solution

Goals

• O(N) checkpoint storage for messages in delta mode; zero data migration
  for existing threads (full → delta is smooth).
• No silent failure modes: every wrong-mode access becomes an explicit error.
• All checkpoint consumers (threads API, branching, regeneration, compaction,
  memory, goal, rollback, client) behave identically in both modes.

Non-goals

• Performance benchmarking (upstream numbers above; local profiling is
  follow-up — snapshot_frequency stays at the upstream default).
• A delta → full conversion tool (the gate error message names it; building it
  is out of scope here).
• Delta channels for other append-only fields (artifacts, etc.) until their
  reducers are proven batch-invariant.

Design

1. Mode selection and process freeze

checkpoint_channel_mode in config.yaml (default full, warn-once when
unset on an LLM-configured deployment). make_lead_agent resolves and freezes
the mode before compiling the graph
(runtime/checkpoint_mode.py::freeze_checkpoint_channel_mode). The mode is
baked into the compiled graph's channel schema, so it is restart-required:
a second, different mode inside one process raises
CheckpointModeReconfigurationError. Switching = edit config + restart.

2. Mode-matched schemas

agents/thread_state.py::get_thread_state_schema(mode) returns ThreadState
(full: messages = add_messages) or DeltaThreadState (delta: messages =
DeltaChannel(merge_message_writes, snapshot_frequency=1000)).
adapt_state_schema_for_mode / normalize_middleware_state_schemas keep
middleware-contributed state on the same channel types, so agent graph and
middleware channels never disagree.

3. Markers and the fail-closed gate

Every write injects the mode into the config (inject_checkpoint_mode);
LangGraph threads it into checkpoint metadata. Delta checkpoints carry
deerflow_checkpoint_channel_mode: "delta"; full checkpoints carry nothing,
so pre-feature databases need no migration. Detection
(checkpoint_tuple_uses_delta) also honors upstream's
counters_since_delta_snapshot.messages marker.

Compatibility is asymmetric and enforced before every state access:

reader \ data	full checkpoint	delta checkpoint
full-mode process	✅	❌ CheckpointModeMismatchError → HTTP 500
delta-mode process	✅ (seeds delta transparently)	✅

A full-mode raw read of a delta checkpoint would silently return empty/partial
messages (sentinel blobs) — the gate converts that into a loud startup/access
error with an actionable message. An explicit config marker takes precedence
over ambient context values.

4. CheckpointStateAccessor — the single choke point

runtime/checkpoint_state.py. Binds graph + checkpointer + mode; every
get/update/history call injects the marker and passes the gate first.
Consumers never call the checkpointer directly for thread state. Gateway
services.py builds it for the threads router; harness consumers (compaction,
run worker, client) receive it. Materialization happens inside LangGraph's
channel machinery (the graph carries the mode-matched schema); the accessor
exists so that no consumer can skip the gate or the schema binding. History
contract: limit=0 means zero items (explicit empty), never passed through to
get_state_history (where 0 means "unlimited" upstream).

5. State-only mutation graphs + Overwrite

Wholesale messages replacement (run rollback, context compaction) cannot go
through update_state plainly: values pass channel reducers (add_messages
merge in full, append in delta), and writing via an agent node schedules
pending tasks. build_state_mutation_graph(as_node, mode) compiles a
single-no-op-node graph (entry = finish) sharing the agent graph's checkpoint
machinery; writes with {"messages": Overwrite([...])} replace the channel
wholesale and leave an idle head. Hand-rolled checkpoint writes
(checkpointer.aput with hand-spliced blobs/versions/metadata) were removed —
context compaction shrank from a shadow checkpoint writer to ~17 lines of
mechanism-reusing code.

6. Writer parentage repair

Raw aput writers (run-duration updates, interrupted title writes, goal
writes) previously minted checkpoints with broken parentage, which severs the
ancestor chain delta replay depends on. Writers now thread the parent
checkpoint through, keeping lineage intact for both modes.

7. Saver patches

checkpoint_patches.py (package root): delta-history folding for
InMemorySaver delegating to the base walk (fixes an upstream override that
dropped the first post-migration write), and stable message IDs across
materialization so UI keys and message-id dedupe don't churn between reads.

Failure modes converted (silent → loud)

Scenario	Before	After
full process opens delta DB	silent empty/partial state	CheckpointModeMismatchError at first access
wrong-mode write	corrupt checkpoint	gate raises before write
hot mode switch in one process	two channel schemas sharing storage	CheckpointModeReconfigurationError
rollback via update_state	messages merged/appended, agent re-triggered	Overwrite + mutation graph, idle head
compaction in delta mode	summarized a sentinel blob, wrote full blob into delta thread	materialized read + mutation-graph write
broken writer parentage	delta replay chain severed	parent threaded through all writers

Risks and mitigations

• Read-side replay cost: delta get_state folds up to
  snapshot_frequency (1000) write batches; history walks touch more rows.
  Acceptable for the target workload (long threads are write/storage-bound);
  profiling is follow-up.
• Dual-mode operational complexity: mitigated by the gate + markers; every
  mixed-mode scenario errors loudly with guidance.
• Upstream beta API: DeltaChannel is marked beta; on-disk format is
  expected to stay readable. The gate contains any format change to reads, and
  schema adaptation is centralized in one module.
• Custom compaction audit metadata removed: removed/preserved counts and
  summary sha256 are no longer stamped into checkpoint metadata (still
  returned to the API caller); checkpoint metadata is now upstream-maintained.

Test evidence

• tests/test_checkpoint_mode.py — freeze, detection, sync/async gate,
  mismatch errors
• tests/test_checkpoint_state.py — accessor, mutation graph
• tests/test_delta_channel_checkpointers.py — saver parity
• tests/test_threads_checkpoint_mode.py, tests/test_gateway_checkpoint_mode.py
  — dual-mode e2e parity (state/history/branch/compress/regenerate, memory +
  sqlite)
• tests/test_context_compaction.py — materialized read, Overwrite write,
  real mutation graph finishes without scheduling
• tests/test_run_worker_rollback.py — capture, restore, pending-write replay,
  both modes
• Full suite: 277 passed; make lint clean.

Rollout

1. Merge behind the default full mode — zero behavior change for existing
   deployments.
2. Opt in via checkpoint_channel_mode: delta + restart. Existing threads
   continue; new writes go delta.
3. Follow-ups: local storage/latency benchmark vs full mode,
   snapshot_frequency tuning, delta → full conversion tool, DeltaChannel for
   other append-only fields after batching-invariance proofs.

Open questions

1. Should new deployments default to delta once local benchmarks confirm
   read-side overhead is negligible?
2. Do we want an offline delta → full conversion command for downgrade
   drills, or is "start a new thread" acceptable operationally?
3. Which fields graduate next (artifacts merge is associative; todos,
   skill_context need batching-invariance proofs)?

Affected area(s)

Agents / LangGraph (graph, prompts, langgraph.json), Backend API (gateway / endpoints / SSE), Config / setup, Docs

Alternatives considered

No response

Additional context

No response

[WillemJiang]

@Vanzeren, I just did some research about the DelaChannel. As it is still in delta status, I suggest we don't enable it by default. We can provide a configuration that the user can enable themselves.

[Vanzeren]

@WillemJiang Excatly, I agree with that. When I implement this feature, I define 2 modes('full' and 'delta') in configurations. "full" means the standard mode of checkpoint storage while 'delta' means delta-channel. The default one is still 'full'.