(wip)feat(beacon-node): add persistent checkpoint state cache - #466
Draft
GrapeBaBa wants to merge 26 commits into
Draft
(wip)feat(beacon-node): add persistent checkpoint state cache#466GrapeBaBa wants to merge 26 commits into
GrapeBaBa wants to merge 26 commits into
Conversation
Start the `beacon_node` module by porting two of lodestar's state-cache pieces under beacon_node/chain/state_cache: - block_state_cache: FIFOBlockStateCache port — a head-pinned FIFO over recent post-states keyed by state root, with read-tracking metrics (size / reads / seconds-since-last-read) and a stable seed state. - cp_datastore: the checkpoint-state byte datastore (file + in-memory backends) the persistent checkpoint cache persists to; the file backend fans batch removal out over `io.async`. - key: checkpoint <-> datastore-key (0x-hex filename) encoding. - metrics: block state-cache metric definitions. Adds the `beacon_node` build module and a `test:beacon_node` CI step. The persistent checkpoint cache (and its metrics) follow in a stacked PR.
Resolve append-list conflicts in build.zig.zon (clock + beacon_node test modules) and CI.yml (clock + beacon-node test steps) after the clock module (#354) merged to main.
Review findings on #452, all independently re-verified (no blocking): - block_state_cache: add() returns the canonical resident state; the duplicate path used to destroy the caller's pointer while the documented add-then-setHeadState flow reused it (use-after-free / double free). get() now documents the no-in-place-mutation contract. - block_state_cache: getSeedState returns null on an empty cache (legal via the debug-API clear() or cold-start) instead of asserting. - block_state_cache: recordRead/scanReadStats use the shared time helpers instead of hand-rolled timestamp/seconds conversions. - cp_datastore: isSafeCheckpointState validates the on-disk epoch via the division inverse before the boundary multiply; a huge foreign epoch used to overflow and panic at boot instead of reading as unsafe. - cp_datastore: writes go temp-then-rename (per-key '<name>.tmp'), so a final name only ever holds complete bytes and a torn write can no longer poison its key; initStore sweeps stale temp debris, tolerant of foreign entries, and closes any prior dir handle before reopening (fd leak). - cp_datastore: readKeys accepts .unknown directory entries (d_type-less filesystems report regular files as unknown, hiding every persisted state); read maps IsDir to absent. - cp_datastore: readLatestSafe scans candidates by a 72-byte prefix read (new readPrefix vtable primitive) and full-reads only the winner; the old scan read entire multi-hundred-MB states per rejected candidate. - metrics: the init test restores the module-global noop metrics on exit, so later tests in the same binary don't observe leaked gauge state. - ssz_bytes: the state byte readers are now tested against real SSZ serialization of electra/phase0 states (the previous fixture hand-wrote the same offset constants, so a wrong constant passed); the module's tests are now aggregated by the state_transition test root. - build.zig.zon: drop unused .config/.fork_types beacon_node imports; add .time. Final files written by the pre-fix code are not repaired retroactively: a truncated final left by an old torn write is still trusted by the exists check. No deployed data dirs exist, so no migration is needed. test:beacon_node 94/94, test:state_transition 111/111.
Port lodestar's PersistentCheckpointsCache under beacon_node/chain/state_cache, completing the state-cache stack started in #452: - checkpoint_state_cache: the two-tier (memory + disk) checkpoint state cache — add/get/getLatest, getOrReload with block-cache-seeded reload, per-epoch persist/prune driven by processState, finality + disk-bound pruning, read-tracking, and the debug dump APIs. Suspension interleaves (adds racing a persist write or reload read) resolve last-writer-wins and are tested with real zio suspensions. - util/buffer_pool: single grow-only serialization buffer with a busy -> fresh-alloc fallback, wired into the persist and reload-validators paths, plus its lodestar_buffer_pool_* metrics. - metrics: the full checkpoint-cache metric set alongside the block cache's. - fork_types: serializedSize/serializeIntoBytes (+ validators variants) on AnyBeaconState for pool-backed serialization. - state_transition: loadOtherState reload entry point; the test-utils state factory gains an Options parameter (electra fork epoch override) so the cache tests run on the TS test fixture's epoch axis. Adds the zio dependency to the beacon_node build module.
Its assertions (re-add over a persisted entry yields in_memory with the disk key carried) are pinned verbatim by the re-added on-disk boundary test's re-add step.
loadOtherState's preload warmed the validator/balance views via getAllReadonly/getAll and freed the results — on mainnet ~16 MB of element pointers plus ~16 MB of decoded balances allocated and discarded per reload. Add void-returning prefetchAll to the list views (composite: the getReadonly loop that warms the per-element view cache; basic: populateAllNodes, a no-op under chunked_leaf where bulk reads cache nothing) and switch the preload to it. The balances half was in fact warming nothing (chunked-leaf getAll decodes into the output without populating any cache), so its decode pass is dropped along with the allocation.
- Rename the global `cache_metrics` to `block_metrics` so it reads unambiguously against the checkpoint cache's metrics in the stacked follow-up. - Reword add()'s doc to describe the state handoff as an ownership transfer (transfers on success; stays with the caller on hashTreeRoot failure), matching the ownership vocabulary used elsewhere.
Use the reviewer's suggested block_cache_metrics — it reads as "the block cache's metrics" and matches the BlockStateCacheMetrics type; the stacked checkpoint cache follows with checkpoint_cache_metrics for symmetry.
Rename block_metrics -> block_cache_metrics and checkpoint_metrics -> checkpoint_cache_metrics so both read as "the <X> cache's metrics" and stay a symmetric pair.
The cache holds its size flat (every add evicts once full), so std.HashMap's grow() - the only point that clears tombstones - never fires, and tombstone-based deletion degrades every probe permanently under the add/evict churn (ziglang/zig#17851; the std rehash() doc names exactly this long-lived insert+delete pattern). ArrayHashMap's index deletes by backward shift and cannot accumulate tombstones; map iteration order is unused (both orderings live in the intrusive lists), so eviction uses swapRemove. Measured with an isolated ordering microbenchmark (landing separately): the HashMap variant settles at ~10x its clean probe cost at the real N=64, the ArrayHashMap variant stays flat. Both are far below the per-add state clone+rehash cost - this removes a structural degradation, not a user-visible slowdown.
prefetchAll's chunked_leaf branch was a no-op on the assumption that those lists are only bulk-read. Balances are read AND written per-element during block processing (increase/decreaseBalance -> balances.get(i)/set(i)), so a reloaded state does pay the lazy per-access navigation on first touch of each leaf. Populate children_nodes at chunked_leaf_depth via one batch getNodesAtDepth: one entry per chunked leaf (each serving K*items_per_chunk elements, ~6k entries for 1.5M balances), so a later per-element get/set skips the O(depth) walk from the root. getAllInto is unaffected (it batches its own walk and does not read this cache). Sub-noise against block processing's BLS/hash cost, but it makes the preload actually warm what block replay touches.
The earlier commit made chunked_leaf prefetchAll warm children_nodes for balances' per-element block-processing access. A microbench (bench/beacon_node/prefetch_effect.zig, kept on a bench branch) shows it is a net loss at the realistic access size: the eager full-list warm costs a fixed ~80us (batch walk of all ~5860 leaves), while a lazy navigation is ~80ns, so eager only wins past ~1024 touched leaves. A block touches ~540 balances (sync committee + proposer + deposits + withdrawals), below the crossover, and the epoch bulk read (getAllInto) ignores this cache entirely. So chunked_leaf prefetch returns to a no-op and loadOtherState no longer prefetches balances. Validators keep the prefetch — epoch processing reads every validator per-element, far past the crossover.
…oint-cache # Conflicts: # build.zig.zon # src/beacon_node/chain/state_cache/block_state_cache.zig # src/beacon_node/chain/state_cache/metrics.zig # src/beacon_node/chain/state_cache/root.zig # src/beacon_node/root.zig
The Options I added were electra-bound (electra_fork_epoch + a hardcoded
generateElectraState), so testing another fork (fulu/gloas) would need
an API change. Mirror TS's generateState(opts, getConfig(fork, forkEpoch))
usage instead:
- generateElectraState -> generateState(comptime fork, ...): common
BeaconState fields for all forks, then the altair+ additions
(participation, inactivity scores, sync committees) under a comptime
fork guard. generateElectraState stays as a thin wrapper.
- Options { fork: ForkSeq = .electra, fork_epoch: ?Epoch = null };
init dispatches every fork via an inline-else switch. Default electra +
null epoch keeps all existing callers unchanged; the cp-cache tests move
from .electra_fork_epoch = 0 to .fork_epoch = 0.
All 8 forks (phase0..gloas) instantiate and compile; adding a future fork
is one generateState arm, no API change.
- Remove cross-impl "mirror TS" / "port of TS" references from code comments (they belong in commit messages, not the source) on generateState, Options.fork, and getConfig. - Trim loadOtherState's ownership-narration what-comments; the errdefer/null idiom speaks for itself. - Delete the now-unused ElectraBeaconState import left by the fork-generic refactor.
initFromState split ownership of `state`: the caller held an errdefer to free it, but createCachedBeaconState takes ownership partway through, with no errdefer after it. If a later step (epoch_transition_cache create/init) failed, cached_state leaked and the caller's errdefer freed the state cached_state now owned. Make initFromState own `state` end to end (var owned_state, freed on early failure) and hand off to a cached_state errdefer once createCachedBeaconState succeeds — the same pattern loadOtherState already uses. Test-only path (OOM during test setup); no production impact.
verify-leanness sweep of the PR found residual cross-impl leakage in code comments: a mekong-devnet / lodestar#7255 provenance URL on the untrusted-checkpoint removal (the behavioral WHY is already stated above it), "lodestar debug API" on two debug helpers, and "TS axis" on three test fixtures (the load-bearing WHY is "fork-agnostic, arbitrary absolute epochs are fine"). Provenance/cross-impl belongs in commit messages, not the source. Also drop a doubled lead line on prefetchAll's doc. The untrusted-checkpoint removal at cp_datastore-removeMany was found on mekong devnet (lodestar#7255).
TestCachedBeaconState.init's default (fork=electra, fork_epoch=null) resolved the epoch to the active config's *_FORK_EPOCH. Under the minimal preset that is FAR_FUTURE_EPOCH (maxInt) for post-altair forks, so generateState's `activation_epoch * SLOTS_PER_EPOCH` overflowed and panicked — 53 crashes across test:state_transition -Dpreset=minimal (38) and test:beacon_node -Dpreset=minimal (15). Pre-existing (the old generateElectraState had the same arithmetic), but the fork-generic factory makes it reachable for any fork and the minimal gate can't run these modules. Resolve an unscheduled fork's epoch to genesis (0) instead — it then activates at epoch 0 (all priors too), giving a low-slot state of that fork. Mainnet is unchanged (electra is scheduled). Crashes go to zero; the remaining minimal failures are pre-existing test-side preset assumptions (mainnet SLOTS_PER_HISTORICAL_ROOT window), not defects.
GrapeBaBa
marked this pull request as draft
July 9, 2026 08:27
Measured on the checkpoint reload path (10k validators, electra, ReleaseFast): preloading the validators view costs +33ms on a 51.8ms reload (+64%; ~+3.3s extrapolated to mainnet's 1M validators) while the warmed children_data cache is never read by the epoch path — it stays empty through a full processEpoch, because validatorsPtrSlice iterates the tree directly. Sparse per-element block reads save only ~4.7us each; break-even needs a ~70% table touch rate, real block touch rate is <0.1%. TS's preloadValidatorsAndBalances pays off because ViewDU's this.nodes serves both sparse gets and getAll full scans; our bulk paths use per-call flat scratch / pointer iteration, so a warmed navigation cache has no bulk consumer. With the only consumer gone, prefetchAll (list_basic/list_composite/ chunks) and LoadOtherStateOpts are removed. The errdefer after the heap_state disarm guarded only the deleted preload try and was dead.
Fix four comments that contradicted the code: the file doc referenced shouldReload/reload()/updateHeadState (no such APIs here — renamed to the local getOrReload split), get was described as memory-blind (it reads only memory, i.e. disk-blind), make()'s doc described makeWithBlockRoot's root stamping, and one comment referenced a discarded top_k design. Strip session vocabulary a cold reader cannot resolve (Model B, spine, S1, anti-wedge swallow, vanish re-resolve, non-vacuous guard, dynamic path), fix ASCII diagram axis labels to the real field names, dedupe comments restated from nearby fn docs (persist throttle, skip-and- continue, PULL-gauge contract), and drop what-comments, changelog narration, a thread-safety note, and two nonsensical read-only caveats on serializers. Delete dead commented-out code in generate_state. Comment/test-name changes only; no executable code touched.
…, memset initFromState reverts to success-only ownership transfer (the repo-wide contract): the owned_state errdefer freed the passed state on failure while the spec-test callers still held their own deinit/destroy errdefers, a double-free on any setup error. A partial-teardown errdefer (CachedBeaconState.deinit minus the state parts) now covers the window after createCachedBeaconState without touching the caller's state; initForFork cleans its generated state itself. pruneFinalized iterates an owned snapshot of the epoch keys: its backwards-cursor walk only accounted for its own swapRemoves, so a concurrent prune shrinking epoch_index during the datastore-remove suspension could leave the cursor out of bounds. acquireScratch uses allocUnsafe: both call sites lease the exact serialized size and immediately overwrite every byte, so alloc's zeroing was a wasted full pass over multi-hundred-MB buffers (and the non-pool fallback never zeroed, proving it unneeded). Cleanups: metrics.write/scrape now include the buffer-pool series (registry parity — TS expositions include them); beaconStateSsz replaced by the existing ForkTypes(fork).BeaconState; dead AnyBeaconState.serializeValidators and the dead ANY_BEACON_STATE_SLOT_OFFSET re-export removed; one changelog-style comment reworded to present tense. Not applied, deliberately: the pubkey-registration loop exists only in TS's legacy zero-caller loadCachedBeaconState — the live reference (beaconStateView.ts loadOtherState) drops modifiedValidators exactly as we do, with a dated rationale comment; the insertEntry/removeEntry full-epoch assert loops are intentional coherence checks.
GrapeBaBa
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Port lodestar's
PersistentCheckpointsCacheunderbeacon_node/chain/state_cache, completing the state-cache stack started in #452 (stacked on it):processState, finality + disk-bound pruning, read-tracking, and the debug dump APIs. Suspension interleaves (adds racing a persist write or reload read) resolve last-writer-wins and are tested with real zio suspensions.lodestar_buffer_pool_*metrics.serializedSize/serializeIntoBytes(+ validators variants) onAnyBeaconStatefor pool-backed serialization.loadOtherStatereload entry point; the test-utils state factory gains anOptionsparameter (electra fork epoch override) so the cache tests run on the TS test fixture's epoch axis.Adds the zio dependency to the
beacon_nodebuild module.