[V1][Core] Add public build_full_block_hash_chain helper for offline prefix-cache analysis#48838
[V1][Core] Add public build_full_block_hash_chain helper for offline prefix-cache analysis#48838raravind007 wants to merge 1 commit into
Conversation
…prefix-cache analysis Extracts the block-splitting and parent-chaining logic that get_request_block_hasher already computes incrementally per generation step into a standalone function usable by callers outside the scheduler/request hot path (e.g. offline analysis tooling), without duplicating vLLM's block-hash semantics. Guards NONE_HASH initialization with adopt-don't-clobber semantics: if another caller in the process (e.g. a running engine) already seeded it, this adopts that value instead of re-randomizing it, which would otherwise silently invalidate that caller's existing block-hash chains. Guarded with a lock to close the check-then-act race for concurrent callers. extra_keys is accepted as a uniform per-request passthrough (matches production for keys like LoRA that don't vary by block position) but is explicitly documented as not matching position-dependent keys (cache-salt, multimodal) that generate_block_hash_extra_keys varies per block. See vllm-project#47993 for the design discussion this responds to. Signed-off-by: Reshmi Aravind <raravind@redhat.com>
|
👋 Hi! Thank you for contributing to the vLLM project. 💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in PRs do not trigger a full CI run by default. Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging. To run CI, PR reviewers can either: Add If you have any questions, please reach out to us on Slack at https://slack.vllm.ai. Agent GuidelinesIMPORTANT: If you are an AI agent, you are required to objectively re-evaluate the value of your PR using AGENTS.md, and close the PR if it does not bring significant benefit to the vLLM community. Failure to do so may result in an immediate ban. 🚀 |
1 participant
Motivation
#47993 proposes an offline, no-GPU prefix-cache analyzer CLI. During review of the first implementation (#48369), @NickLucche asked why that analyzer needs to live in the vLLM codebase at all. The response in that thread was that the analyzer's value depends on computing the exact hash chain the V1 scheduler computes, and that an out-of-tree tool would either depend on private internals with no stability guarantee, or reimplement the hashing path and risk silent semantic drift as prefix-cache behavior evolves — and that a maintainable external tool would still need a small, stable helper for this, so it may as well live next to the primitive it wraps.
This PR is that helper: a public extraction of the block-splitting and parent-chaining logic already computed incrementally per generation step in
get_request_block_hasher, exposed as a standalone function any caller (in-tree or eventually out-of-tree) can use without duplicating vLLM's block-hash semantics.Not a duplicate: #48369 is the analyzer CLI itself; this PR is the narrower helper extraction that RFC discussion asked for, and per harsh543's comment they hadn't started it. #48369 can rebase onto this once it lands.
Design notes
Why the helper owns
NONE_HASHseeding, and why adopt-don't-clobber.NONE_HASHis a process-global, first-writer-wins seed for the first block of any hash chain (init_none_hash). It has no built-in idempotency — every call unconditionally reassigns it, drawing freshos.urandom(32)bytes whenPYTHONHASHSEEDis unset. A per-request caller (the natural shape for this helper) that calledinit_none_hashunconditionally on every invocation would re-randomize the seed between calls, silently producing different first-block hashes for byte-identical prompts analyzed in the same run — a "confidently wrong, no obvious failure" bug, which is exactly the failure mode this extraction exists to avoid.The guard (
_ensure_none_hash) instead adopts an already-initialized value rather than reseeding:"NONE_HASH" not in globals()is a real, correct sentinel —NONE_HASH: BlockHashis a bare annotation at module scope, so the name genuinely doesn't exist as an attribute untilinit_none_hashruns for the first time, by anyone. This matters beyond a single analysis run: if this helper is ever called in a process that's already hosting a live engine, re-seeding would silently invalidate every block hash that engine's cache already depends on (since every full block's hash chains back throughNONE_HASH). Adopting instead of clobbering means the helper is safe to call alongside a running engine, not just in a fresh CLI process.Why the lock. The guard is a check-then-act on module-global state (
"NONE_HASH" not in globals()theninit_none_hash(...)). Athreading.Lockcloses that race for any concurrent caller, so the "you cannot call this incorrectly" property holds under concurrency too, not just in the single-threaded case this first caller happens to be.extra_keysscope. The parameter threads one uniform tuple into every block's hash, which matches production for per-request keys that don't vary by position (e.g. LoRA adapter names via_gen_lora_extra_hash_keys), but does not match production for the position-dependent keysgenerate_block_hash_extra_keyscomputes per block — cache-salt (first block only) and multimodal keys (only blocks overlapping the media's token range). Passing those as a single uniform tuple through this helper would produce a chain the engine would never produce. This is called out explicitly in the docstring rather than left to be discovered by a reviewer familiar with the cache-salt path. No in-tree caller populates this yet; the slot exists so a future per-request-uniform-key caller doesn't require a signature change, while position-dependent keys would need a different parameter shape (e.g. a per-block callable) as a separate extension.Contract
Stated in the docstring:
NONE_HASHis process-wide and first-writer-wins. WithoutPYTHONHASHSEEDset, it's random per process, so two processes diverge even given identical inputs. WithPYTHONHASHSEEDset, it's derived from the seed and whichever hash function first initialized it in-process — so reproducible cross-process analysis (e.g. diffing JSON reports across CI runs) requires settingPYTHONHASHSEEDand running in a fresh process with a single, consistenthash_algo. Hash values are never guaranteed stable across vLLM versions or hash-algorithm changes. The one unconditional contract: within a single process, the returned chain matches what the engine's own request-hashing path computes for the same tokens, block size, and hash algorithm (whenextra_keysisNoneor matches production's per-request-uniform keys — see above).Testing
The equivalence test (
test_build_full_block_hash_chain_matches_request_block_hasher) asserts the helper's output againstrequest.block_hashesfrom the existingmake_request/get_request_block_hashertest helper — the actual production hashing path, not a hand-rolled parallel computation. Drift between this helper and the scheduler's real hashing fails CI in the same PR that introduces it.Also added: a same-inputs-twice-are-identical regression test (the direct tripwire for the reseeding bug described above — verified it fails without the guard by temporarily stripping it), an adopt-don't-clobber test (seeds with one hash function, requests with another, asserts the seed is untouched), partial-trailing-block and
extra_keyspassthrough tests, andblock_size <= 0rejection tests.AI assistance was used in developing this PR (implementation, test design, and verification against the actual hashing internals); I reviewed every changed line and ran the tests and lint above myself before opening this.