[Core] Refactor omni prefix cache into Manager/Controller

[ZhengWG]

Purpose

Related to #6039

Replace the monolithic vllm_omni/core/prefix_cache.py with vllm_omni/core/prefix_cache/, split along vLLM v1/core lines:

• OmniPrefixCacheManager owns (slot, key) occupancy, hit spans, and merge.
• OmniPrefixCacheController owns staging D2H, copy queues, and scatter into PrefixBlockPool.
• The state lock covers those tables only — never a join, cap flush, or copy.
  Writes are a key split, not a token-count split: hidden / non-deferred mm use JOIN_NEXT_STEP (whole-step D2H at save_outputs); deferred_keys use JOIN_ON_FINISH. Each save_outputs returns a step_id that must be consumed exactly once (materialize XOR discard_step). materialize may run on the async output builder after the engine has entered the next step, so prefix cache and async Omni output can run together.
  Miss is not an error. A hit span that resolves to absent slots, a leaked step_id, or a poisoned save fails fast (OmniPrefixCacheUnmatchError). Abort still writes once a hash is in this step's batch.

cc @amy-why-3459 @alex-jw-brooks @hsliuustc0106

Test Plan

Hardware: 2×H20 pod. Thinker on cuda:0; Talker + Code2Wav on cuda:1; 3 engine processes.
Serve: vllm serve <model> --omni --deploy-config <yaml> --port 8000. One independent boot per ladder level. Ready = HTTP 200 and 3 engines.
Load (same for text and audio): random dataset, --random-input-len 1024 --random-prefix-len 512 --random-output-len 128, 40 prompts, --max-concurrency 8, --ignore-eos --seed 1234, request_rate=inf. --extra-body '{"max_tokens":128}' on the text path (serve currently ignores max_completion_tokens).
Text: vllm-omni bench serve --backend openai-chat --endpoint /v1/chat/completions
metrics: ttft,tpot,itl,e2el at p50/p90/p99.
Audio (full 3-stage): vllm-omni bench serve --backend openai-chat-omni with
--extra-body '{"modalities":["text","audio"],"audio":{"voice":"Chelsie","format":"wav"}}'
metrics: same plus audio_ttfp. E2EL includes the audio stream. TPOT/ITL are the text stream under audio mode and are not comparable to the text-only table.
Rounds / “r2–4 median”:

1. Warmup boot: seed 9999, 8 prompts (not reported).
2. Four measured rounds with seed 1234 (identical prompts).
3. Drop r1 (cold cache / ramp). For each metric, take the median of r2, r3, r4.
4. Per-round audit: input tokens = 61760, output tokens = 5120 (40×128), completed = 40. Inter-round noise ~±1.5%.
   prefix512 is ~33% shared-prefix hit inside a round. With a fixed seed, r2+ is a full-prompt hot cache; L4 numbers in the tables are that hot median, not a cold start. Cold vs hot TTFT is reported separately.
   Ladder: L0 = sync + no chunk + no cache + eager → L1 +async_sched → L2 +async_chunk → L3 +async_omni_output → L4 +prefix_cache (prod yaml).
   Accuracy: golden 5 prompts byte-identical; tests/core/test_prefix_cache.py.

vLLM Version: 0.26.0
vLLM-Omni Commit: 1ca0dfd46acbd442f3481d59a93c76a88c858415

Test Result

Accuracy: golden 5 prompts ≡ baseline (eager ×3, async builder ×2, default tree ×2; §6 2-pass). Unit 38/38; pod tests/core 134/134. No staging exhaustion, no UnmatchError.

Text (r2–4 median)

	L0	L3	L4 (prod)
tok/s	111.2	103.8	104.9
TTFT p50 / p90 (ms)	120.4 / 736	121.3 / 662	45.5 / 94
TPOT p50 (ms)	65.4	60.8	61.0
E2EL p50 (s)	8.64	9.25	9.15

L4 vs L3: tok/s +1.1%, TPOT +0.3%, TTFT p50 −62%, p90 −86%.
L4 cold (33% prefix hit) TTFT p50 121.9 / p90 545.7; hot (full hit) 45.5 / 94.2.

Audio (full 3-stage, r2–4 median)

	L0	L2	L3	L4
TTFP p50 / p90 (ms)	8542 / 9638	392 / 1035	366 / 1010	292 / 459
audio-s/s	35.0	30.6	33.4	33.4
text tok/s	115.0	95.9	105.5	104.9
E2EL p50 (s)	8.55	10.29	9.33	9.15

L4 vs L0: TTFP p50 8542 → 292 ms (−97%), TTFT p90 −87%, E2EL/throughput roughly flat.

BEFORE SUBMITTING: read CONTRIBUTING.md and run the precheck-pr skill with the code agent for a self-check against project conventions.
(anything written below this line will be removed by GitHub Actions)

[chatgpt-codex-connector]

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

[vllm-omni-review-bot]

This PR appears to belong to: docs/design/module/cache_management.md.

Module owners: @Isotr0py @princepride @SamitHuang

@ZhengWG, please review your own changes and leave a short self-review comment describing what you checked. PRs without author self-review may not be assigned a reviewer.

Please take a look when you have a chance. If you would like an automated review, mention @vllm-omni-review-bot in a comment.

[hsliuustc0106]

This PR appears to belong to: docs/design/module/cache_management.md.

Module owners: @Isotr0py @princepride @SamitHuang

@ZhengWG, please review your own changes and leave a short self-review comment describing what you checked. PRs without author self-review may not be assigned a reviewer.

Please take a look when you have a chance. If you would like an automated review, mention @vllm-omni-review-bot in a comment.

this seems not correct, the related maintainers are @amy-why-3459 @yuanheng-zhao @Gaohan123 @tzhouam