[Bugfix][KV Offloading] Handle queued request aborts without allocated KV blocks by chaunceyjiang · Pull Request #49146 · vllm-project/vllm

chaunceyjiang · 2026-07-20T03:48:24Z

Purpose

Handle queued request aborts without allocated KV blocks

Test Plan

CUDA_VISIBLE_DEVICES=4,5,6,7 \
  vllm serve /mnt/data3/models/Qwen/Qwen3.6-35B-A3B \
    --served-model-name Qwen3.6-35B-A3B \
    --host 0.0.0.0 \
    --port 8840 \
    --tensor-parallel-size 4 \
    --language-model-only \
    --dtype bfloat16 \
    --max-model-len 180000 \
    --max-num-seqs 4 \
      --max-num-batched-tokens 4224 \
    --gpu-memory-utilization 0.90 \
    --kv-cache-dtype fp8_e4m3 \
    --enable-prefix-caching \
    --enable-chunked-prefill \
    --reasoning-parser qwen3 \
    --kv-transfer-config '{
      "kv_connector": "OffloadingConnector",
      "kv_role": "kv_both",
      "kv_connector_extra_config": {
        "cpu_bytes_to_use": 8589934592
      }
    }'


import json
import socket
import sys
import time
import urllib.error
import urllib.request
from concurrent.futures import ThreadPoolExecutor

BASE_URL = "http://127.0.0.1:8840"
MODEL = "Qwen3.6-35B-A3B"

PROMPT_CHARS = 220_000
REQUEST_TIMEOUT = 900


def make_prompt(seed: str, chars: int = PROMPT_CHARS) -> str:
    fragment = f"[{seed}] filler text for a long queued request. "
    return (fragment * (chars // len(fragment) + 1))[:chars]


def chat(prompt: str, timeout: float = REQUEST_TIMEOUT) -> dict:
    payload = {
        "model": MODEL,
        "messages": [{"role": "user", "content": prompt}],
        "max_tokens": 16,
        "temperature": 0,
    }
    request = urllib.request.Request(
        f"{BASE_URL}/v1/chat/completions",
        data=json.dumps(payload).encode(),
        headers={"Content-Type": "application/json"},
        method="POST",
    )
    with urllib.request.urlopen(request, timeout=timeout) as response:
        return json.load(response)


def abort_queued_request(prompt: str) -> str:
    try:
        chat(prompt, timeout=1.0)
    except (
        urllib.error.URLError,
        socket.timeout,
        TimeoutError,
        ConnectionError,
    ) as exc:
        return f"aborted ({type(exc).__name__})"
    return "unexpectedly completed"


def check_health() -> None:
    with urllib.request.urlopen(f"{BASE_URL}/health", timeout=5) as response:
        if response.status != 200:
            raise RuntimeError(f"health check returned {response.status}")


def main() -> int:
    check_health()
    print("Server is healthy; starting queued-abort reproduction.")
    busy_errors = []
    # max-num-seqs=4:

    with ThreadPoolExecutor(max_workers=8) as executor:
        busy = [
            executor.submit(chat, make_prompt(f"busy-{index}")) for index in range(5)
        ]
        time.sleep(0.5)
        aborted = [
            executor.submit(
                abort_queued_request,
                make_prompt(f"abort-{index}"),
            )
            for index in range(3)
        ]
        for index, future in enumerate(aborted):
            try:
                print(f"abort-{index}: {future.result()}")
            except Exception as exc:
                print(f"abort-{index}: client error: {exc!r}")

        for index, future in enumerate(busy):
            try:
                future.result()
                print(f"busy-{index}: completed")
            except Exception as exc:
                busy_errors.append(exc)
                print(f"busy-{index}: failed: {exc!r}")
    time.sleep(1)
    try:
        check_health()
    except Exception as exc:
        print()
        print("REPRODUCED: EngineCore is no longer healthy.")
        print(f"Health error: {exc!r}")
        return 1
    if busy_errors:
        print()
        print("Server is healthy, but one or more in-flight requests failed.")
        return 2
    final = chat("Reply with only the digit 4: what is 2+2?", timeout=60)
    content = final["choices"][0]["message"]["content"]
    print()
    print(f"PASS: EngineCore survived. Final response: {content!r}")
    return 0


if __name__ == "__main__":
    sys.exit(main())

Test Result

before:

(EngineCore pid=909178) ERROR 07-20 11:15:01 [core.py:1332] EngineCore encountered a fatal error.
(EngineCore pid=909178) ERROR 07-20 11:15:01 [core.py:1332] Traceback (most recent call last):
(EngineCore pid=909178) ERROR 07-20 11:15:01 [core.py:1332]   File "/mnt/data4/jxy/vllm/vllm/v1/engine/core.py", line 1323, in run_engine_core
(EngineCore pid=909178) ERROR 07-20 11:15:01 [core.py:1332]     engine_core.run_busy_loop()
(EngineCore pid=909178) ERROR 07-20 11:15:01 [core.py:1332]   File "/mnt/data4/jxy/vllm/vllm/v1/engine/core.py", line 1364, in run_busy_loop
(EngineCore pid=909178) ERROR 07-20 11:15:01 [core.py:1332]     self._process_engine_step()
(EngineCore pid=909178) ERROR 07-20 11:15:01 [core.py:1332]   File "/mnt/data4/jxy/vllm/vllm/v1/engine/core.py", line 1403, in _process_engine_step
(EngineCore pid=909178) ERROR 07-20 11:15:01 [core.py:1332]     outputs, model_executed = self.step_fn()
(EngineCore pid=909178) ERROR 07-20 11:15:01 [core.py:1332]                               ^^^^^^^^^^^^^^
(EngineCore pid=909178) ERROR 07-20 11:15:01 [core.py:1332]   File "/mnt/data4/jxy/vllm/vllm/v1/engine/core.py", line 645, in step_with_batch_queue
(EngineCore pid=909178) ERROR 07-20 11:15:01 [core.py:1332]     scheduler_output = self.scheduler.schedule(self._should_throttle_prefills())
(EngineCore pid=909178) ERROR 07-20 11:15:01 [core.py:1332]                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=909178) ERROR 07-20 11:15:01 [core.py:1332]   File "/mnt/data4/jxy/vllm/vllm/v1/core/sched/scheduler.py", line 1170, in schedule
(EngineCore pid=909178) ERROR 07-20 11:15:01 [core.py:1332]     meta = self._build_kv_connector_meta(self.connector, scheduler_output)
(EngineCore pid=909178) ERROR 07-20 11:15:01 [core.py:1332]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=909178) ERROR 07-20 11:15:01 [core.py:1332]   File "/mnt/data4/jxy/vllm/vllm/v1/core/sched/scheduler.py", line 1192, in _build_kv_connector_meta
(EngineCore pid=909178) ERROR 07-20 11:15:01 [core.py:1332]     return connector.build_connector_meta(scheduler_output)
(EngineCore pid=909178) ERROR 07-20 11:15:01 [core.py:1332]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=909178) ERROR 07-20 11:15:01 [core.py:1332]   File "/mnt/data4/jxy/vllm/vllm/distributed/kv_transfer/kv_connector/v1/offloading_connector.py", line 157, in build_connector_meta
(EngineCore pid=909178) ERROR 07-20 11:15:01 [core.py:1332]     return self.connector_scheduler.build_connector_meta(scheduler_output)
(EngineCore pid=909178) ERROR 07-20 11:15:01 [core.py:1332]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=909178) ERROR 07-20 11:15:01 [core.py:1332]   File "/mnt/data4/jxy/vllm/vllm/distributed/kv_transfer/kv_connector/v1/offloading/scheduler.py", line 1157, in build_connector_meta
(EngineCore pid=909178) ERROR 07-20 11:15:01 [core.py:1332]     store_jobs=self._build_store_jobs(scheduler_output),
(EngineCore pid=909178) ERROR 07-20 11:15:01 [core.py:1332]                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=909178) ERROR 07-20 11:15:01 [core.py:1332]   File "/mnt/data4/jxy/vllm/vllm/distributed/kv_transfer/kv_connector/v1/offloading/scheduler.py", line 975, in _build_store_jobs
(EngineCore pid=909178) ERROR 07-20 11:15:01 [core.py:1332]     assert len(offload_keys) == len(offload_block_ids)
(EngineCore pid=909178) ERROR 07-20 11:15:01 [core.py:1332]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=909178) ERROR 07-20 11:15:01 [core.py:1332] AssertionError
(Worker_TP0 pid=911359) INFO 07-20 11:15:01 [multiproc_executor.py:793] Parent process exited, terminating worker queues
(EngineCore pid=909178) INFO 07-20 11:15:01 [multiproc_executor.py:429] [shutdown] Executor: waiting for worker exit count=4
(APIServer pid=905233) ERROR 07-20 11:15:01 [async_llm.py:704] AsyncLLM output_handler failed.
(APIServer pid=905233) ERROR 07-20 11:15:01 [async_llm.py:704] Traceback (most recent call last):
(APIServer pid=905233) ERROR 07-20 11:15:01 [async_llm.py:704]   File "/mnt/data4/jxy/vllm/vllm/v1/engine/async_llm.py", line 660, in output_handler
(APIServer pid=905233) ERROR 07-20 11:15:01 [async_llm.py:704]     outputs = await engine_core.get_output_async()
(APIServer pid=905233) ERROR 07-20 11:15:01 [async_llm.py:704]               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=905233) ERROR 07-20 11:15:01 [async_llm.py:704]   File "/mnt/data4/jxy/vllm/vllm/v1/engine/core_client.py", line 1061, in get_output_async
(APIServer pid=905233) ERROR 07-20 11:15:01 [async_llm.py:704]     raise self._format_exception(outputs) from None
(APIServer pid=905233) ERROR 07-20 11:15:01 [async_llm.py:704] vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause.
(APIServer pid=905233) INFO:     127.0.0.1:60860 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
(APIServer pid=905233) INFO:     127.0.0.1:60870 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
(APIServer pid=905233) INFO:     127.0.0.1:60884 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
(APIServer pid=905233) INFO:     127.0.0.1:60890 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
(APIServer pid=905233) INFO:     127.0.0.1:60892 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
(APIServer pid=905233) INFO:     Shutting down
(APIServer pid=905233) INFO:     Waiting for application shutdown.
(APIServer pid=905233) INFO:     Application shutdown complete.

after:

(APIServer pid=1344642) INFO:     127.0.0.1:48516 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1344642) INFO 07-20 11:42:12 [loggers.py:310] Engine 000: Avg prompt throughput: 11919.6 tokens/s, Avg generation throughput: 1.8 tokens/s, Running: 2 reqs, Waiting: 2 reqs, GPU KV cache usage: 0.6%, Prefix cache hit rate: 0.0%, External prefix cache hit rate:0.0%
(APIServer pid=1344642) INFO 07-20 11:42:12 [metrics.py:103] KV Transfer metrics: vllm:kv_offload_lookup_sync_delay_seconds_count=22, vllm:kv_offload_lookup_sync_delay_seconds_sum=0.00020857341587543488, vllm:kv_offload_cpu_cache_usage_perc=0.030226700251889168, vllm:kv_offload_cpu_allocation_size_count=35, vllm:kv_offload_cpu_allocation_size_sum=226, vllm:kv_offload_cpu_cache_write_usage_perc=0.030226700251889168, vllm:kv_offload_cpu_cache_read_usage_perc=0.0, vllm:kv_offload_store_bytes=4611932160, vllm:kv_offload_store_time=0.09642745602130891, vllm:kv_offload_store_size_count=132, vllm:kv_offload_store_size_sum=4611932160
(APIServer pid=1344642) INFO:     127.0.0.1:48530 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1344642) INFO:     127.0.0.1:48538 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1344642) INFO:     127.0.0.1:48554 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1344642) INFO:     127.0.0.1:48566 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1344642) INFO:     127.0.0.1:59012 - "GET /health HTTP/1.1" 200 OK
(APIServer pid=1344642) INFO:     127.0.0.1:59016 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1344642) INFO 07-20 11:42:22 [loggers.py:310] Engine 000: Avg prompt throughput: 17879.9 tokens/s, Avg generation throughput: 7.8 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%, External prefix cache hit rate:0.0%
(APIServer pid=1344642) INFO 07-20 11:42:22 [metrics.py:103] KV Transfer metrics: vllm:kv_offload_lookup_sync_delay_seconds_count=32, vllm:kv_offload_lookup_sync_delay_seconds_sum=0.00022306758910417557, vllm:kv_offload_store_bytes=6981058560, vllm:kv_offload_store_time=0.14641644805669782, vllm:kv_offload_store_size_count=212, vllm:kv_offload_store_size_sum=6981058560, vllm:kv_offload_cpu_cache_usage_perc=0.0, vllm:kv_offload_cpu_allocation_size_count=51, vllm:kv_offload_cpu_allocation_size_sum=312, vllm:kv_offload_cpu_cache_write_usage_perc=0.0, vllm:kv_offload_cpu_cache_read_usage_perc=0.0
(APIServer pid=1344642) INFO 07-20 11:42:32 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%, External prefix cache hit rate: 0.0%
(APIServer pid=1344642) INFO:     127.0.0.1:47952 - "GET /health HTTP/1.1" 200 OK
(APIServer pid=1344642) INFO:     127.0.0.1:47962 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1344642) INFO:     127.0.0.1:47972 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1344642) INFO:     127.0.0.1:47978 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1344642) INFO:     127.0.0.1:47984 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1344642) INFO:     127.0.0.1:48000 - "POST /v1/chat/completions HTTP/1.1" 200 OK

Essential Elements of an Effective PR Description Checklist

The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
The test plan, such as providing test command.
The test results, such as pasting the results comparison before and after, or e2e results
(Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

claude

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

chaunceyjiang · 2026-07-20T08:47:12Z

/cc @orozery PTAL.

mergify · 2026-07-20T14:39:52Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @chaunceyjiang.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

orozery

Thanks @chaunceyjiang !
Could you rebase and address my comment?
I can also push, just let me know.

orozery · 2026-07-20T14:47:30Z

@@ -327,6 +327,21 @@ def storable_chunks(
            num_chunks = max(0, num_chunks - 1)
        return num_chunks

+    def storable_allocated_chunks(


Can we please merge the logic here into the existing storable_chunks function?

…d KV blocks Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>

orozery

Thanks @chaunceyjiang !

…equest_finished Root-cause fix for issue vllm-project#49118, complementary to PR vllm-project#49146's clamp. Changes: 1. preempt path: sync clear offload_keys with block_ids - Maintains invariant: len(offload_keys) * blocks_per_chunk <= len(block_ids) - Prevents offload_keys residue after preempt 2. request_finished: check block_ids before calling update_offload_keys - Only populate offload_keys if request was actually scheduled - Prevents offload_keys without corresponding block_ids for queued aborts 3. test: update test_abort_queued_request_does_not_build_store_job - Assert offload_keys is empty for never-scheduled request - Reflects the source fix in request_finished Why this matters: - preempt sync clear: defensive fix for invariant consistency - request_finished check: source fix for queued abort (no longer need clamp) - mid-prefill abort: still benefits from PR vllm-project#49146's clamp (prefix cache reuse) Co-authored-by: Claude <noreply@anthropic.com> Signed-off-by: Alex <jihuihuang@example.com> Signed-off-by: Alex <jihui.huang@daocloud.io> Signed-off-by: Alex <alex.tech.lab@outlook.com> Signed-off-by: Alex <jihui.huang@daocloud.io>

…d KV blocks (vllm-project#49146) Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>

chaunceyjiang requested review from ApostaC, NickLucche, ivanium, orozery and xuechendi as code owners July 20, 2026 03:48

claude Bot reviewed Jul 20, 2026

View reviewed changes

mergify Bot added v1 bug Something isn't working kv-connector labels Jul 20, 2026

mergify Bot added the needs-rebase label Jul 20, 2026

orozery reviewed Jul 20, 2026

View reviewed changes

chaunceyjiang added 2 commits July 20, 2026 22:52

[Bugfix][KV Offloading] Handle queued request aborts without allocate…

3ae4614

…d KV blocks Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>

[Bugfix][KV Offloading] Handle queued request aborts without allocate…

95aabf6

…d KV blocks Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>

chaunceyjiang force-pushed the kv_offload_abort_req branch from 1d24e2e to 95aabf6 Compare July 20, 2026 14:58

orozery added the ready ONLY add when PR is ready to merge/full CI is needed label Jul 20, 2026

mergify Bot removed the needs-rebase label Jul 20, 2026

orozery approved these changes Jul 20, 2026

View reviewed changes

chaunceyjiang merged commit 94ed0bf into vllm-project:main Jul 21, 2026
94 checks passed

Alex-ai-future mentioned this pull request Jul 21, 2026

[KV Offload] Fix num_tokens_after_batch for different termination types #49285

Merged

ArjunPakhan pushed a commit to ArjunPakhan/vllm that referenced this pull request Jul 21, 2026

[Bugfix][KV Offloading] Handle queued request aborts without allocate…

9ee1d6e

…d KV blocks (vllm-project#49146) Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>

This was referenced Jul 21, 2026

[GG] fix(kv-offload): defer stores until metadata is ready local-inference-lab/vllm#153

Merged

[Bugfix][KV Offloading] Defer stores until block IDs are tracked local-inference-lab/vllm#133

Closed

Palaiologos1453 mentioned this pull request Jul 24, 2026

[Bugfix][KV Offloading] Defer request finalization until final store #49671

Merged

mueslo mentioned this pull request Aug 4, 2026

[Bug]: EngineCore Fatal Error on KV offloading attempt #51013

Closed

1 task

yifjiang mentioned this pull request Aug 21, 2026

[Bugfix][KV Offload] Do not let a recurrent group's unhashed block truncate the load boundary #52807

Merged

Navigation Menu

Uh oh!

Uh oh!

[Bugfix][KV Offloading] Handle queued request aborts without allocated KV blocks - #49146

[Bugfix][KV Offloading] Handle queued request aborts without allocated KV blocks#49146
chaunceyjiang merged 2 commits into
vllm-project:mainfrom
chaunceyjiang:kv_offload_abort_req

chaunceyjiang commented Jul 20, 2026 •

edited

Loading

Uh oh!

claude Bot left a comment

Uh oh!

chaunceyjiang commented Jul 20, 2026

Uh oh!

mergify Bot commented Jul 20, 2026

Uh oh!

orozery left a comment

Uh oh!

orozery Jul 20, 2026

Uh oh!

chaunceyjiang Jul 20, 2026

Uh oh!

orozery left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Uh oh!

Conversation

chaunceyjiang commented Jul 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Test Plan

Test Result

Uh oh!

claude Bot left a comment

Choose a reason for hiding this comment

Claude Code Review

Uh oh!

chaunceyjiang commented Jul 20, 2026

Uh oh!

mergify Bot commented Jul 20, 2026

Uh oh!

orozery left a comment

Choose a reason for hiding this comment

Uh oh!

orozery Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

chaunceyjiang Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

orozery left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

chaunceyjiang commented Jul 20, 2026 •

edited

Loading