[chore] bump vllm to 0.26.0 by erictang000 · Pull Request #1854 · NovaSky-AI/SkyRL

erictang000 · 2026-07-01T00:10:58Z

^

Note

High Risk
Touches critical inference startup and weight-sync paths (NCCL + delta), plus the core vLLM/flashinfer dependency stack used by all GPU training.

Overview
Bumps vLLM 0.23 → 0.26.0 (cu129 wheel) and applies the compatibility fixes needed to keep install, server startup, and weight sync working.

Dependency / install fixes: pins flashinfer* to 0.6.13 (MNNVL layout_code support; cubin has no 0.6.14), overrides xgrammar==0.2.3 for a cp312 linux wheel, and poisons flash_attn.cute at import time so megatron-core's FA4 probe no longer crashes on cutlass-dsl 4.6.

Server startup: passes the new required reuse_port=False to create_server_socket, and seeds VLLM_DP_MASTER_PORT with a free ephemeral port so RayExecutorV2 no longer tries privileged port 100 when DP is off.

Weight sync API drift: NCCL sync now retargets the engine via set_weight_update_target + a _LoadWeightsProxy (0.26 dropped the load_weights callback). DeltaWeightTransferEngine adopts the 4-arg constructor and the full update_weights / start / finish lifecycle.

Tests / misc: multimodal mm_serde import falls back across the moved module path; GLM MLA coverage swaps the unservable tiny fixture for real zai-org/GLM-4.7-Flash in the h100 job. Adds VLLM_0.26_UPGRADE_NOTES.md documenting each required fix.

^{Reviewed by Cursor Bugbot for commit 8d61888. Bugbot is set up for automated code reviews on this repo. Configure here.}

gemini-code-assist · 2026-07-01T00:13:52Z

Warning

Gemini encountered an error creating the review. You can try again by commenting /gemini review.

…_0.24.0

SumanthRH · 2026-08-12T19:13:17Z

+    No-op when DP is enabled (vLLM assigns the master port itself on that path)
+    or when the caller already set the variable.
+    """
+    os.environ.setdefault("VLLM_DP_MASTER_PORT", str(pick_free_port()))


This can have race conditions when running multiple engines on the same node right? Say TP=2, num_engines=4, we might grab the same port on all the VLLMServerActors with pick_free_port because we release the port immediately. The local torch.distributed group per engine might end up trying to use the bind to the same port.

Why can't we use the same reservation based setup as in find_and_reserve_port? Reserve the port first and then close it just before engine initialization.

Hmm the actual port that gets used is VLLM_DP_MASTER_PORT + 100. So ideally we need to ensure that even this port is free.

I feel like the best fix for this is inside vLLM, but for now using find_and_reserve_port should atleast prevent the case of base port collision across engines.

hmm seems like vllm-project/vllm#50969 might address this

Oh nice. We will need to have a workaround meanwhile though

yep added a workaround for now and inlined comments so an agent can delete it on bump to like 0.28.0 probably

SumanthRH · 2026-08-12T19:32:50Z

These changes seem reasonable to me! Have you run GPU CI already?

yeah ran offline on h100 node overnight, all passing (report in the .md file)

let me just run them here again

SumanthRH · 2026-08-12T19:34:02Z

Is this custom endpoint still needed in vllm 0.26.0 btw?

https://github.com/erictang000/SkyRL/blob/752820bdd324327db8dec0b816172ecace44a558/skyrl/backends/skyrl_train/inference_servers/vllm_server_actor.py#L411-L417

If we can remove it, we can also do so in a follow-up, but good to check

yeah probably not needed anymore but i'm not familiar

created an issue to track: #2020

…_0.24.0

… infra log on failure (#2051) # Fix colocated tinker E2E nightly: vLLM startup OOM on L4 The `SkyRL-GPU-E2E-CI-Tinker` nightly has failed on every run since 2026-08-14 (the first nightly after the vLLM 0.26.0 bump in #1854) with: ``` RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {} ``` raised from `VLLMServerActor.start()` at the first `save_weights_for_sampler`. (Failures on 08-08..08-13 were the separate unpinned tinker SDK break, fixed by #2022.) ## Root cause Reproduced on a 1x L4 workspace with a scaled-down (1-GPU) version of the CI backend config. The real error never reaches the job log (see "Log visibility" below); it is a CUDA OOM during vLLM engine startup, at the flashinfer sampler warmup that runs after CUDA graph capture: ``` torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 594.00 MiB. GPU 0 has a total capacity of 22.03 GiB of which 582.00 MiB is free. Process 29529 has 796.00 MiB memory in use. ... this process has 20.67 GiB in use. ``` Two factors combine: 1. **Engine startup ordering.** The main trainer entrypoint starts inference engines on empty GPUs before `build_models()`. The tinker `SkyRLTrainBackend` does the reverse: FSDP workers initialize first, and even after `offload_to_cpu()` each leaves ~800 MiB of unreclaimable CUDA context on the GPU that vLLM later profiles on. `gpu_memory_utilization` budgets a fraction of *total* (not free) memory, so vLLM still sizes its KV cache as if it had the whole card. 2. **vLLM 0.26.0's larger startup peak.** FULL_AND_PIECEWISE cudagraph capture plus the flashinfer sampler warmup transiently allocate ~3 GiB beyond the utilization budget. At 0.8 on a 22 GiB L4 with the FSDP context resident, this misses by ~12 MiB. This explains why only this nightly regressed: the fully-async tinker E2E is non-colocated (engines get empty GPUs), and the main colocated E2E starts engines before building models. ## Changes - `gsm8k_tinker.sh`: `gpu_memory_utilization` 0.8 → 0.7. Costs ~2 GiB of KV cache, irrelevant for 512-token GSM8K rollouts. - Both tinker E2E scripts: on any non-zero exit, the cleanup trap now dumps the tails of `server.log` **and** the newest `/tmp/skyrl-logs/infra-*.log`. ### Log visibility `VLLMServerActor` calls `redirect_actor_output_to_file()`, which sends actor output — including the vLLM engine's real traceback — to `/tmp/skyrl-logs/infra-*.log` on the cluster. The scripts previously only tailed `server.log`, and only when the server failed to boot, so a client-visible failure like this one showed nothing but the opaque re-raised `RayTaskError` (the empty `Failed core proc(s): {}` is a vLLM race where the engine-core proc dies before its exit code is captured). ## Verification On a 1x L4 with the 1-GPU equivalent of the CI backend config (`colocate_all=true`, 1 engine): - at `gpu_memory_utilization=0.8`, `save_weights_and_get_sampling_client()` fails with the exact CI error (OOM in flashinfer sampler warmup); - at `0.7`, the same call succeeds: engine starts, weights sync, KV cache wakes at 18.5/22 GiB. ## Note Any tinker + `colocate_all` user on ~24 GB GPUs will hit this same OOM at the default 0.8 utilization, since the lazy engine startup ordering is inherent to the backend. A follow-up could account for the training workers' CUDA context when sizing the KV budget, or start engines eagerly once the LoRA config is known. 🤖 Generated with [Claude Code](https://claude.com/claude-code)  --- > [!NOTE] > **Low Risk** > CI-only shell script changes; no production trainer or auth paths touched. > > **Overview** > Fixes the **colocated** tinker GSM8K nightly (`gsm8k_tinker.sh`) by lowering vLLM **`gpu_memory_utilization` from 0.8 to 0.7**, with comments explaining lazy engine startup after FSDP leaves CUDA context and vLLM 0.26’s larger startup peak OOMing 22 GiB L4s. The fully-async script keeps **0.8** (non-colocated engines). > > **Both** tinker E2E scripts replace the simple EXIT trap with a **`cleanup()`** that on any non-zero exit tails **`server.log`** and the newest **`/tmp/skyrl-logs/infra-*.log`** (Ray/vLLM errors) before tearing down the server process group—so CI logs show real OOM/tracebacks instead of opaque `RayTaskError`. > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit 77ff96c. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup>  Signed-off-by: Avi Basnet <avigyabb@stanford.edu> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

vLLM 0.26 (now on main via NovaSky-AI#1854) added an annotation-only lora_manager member to the SupportsLoRA protocol. The patched KimiK25ForConditionalGeneration is checked *structurally* (unlike native nominal subclasses), so the member must exist on the class; without it the supports_lora() gate rejects the model. Set it to None, the documented 'not yet set' state. Also future-proof KimiK25TextBridge.build_conversion_tasks for megatron-bridge 0.7.0 (NovaSky-AI#2042), which grows the base signature with a weight_dtype keyword: forward *args/**kwargs instead of pinning the two-argument shape. Co-authored-by: Cursor <cursoragent@cursor.com>

x

d9034bb

erictang000 added run_train_gpu_ci run_train_megatron_gpu_ci run_train_megatron_gpu_ci_models run_h100_gpu_ci Run H100 GPU CI labels Jul 1, 2026

bump to 0.25.0

631f4a2

erictang000 changed the title ~~[chore] bump vllm to 0.24.0~~ [chore] bump vllm to 0.25.0 Jul 13, 2026

Merge branch 'main' of https://github.com/erictang000/SkyRL into vllm…

fd560ec

…_0.24.0

erictang000 removed run_train_gpu_ci run_train_megatron_gpu_ci run_train_megatron_gpu_ci_models run_h100_gpu_ci Run H100 GPU CI labels Jul 13, 2026

Merge branch 'main' of https://github.com/erictang000/SkyRL into vllm…

54ee6fb

…_0.24.0

erictang000 added run_train_gpu_ci run_train_megatron_gpu_ci run_train_megatron_gpu_ci_models labels Jul 13, 2026

SumanthRH and others added 2 commits August 5, 2026 00:25

ok 0.26.0 now..

544d529

Merge branch 'main' of https://github.com/erictang000/SkyRL into vllm…

ce8e5dd

…_0.24.0

erictang000 added run_train_gpu_ci run_train_megatron_gpu_ci run_train_megatron_gpu_ci_models and removed run_train_gpu_ci run_train_megatron_gpu_ci run_train_megatron_gpu_ci_models labels Aug 5, 2026

erictang000 changed the title ~~[chore] bump vllm to 0.25.0~~ [chore] bump vllm to 0.26.0 Aug 5, 2026

erictang000 added run_train_megatron_gpu_ci_models and removed run_train_gpu_ci run_train_megatron_gpu_ci run_train_megatron_gpu_ci_models labels Aug 10, 2026

erictang000 added 2 commits August 12, 2026 00:45

clean up comments a little

c5e99f8

x

752820b

erictang000 requested a review from hao-aaron August 12, 2026 00:47

SumanthRH reviewed Aug 12, 2026

View reviewed changes

Comment thread VLLM_0.26_UPGRADE_NOTES.md Outdated

SumanthRH reviewed Aug 12, 2026

View reviewed changes

erictang000 mentioned this pull request Aug 12, 2026

[vllm] Remove old inference worker wrap code after 0.26.0 update #2020

Open

fix port race and add comments about when it can be deleted

963412a

erictang000 added run_train_gpu_ci run_train_megatron_gpu_ci run_train_megatron_gpu_ci_models run_h100_gpu_ci Run H100 GPU CI and removed run_train_megatron_gpu_ci_models labels Aug 12, 2026

Merge branch 'main' of https://github.com/erictang000/SkyRL into vllm…

043309e

…_0.24.0

erictang000 added run_train_gpu_ci run_train_megatron_gpu_ci run_train_megatron_gpu_ci_models run_h100_gpu_ci Run H100 GPU CI and removed run_train_gpu_ci run_train_megatron_gpu_ci run_train_megatron_gpu_ci_models run_h100_gpu_ci Run H100 GPU CI labels Aug 12, 2026

x

6042f20

erictang000 merged commit 7b9c72b into NovaSky-AI:main Aug 13, 2026
3 of 5 checks passed

erictang000 deleted the vllm_0.24.0 branch August 13, 2026 17:06

avigyabb mentioned this pull request Aug 17, 2026

[fix][ci] Lower gpu_memory_utilization for colocated tinker E2E; dump infra log on failure #2051

Merged

casper-hansen mentioned this pull request Aug 19, 2026

[megatron] 5/n towards Kimi K2.6: Kimi K2.5-family (KimiK25ForConditionalGeneration) support with INT4 QAT + LoRA #2027

Draft

Navigation Menu

Uh oh!

[chore] bump vllm to 0.26.0 - #1854

[chore] bump vllm to 0.26.0#1854
erictang000 merged 12 commits into
NovaSky-AI:mainfrom
erictang000:vllm_0.24.0

erictang000 commented Jul 1, 2026 •

edited by cursor Bot

Loading

Uh oh!

gemini-code-assist Bot commented Jul 1, 2026

Uh oh!

SumanthRH Aug 12, 2026

Uh oh!

SumanthRH Aug 12, 2026 •

edited

Loading

Uh oh!

erictang000 Aug 12, 2026

Uh oh!

SumanthRH Aug 12, 2026

Uh oh!

erictang000 Aug 12, 2026

Uh oh!

SumanthRH Aug 12, 2026

Uh oh!

erictang000 Aug 12, 2026

Uh oh!

erictang000 Aug 12, 2026

Uh oh!

Uh oh!

SumanthRH Aug 12, 2026 •

edited

Loading

Uh oh!

erictang000 Aug 12, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

erictang000 commented Jul 1, 2026 • edited by cursor Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gemini-code-assist Bot commented Jul 1, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SumanthRH Aug 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

SumanthRH Aug 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

erictang000 commented Jul 1, 2026 •

edited by cursor Bot

Loading

SumanthRH Aug 12, 2026 •

edited

Loading

SumanthRH Aug 12, 2026 •

edited

Loading