Skip to content

refactor(server): own per-model engine launch in model crates - #404

Merged
xiaguan merged 2 commits into
mainfrom
refactor/model-crate-launch
Jun 16, 2026
Merged

xiaguan merged 2 commits into
mainfrom
refactor/model-crate-launch

Conversation

@xiaguan

@xiaguan xiaguan commented Jun 16, 2026

Copy link
Copy Markdown
Collaborator

What

load_engine was a per-model match that hardcoded each model's GPU topology, CUDA-Graph capability, and cross-arg validation, then built EngineLoadOptions inline — every new model fattened the server binary with model-specific knowledge.

Push that policy down into each model crate. Each crate now exposes a server-facing launch(...) that owns:

  • its GPU topology defaults (V4 = 8 GPUs, V2-Lite = 2 GPUs, Qwen3 tp→device mapping, Kimi EP-world derivation),
  • its CUDA-Graph capability handling (Qwen3 graph-vs-LoRA exclusion; V4/V2-Lite ignore+warn),
  • its cross-arg validation (Kimi tp/dp; the old kimi_parallel_config moved here).

The server forwards CLI flags uniformly and no longer judges model capabilityload_engine is now pure dispatch (≈145 lines → forwarding).

crate new entry
qwen3-4b Qwen3LaunchOptions + launch
qwen35-4b launch(path, device_ordinal, cuda_graph)
kimi-k2 KimiLaunchOptions + launch
deepseek-v4 launch(path, cuda_graph, prefill_profile)
deepseek-v2-lite launch(path, cuda_graph)

Server-crate slim

  • Extract CLI Args into config.rs.
  • Delete dead modules ops / tensor / weight_loader / trace_reporter and the unrun benches/ (only consumers were the benches + the deleted ops tests). sampler / scheduler kept — bench_serving uses them.
  • Net ≈ -1000 LOC.

Also in this PR (separate commit)

refactor(frontend): tighten vllm-frontend serve-path visibility (lora_routes → crate-internal), extract a max-model-len helper, reshape serve_model; adapt openinfer-sim. Pre-existing working-tree work, bundled here at the author's request.

Verification

  • Compiles clean (zero warnings): server+qwen3-4b (default), qwen35-4b, deepseek-v2-lite, and the bench_serving bin.
  • kimi-k2 / deepseek-v4 cannot be built on the dev box — they fail inside openinfer-kernels' build.rs (missing tilelang / CUDA kernel toolchain), before any of this PR's Rust is reached. Their launch is trivial and was reviewed by inspection.
  • Config arg-parsing unit tests pass (the GPU ops tests were removed with src/ops).

Deferred (not in this PR)

config.rs max_prefill_tokens default references openinfer_qwen3_4b::… without a #[cfg(feature = "qwen3-4b")] gate (its sibling LoRA fields are gated), so --no-default-features won't build. Left as-is per owner.

🤖 Generated with Claude Code

xiaguan and others added 2 commits June 16, 2026 16:41
load_engine was a per-model match that hardcoded each model's GPU topology, CUDA-Graph capability, and cross-arg validation, then built EngineLoadOptions inline. Each model crate now exposes a server-facing launch() that owns its own topology defaults, capability handling (graph-vs-LoRA, graph-unsupported), and validation. The server forwards CLI flags uniformly and no longer judges model capability, so load_engine is pure dispatch.

Also slim the server crate: extract CLI Args into config.rs and delete dead modules (ops, tensor, weight_loader, trace_reporter) plus the unrun benches/, whose only consumers were the benches and the deleted ops tests. sampler/scheduler stay (used by bench_serving).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Restrict lora_routes to crate-internal, fold max-model-len resolution into a helper, and reshape serve_model accordingly; update openinfer-sim to the adjusted frontend surface.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@xiaguan
xiaguan merged commit 6b7c201 into main Jun 16, 2026
1 check failed
@xiaguan
xiaguan deleted the refactor/model-crate-launch branch June 16, 2026 08:43
xiaguan added a commit that referenced this pull request Jun 16, 2026
Two independent breakages had main red:

1. The vllm git-dep bump (#402) made the Rust text backend require a
   `vocab_size` in the model config; the simulated frontend's tiny
   `config.json` fixture omitted it, so every `frontend_e2e` test failed
   with "the model config does not define `vocab_size`". Add
   `vocab_size: 3` to match the 3-token tiny tokenizer.

2. #404 landed with rustfmt violations (import order, signature/match-arm
   wrapping) across kimi-k2, server, and the vllm frontend, so
   `cargo fmt --all --check` failed before the e2e step even ran. Apply
   rustfmt.

Verified locally: `cargo fmt --all --check`, `cargo metadata --locked`,
and `cargo test --release -p openinfer-sim --test frontend_e2e` (5/5) all pass.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant