refactor(server): own per-model engine launch in model crates - #404
Merged
Merged
Conversation
load_engine was a per-model match that hardcoded each model's GPU topology, CUDA-Graph capability, and cross-arg validation, then built EngineLoadOptions inline. Each model crate now exposes a server-facing launch() that owns its own topology defaults, capability handling (graph-vs-LoRA, graph-unsupported), and validation. The server forwards CLI flags uniformly and no longer judges model capability, so load_engine is pure dispatch. Also slim the server crate: extract CLI Args into config.rs and delete dead modules (ops, tensor, weight_loader, trace_reporter) plus the unrun benches/, whose only consumers were the benches and the deleted ops tests. sampler/scheduler stay (used by bench_serving). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Restrict lora_routes to crate-internal, fold max-model-len resolution into a helper, and reshape serve_model accordingly; update openinfer-sim to the adjusted frontend surface. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This was referenced Jun 16, 2026
xiaguan
added a commit
that referenced
this pull request
Jun 16, 2026
Two independent breakages had main red: 1. The vllm git-dep bump (#402) made the Rust text backend require a `vocab_size` in the model config; the simulated frontend's tiny `config.json` fixture omitted it, so every `frontend_e2e` test failed with "the model config does not define `vocab_size`". Add `vocab_size: 3` to match the 3-token tiny tokenizer. 2. #404 landed with rustfmt violations (import order, signature/match-arm wrapping) across kimi-k2, server, and the vllm frontend, so `cargo fmt --all --check` failed before the e2e step even ran. Apply rustfmt. Verified locally: `cargo fmt --all --check`, `cargo metadata --locked`, and `cargo test --release -p openinfer-sim --test frontend_e2e` (5/5) all pass. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
load_enginewas a per-modelmatchthat hardcoded each model's GPU topology, CUDA-Graph capability, and cross-arg validation, then builtEngineLoadOptionsinline — every new model fattened the server binary with model-specific knowledge.Push that policy down into each model crate. Each crate now exposes a server-facing
launch(...)that owns:kimi_parallel_configmoved here).The server forwards CLI flags uniformly and no longer judges model capability —
load_engineis now pure dispatch (≈145 lines → forwarding).Qwen3LaunchOptions+launchlaunch(path, device_ordinal, cuda_graph)KimiLaunchOptions+launchlaunch(path, cuda_graph, prefill_profile)launch(path, cuda_graph)Server-crate slim
Argsintoconfig.rs.ops/tensor/weight_loader/trace_reporterand the unrunbenches/(only consumers were the benches + the deletedopstests).sampler/schedulerkept —bench_servinguses them.Also in this PR (separate commit)
refactor(frontend): tightenvllm-frontendserve-path visibility (lora_routes→ crate-internal), extract a max-model-len helper, reshapeserve_model; adaptopeninfer-sim. Pre-existing working-tree work, bundled here at the author's request.Verification
server+qwen3-4b(default),qwen35-4b,deepseek-v2-lite, and thebench_servingbin.kimi-k2/deepseek-v4cannot be built on the dev box — they fail insideopeninfer-kernels' build.rs (missingtilelang/ CUDA kernel toolchain), before any of this PR's Rust is reached. Theirlaunchis trivial and was reviewed by inspection.opstests were removed withsrc/ops).Deferred (not in this PR)
config.rsmax_prefill_tokensdefault referencesopeninfer_qwen3_4b::…without a#[cfg(feature = "qwen3-4b")]gate (its sibling LoRA fields are gated), so--no-default-featureswon't build. Left as-is per owner.🤖 Generated with Claude Code