(8) Two layers of V1/V2 in vLLM 0.25.1: Engine namespace vs Model Runner, and where vLLM-Kunlun lands

1. Name collision: two layers of V1/V2

The easiest place to get lost when reading vLLM 0.25.1 (and its Kunlun fork, vLLM-Kunlun) is that the code contains two completely different “V1 / V2” concepts, on two different levels.

• Layer 1 (engine level): vllm.v1 namespace = Engine V1. This is the new unified engine that replaced V0 starting in vLLM ~0.8, unifying scheduling, KV management, and continuous batching. If your code does import vllm.v1, you are on Engine V1 — unrelated to the “V2” discussed here.
• Layer 2 (executor level): GPU Model Runner V1 / V2. Inside Engine V1, at the GPU worker layer, there are two parallel model executors: ModelRunner (the old V1 runner) and ModelRunnerV2 (the new V2 runner). These are not engine versions, but two GPU forward-execution paths under the same engine.

One sentence to tell them apart: Engine V1 is the “commander”, ModelRunner(V1)/ModelRunnerV2(V2) are “two playbooks”. The Kunlun fork’s method:dspark (DeepSeek-style draft–verify speculative decoding) forces the V2 playbook — the reason is in Section 4.

2. File layout: both runners in one subpackage

Engine V1’s GPU worker keeps all executor code under vllm/v1/worker/gpu/. In vLLM-Kunlun 0.25.1 it looks roughly like this:

vllm/v1/worker/gpu/
├── model_runner.py          # ModelRunner (V1, old executor)
├── model_runner_v2.py       # ModelRunnerV2 (V2, new executor) ← all 6 Kunlun DSpark opts land here
├── gpu_worker.py            # GPUWorker, picks which runner to instantiate
└── ...                       # other kernels / utilities

Both runners expose the same contract (same execute_model entry, same input structure); the difference is internal implementation: V2 bakes the draft–verify speculative-decoding scaffolding (draft generation, accept decision, KV reuse) into the forward loop, whereas V1’s support for these advanced features is retrofitted and limited.

3. Runner dispatch logic

The GPU worker decides which runner to use at construction time, based on two signals:

1. Config flag use_v2_model_runner (explicitly set by user/platform);
2. Speculative method method: dspark — once enabled, it is force-promoted to use_v2_model_runner = True.

Pseudocode (illustrative, not verbatim):

# gpu_worker.py (illustrative)
def _build_model_runner(self, vllm_config):
    spec_config = vllm_config.speculative_config
    method = spec_config.method if spec_config else None

    use_v2 = vllm_config.use_v2_model_runner
    # Key line: dspark forces the V2 executor
    if method == "dspark":
        use_v2 = True

    if use_v2:
        return ModelRunnerV2(...)   # vllm/v1/worker/gpu/model_runner_v2.py
    return ModelRunner(...)         # vllm/v1/worker/gpu/model_runner.py

4. vLLM-Kunlun’s 6 DSpark optimizations: all in the V2 runner subpackage

vLLM-Kunlun added 6 XPU-targeted optimizations for dspark (draft–verify speculative decoding) in 0.25.1. Their common trait: all of them land in vllm/v1/worker/gpu/model_runner_v2.py (and its directly referenced V2 submodules), not in the Engine layer or the V1 runner. This matters — it explains why these optimizations are invisible to the V1 runner, and why “you only get them when dspark is on”.

The 6 switches (env vars) and landing points (verify exact names against your original material; category and the confirmed anchor are given here):

Note: KUNLUN_HOSTVEC is the env var named in your original material, semantics confirmed (host control-plane vectorization). The exact env names and precise semantics of the other 5 should follow your original paste / diff — the names above are illustrative category names; the architectural location (all in the V2 runner subpackage) is certain.

5. Root cause: host control-plane overhead

Why does Kunlun make almost all 6 optimizations “host-side / control-plane” related? Because the root cause is that the decode-phase bottleneck has shifted from GPU compute to host (CPU) control-plane overhead.

In per-token decoding, every step requires preparing a pile of metadata on the CPU side: positional encodings, attention masks, paged-KV block tables, speculative-draft candidates… These operations don’t run on the GPU, but they are prerequisites for the GPU to start. When the batch mixes speculative drafts and the XPU’s kernel-launch cost is non-trivial, the host time spent preparing this metadata drags down the whole-step latency, creating “GPU waiting for the CPU to prep materials” bubbles.

KUNLUN_HOSTVEC’s approach: vectorize + batch the per-token metadata prep — assemble mask/position/block-tables for a group of tokens at once, reducing repeated Python/scheduling overhead. This corresponds exactly to the “host-side vectorization” row in the table of §4. The other items (batched metadata, lazy KV, mask fusion) are essentially different facets of cutting the count and size of per-step control-plane work — all slices of the same root cause.

6. One-sentence summary

The “V1/V2” in vLLM 0.25.1 is two different things: Engine V1 is the engine namespace, while ModelRunner(V1)/ModelRunnerV2(V2) are GPU executors; method:dspark forces use_v2_model_runner=True, and all 6 of vLLM-Kunlun’s DSpark optimizations land in the V2 runner subpackage, dissolving the CPU prep bottleneck of the decode phase by vectorizing/batching the host control-plane metadata.

7. Top-level framework diagram

An end-to-end framework diagram of the dispatch and landing points (corresponds to the mermaid flowchart TB in your original material):

8. Code comparison with DeepSpec

“DeepSpec” here refers to the community direction of standardizing speculative decoding into a spec_decode module — unifying the draft, verify, and accept interfaces. method:dspark is one draft strategy within the DeepSpec family (fast draft generation based on n-gram / local repetition patterns).

Key points of the code comparison:

• DeepSpec (community mainline): the spec_decode module provides a generic SpecDecoder interface, with draft and verify decoupled, in principle attachable to both V1/V2 runners;
• vLLM-Kunlun 0.25.1: bakes the dspark draft strategy into the V2 runner’s forward loop (model_runner_v2.py), rather than going through the decoupled spec_decode path. The upside is that draft gen, accept decision, and KV reuse can be fused more aggressively inside V2; the cost is that this dspark implementation is V2-exclusive and is not the same code as mainline spec_decode.

So the core difference in the “DeepSpec code comparison” is: mainline puts speculative decoding in an independent module above the runner, while the Kunlun fork welds dspark into the V2 runner internals. This is also why Kunlun’s 6 optimizations can only act on V2 — they depend on the execution hooks inside the V2 runner.

9. V2 throughput / DSpark acceptance conclusions

The payoff splits into two layers:

1. Architectural layer (certain): as long as you use method:dspark (→ V2 runner), you get all 6 Kunlun optimizations, of which KUNLUN_HOSTVEC and friends directly cut the per-step CPU prep tax in decode — this is the root cause of V2’s higher throughput over the V1 runner on Kunlun XPU.
2. Data layer (verify against your original material): the specific acceptance rate and end-to-end throughput of V2 + dspark (your paste should contain measured comparisons) depend on model, batch, and sequence length. The verifiable qualitative relation: the higher the acceptance rate, the more tokens produced per step at equal compute → throughput approaches linear amplification; and V2, by minimizing control-plane overhead, lets a high acceptance rate actually convert into throughput gains instead of being eaten by CPU prep.

Further reading

• vLLM source: vllm/v1/worker/gpu/model_runner.py and model_runner_v2.py (note the use_v2_model_runner dispatch)
• vLLM-Kunlun repo’s dspark / KUNLUN_* env switches (follow your fork’s README and diff)
• vLLM docs: Speculative Decoding
• Previous in this series: (7) Graph mode: CUDA Graph in vLLM / SGLang