A co-located orchestrator + coder agent, tuned to roughly 2× on both ends — then benchmarked properly: decode curves, time-to-first-token vs context, concurrency scaling, and real speculative-decoding acceptance rates.
Single-stream, temperature 0, on the co-located box. Hover the charts — every point is a real run.
Aggregate throughput scales hard on both models — while they share the same 121 GiB. All 16/16 requests succeeded at every level.
Straight from vLLM's Prometheus counters over the whole suite — how often the draft head's guesses survive verification.
31,309 / 43,927 draft tokens accepted. ≈ 2.14 bonus tokens per verified step — the engine that makes 105+ tok/s possible.
7,372 / 13,503 draft tokens accepted on mixed work. On code it's higher — which is exactly why the coder hits 30+ tok/s there.
Not every knob pays off — and the ones that do aren't always the ones you'd expect. Both measured on the box, temperature 0.
vLLM warned the MTP scheduler caps max_num_scheduled_tokens at 2048. So we raised --max-num-batched-tokens to 8192 and re-measured single-stream code decode. Result: statistical noise — and it ate memory headroom, so we reverted it.
Lesson: single-stream MTP decode is already at the memory-bandwidth ceiling. The batch budget only helps prefill and concurrency, not one-request tok/s.
The literature calls it the "reasoning trap": a verbose chain-of-thought steals the attention an orchestrator needs to read requests and route. We A/B'd our orchestrator's thinking on vs off across 3 plan-and-delegate tasks.
Lesson: ~half the tokens, ~46% lower latency, and delegation quality held at 3/3 clean. Light, decisive routing beats heavy deliberation — we shipped thinking-off for the orchestrator, kept the coder at medium (it's the one that hallucinates).
The NVFP4 weights ship a trained multi-token-prediction head. Enabling it — method:mtp, n=3 — nearly tripled code decode, with tool-calls intact. No new image needed.
Replaced a 30B reasoning model (forced-eager, ~50 tok/s) with a 3B-active MoE running CUDA graphs + MTP. Same clean tool-calls, half the latency per turn.
Reasoning lives on the 105 tok/s orchestrator (thinking: high); the 30 tok/s coder thinks medium and executes. The delegation turn collapsed.
The MoE drafts; the stronger 27B critiques & improves — complex code only. First trial: it caught 3 real bugs the draft missed (an unlocked __repr__ race, a bool-is-int gap, a liveness-only test).
Two NVFP4 models resident on one unified-memory GB10. The whole fight is memory + startup order.
# Qwen3.6-35B-A3B · MoE (3B active) · NVFP4 vllm serve Qwen3.6-35B-A3B-NVFP4 \ --quantization modelopt \ --moe-backend marlin \ --speculative-config '{"method":"mtp", "num_speculative_tokens":3}' \ --tool-call-parser qwen3_coder \ --reasoning-parser qwen3 \ # enable_thinking: true — reasons hard --gpu-memory-utilization 0.42 \ --max-model-len 131072
# Qwen3.8-27B · Qwen3_5 hybrid · NVFP4 vllm serve Qwen3.8-27B-NVFP4 \ --speculative-config '{"method":"mtp", "num_speculative_tokens":3}' \ --attention-backend flashinfer \ --enable-prefix-caching \ --tool-call-parser qwen3_coder \ --reasoning-parser qwen3 \ # enable_thinking: true — it critiques --gpu-memory-utilization 0.45 \ --max-model-len 131072 --max-num-seqs 32
--restart no + one ordered boot script — Docker auto-restart re-introduces the race on reboot.At 105 tok/s, reasoning is nearly free. It plans, decomposes, writes the first draft, and hands the coder a precise spec — signatures, file paths, test expectations.
At 30 tok/s, tokens are precious. Medium thinking is the sweet spot for its critic role — deep enough to spot an unlocked __repr__, cheap enough to stay responsive.
Trivial code → the orchestrator just writes it. Complex code → fast draft, then a correctness/edge-case/perf critique from the 27B, then integration. Reasoning lives where it's cheap; depth is applied where it counts.