225 unattended agent tasks, ~3,100 delegated LLM calls, two overnight runs, one real vLLM bug — every chart below is drawn from the CSVs in this repo.
150 back‑to‑back delegated coding tasks. The bulk lands in 60–90 s; the long tail is the coder thinking harder, not the stack degrading — memory stayed flat all night and the self‑healing never fired.
Each cycle: the orchestrator gets an intense coding task, we run the result, then feed the real error back for a revision. The delegation counters tell a story nobody wrote in a model card:
Failures weren’t random: the same few algorithms made the coder over‑iterate past its 600 s budget (17–36 calls instead of ~3), while it aced everything else across both nights.
n‑gram speculative decoding needs no draft model: vLLM guesses continuations from patterns already in the context, and code is full of patterns. Acceptance by draft position, measured on 900‑token generations:
qwen3_coder tool‑calls came back mangled — the function name absorbed the arguments (reproduced at both k, temp 0; clean without). For an agent coder that writes files via tools, that’s disqualifying — so the final config keeps prefix caching and trades the 1.9× back. Pure codegen? Turn it on.The honest engineering log — each one cost real time, each one is reproducible, each one has a fix in the recipes.
vLLM’s --gpu-memory-utilization is a per‑process reservation checked against free memory at startup. Two instances launched together fight the profiler; the loser dies with No available memory for the cache blocks.
Solved: fractions sum ≲ 0.88, sequential startup, coder first — codified in vllm-stack-up.sh.
Nemotron’s torch.compile / graph‑capture path froze at boot every time a second model was resident — banner printed, memory flat, nothing loaded, no error. The nastiest kind of failure: silent.
Solved: --enforce-eager for the orchestrator. Costs decode speed, boots 100% of the time.
On vLLM v0.26.0, n‑gram spec‑decode + the qwen3_coder parser returns name="get_weather(city=…)" with empty arguments — at temperature 0, reproduced at k=3 and k=5, clean without spec. A genuine serving‑stack interaction bug.
Solved (worked around): agent config ships without spec‑decode; documented as pure‑codegen‑only. 1.9× left on the table until the parser interaction is fixed upstream.
On Myers diff, min‑max heaps, and thread‑safe pools, the delegated coder loops — 17–36 calls refining instead of converging — and blows its time budget. 100% of both nights’ failures trace to this one behavior.
Mitigable: iteration caps or per‑task budgets; the per‑problem data makes hard tasks routable.
Reasoning models reason — even on trivial prompts. Hidden thinking consumed whole token budgets and made early benchmarks look like empty responses.
Solved: reasoning_effort: none at the agent layer + generous budgets; “detailed thinking off” only trims it.
Benchmark harnesses that parse the orchestrator’s chat output see summaries, not code — the sub‑agent wrote solution.py to disk all along. Cost three harness rewrites at 2 a.m.
Solved: make the file the artifact — instruct a canonical path, read the file, keep round‑2 prompts short so the orchestrator delegates instead of over‑reasoning.
273 GB/s ÷ ~14 GB of active NVFP4 weights ≈ 20 tok/s ceiling; we measured 11.6 co‑located. No flag fixes bandwidth — only fewer active bytes (MoE) or accepted drafts (speculation) do.
Accepted: the orchestrator fronts the conversation at ~50 tok/s while the strong‑but‑slow coder works in the background — the architecture absorbs the physics.