One AI model too big for any single desktop supercomputer. Two machines. One very fast cable. Here's how a million-token brain came to live in a home rack — and what surprised us when it did.
In a home lab running on ordinary wall current, two NVIDIA DGX Spark computers now share a single mind. Each one has 128 gigabytes of memory. The model they wanted to run — DeepSeek-V4-Flash, a reasoning engine whose weights alone fill 156 gigabytes and whose memory stretches to a million tokens — fits in neither. The solution was as old as it is satisfying: cut the thing in half.
Rank zero sits on the machine called sparky1; rank one on sparky2. Every layer's arithmetic is divided across the two graphics processors, and the partial results are reconciled many thousands of times a second over a dedicated 200-gigabit ConnectX link — a private thread of copper and light that never touches the household network.
Loaded, each processor carries roughly 79 gigabytes. Together the pair pools 256 gigabytes of unified memory, enough to hold both the enormous checkpoint and the million-token cache that gives the model its long memory. It is, in effect, one large accelerator assembled from two small ones.
The hardware reads like a contradiction: a supercomputer topology on a desk. Two Grace-Blackwell GB10 modules, each a Grace CPU and a Blackwell GPU fused on one die, linked by a fabric normally found in data centres.
That fabric is not decoration. In a tensor-parallel arrangement the interconnect is the true bottleneck, and buying it down is the entire point of the second machine. The all-reduce that meets after every layer rides RoCEv2 on adapter rocep1s0f0, and the household's slow one-gigabit management wire is kept well clear of it.
The engine is MiaAI-Lab's DSpark build — a vLLM tuned for GB10 with the DeepSeek-V4 fixes already applied. It serves NVFP4 weights, a compact nvfp4_ds_mla key-value cache, and DeepSeek's own speculative-decode head, which drafts five tokens ahead of each verified step.
A single command, issued on the head, brings up both nodes; the head simply reaches across the fabric and starts the worker's container itself.
# DeepSeek-V4-Flash-0731 · NVFP4 · anemll dspark-vllm-gx10:0.1.1 vllm serve DeepSeek-V4-Flash-0731 \ --tensor-parallel-size 2 --nnodes 2 --node-rank 0 \ --master-addr <head-fabric-ip> --kv-cache-dtype nvfp4_ds_mla \ --max-model-len 1048576 # one million tokens --speculative-config '{"method":"dspark","num_speculative_tokens":5}' \ --moe-backend flashinfer_b12x --async-scheduling \ --gpu-memory-utilization 0.835
Inference has two speeds, and they are nothing alike. Prefill — reading the prompt — happens all at once, a single parallel matrix crunch that builds the model's working memory and yields the first token; it is bound by raw compute, and here it runs at about 1,020 tokens a second, flat from a two-thousand-token prompt to sixteen thousand. Decode — writing the reply — comes one token at a time, each re-reading all 156 gigabytes of weights, so it is bound by memory bandwidth and runs at about 37 tokens a second warm. Reading the question is roughly twenty-eight times faster than answering it.
That first decode number is remarkable in its own right for a 156-gigabyte reasoning model sharded over a network cable while holding a million-token window open; the first cold request, before the compute graphs settle, limps in at eight. But the measurement that surprised us came under load, and it was not about either speed — it was about the graphics chip not being the hottest thing in the rack.
| Reading | Value | Notes |
|---|---|---|
| Prefill (prompt ingest) | ~1,020 tok/s | flat 2k–16k · compute-bound |
| Decode, single stream (warm) | ~37 tok/s | best 38 · bandwidth-bound |
| Decode, first cold request | 8.1 tok/s | graphs still warming |
| Weights per GPU | 79 GiB | 156 GB checkpoint, halved |
| Fabric copy, node → node | ~690 MB/s | 200 Gb RoCE, provisioning |
| vLLM CPU, head — idle | 3.4% | quiet |
| vLLM CPU, head — under load | 377% | ≈ 4 cores spinning |
| SoC temperature, head — load | 86 °C | GPU only ~42 W / 66 °C |
| Spec-decode acceptance (MTP-5) | 74.2% | drafter guesses that survive verification |
| Long-context quality (RULER-lite) | 12 / 12 | 8k · 32k · 131k — retrieval, tracking, aggregation |
The recipe's achievement — a 1M-context DeepSeek that runs on GB10 at all — belongs to its authors. What follows is what this bureau added on top, reported plainly, including the improvement that barely moved the needle.
Download once, copy over the fabric. The obvious path pulls the 156-gigabyte checkpoint on both machines — some 800 gigabytes of internet over a run's life. Instead it was fetched once on the head and sent to the worker over the 200-gigabit link at roughly 690 megabytes a second. The container image travelled the same way, streamed node-to-node with docker save; a digest re-pin fetched only the manifest, so not one layer was downloaded twice.
The spinning cores. Following nacyot's analysis, vLLM's shared-memory message queue was found to spin for a full second before yielding — and during decode, with messages arriving every few milliseconds, it never yielded at all. A one-line change (busy_loop_s → 0.002) was baked into the launch and the same load re-run. The head shed a full CPU core, from 377 percent to 281, at no cost to throughput. Modest here — this configuration already runs an asynchronous scheduler — but a free core is a free core, and the head shares that silicon with the agent.
The clock at the knee — evaluated, and skipped. A second thermal lever, from @pbastowski's gb10-clock-cap, pins the GPU at 2,200 megahertz — its efficiency knee — for a large cut in watts at heavy sustained load. We measured our actual load first, and the numbers argued against it: under real cluster decode the GPUs draw only ~42 and ~34 watts at ~2.4 GHz — nowhere near the worst case the cap protects against — and the package heat turned out to be CPU-side, which the spin-wait fix had already eliminated for free. The cap's one real cost (about eight percent at high concurrency) would land exactly where benchmarks run. Sometimes the best tuning move is the one you decline; it's two commands and instantly reversible if a summer soak ever pushes the SoC toward 90 °C.
And it earns its keep. The cluster is not a benchmark rig. It is the brain of a home assistant named Kaaval, an agent lately repointed from a two-model Qwen stack to this single, stronger endpoint. Asked what it runs on, it now answers without prompting: "DeepSeek-V4-Flash-0731, tensor-parallel across your two-node DGX Spark cluster." One reasoning model fills every role the two smaller ones used to divide between them.
A follow-up experiment, prompted by llm-as-a-verifier — a framework whose headline result is that deepseek-v4-flash, asked to judge its own candidate solutions, lifts Terminal-Bench scores from 79% to 86.5% (best-of-3) and 88% (best-of-5). No training, no second model: generate several attempts, let the same model referee, keep the winner. Our cluster runs exactly that model, so the promise was too good not to test.
The first run returned nonsense: every score a flat 0.5, the "winner" arbitrary. The trail led somewhere interesting. The library scores candidates by pre-filling a score tag and reading a one-token probability distribution — a clever trick our server supports fine. But a server running a reasoning parser (as every self-hosted DeepSeek does) files that continued token under reasoning, not content. The library saw an empty letter, its last-match rule re-matched the tag one position late, and the placeholder for the closing tag silently shadowed the real distribution. Every self-hoster of the library's own flagship model hits this; nobody sees an error.
Three lines fixed it — read the letter from the reasoning field, and never let an empty token shadow a score — submitted upstream as PR #7. After the patch, judged on this cluster by the model itself: a correct string-reversal scored 0.91 against a broken one's 0.73, and a best-of-3 over Fibonacci implementations picked the right winner — provided the verifier got four evaluation passes per criterion. One pass was noise; four took 26 minutes and 120 calls, with the library's prefix-friendly prompts hitting our cache 53% of the time.
The verdict writes itself: at desktop-cluster speeds, self-verification is an overnight tool, not an interactive one. It now lives here as a small utility that generates N candidates at spread temperatures and lets the model referee while nobody is watching — the cluster's idle hours put to work grading its own homework.
Five moves reproduce the whole affair. First, exchange SSH keys between the machines over the fast link, not merely the household LAN — the launch drives the worker across it. Second, download the checkpoint once (with Xet storage disabled, or old clients hang) and rsync it to the worker.
Third, stage the image over the fabric and re-register its digest on the far side. Fourth, fill the environment file with the fabric addresses and NCCL adapter, then launch from the head and wait for /health. Fifth, point any agent's base_url at port 8888 — and a million-token reasoning model, on hardware you own, is yours to command.
dspark-vllm-gx10 runtime image that makes DeepSeek-V4 serve on Grace-Blackwell at all.