▲ UPDATE · 2026‑08‑14 · the stack got ~2× faster on both ends
Orchestrator swapped Nemotron‑3.5 → Qwen3.6‑35B‑A3B (MoE + MTP). Head‑to‑head on this box, Qwen3.6 ran the delegation turn at 105.6 tok/s vs 51.7 and cut turn latency 6.85 s → 2.72 s (2.5×) with clean tool‑calls — so it's now the harness. And the Qwen3.8‑27B coder gained its built‑in MTP head: 11.6 → 30.4 tok/s (2.6×), tool‑calls intact, on our own v0.26 image (no third‑party build). Details in Recipe 02. Nemotron kept cached for rollback. Hat‑tip @drowzeys & @eugr for the MTP‑on‑Spark lead.

🔐 Also built: a $10 remote lock
A BLE-only August smart lock controlled over WiFi via an ESP32‑C6 Bluetooth proxy + Home Assistant — no subscription bridge. Full writeup →

hsrakri / sparky · home-lab inference engineering

Two brains.
One DGX Spark.

A reasoning orchestrator and a heavyweight coder, co-located on a single GB10 — wired into a live agent, soaked overnight, and written up with every crash and every number.

0×30B

models resident

0 GiB

unified memory

overnight agent tasks

api cost

▾

The architecture

Orchestrator thinks.
Coder builds.

Hermes fronts every conversation with a fast reasoning model, and delegates implementation to a stronger coder over the LAN. Both live on the same box — the whole fight is memory and startup order.

Orchestrator · :8002

Qwen3.6‑35B‑A3B

MoE · 3B active · NVFP4 · MTP · util 0.42

105.6 tok/s131K ctx2.72 s / turnclean tools

delegates

returns code

Coder · :8000

Qwen3.8‑27B dense

dense 27B · NVFP4 · util 0.45 · prefix‑cached

all 27B fire/token131K ctxtool‑calls ✓

0.45+0.42=0.87— gpu‑memory‑utilization is a per‑process share of the whole pool, not a ceiling. Sum ≲ 0.88, start sequentially, coder first.

coder 0.45

orchestrator 0.42

The recipes

Everything measured.
Nothing hand‑waved.

Two deep‑dive writeups — the exact flags, the memory math, and the failure modes that cost real hours.

RECIPE / 01

Co‑locating 2×30B on one GB10

vLLM treats --gpu-memory-utilization as a per‑process reservation that must fit in free memory — so “0.45 + 0.45” doesn’t partition, it races. The recipe: fractions summing ≲ 0.88, sequential startup (coder first), enforce‑eager for the hybrid Mamba‑MoE, and a boot script that makes reboots deterministic.

Read the recipe →

field notes

models resident2 × 30B NVFP4

memory at steady state~115 / 121.7 GiB

startup ordercoder → orchestrator

CUDA‑graph capture, co‑locateddeadlocks → eager

reboot racefixed: ordered boot script

documented failure modes4

RECIPE / 02

Qwen3.8‑27B dense — beating the bandwidth roofline

A dense 27B streams all ~14 GB of active weights per token: 273 GB/s ÷ 14 GB ≈ a hard ~20 tok/s ceiling. n‑gram speculative decoding (no draft model) hit 22.1 tok/s — 1.9× — but broke qwen3_coder tool parsing, so the agent config trades it back. The recipe shows both paths, with acceptance stats.

Read the recipe →

measured, day of release

roofline ceiling (273/14)~20 tok/s

baseline dense decode11.6 tok/s

+ n‑gram spec k=522.1 tok/s (1.9×)

tool calls under spec‑decodemangled args

final agent configprefix‑cache, tools clean

swap from 3.6 MoE3‑line repoint

Field guide

Four ways it crashed
before it worked.

Every one reproduced, root‑caused, and fixed. Tap to expand.

CRASH Both models started together FIXED›

Launching two vLLM instances simultaneously at 0.45 + 0.45 races the memory profiler — whoever profiles second sees the other’s memory as used: ValueError: No available memory for the cache blocks.

Fix: start sequentially — coder up, wait for /v1/models 200, then orchestrator.

CRASH “Just give the 2nd model 0.9” FIXED›

Utilization is a share of total that must fit in free memory: Free memory on device (54.22/121.69 GiB) on startup is less than desired GPU memory utilization (0.9, 109.52 GiB).

Fix: size to free memory → 0.42 for the second model. Sum ≲ 0.88.

CRASH Mamba cache “align” mode FIXED›

NVIDIA’s recipe flag forces an oversized attention block: AssertionError: In Mamba cache align mode, block_size (4176) must be <= max_num_batched_tokens (2048).

Fix: drop --mamba-cache-mode align (or raise --max-num-batched-tokens ≥ 4176).

HANG Silent freeze before weight load FIXED›

Co‑located, Nemotron’s torch.compile / CUDA‑graph capture deadlocks at boot — banner prints, memory stays flat, nothing ever loads.

Fix: --enforce-eager. Costs decode speed; boots every time.

Single‑stream decode

MoE sprints.
Dense grinds. Physics.

Decode is memory‑bandwidth‑bound: tok/s tracks active bytes per token, not total parameters. A 3B‑active MoE flies; a dense 27B pays for every parameter, every token.

tokens / second · measured on this GB10

Qwen3.6‑35B MoE · 3B active · MTP

114

Nemotron‑3.5 30B hybrid MoE · eager

~50

Qwen3.8‑27B + MTP‑3 built‑in draft head · clean tools

30.4

Qwen3.8‑27B + n‑gram fast but mangles tools

22.1

Qwen3.8‑27B baseline · no spec

11.6

The fix: the checkpoint ships a trained MTP head — enabling it (method:mtp, n=3) gives 2.6× with clean tool‑calls, on our own v0.26 image, no third‑party build. n‑gram is faster‑looking but mangles tool parsing. Recipe 02 has the correction + data. Hat‑tip @drowzeys & @eugr.

Overnight, unattended, $0

Then we let it run
all night. Twice.

Not demos — hours of unattended delegation through the live agent, with every task logged to CSV.

Overnight delegation soak

150 back‑to‑back coding tasks, orchestrator → coder

150/150

delegated

1,348

llm calls

infra failures

full data + CSV →

Multi‑turn correction stress

75 cycles: write intense code → run it → feed the real error back → revise

1,785

coder calls

~22

calls / draft

crashes

full data + CSV →

The standout finding: the orchestrator delegates the hard first draft (~22 coder calls) but patches corrections itself — and every failure across both nights was the coder over‑thinking a hard algorithm past its timeout. Never the pipeline.

Four days in August

The build log.

A weekend hack with my kid that got out of hand, in the best way.

Aug 11 · night

Nemotron‑3.5‑Lightning drops — tested that night

Downloaded, co‑located next to the Qwen coder, survived four crash modes, wired into Hermes as the orchestrator by midnight.

Aug 12

150‑task overnight soak

97.3% pass, 100% delegation, memory flat all night. Reboot ordering hardened with a sequential boot script.

Aug 13

Correction stress — 75 two‑round cycles

Real errors fed back for revision. 1,785 delegated calls surfaced how the orchestrator actually divides labor.

Aug 14 · 8am

Qwen3.8‑27B releases — swapped in the same morning

Detected by a 10‑minute watch cron, downloaded, serving, Hermes repointed, roofline measured, spec‑decode bug found and documented. Recipe shipped by lunch.

Orchestrator thinks.Coder builds.