GLM-5.3-Flash on 2× NVIDIA DGX Spark — bring-up, tuning, a two-lane serving design, and thermal engineering.
Everything below was measured on this cluster, failures included.
Day-0 models on day-0 hardware means every layer of the stack can betray you. Click each attempt — the failures taught us exactly what the success needed.
marlin kernel runs, and it repacks weights at ~2× memory. Day-old engine, last-mile failure. Reverted cleanly.NVRM: Out of memory with ~117 GB free per node. A knife-edge fit that fit the author's kit and not ours. Lesson: on unified memory, your free RAM is the spec that matters.reasoning_effort to max. A simple question burned 33 s and 2,600 characters of hidden reasoning — then returned an empty answer (budget exhausted). Setting effort to low for chat: same question in 1.3 s. One default, 25× latency. We now route: low effort for conversation, high + thinking for delegated hard work.We started on NVFP4 and migrated to EXL3 4-bit. KLD (divergence vs the full-precision model — lower is better) tells the story better than vibes:
One engine can't be cloned (2 × 164 GB doesn't fit), and tensor-parallel already balances the GPUs perfectly. What isn't balanced is workload priority: an interactive assistant shares the engine with long-running batch jobs. So: one engine, one shared KV pool, two QoS lanes.
priority 0 · reasoning_effort low → snappy voice/chat replies. Measured 0.6–2.8 s while the batch lane was saturated.
priority 10 · reasoning_effort high + thinking → deep thinking for delegated heavy work, big token budgets.
Running inference 24/7 made the head node average 89–90 °C. Diagnosis before levers: where is the heat?
| Finding | Evidence |
|---|---|
| Governor pinned all 20 Grace cores at max, 24/7 | performance governor; idle cores at 2.8 GHz doing nothing |
| Fast cores boost to 3.9 GHz under load | per-policy cpuinfo_max_freq — invisible if you only read cpu0 |
| GPU at ~96% util is the floor heat | GPU die 65 °C, 96%, ~33 W — soaks SoC zones |
| Busy-wait polling burned ~2 cores | Worker_TP 198% CPU with a 1 s spin window — the single biggest lever |
VLLM::Worker_TP sat at 198% CPU — nearly two cores busy-waiting. vLLM's SpinCondition reader polls with a 1-second window by default; the upstream TP=2 sweep selected 16 ms. One env var (GLM53_SPINWAIT_MS=16) took the head node from a 93.7 °C peak to 80.3 °C (74 °C average) under identical 96% GPU load, with worker CPU dropping to 123% — and no throughput cost. Lesson: before blaming the workload, check what your stack is doing while it waits. Settings re-assert after reboot via the watchdog.
# the whole fix, applied via a privileged container (no sudo on the box)
for g in /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor; do echo schedutil > $g; done
# cap only the 3.9 GHz-capable policies at 3.3 GHz; efficiency cores untouched
for p in /sys/devices/system/cpu/cpufreq/policy*/scaling_max_freq; do
hw=$(cat ${p%scaling_max_freq}cpuinfo_max_freq)
[ "$hw" -gt 3300000 ] && echo 3300000 > "$p"
done
reasoning_effort: low (25× latency difference) — keep max for genuinely hard delegated work.build.gradle next to an existing build.gradle.kts, the model answered: training priors favor generating a fresh canonical file over editing what exists — "the same prior kept winning." The fix it prescribed for itself now leads every task prompt: edit in place; a new file needs proof no equivalent exists.max-num-batched-tokens) isn't page-aligned, prefix caching silently reports 0% — and one config knob isn't enough; the fix needs engine patches. Know your model's cache granularity.pkill -f "name[.]py": the bracket class can't match its own literal text.