Mouchak TensorHive engine
Can two NVIDIA DGX Sparks run a 321B-parameter model?
Yes. TensorHive serves GLM-5.3-Flash, a 321-billion-parameter mixture-of-experts model, split across two DGX Sparks (GB10, 128 GB each) joined by a 200 Gb/s ConnectX-7 cable. In a 2,032-step run on 30 October 2026, both machines agreed on every token and all 16 replies matched the reference run. Per-token speed is not published yet.
Last updated · by Mouchak
The setup
| Model | GLM-5.3-Flash, 321B-parameter mixture-of-experts |
|---|---|
| Weights | 4-bit class: Q4 dense projections, ~4 bpw EXL3 routed experts, FP8 LM head |
| KV cache | FP8 |
| Hardware | 2× NVIDIA DGX Spark (GB10), 128 GB unified memory each, 273 GB/s |
| Link | ConnectX-7, 200 Gb/s, RDMA |
| Parallelism | Tensor parallel across 2 ranks (TP=2) |
| Exchanges per token | 91 (90 partial all-gathers + 1 sample gather) |
| Weights read per token, per rank | ~5.1 GB (computed from the active weights at ~4 bits, split over 2 ranks) |
Why it takes two Sparks
At about 4 bits per weight, a 321B-parameter model needs well over 128 GB, more than one Spark holds. Split in two, each Spark keeps half the weights, and the two exchange small partial results 91 times per token over the ConnectX-7 link.
That exchange is where speed is won or lost, so TensorHive uses its own RDMA transport: a 16 KiB exchange takes 6.0–6.5 µs at the median, against 18.1–19.1 µs for the baseline engine’s all-gather on the same machines.
How fast is it?
Reading each rank’s ~5.1 GB of active weights at the measured 240 GB/s takes about 21 ms, so no engine can produce a token faster than that on this hardware. Our target is 24 ms or less per token for a single user. The baseline engine measured 25.44 ms on 16 October 2026. We will publish TensorHive’s measured number when a benchmark run shows it.
Sources
- Parity run: 30 Oct 2026, 2,032 decode steps, 16 / 16 replies match. How we measure
- Exchange latency: 22 Oct 2026 microbenchmark, both ranks. Benchmarks