Why build an LLM inference engine in Rust?

An inference server is long-running code that juggles GPU memory, network buffers and many users at once, where one memory bug can corrupt output or crash a run. Rust catches that class of bug when the code is compiled. TensorHive uses Rust for its server, scheduler, network transport and GPU kernels, built with NVIDIA’s cuda-oxide compiler.

Last updated 31 October 2026 · by Mouchak

What is in Rust today

• Server and scheduler: an OpenAI-compatible API with streaming and cancellation.
• Network transport: our own RDMA exchange over ConnectX-7 between the two Sparks.
• GPU kernels: written in Rust and compiled with NVIDIA’s cuda-oxide; memory-bound kernels reach 240–245 GB/s on GB10, bit-exact against a CPU reference.
• Still borrowed: a few prefill kernels come from existing CUDA code, clearly labelled, and are being replaced one by one.

What it buys

• One toolchain and one binary, with no Python in the serving path.
• Memory errors in host code are caught at compile time instead of in production.
• Every change lands with evidence: outputs are checked step by step against a reference run.

What it costs

The GPU side of Rust is young. TensorHive pins one nightly toolchain together with the cuda-oxide revision it builds against, and some kernels are still easier to borrow than to write. We accept that trade for an engine whose correctness we can show.