Measured: Local MoE Inference on a Single GPU

How tensor role-aware offloading achieved 31x throughput over naive layer splitting.

Core Finding:Can an 80B-parameter Mixture-of-Experts (MoE) coding model run locally on a single consumer GPU? Naive layer-based offloading across PCIe choked at 0.9 tokens per second. By offloading by tensor role rather than layer number, we measured a 31x speedup on identical hardware.

Ollama Automatic Split

0.9 tok/s

Layer-wise spill across PCIe; unusable for interactive coding.

llama.cpp CPU Only

7.4 tok/s

Pure CPU inference with zero VRAM allocation; 8.2x faster than Ollama split.

Role Offload (--cpu-moe)

13.2 tok/s

Attention tensors on GPU, expert FFNs in RAM; 4.7 GB VRAM consumed.

Role Offload (--n-cpu-moe 20)

27.7 tok/s

Saturated GPU VRAM (31.7 GB); 31x speedup over naive layer splitting.

Empirical Findings Table

Token throughput across placement strategies on identical RTX 5090 hardware

Telemetry Metric	Measured Value	Context & Observability
Ollama Automatic Split	0.9 tok/s	Layer-wise spill across PCIe; unusable for interactive coding.
llama.cpp CPU Only	7.4 tok/s	Pure CPU inference with zero VRAM allocation; 8.2x faster than Ollama split.
Role Offload (--cpu-moe)	13.2 tok/s	Attention tensors on GPU, expert FFNs in RAM; 4.7 GB VRAM consumed.
Role Offload (--n-cpu-moe 20)	27.7 tok/s	Saturated GPU VRAM (31.7 GB); 31x speedup over naive layer splitting.

Engineering Action Directives

For Mixture-of-Experts architectures on consumer hardware, memory bus bandwidth across PCIe is typically the primary performance bottleneck rather than raw compute.
Offloading weights based on functional tensor role (keeping dense attention in VRAM) preserves interactive throughput better than layer-index splits.

Methodology Boundaries & Limits

Hardware specific: Tested on a single system configuration (AMD Ryzen 7 9800X3D and NVIDIA RTX 5090); PCIe bus topology variations across motherboards will alter throughput.
Single model evaluated: Benchmarks conducted on qwen3-coder-next (80B/3B active); other MoE parameter and quantization configurations may exhibit different transfer dynamics.