Published by Sean Carr
Measured: Local MoE Inference on a Single GPU
How tensor role-aware offloading achieved 31x throughput over naive layer splitting.
At a Glance
Ollama Automatic Split
0.9 tok/s
Layer-wise spill across PCIe; unusable for interactive coding.
llama.cpp CPU Only
7.4 tok/s
Pure CPU inference with zero VRAM allocation; 8.2x faster than Ollama split.
Role Offload (--cpu-moe)
13.2 tok/s
Attention tensors on GPU, expert FFNs in RAM; 4.7 GB VRAM consumed.
Role Offload (--n-cpu-moe 20)
27.7 tok/s
Saturated GPU VRAM (31.7 GB); 31x speedup over naive layer splitting.
| Telemetry Metric | Measured Value | Context & Observability |
|---|---|---|
| Ollama Automatic Split | 0.9 tok/s | Layer-wise spill across PCIe; unusable for interactive coding. |
| llama.cpp CPU Only | 7.4 tok/s | Pure CPU inference with zero VRAM allocation; 8.2x faster than Ollama split. |
| Role Offload (--cpu-moe) | 13.2 tok/s | Attention tensors on GPU, expert FFNs in RAM; 4.7 GB VRAM consumed. |
| Role Offload (--n-cpu-moe 20) | 27.7 tok/s | Saturated GPU VRAM (31.7 GB); 31x speedup over naive layer splitting. |
Published by Sean Carr
View & print executive brief version →Executive Summary & Core Finding
Can an 80B-parameter Mixture-of-Experts (MoE) coding model run locally on a single consumer GPU? Naive layer-based offloading across PCIe choked at 0.9 tokens per second. By offloading by tensor role rather than layer number, we measured a 31x speedup on identical hardware.
Method
Hardware testbed: NVIDIA RTX 5090 (31.8 GB VRAM, 61.6 GB system RAM).
Target model: qwen3-coder-next (48.2 GB GGUF quantization, 80B total parameters, 3B active parameters per token).
Tested runtime configurations: Ollama standard layer splitting, llama.cpp CPU-only execution, and llama.cpp role-aware offloading via custom tensor placement flags.
All benchmarks executed on identical hardware under sustained token generation workloads.
What We Measured
Token throughput across placement strategies on identical RTX 5090 hardware
Ollama Automatic Split
0.9 tok/s
Layer-wise spill across PCIe; unusable for interactive coding.
llama.cpp CPU Only
7.4 tok/s
Pure CPU inference with zero VRAM allocation; 8.2x faster than Ollama split.
Role Offload (--cpu-moe)
13.2 tok/s
Attention tensors on GPU, expert FFNs in RAM; 4.7 GB VRAM consumed.
Role Offload (--n-cpu-moe 20)
27.7 tok/s
Saturated GPU VRAM (31.7 GB); 31x speedup over naive layer splitting.
Experimental Notes & Observability Details
- For contrast, a model that fits entirely within VRAM (qwen3-coder:30b) generates 204 tokens/sec cold and 294 tokens/sec warm on this hardware.
- CPU-only execution outperforming GPU-assisted layer splitting by 8.2x revealed that automatic placement was actively worse than leaving the GPU idle.
What We Think It Suggests
- For Mixture-of-Experts architectures on consumer hardware, memory bus bandwidth across PCIe is typically the primary performance bottleneck rather than raw compute.
- Offloading weights based on functional tensor role (keeping dense attention in VRAM) preserves interactive throughput better than layer-index splits.
Limits
- Hardware specific: Tested on a single system configuration (AMD Ryzen 7 9800X3D and NVIDIA RTX 5090); PCIe bus topology variations across motherboards will alter throughput.
- Single model evaluated: Benchmarks conducted on qwen3-coder-next (80B/3B active); other MoE parameter and quantization configurations may exhibit different transfer dynamics.
Detailed Analysis & Field Observations
Mixture-of-Experts (MoE) architectures are theoretically well-suited for local developer workstations. Because only a small subset of experts activate on each token (for example, 3 billion active parameters out of 80 billion total), the floating-point compute requirements are relatively modest. However, the model's total memory footprint exceeds the VRAM capacity of any single consumer graphics card.
On August 24, 2026, we benchmarked the 48.2 GB quantized qwen3-coder-next MoE model on an NVIDIA RTX 5090 equipped with 31.8 GB of VRAM and 61.6 GB of host RAM. Using standard runtime defaults in Ollama, the model split automatically across memory domains. Generation throughput stalled at an abysmal 0.9 tokens per second. An initial assumption might be that offloading a spilled MoE is fundamentally unviable.
The breakthrough came from diagnosing what was actually traversing the PCIe bus. Standard offloaders split models by layer index: the first N layers reside in GPU VRAM, while the remaining layers spill into host system RAM. In a transformer architecture, attention tensors execute on every single token. Splitting by layer forced attention tensors to shuttle across the PCIe bus on every forward pass, saturating the bus with transfer overhead.
To test this hypothesis, we switched to llama.cpp and tested pure CPU execution with zero GPU offload. Pure CPU execution achieved 7.4 tokens per second, over eight times faster than the GPU-assisted layer split. The GPU split was slower than running entirely on host CPU because the PCIe bus had become the critical system bottleneck.
We then applied tensor role-aware offloading. Rather than splitting by layer index, llama.cpp's expert offload flags (--cpu-moe and --n-cpu-moe) split by functional tensor role. Attention tensors and shared weights are permanently pinned to GPU VRAM where they execute with full tensor core memory bandwidth. Only the sparsely activated expert feed-forward network (FFN) tensors spill to system RAM. Pinning attention to GPU while offloading 20 expert blocks produced 27.7 tokens per second, a 31x acceleration over the default split on the exact same silicon.
This benchmark in this test indicates that active parameter count predicts compute latency, but tensor placement predicts bus saturation. Effective local agent architectures require role-aware offload policies that respect tensor access frequencies.