The Local Coder Ceiling: 100% on Single Functions, 0% on Multi-File Bug Fixes Under Default Config
Coordinate ascent benchmarks on 30B and 80B MoE models: 0/24 under the default config; across all configs, 7 of 8 repo tasks never passed, with task 003 passing 1/3 at the best config.
Tier 1: Single Function
30 / 30 (100%)
Both 30B and 80B MoE achieved perfect pass rates on isolated function edits.
Tier 2: Single File
12 / 12 (100%)
Flawless execution on multi-function refactors contained within a single file.
Tier 3: Repo Fixes (Default)
0 / 24 (0%)
0/24 under the default config (best_of_n=1, temp=0.2); across all configs, 7 of 8 repo tasks never passed, with task 003 passing 1/3 at best config (best-of-3, temp 0.6) and 3/3 under the surgical prompt.
In-VRAM Throughput
315-340 tok/s
30B achieved 315 to 340 tok/s decode; 80B MoE achieved 214 tok/s in VRAM.
Empirical Findings Table
Pass rates across task difficulty tiers on identical hardware| Telemetry Metric | Measured Value | Context & Observability |
|---|---|---|
| Tier 1: Single Function | 30 / 30 (100%) | Both 30B and 80B MoE achieved perfect pass rates on isolated function edits. |
| Tier 2: Single File | 12 / 12 (100%) | Flawless execution on multi-function refactors contained within a single file. |
| Tier 3: Repo Fixes (Default) | 0 / 24 (0%) | 0/24 under the default config (best_of_n=1, temp=0.2); across all configs, 7 of 8 repo tasks never passed, with task 003 passing 1/3 at best config (best-of-3, temp 0.6) and 3/3 under the surgical prompt. |
| In-VRAM Throughput | 315-340 tok/s | 30B achieved 315 to 340 tok/s decode; 80B MoE achieved 214 tok/s in VRAM. |
Engineering Action Directives
- Local coder models (30B and 80B MoE) have reached high reliability on single-function and single-file refactors, but drop precipitously when required to reason across multiple repository files.
- Autonomous agent workflows should decompose large tasks into localized single-file subagent briefs when targeting local model execution.
Methodology Boundaries & Limits
- Model sample: Evaluated on Qwen3-Coder-30B and Qwen3-Coder-Next-80B MoE; other open weight architectures may exhibit different tier boundaries.
- Benchmark task set: 30 single-function tasks, 12 single-file tasks, and 24 repository-level tasks in our coordinate ascent harness; broader benchmark suites like SWE-bench provide additional difficulty gradations.