Published by Sean Carr
The Local Coder Ceiling: 100% on Single Functions, 0% on Multi-File Bug Fixes Under Default Config
Coordinate ascent benchmarks on 30B and 80B MoE models: 0/24 under the default config; across all configs, 7 of 8 repo tasks never passed, with task 003 passing 1/3 at the best config.
At a Glance
Tier 1: Single Function
30 / 30 (100%)
Both 30B and 80B MoE achieved perfect pass rates on isolated function edits.
Tier 2: Single File
12 / 12 (100%)
Flawless execution on multi-function refactors contained within a single file.
Tier 3: Repo Fixes (Default)
0 / 24 (0%)
0/24 under the default config (best_of_n=1, temp=0.2); across all configs, 7 of 8 repo tasks never passed, with task 003 passing 1/3 at best config (best-of-3, temp 0.6) and 3/3 under the surgical prompt.
In-VRAM Throughput
315-340 tok/s
30B achieved 315 to 340 tok/s decode; 80B MoE achieved 214 tok/s in VRAM.
| Telemetry Metric | Measured Value | Context & Observability |
|---|---|---|
| Tier 1: Single Function | 30 / 30 (100%) | Both 30B and 80B MoE achieved perfect pass rates on isolated function edits. |
| Tier 2: Single File | 12 / 12 (100%) | Flawless execution on multi-function refactors contained within a single file. |
| Tier 3: Repo Fixes (Default) | 0 / 24 (0%) | 0/24 under the default config (best_of_n=1, temp=0.2); across all configs, 7 of 8 repo tasks never passed, with task 003 passing 1/3 at best config (best-of-3, temp 0.6) and 3/3 under the surgical prompt. |
| In-VRAM Throughput | 315-340 tok/s | 30B achieved 315 to 340 tok/s decode; 80B MoE achieved 214 tok/s in VRAM. |
Published by Sean Carr
View & print executive brief version →Executive Summary & Core Finding
Can current consumer-run local coding models replace frontier cloud models for autonomous software engineering? In rigorous coordinate ascent benchmarks across 30B and 80B MoE models, we measured 100% pass rates on single functions and single-file refactors, but 0/24 under the default config on repository-level bug fixes. Across all tested configs, 7 of 8 repo tasks never passed; task 003 reached 1/3 at the best config (best-of-3, temp 0.6) and 3/3 under a surgical system prompt.
Method
Evaluated two high-capability local coder models: Qwen3-Coder-30B (dense) and Qwen3-Coder-Next-80B MoE (3B active parameters).
Designed a three-tier difficulty ladder: Tier 1 (30 single-function algorithmic tasks), Tier 2 (12 multi-function single-file refactors), and Tier 3 (24 repository-level bug fixes requiring multi-file edits and test adaptations).
Executed coordinate ascent sweeps testing prompt framing, raw diff vs whole-file replacement, and multi-turn iterative repair.
All executions evaluated against deterministic unit test suites under identical VRAM allocations.
What We Measured
Pass rates across task difficulty tiers on identical hardware
Tier 1: Single Function
30 / 30 (100%)
Both 30B and 80B MoE achieved perfect pass rates on isolated function edits.
Tier 2: Single File
12 / 12 (100%)
Flawless execution on multi-function refactors contained within a single file.
Tier 3: Repo Fixes (Default)
0 / 24 (0%)
0/24 under the default config (best_of_n=1, temp=0.2); across all configs, 7 of 8 repo tasks never passed, with task 003 passing 1/3 at best config (best-of-3, temp 0.6) and 3/3 under the surgical prompt.
In-VRAM Throughput
315-340 tok/s
30B achieved 315 to 340 tok/s decode; 80B MoE achieved 214 tok/s in VRAM.
Experimental Notes & Observability Details
- Under the default config (best_of_n=1, temp=0.2), Tier 3 scored 0/24 across 8 repo tasks with failure modes including editing the wrong file, omitting imports in adjacent files, or introducing syntax regressions.
- Across all tested configurations, 7 of 8 repo tasks (004-010) never passed. Task 003 passed 1/3 at the best config (best-of-3, temp 0.6) and 3/3 under the surgical system prompt (which broke task 002 from 3/3 to 0/3).
What We Think It Suggests
- Local coder models (30B and 80B MoE) have reached high reliability on single-function and single-file refactors, but drop precipitously when required to reason across multiple repository files.
- Autonomous agent workflows should decompose large tasks into localized single-file subagent briefs when targeting local model execution.
Limits
- Model sample: Evaluated on Qwen3-Coder-30B and Qwen3-Coder-Next-80B MoE; other open weight architectures may exhibit different tier boundaries.
- Benchmark task set: 30 single-function tasks, 12 single-file tasks, and 24 repository-level tasks in our coordinate ascent harness; broader benchmark suites like SWE-bench provide additional difficulty gradations.
Reproduce It (Data & Benchmarks)
- Harness: ModelHost coordinate ascent sweeps with clean instruction scaffolding.
- Tasks: Tier 1 (30 single-function edits), Tier 2 (12 single-file refactors), Tier 3 (24 multi-file production bug fixes).
- Models: Qwen3-Coder-30B and Qwen3-Coder-Next-80B MoE UD-Q2_K_XL in VRAM.
- Raw pass counts: Tier 1 30/30 (100%), Tier 2 12/12 (100%), Tier 3 0/24 (0%) under default config; across all configs, 7 of 8 repo tasks never passed (task 003 passed 1/3 at best config, 3/3 with surgical prompt).
Data and prompt templates only; no proprietary code is published.
Detailed Analysis & Field Observations
With the release of capable 30B and 80B open-weight coding models, there is substantial enthusiasm for running autonomous software engineering entirely on local hardware. The appeal is obvious: zero API token costs, complete data privacy, and fast generation speeds exceeding 300 tokens per second.
To determine the operational boundary of local coding models, we constructed a coordinate ascent benchmark evaluating Qwen3-Coder-30B and Qwen3-Coder-Next-80B MoE across three difficulty tiers.
The results in this test demonstrated a sharp capability boundary. On Tier 1 and Tier 2 tasks, both models performed reliably when configured with clean instruction scaffolding. However, on Tier 3 tasks, where a bug in one module required updating an import, adapting a downstream fixture, and fixing an adjacent regression, both models scored 0/24 under the default configuration (best_of_n=1, temp=0.2). Across all configurations tested in coordinate ascent, 7 of 8 repository tasks never passed a single attempt. Task 003 (dotted-key merge write) was the sole exception, passing 1/3 repeats under the best configuration (best-of-3, temp 0.6) and 3/3 under a surgical system prompt (which regressed task 002 from 3/3 to 0/3).
The primary failure mode on repository-level tasks was attention fragmentation. When presented with multi-file contexts, local models frequently hallucinated nonexistent methods on imported classes, introduced subtle indentation errors, or repeatedly reverted edits made in earlier turns.
This benchmark defines the current envelope for local coding agents: they are exceptional copilots for localized implementation, but cannot currently be trusted with unsupervised repository refactoring. Real-world agent systems must partition complex features into single-file tasks before delegating to local models.
Correction (October 8, 2026): An earlier version of this article reported 0/24 (0%) on Tier 3 repository-level bug fixes without specifying that this figure applied only to the default configuration (best_of_n=1, temp=0.2). In coordinate ascent sweeps across alternative configurations, task 003 passed 1/3 at the best configuration (best-of-3, temp 0.6) and 3/3 under a surgical system prompt (which regressed task 002 from 3/3 to 0/3). Across all configurations, 7 of 8 repository tasks never passed.