Published by Sean Carr
When the Edit Harness Makes a Valid Model Patch Fail
A format sweep found fewer application failures, then a splice bug showed why benchmark plumbing needs its own diagnosis.
At a Glance
| Telemetry Metric | Measured Value | Context & Observability |
|---|
Published by Sean Carr
View & print executive brief version →Executive Summary & Core Finding
An edit-format sweep initially favored replacing whole functions over asking a local model for a unified diff. A later harness defect complicated the story: the tool applying replacements could corrupt valid generated code. The result extends our published local-coder study by separating representation failures, application failures, and code that applies but fails its tests.
Method
Data from September 2026, analyzed October 2026. The evaluation used a local coding model on a fixed set of repository tasks. A greedy coordinate-ascent loop changed a configuration choice, ran the tasks, and retained the better observed setting as the base for later choices. This was an engineering search, not a fully crossed controlled experiment.
The format comparison included unified diffs, search-and-replace blocks, and whole-function replacement. These are different contracts between a model and the editing tool. A useful answer has to satisfy both the representation contract and the software task. Failure before the edit is applied tells us something different from a test failure after application.
We read the dated evaluation ledger, including corrections and the later repeated runs. We also checked the published local-coder article for overlap. This draft concentrates on the edit application's behavior rather than repeating the broader claim about model capability at different task difficulties.
What We Measured
After an extraction correction, the initial format sweep recorded 2/10 for unified diff, 1/10 for search-and-replace, and 3/10 for whole-function replacement. The whole-function row reports that four of eight patch failures changed from unapplied or missing edits to edits that applied and then failed tests.
Experimental Notes & Observability Details
- That transition is useful diagnostic progress. It is not four additional correct solutions. The representation became usable for those cases, revealing problems in the proposed changes that the earlier failure had hidden. A summary that reports only final passes loses this distinction.
- A later repeated sweep exposed a separate splice defect. The harness treated a dedented line within a multiline function signature as if it ended the function. It could therefore show incomplete context to the model and leave part of the old function behind when inserting a replacement. The ledger identifies four of eight patch tasks as affected by this boundary error.
- The record explicitly supersedes the repeated-run rows marked PRE-FIX. The initial single-run format comparison also predates the splice repair and should be treated as potentially confounded. In a clean post-fix reference run, the whole-function configuration recorded 3/30, with the affected tasks now reaching logic failures rather than harness-induced syntax errors. That repeated-run result used a different configuration and protocol from the initial format sweep. It must not be presented as the later half of a controlled before-and-after accuracy comparison.
What We Think It Suggests
- A coding benchmark measures a pipeline, not a model in isolation. The pipeline selects context, asks for an edit, extracts the answer, applies it, and runs a checker. A defect in any of those steps can be assigned to the model unless intermediate evidence remains available.
- Whole-function replacement can make an editing contract easier to satisfy for a particular model and task set. It also transfers responsibility to the tool that identifies and replaces the target. A simpler output format does not remove complexity; it changes where correctness must be checked.
- For AI-built software, the strongest lesson is to retain failure classes and inspect the applied artifact. If a model generated valid code but the tool produced a syntax error, another prompt revision is aimed at the wrong problem. If the patch applies but tests fail, improving patch extraction will not repair the logic.
- The small initial pass-rate advantage is consequently a tentative engineering observation. The harness diagnosis is the more durable contribution: a benchmark can become better at distinguishing errors without becoming more flattering to the model. Correcting the tool did not suddenly make the repository tasks reliable.
Limits
- The initial comparison is a small sample from one local model and a fixed task set. Sampling noise was recorded, and the winning format was not independently confirmed on fresh tasks.
- Greedy configuration choices compound. Later runs are not isolated estimates of the causal effect of edit format.
- Pre-fix rows contain harness-caused failures and cannot support an unqualified model capability claim. The clean repeated reference has a different protocol from the original sweep.
- Passing the application step does not establish semantic correctness, completeness, or freedom from regressions beyond the executed checks.