When the Edit Harness Makes a Valid Model Patch Fail

A format sweep found fewer application failures, then a splice bug showed why benchmark plumbing needs its own diagnosis.

Core Finding:An edit-format sweep initially favored replacing whole functions over asking a local model for a unified diff. A later harness defect complicated the story: the tool applying replacements could corrupt valid generated code. The result extends our published local-coder study by separating representation failures, application failures, and code that applies but fails its tests.

Empirical Findings Table

After an extraction correction, the initial format sweep recorded 2/10 for unified diff, 1/10 for search-and-replace, and 3/10 for whole-function replacement. The whole-function row reports that four of eight patch failures changed from unapplied or missing edits to edits that applied and then failed tests.

Telemetry Metric	Measured Value	Context & Observability

Engineering Action Directives

A coding benchmark measures a pipeline, not a model in isolation. The pipeline selects context, asks for an edit, extracts the answer, applies it, and runs a checker. A defect in any of those steps can be assigned to the model unless intermediate evidence remains available.
Whole-function replacement can make an editing contract easier to satisfy for a particular model and task set. It also transfers responsibility to the tool that identifies and replaces the target. A simpler output format does not remove complexity; it changes where correctness must be checked.
For AI-built software, the strongest lesson is to retain failure classes and inspect the applied artifact. If a model generated valid code but the tool produced a syntax error, another prompt revision is aimed at the wrong problem. If the patch applies but tests fail, improving patch extraction will not repair the logic.
The small initial pass-rate advantage is consequently a tentative engineering observation. The harness diagnosis is the more durable contribution: a benchmark can become better at distinguishing errors without becoming more flattering to the model. Correcting the tool did not suddenly make the repository tasks reliable.

Methodology Boundaries & Limits

The initial comparison is a small sample from one local model and a fixed task set. Sampling noise was recorded, and the winning format was not independently confirmed on fresh tasks.
Greedy configuration choices compound. Later runs are not isolated estimates of the causal effect of edit format.
Pre-fix rows contain harness-caused failures and cannot support an unqualified model capability claim. The clean repeated reference has a different protocol from the original sweep.
Passing the application step does not establish semantic correctness, completeness, or freedom from regressions beyond the executed checks.