Published by Sean Carr
A Fine-Tuned Code Author Failed Its Own Pre-Registered Test
An adversarial audit of the training corpus found a poisoned pattern, and a four-arm eval then showed the tuned model flagged unknowns far less often than the untuned base.
At a Glance
Tuned, no specifics (arm A)
31% (4/13)
Arm A (tuned, no specifics): halt-and-flag 31% (4/13), average rubric 0.69, 7 hallucinations, 1 of 61 cases errored.
Untuned base, no specifics (arm B)
100% (13/13)
Arm B (untuned base, no specifics): halt-and-flag 100% (13/13), average rubric 0.80.
Thesis check against 80% margin
30.77%
(C - A) correctness delta: +4.85%, inside the 10% margin. Arm A halt-and-flag 30.77% vs the 80% threshold: thesis FALSIFIED.
Defective idempotency-key records
25 records
25 records taught a broken idempotency-key derivation as the correct pattern: 16 of one shape and 9 of another, converged on independently by three separate auditors.
| Telemetry Metric | Measured Value | Context & Observability |
|---|---|---|
| Tuned, no specifics (arm A) | 31% (4/13) | Arm A (tuned, no specifics): halt-and-flag 31% (4/13), average rubric 0.69, 7 hallucinations, 1 of 61 cases errored. |
| Untuned base, no specifics (arm B) | 100% (13/13) | Arm B (untuned base, no specifics): halt-and-flag 100% (13/13), average rubric 0.80. |
| Thesis check against 80% margin | 30.77% | (C - A) correctness delta: +4.85%, inside the 10% margin. Arm A halt-and-flag 30.77% vs the 80% threshold: thesis FALSIFIED. |
| Defective idempotency-key records | 25 records | 25 records taught a broken idempotency-key derivation as the correct pattern: 16 of one shape and 9 of another, converged on independently by three separate auditors. |
| Training corpus size | 1,069 records | Corpus size on disk at audit time: 1,069 records (a header plus 1,068 examples). |
| Counter-examples for the 16-record shape | 16:0 | The 16-record shape had 0 counter-examples in the corpus labeling it wrong (16:0). |
| Defective vs correct key-derivation teachers | 25 to 19 | Among records that teach key-derivation doctrine, defective teachers outnumbered correct ones 25 to 19. |
| Poison rate, any idempotency key | 19% | Across all records showing any idempotency key: about 105 correct vs 25 defective, roughly a 19% poison rate for that pattern. |
| Distinct symbols verified real | 200+ | About 200+ distinct symbols were verified real; exact duplicates within the corpus: 0. |
| Tuned, specifics in context (arm C) | 15% (2/13) | Arm C (tuned, specifics in context): halt-and-flag 15% (2/13), average rubric 0.74, 9 hallucinations. |
| Untuned base, specifics in context (arm D) | 100% (13/13) | Arm D (untuned base, specifics in context): halt-and-flag 100% (13/13), average rubric 0.88. |
| Halt-and-flag share of training data | 4.4% | Post-run diagnosis: halt-and-flag examples were only about 4.4% of the training data. |
Published by Sean Carr
View & print executive brief version →Executive Summary & Core Finding
We tested whether a small fine-tuned model could write platform-correct code without the volatile specifics (endpoints, field keys, schemas) in its context, and halt and flag when it did not know one. Before tuning, a seven-auditor review of the 1,069-record training corpus found 25 records teaching a broken idempotency-key pattern as correct. After tuning, the pre-registered eval falsified the thesis: the tuned model flagged 4 of 13 withheld unknowns, while the untuned base flagged 13 of 13.
Method
Data collected Jul 1-2, 2026; written up October 2026.
Two checks on one fine-tuning project.
(1) Corpus audit, July 1, 2026: every example in the supervised fine-tuning corpus was read in full by 7 independent auditor agents (5 slice auditors, 1 dedicated hunter for the known defect class, 1 cross-corpus consistency analyst), and every platform-specific symbol was checked against real code and docs.
(2) Model eval, July 2, 2026: a Gemini Flash-class model was fine-tuned on the corpus and run through a 4-arm design on 61 golden cases: A tuned with no specifics in context (the thesis arm), B untuned base with no specifics, C tuned with specifics retrieved into context, D untuned base with specifics in context. 13 of the cases withheld a specific the model could not know.
Falsification margins were committed to the plan on June 22, 2026, before any run: the thesis fails if arm A's halt-and-flag rate is below 0.80 or if C beats A on correctness by more than 0.10.
What We Measured
Corpus audit and four-arm eval of one fine-tuned code author
Tuned, no specifics (arm A)
31% (4/13)
Arm A (tuned, no specifics): halt-and-flag 31% (4/13), average rubric 0.69, 7 hallucinations, 1 of 61 cases errored.
Untuned base, no specifics (arm B)
100% (13/13)
Arm B (untuned base, no specifics): halt-and-flag 100% (13/13), average rubric 0.80.
Thesis check against 80% margin
30.77%
(C - A) correctness delta: +4.85%, inside the 10% margin. Arm A halt-and-flag 30.77% vs the 80% threshold: thesis FALSIFIED.
Defective idempotency-key records
25 records
25 records taught a broken idempotency-key derivation as the correct pattern: 16 of one shape and 9 of another, converged on independently by three separate auditors.
Training corpus size
1,069 records
Corpus size on disk at audit time: 1,069 records (a header plus 1,068 examples).
Counter-examples for the 16-record shape
16:0
The 16-record shape had 0 counter-examples in the corpus labeling it wrong (16:0).
Defective vs correct key-derivation teachers
25 to 19
Among records that teach key-derivation doctrine, defective teachers outnumbered correct ones 25 to 19.
Poison rate, any idempotency key
19%
Across all records showing any idempotency key: about 105 correct vs 25 defective, roughly a 19% poison rate for that pattern.
Distinct symbols verified real
200+
About 200+ distinct symbols were verified real; exact duplicates within the corpus: 0.
Tuned, specifics in context (arm C)
15% (2/13)
Arm C (tuned, specifics in context): halt-and-flag 15% (2/13), average rubric 0.74, 9 hallucinations.
Untuned base, specifics in context (arm D)
100% (13/13)
Arm D (untuned base, specifics in context): halt-and-flag 100% (13/13), average rubric 0.88.
Halt-and-flag share of training data
4.4%
Post-run diagnosis: halt-and-flag examples were only about 4.4% of the training data.
What We Think It Suggests
- In this run, fine-tuning on a corpus dominated by confident code examples appears to have trained the model away from saying "I don't know". The untuned base flagged every withheld unknown in both of its arms, and both tuned arms flagged far fewer. Our working explanation is class imbalance in the training data, not a ceiling on the method, but we have not yet re-run with a rebalanced corpus.
- The audit result points the same direction from the data side: a pattern that appears many times with no counter-example will be learned as canon. A sampling review would likely have missed it, because the defect was a one-line key derivation inside otherwise correct examples.
Limits
- One tuning run, one base model family, 61 cases with only 13 halt-and-flag cases. The 4/13 vs 13/13 gap is large, but the sample is small.
- The tuned model's dataset snapshot appears to predate the audit's fix pass, so this eval does not measure whether the corpus fixes would have changed the outcome.
- The class-imbalance explanation is a post-hoc diagnosis, not a tested intervention.
- An earlier eval run the same night produced all-zero rubric scores across every arm, which looks like a scoring failure rather than a model result; only the re-run is reported here.
- Golden cases and the rubric were authored in-house; there is no external benchmark here.
Detailed Analysis & Field Observations
We tested whether a small fine-tuned model could write platform-correct code without the volatile specifics (endpoints, field keys, schemas) in its context, and halt and flag when it did not know one. Before tuning, a seven-auditor review of the 1,069-record training corpus found 25 records teaching a broken idempotency-key pattern as correct. After tuning, the pre-registered eval falsified the thesis: the tuned model flagged 4 of 13 withheld unknowns, while the untuned base flagged 13 of 13.
Data collected Jul 1-2, 2026; written up October 2026.