Published by Sean Carr
Our Determinism Hash Was Wrong on 74.3% of Steps
A cache keyed on buffer identity instead of content served stale digests, and our own falsifier tested the one component that could not fail.
At a Glance
Wrong state hash before fix
74.3%
Before fix: 22,290 of 30,000 steps had a wrong state hash (74.3%), the first at step 7,710.
Wrong state hash after fix
0 of 30,000
After fix: 0 of 30,000.
Root trigger, component size
21,508 B
Root trigger: one component grew past the cache's size floor by step ~1,470, saturated at 21,508 B by step 10,000, and then held a constant length with changing content every 30 steps.
Differential test, cache on
tick 12
Differential test before fix with cache on: first divergence at tick 12 instead of the expected 6; cache off: 6. After fix: 6 in both arms.
| Telemetry Metric | Measured Value | Context & Observability |
|---|---|---|
| Wrong state hash before fix | 74.3% | Before fix: 22,290 of 30,000 steps had a wrong state hash (74.3%), the first at step 7,710. |
| Wrong state hash after fix | 0 of 30,000 | After fix: 0 of 30,000. |
| Root trigger, component size | 21,508 B | Root trigger: one component grew past the cache's size floor by step ~1,470, saturated at 21,508 B by step 10,000, and then held a constant length with changing content every 30 steps. |
| Differential test, cache on | tick 12 | Differential test before fix with cache on: first divergence at tick 12 instead of the expected 6; cache off: 6. After fix: 6 in both arms. |
| Determinism battery after fix | 21/21 | Determinism battery after fix: 21/21 green; pinned hash sequence unchanged. |
Published by Sean Carr
View & print executive brief version →Executive Summary & Core Finding
The engine's per-step state hash, which replicas and replays compare against, was served from a digest cache whose guard keyed on a buffer's pointer and size rather than its content. Over a 30,000-step run, 22,290 steps (74.3%) carried a wrong hash, starting at step 7,710. A content-version counter fixed it to 0 of 30,000, and no committed golden moved.
Method
Data collected Aug 7, 2026; written up October 2026.
August 7, 2026.
A 30,000-step headless server run compared the cached hash against a full re-walk of state (cache disabled) at every step, and a 240,000-step census tracked component sizes over time.
A differential test committed the same constant-length write in two arms and checked that the hash moved at the expected step.
The fix replaced the pointer in the cache key with a per-store content version stamped at the store's single component-write site.
What We Measured
A 30,000-step run comparing cached and re-walked state hashes
Wrong state hash before fix
74.3%
Before fix: 22,290 of 30,000 steps had a wrong state hash (74.3%), the first at step 7,710.
Wrong state hash after fix
0 of 30,000
After fix: 0 of 30,000.
Root trigger, component size
21,508 B
Root trigger: one component grew past the cache's size floor by step ~1,470, saturated at 21,508 B by step 10,000, and then held a constant length with changing content every 30 steps.
Differential test, cache on
tick 12
Differential test before fix with cache on: first divergence at tick 12 instead of the expected 6; cache off: 6. After fix: 6 in both arms.
Determinism battery after fix
21/21
Determinism battery after fix: 21/21 green; pinned hash sequence unchanged.
What We Think It Suggests
- Cache guards that key on where data lives rather than what it contains can pass every test that changes data size, and fail on the common case of same-length overwrites. Here the existing falsifier mutated the one component that had its own dirty tracking, so its red result showed the guard working only where the guard was not needed.
- The bug only became live when an earlier fix made the cache start hitting. A cache that never hits cannot serve stale data, so improving hit rate can arm latent unsoundness. Our goldens were all pinned in the first few steps, about 7,700 steps before corruption began, which is why none of them caught it.
Limits
- One engine, one compiler (the pointer-preserving copy behavior is toolchain-specific), one long-run scenario.
- The 74.3% rate depends on how long the run lasts and how quickly components saturate; a different workload could show a very different rate.
- A second consumer of the same cache, in replica comparison, was found and fixed in the same change; we have not measured how often it produced false agreement in practice.
Detailed Analysis & Field Observations
The engine's per-step state hash, which replicas and replays compare against, was served from a digest cache whose guard keyed on a buffer's pointer and size rather than its content. Over a 30,000-step run, 22,290 steps (74.3%) carried a wrong hash, starting at step 7,710. A content-version counter fixed it to 0 of 30,000, and no committed golden moved.
Data collected Aug 7, 2026; written up October 2026.