Our Determinism Hash Was Wrong on 74.3% of Steps

A cache keyed on buffer identity instead of content served stale digests, and our own falsifier tested the one component that could not fail.

Core Finding:The engine's per-step state hash, which replicas and replays compare against, was served from a digest cache whose guard keyed on a buffer's pointer and size rather than its content. Over a 30,000-step run, 22,290 steps (74.3%) carried a wrong hash, starting at step 7,710. A content-version counter fixed it to 0 of 30,000, and no committed golden moved.

Wrong state hash before fix

74.3%

Before fix: 22,290 of 30,000 steps had a wrong state hash (74.3%), the first at step 7,710.

Wrong state hash after fix

0 of 30,000

After fix: 0 of 30,000.

Root trigger, component size

21,508 B

Root trigger: one component grew past the cache's size floor by step ~1,470, saturated at 21,508 B by step 10,000, and then held a constant length with changing content every 30 steps.

Differential test, cache on

tick 12

Differential test before fix with cache on: first divergence at tick 12 instead of the expected 6; cache off: 6. After fix: 6 in both arms.

Empirical Findings Table

A 30,000-step run comparing cached and re-walked state hashes

Telemetry Metric	Measured Value	Context & Observability
Wrong state hash before fix	74.3%	Before fix: 22,290 of 30,000 steps had a wrong state hash (74.3%), the first at step 7,710.
Wrong state hash after fix	0 of 30,000	After fix: 0 of 30,000.
Root trigger, component size	21,508 B	Root trigger: one component grew past the cache's size floor by step ~1,470, saturated at 21,508 B by step 10,000, and then held a constant length with changing content every 30 steps.
Differential test, cache on	tick 12	Differential test before fix with cache on: first divergence at tick 12 instead of the expected 6; cache off: 6. After fix: 6 in both arms.
Determinism battery after fix	21/21	Determinism battery after fix: 21/21 green; pinned hash sequence unchanged.

Engineering Action Directives

Cache guards that key on where data lives rather than what it contains can pass every test that changes data size, and fail on the common case of same-length overwrites. Here the existing falsifier mutated the one component that had its own dirty tracking, so its red result showed the guard working only where the guard was not needed.
The bug only became live when an earlier fix made the cache start hitting. A cache that never hits cannot serve stale data, so improving hit rate can arm latent unsoundness. Our goldens were all pinned in the first few steps, about 7,700 steps before corruption began, which is why none of them caught it.

Methodology Boundaries & Limits

One engine, one compiler (the pointer-preserving copy behavior is toolchain-specific), one long-run scenario.
The 74.3% rate depends on how long the run lasts and how quickly components saturate; a different workload could show a very different rate.
A second consumer of the same cache, in replica comparison, was found and fixed in the same change; we have not measured how often it produced false agreement in practice.