Falsifying Transformer Activation Cache Reuse for Code Edits

Empirical measurement of hidden state divergence indicates mid-network edit-delta injection fails downstream.

Core Finding:Can language models avoid recomputing activations after small code edits by injecting difference vectors into cached hidden states? In an empirical study across Qwen2.5 models on 24 code contexts, we falsified activation delta caching: hidden state divergence crossed 90% by layers 12 to 15, yielding only 3-4% net compute savings while corrupting downstream token agreement.

Residual Dirtiness

>90% at Layer 12

Hidden state divergence propagated to >90% of subsequent tokens by layers 12 to 15.

Delta Cosine Similarity

0.06 - 0.13

Final-position cross-context delta cosine collapsed, indicating complete decorrelation.

Top-1 Token Agreement

0.91 - 0.98

Downstream token agreement dropped below the 0.99 correctness threshold required for code generation.

Net Compute Saved

~3 - 4%

Net compute savings was only 3 to 4% at layer 1, completely erased by injection overhead.

Empirical Findings Table

Hidden state divergence and top-1 agreement degradation across 24 code contexts

Telemetry Metric	Measured Value	Context & Observability
Residual Dirtiness	>90% at Layer 12	Hidden state divergence propagated to >90% of subsequent tokens by layers 12 to 15.
Delta Cosine Similarity	0.06 - 0.13	Final-position cross-context delta cosine collapsed, indicating complete decorrelation.
Top-1 Token Agreement	0.91 - 0.98	Downstream token agreement dropped below the 0.99 correctness threshold required for code generation.
Net Compute Saved	~3 - 4%	Net compute savings was only 3 to 4% at layer 1, completely erased by injection overhead.

Engineering Action Directives

Mid-network hidden state differences from small text edits diverge rapidly across transformer layers, rendering activation delta caching ineffective for code editing.
Standard KV-cache prefix reuse remains the mathematically sound strategy for incremental code generation.

Methodology Boundaries & Limits

Model family tested: Evaluated across Qwen2.5 model sizes (0.5B, 1.5B, 3B); larger 70B+ architectures with deeper residual streams may exhibit slightly different divergence rates.
Edit size: Tested on single-token to small-span modifications; larger architectural edits would cause even faster divergence.