Published by Sean Carr
Falsifying Transformer Activation Cache Reuse for Code Edits
Empirical measurement of hidden state divergence indicates mid-network edit-delta injection fails downstream.
At a Glance
Residual Dirtiness
>90% at Layer 12
Hidden state divergence propagated to >90% of subsequent tokens by layers 12 to 15.
Delta Cosine Similarity
0.06 - 0.13
Final-position cross-context delta cosine collapsed, indicating complete decorrelation.
Top-1 Token Agreement
0.91 - 0.98
Downstream token agreement dropped below the 0.99 correctness threshold required for code generation.
Net Compute Saved
~3 - 4%
Net compute savings was only 3 to 4% at layer 1, completely erased by injection overhead.
| Telemetry Metric | Measured Value | Context & Observability |
|---|---|---|
| Residual Dirtiness | >90% at Layer 12 | Hidden state divergence propagated to >90% of subsequent tokens by layers 12 to 15. |
| Delta Cosine Similarity | 0.06 - 0.13 | Final-position cross-context delta cosine collapsed, indicating complete decorrelation. |
| Top-1 Token Agreement | 0.91 - 0.98 | Downstream token agreement dropped below the 0.99 correctness threshold required for code generation. |
| Net Compute Saved | ~3 - 4% | Net compute savings was only 3 to 4% at layer 1, completely erased by injection overhead. |
Published by Sean Carr
View & print executive brief version →Executive Summary & Core Finding
Can language models avoid recomputing activations after small code edits by injecting difference vectors into cached hidden states? In an empirical study across Qwen2.5 models on 24 code contexts, we falsified activation delta caching: hidden state divergence crossed 90% by layers 12 to 15, yielding only 3-4% net compute savings while corrupting downstream token agreement.
Method
Evaluated Mode D Activation Delta Caching across three Qwen2.5 model scales: 0.5B, 1.5B, and 3B parameters.
Tested across 24 real-world code contexts with 7 realistic code edit archetypes (renaming, adding arguments, modifying conditionals).
Measured layer-by-layer hidden state dirtiness, final-position cross-context cosine similarity, and downstream top-1 token agreement.
Tracked net compute savings after accounting for delta extraction and injection overhead.
What We Measured
Hidden state divergence and top-1 agreement degradation across 24 code contexts
Residual Dirtiness
>90% at Layer 12
Hidden state divergence propagated to >90% of subsequent tokens by layers 12 to 15.
Delta Cosine Similarity
0.06 - 0.13
Final-position cross-context delta cosine collapsed, indicating complete decorrelation.
Top-1 Token Agreement
0.91 - 0.98
Downstream token agreement dropped below the 0.99 correctness threshold required for code generation.
Net Compute Saved
~3 - 4%
Net compute savings was only 3 to 4% at layer 1, completely erased by injection overhead.
Experimental Notes & Observability Details
- Reusing cached deltas reduced KL divergence by 4x to 27x in initial layers, but non-linear feedforward projections quickly magnified residual errors.
- The theoretical premise that small code edits produce localized, reusable activation shifts was refuted by the empirical data.
What We Think It Suggests
- Mid-network hidden state differences from small text edits diverge rapidly across transformer layers, rendering activation delta caching ineffective for code editing.
- Standard KV-cache prefix reuse remains the mathematically sound strategy for incremental code generation.
Limits
- Model family tested: Evaluated across Qwen2.5 model sizes (0.5B, 1.5B, 3B); larger 70B+ architectures with deeper residual streams may exhibit slightly different divergence rates.
- Edit size: Tested on single-token to small-span modifications; larger architectural edits would cause even faster divergence.
Detailed Analysis & Field Observations
In high-frequency developer workflows, language models repeatedly process large source files where only a single line or identifier has changed. Standard KV-cache reuse allows caching tokens preceding the edit, but tokens after the edit must typically be recomputed from scratch. A theoretical proposal, which we designated 'Mode D Activation Delta Caching', hypothesized that the difference in hidden states (the delta) introduced by a small edit could be added directly to cached activations mid-network, bypassing full recomputation.
To test this hypothesis rigorously, we instrumented Qwen2.5 (0.5B, 1.5B, and 3B) across 24 code contexts, applying 7 common code edit archetypes. We tracked how activation deltas propagated through self-attention and MLP blocks layer by layer.
The empirical data conclusively refuted the hypothesis. While delta injection suppressed initial KL divergence in early layers, non-linear activation functions (SwiGLU) and multi-head attention mixing rapidly amplified divergence. By layers 12 to 15, hidden state dirtiness exceeded 90%, meaning over 90% of downstream token representations were significantly perturbed.
At the final transformer layer, cross-context cosine similarity of the delta vectors collapsed to between 0.06 and 0.13, indicating near-total decorrelation. Consequently, downstream top-1 token agreement fell to 0.91 to 0.98, far below the 0.99 threshold required to prevent code syntax errors.
Furthermore, calculating, scaling, and injecting the activation deltas added memory bandwidth overhead that eroded compute gains. Net compute savings was measured at only 3 to 4% at layer 1. We formally closed the research track: prefix-based KV caching remains the only viable strategy for incremental code inference.