83% Fewer Tokens, Matched Accuracy on 100 Questions: Semantic Memory vs Pasting the Whole History

Detailed Analysis & Field Observations

In modern conversational AI and agent engineering, the prevailing default for handling extended interactions is brute-force context stuffing: as long as a model supports a 1M+ token context window, applications simply paste the entire historical chat transcript into the prompt on every turn. While technically straightforward, this approach imposes severe operational costs: sending 100,000+ tokens on every turn causes linear cost explosion, multi-second latency inflation, and degradation where reasoning models struggle to attend reliably across massive context spans.

The standard alternative has been naive chunk-based Retrieval-Augmented Generation (RAG): chunking text into 400-token blocks and retrieving top-k chunks via dense vector similarity. However, on complex multi-session reasoning, such as tracking state progression over time, resolving user preferences, or synthesizing temporal sequences, naive chunk RAG consistently fails because chunk boundaries fracture discourse structure.

To resolve this trade-off, we engineered a semantic memory architecture that converts ~115,000-token chat histories once into an append-only SQLite store of structured semantic events (recording agent, action, patient, timestamps, certainty, evidence quotes, and resolved entity IDs) backed by local vector embeddings. Questions are answered through a multi-tier fallback ladder designed to satisfy queries with minimal token expenditure.

On the public LongMemEval_S cleaned benchmark, evaluating 100 never-seen questions over ~115k-token chat histories with an independent Claude Sonnet judge, our semantic memory architecture (Arm S) achieved 88/100 accuracy, matching the full-context baseline (Arm A, 88/100) on this 100-question set while consuming only 19,500 tokens per question (17.2% of Arm A's 113,600 tokens). This represents an 82.8% (~83%) reduction in token consumption with zero loss in task accuracy on this sample. Meanwhile, standard chunk RAG (Arm B) collapsed to 77/100, trailing semantic memory by 11 points (a statistically significant margin under paired McNemar testing, p = 0.027).

The efficiency of the semantic memory system is governed by its multi-tier fallback ladder. Rather than guessing or committing to a single representation, retrieval executes sequentially: Rung 1 constructs a compact working set of structured events and cited source spans within a strict 4k-token budget. If the model determines evidence is insufficient or cannot cite a valid event ID, it escalates to Rung 2, retrieving the full source sessions of the top candidate events (<=12k tokens). Only if Rung 2 remains insufficient does execution fall back to Rung 3 (full context).

In total, 82% of all questions were answered cleanly from Rung 1 alone, 10% required whole sessions at Rung 2, and only 8% required full-context fallback at Rung 3. Furthermore, 4 questions triggered an early L2 'not-in-memory' exit before Rung 3, and independent audit confirmed that all 4 were correct abstentions on unanswerable questions.

In adherence to our Empirical Verification Standard, all quantitative hypotheses were pre-registered and committed to version control prior to scoring. Clause 1 (Arm S accuracy within 2 points of full context) was CONFIRMED (88 vs 88). Clause 3 (Arm S outperforming chunk RAG by >= 10 points) was CONFIRMED (+11 points, paired McNemar test, p = 0.027). However, Clause 2, which predicted that Arm S tokens would be <= 15% of full context, was explicitly REFUTED by the empirical data at 17.2%. Transparently publishing refuted prediction clauses is essential to maintaining scientific integrity across AI systems benchmarks.

To test whether these findings were stable beyond a single test split, we conducted a follow-up holdout evaluation (freshC) across an additional 100 fresh, never-seen questions with Sonnet judging. The merged pipeline reproduced the same performance characteristics on this second sample, achieving 89/100 accuracy at approximately 16% of full-context tokens with an estimated retrieval recall of ~99%. For long-running conversational systems, semantic event memory with fallback ladders eliminates the economic and latency burden of pasting whole histories without compromising reasoning accuracy on these benchmarks.

Current synthesis: Semantic Memory Living Paper v1.