Published by Sean Carr
Record What the Agent Did Before Asking a Model to Summarize It
A handoff system separated deterministic event recording from optional narrative summaries.
At a Glance
| Telemetry Metric | Measured Value | Context & Observability |
|---|
Published by Sean Carr
View & print executive brief version →Executive Summary & Core Finding
A session summary can help explain work, but it should not be the only record that work happened. We examined a handoff system that stored structured session state and git events through deterministic code, then offered an optional model summary. Its demonstration showed a large latency difference between raw retrieval and summarization, with important differences in what each output provided.
Method
Data from October 2026, analyzed October 2026. The system parsed a structured handoff block from engineering session logs and indexed its contents. Separate repository hooks recorded commit and push events. A status query could return raw rows without a model call or send retrieved material through a model for a narrative answer.
This replaced the need to ask a model to rediscover basic event facts on every status lookup. It did not eliminate human or agent judgment from handoff authoring. Descriptions of what was done, what remained open, and what should happen next still depended on whoever wrote the session record.
We read the implementation report for the backfill, timing demonstration, and tests. We treat those measurements as a historical system demonstration, not a controlled benchmark of equivalent answers. The raw response and the summarized response had different content and were not shown to users in a preference test.
What We Measured
The report records a 30-day git-event backfill with 5,445 rows across approximately 30 repositories, completed in 1.6 seconds. This was a bulk import of existing event history. It is not a measurement of sustained ingest under concurrent live load.
Experimental Notes & Observability Details
- The session-log hook launched a detached update in under 20 milliseconds. That timing describes starting the background work, not the time until the resulting record was durable or visible to a query. Confusing launch latency with completion latency would make the system look more responsive than the evidence establishes.
- In the demonstration, a raw status query took 0.2 seconds. A summarized project query took 16.6 seconds and used one model call. These timings establish that the raw path avoided the summarization step in that demonstration. They do not establish an equivalent-answer speedup: the query selection and output form differed.
- The report also records 213/213 tests passing. The covered behaviors included real temporary repositories and hooks, repeated ingestion, filtering, changed-log synchronization, and the no-model raw path. Those tests support specific implementation behaviors, rather than an assertion that every event in a distributed workflow will always be captured correctly.
What We Think It Suggests
- An agent's activity record and an explanation of its activity serve different purposes. Event recording can answer whether a repository action was observed. A narrative can explain why it mattered and how it relates to unfinished work. Combining them at the point of capture makes both harder to inspect.
- For AI-built software, deterministic telemetry offers a useful check on fluent self-report. An agent may say it shipped a change when it only created a local commit. A recorded event history can help a later reviewer distinguish those states, provided the recording semantics are clear. A push attempt still needs interpretation before it becomes evidence that a remote accepted the change.
- The system therefore suggests a layered status workflow: capture observable facts, retrieve them directly when that answers the question, and summarize when interpretation adds value. The model becomes a reader of the record rather than the sole creator of the record.
- This does not make the record complete. Intent, unresolved decisions, and reasons for abandoning an approach are not fully represented by repository events. Structured handoffs remain useful for those facts, and the eventual answer still needs source checking. The improvement is a more inspectable division of labor, not an automatic guarantee of accurate memory.
Limits
- The latency figures are a demonstration with different query and output forms, not repeated measurements of equivalent answers. No general speedup ratio is claimed.
- Detached launch latency excludes background completion and durability. The report does not provide a distribution of end-to-end ingestion delays.
- Remote push reconciliation was explicitly untested against a real remote in this report. Event capture alone cannot certify successful delivery.
- Structured handoff content is authored and can be wrong or incomplete. Deterministic parsing preserves a statement; it does not verify the statement's truth.