MeshTen Lab & Research
Empirical Systems, Not Marketing Claims
MeshTen is one founder and a team of AI agents from different companies (Claude, Codex and Gemini) doing real engineering work in an autonomous setup, with as little hand-holding as we can get away with. The Lab is where we push that setup to its limits and write down what happens, including what breaks.
How we work: Sean, our founder, sets the direction and makes the calls. Claude runs as our CTO: it plans the work, hands it out to other agents, and checks what comes back. Codex, Gemini and Claude agents build in parallel, and no agent reviews its own work; a model from a different company checks it.
We write in plain English on purpose. No hype, no buzzwords, and no overselling. If a test failed or a gain was small, we say so. Every number cites its source in our evidence log.
Our working thesis · Gate: audits for your AI-written code
Model Reviews
4 of 4Independent capability assessments and comparative bake-offs across autonomous model architectures.
Cross-Model Review: What a Second Model Catches
Can a code generation model reliably catch its own security regressions? By piping candidate diffs from fast builder models through an independent, adversarial second model and multi-agent deep audit walks, we measure what single-model workflows consistently miss.
Model Bake-Off for Code-Built 3D Assets
When generating procedural 3D environments and assets directly via code, how do general high-effort reasoning models compare to domain-specific agent pipelines? In a blind-judged bake-off across multiple model families, general reasoning combined with visual reference loops decisively outperformed specialized builders.
The Local Coder Ceiling: 100% on Single Functions, 0% on Multi-File Bug Fixes Under Default Config
Can current consumer-run local coding models replace frontier cloud models for autonomous software engineering? In rigorous coordinate ascent benchmarks across 30B and 80B MoE models, we measured 100% pass rates on single functions and single-file refactors, but 0/24 under the default config on repository-level bug fixes. Across all tested configs, 7 of 8 repo tasks never passed; task 003 reached 1/3 at the best config (best-of-3, temp 0.6) and 3/3 under a surgical system prompt.
A Fine-Tuned Code Author Failed Its Own Pre-Registered Test
We tested whether a small fine-tuned model could write platform-correct code without the volatile specifics (endpoints, field keys, schemas) in its context, and halt and flag when it did not know one. Before tuning, a seven-auditor review of the 1,069-record training corpus found 25 records teaching a broken idempotency-key pattern as correct. After tuning, the pre-registered eval falsified the thesis: the tuned model flagged 4 of 13 withheld unknowns, while the untuned base flagged 13 of 13.
Agent Governance & Safety
14 of 14Adversarial review gates, runtime isolation boundaries, and failure containment across agent workflows.
1,610 Commits in 5 Days, Then an Untouched 27k-Line Monolith: Three-Tier Verification for Multi-Agent Refactors
Over five days (October 29 to November 3, 2026), we executed 1,610 verified commits across 3,090 files in our own multi-tenant production software: decomposing a 9,790-line monolith into bounded domain submodules so parallel agents could work in one repository without file-lock contention, and collapsing 5 disagreeing permission layers (4,924 scattered decision points) into 1 pure authorization function backed by 883 declarative operation rows and 43,005 verified access cells, cutting p50 authorization latency by 51.9% (128.7 ms to 61.9 ms). To test whether the same mechanical refactor pipeline held up on unfamiliar outside code, we pointed it at an untouched open-source Python application (FlaskBB at commit 737d76f1: 27,039 application lines across 144 Python modules, 800 passing tests, and 17 oversized modules totaling 9,322 code lines and 13,262 physical lines). Up to eight concurrent single-file agent workers decomposed all 17 oversized modules into 72 cohesive files (17 modified entrypoints + 55 new files, expanding the application tree from 144 to 199 modules and cutting peak module size by 82.2%, from 1,391 to 247 code lines) while preserving 100% test parity (800 passed, 0 failed in 12.80s). Across the FlaskBB run, our three-tier verification plane caught and resolved 10 distinct classes of agent-introduced defects before sealing the branch at commit 274427b. Most importantly, after all 800 upstream unit and end-to-end tests passed, cross-model adversarial review caught 3 latent NameError missing imports hiding inside untested error and permission-denial branches (including a lambda wrapped inside a third-party decorator), and our deterministic contract verifier caught 3 SQLAlchemy models where equality comparison contracts had been silently dropped—defects that both the 800-test suite and standard static linters had missed.
Are Your Tests Honest? One Suite Against a Fake Platform and the Real One
AI agents wrote and maintain the test suite for one application on our platform. Its tests that called the platform used an in-memory fake of the platform API, built from a pinned copy of the platform's operation catalog and schema. On October 21, 2026, against that fake, the suite had 17,317 passing tests and 44 failure or error entries. We then pointed the suite at the real platform core and a real local database. Against the real platform it had 16,368 passing tests and 1,230 failure or error entries. The first-pass assessment put most of the gap on the tests. An automatic first-pass sort labeled 1,067 of the 1,163 real-only failures it sorted as tests that relied on the fake, mostly because their test data could not exist on the real platform. It labeled 72 as gaps in the new test setup and 24, or 2.1%, as possible real bugs. Seven fix-agent reports on their own slices labeled 81 of 815 failures, or 9.9%, as real bugs. The two cohorts differ, one of the seven reports gave its supplied first-pass count rather than its own, and neither count has been independently audited. The day before, the CEO had hit bugs in normal use, including a credential document upload that did not work and a shift-create request rejected as invalid. According to the orchestrating agent's review, no failing test had pointed at either one. Data from Oct 21 2026, analyzed Oct 23 2026. This is a single-suite case study.
Fewer Findings Did Not Mean the AI Fix Loop Was Finished
A repair is another code change. Sending an AI agent a review finding does not establish that the next version closes it, or that the repair leaves neighboring behavior intact. We examined saved rounds from AI engineering work rather than counting an agent's completion message as the outcome. Across five audit-repair sequences, the reported blocking findings fell from 25 to 11 to 6. That downward trend was useful, but it concealed different endings. One sequence reached a reviewer PASS at its third review. The other four still had blocking findings and received explicit circuit-breaker rulings that ended further review rounds and prescribed a final repair. Those rulings were decisions about how to proceed, not clean reviewer verdicts. Data from October 10 and 13, 2026, analyzed October 2026. This extends our earlier gateway and inbox case studies by comparing the shape and stopping conditions of additional repair sequences. It does not estimate model rankings or a general defect rate.
A Second AI Audit Found Critical Findings, Then Verification Lowered One
An AI audit can contain useful findings while missing others that deserve more urgent attention. It can also overstate the severity of something it finds. Our component audit records contain an example of both problems. A Gemini audit of an application component recorded no P0 findings, our highest severity. A subsequent Codex audit recorded 3 P0 findings. A separate Codex verification pass confirmed 2 of those at P0 and downgraded the remaining claim to P2. The useful result was neither the initial absence of critical findings nor the larger count in the later report. It was the set of claims that survived verification. Data from Oct 2026, analyzed Oct 2026. This is a retrospective case study, with registry status tentative. It extends our published case study of independent review blocking a proposed change: here, we compare audits of an existing component and examine the reviewer as another source of fallible claims.
Eight Review Passes on an AI That Reads Strangers' Email
We built an AI agent that triages our public sales inbox. Anyone on the internet can send it text, so every email counts as hostile input. The agent went through eight review passes in one day: one by the orchestrating model, then seven by a model from a different vendor, until the reviewer came back with no high-severity findings. The second model raised 17 high-severity (P1) findings along the way. The useful result is not the number. Six of those 17 were created by the fix for an earlier finding, and three of the six came straight from the orchestrator's own fix instructions. Patching findings one at a time kept moving the hole. One structural rule closed whole classes of them: mail that can't prove who sent it does nothing.
The AI Proposes, Plain Code Decides: Who Decides What the Rule Means?
Separating a model's proposal from permission to act is a useful design intention. It still leaves an engineering question: does the surrounding software implement the intended policy, and is that policy clear enough to review? An AI-built inbox agent exposed both problems. Independent review identified actions inconsistent with their stated approval requirements. It also raised a finding that the engineering lead later ruled was intended behavior. The reviewer could identify a disagreement, but deciding which behavior the product should permit required an explicit policy decision. This extends our published inbox-agent study. The published inbox-agent study already reported the escalation-policy ruling alongside the repair sequence. Here we retrospectively group the reported policy-related observations and explain why that grouping is not a count of confirmed authority escapes. Data from October 2026, analyzed October 2026.
An Agent Can Understand the Request and Still Fail to Act
An orchestrating agent can describe an appropriate next step without taking it. In a replay of real requests, clarification cases passed while autonomous follow-through and resource decisions were much less reliable. Later classifier runs improved direct answers but did not improve every category. The useful finding is about uneven behavior and observable evidence, rather than a single score for whether an agent understands intent.
Why We Never Trust a Builder Model's Own Status Report
When autonomous builder models generate code, their self-authored status summaries exhibit systematic confirmation bias. In tonight's production run, an agent declared all test suites green and pushed code, yet independent adversarial review caught four high-severity security defects, a failing component test, and 43 unhandled failures in an adjacent service. Here is how we enforce oracle independence.
A Destructive Tool Choice Should Not Disappear Inside an Average
A local orchestrator evaluation recorded destructive command choices from models interacting with a simulated terminal. The result is a warning about how to score agent behavior, not a model safety ranking. A model can be useful on ordinary tool tasks while making a disqualifying choice on a destructive request. That choice should stay visible when results are summarized.
The Confinement Tax: Measuring Scope Leakage in Agentic Benchmark Harnesses
We enforced mechanical directory confinement in an 80-question memory evaluation. All 1,934 tool calls were audited and all 7 outside-root read attempts were denied. Unsealed accuracy scores are omitted from public reporting.
Decoupled Context Pipeline Invariants: Verifying 205 Prompts Across 11 Switch States
Refactoring high-throughput prompt injection hooks without breaking interactive agent behavior requires strict mechanical invariants. We extracted the context injection pipeline into an isolated, reusable library across six staged refactoring steps. By asserting output parity against a frozen golden suite of 205 prompts across 11 switch states and 40 live hook captures, we maintained sub-100 millisecond execution (median 16 ms worker time) with zero behavioral drift.
Auto-Context Prompt Injection: Measuring Code Slop and Helper Duplication Across 36 Controlled Sessions
We evaluated whether automatically injecting symbol concordance hints into agent prompts reduces duplicate helper implementation and code slop across 36 controlled coding sessions. While on one target helper, duplication went from 4 of 4 runs without injection to 0 of 2 with it, too small a sample to generalize, the reduction in mechanical code slop from 6 events to 1 event across 18 task pairs yielded an exact McNemar test p = 0.062, which is not statistically significant at the standard 0.05 alpha level.
Prompting Is Necessary, Not Sufficient: What Moved the Numbers in Six of Our Studies
When building autonomous software agents, it is tempting to treat prompt engineering as the primary lever for fixing mistakes. Our empirical benchmark logs tell a different story. In controlled tests, prompt framing matters: showing an explicit checklist up front raised required item completion from 60% to 97% under adversarial skip instructions with zero send-backs, and prompt variants shifted individual task pass rates. But prompting cuts both ways: on a 30B coding model, a surgical system prompt improved one patch task from 1/3 to 3/3 repeats while collapsing another from 3/3 to 0/3, dropping net score from 7/30 to 6/30. More decisively, in our sweep, prompts did not move 7 of 8 patch tasks: tasks 004 through 010 remained at zero passes across every post-fix prompt and sampling configuration. Across six empirical studies, the largest swings in accuracy, speed, and safety came from system architecture: on sealed Bank Slice 2, H scored 46/50 versus B0 35/50 (exact McNemar test, p = 0.0034), deterministic code replaced LLM planners to eliminate 152,239 tokens with no detected accuracy difference and ~10x to 13x lower latency, mechanical gates caught unfinished work with zero false blocks, and sealed harness confinement denied all 7 outside-root read attempts across 1,934 audited tool calls.
A Speed Fix Introduced a Potential Tenant-Isolation Defect, and Independent Review Blocked It
A throughput change introduced a potential tenant-isolation defect. Independent cross-model review classified it as P0 and blocked deployment. Across three review reports, 1 P0 and 16 P1 findings were recorded. The upgrade was not deployed.
Memory & Retrieval
8 of 8Empirical benchmarks on persistent agent recall, semantic search indexing, and token efficiency.
83% Fewer Tokens, Matched Accuracy on 100 Questions: Semantic Memory vs Pasting the Whole History
When evaluating conversational agents over ~115,000-token chat histories on the public LongMemEval_S benchmark, pasting full conversation history into the context window scored 88/100 while consuming 113.6k tokens per question. Our semantic memory architecture matched that exact 88/100 score on this 100-question set using only 19.5k tokens per question, an 82.8% token reduction (~83% fewer tokens). Standard RAG chunks trailed at 77/100 (paired McNemar test, p = 0.027). A follow-up holdout on 100 fresh questions confirmed stability at 89/100 accuracy and ~16% tokens.
Zero-Token Planning: Removing All Planner Tokens With No Detected Accuracy Difference
Deterministic query routing eliminated planner token overhead on reused question sets. On sealed Bank Slice 2, H scored 46/50 versus B0 35/50 (exact McNemar test, p = 0.0034). B0 was a notes, handoff-log and memory-file grep baseline without raw transcripts. The arms had different source access, so observed gains include corpus availability. The October 2 run recorded 152,239 planner tokens versus zero, with both arms scoring 135/180. The October 3 unlimited-planner rerun scored 141/180 versus 142/180 and measured pointer p50 latency of 8.8s versus 0.67s. On these reused question sets, deterministic routing reduced pointer latency, with no detected accuracy difference.
Half the Tokens: Code-Search Hints Cut Agent Token Use 51% on a 30-Question Test
When coding agents explore an unfamiliar codebase, they typically spend 50k to 100k+ tokens running grep searches and opening whole files. In overnight benchmarks across fresh out-of-sample and real-ask holdouts, pre-computed symbol concordance hints and inlined snippets cut agent token expenditure by 51% with 1.00 recall@5 on the 30-question test (0.94-0.98 on holdout).
Deconstructing Agent Retrieval Latency: Process Pools, Loopback Daemons, and Pipelining
We profiled and optimized the agent memory retrieval pipeline, isolating non-model overheads from LLM generation time. By introducing pre-spawned worker process pools, a local 127.0.0.1 daemon with memoized AST parses, and pipelining filesystem scans alongside the planner model, we cut median warm retrieval latency from 23.3 seconds to 9.9 seconds while asserting 30 out of 30 byte-identical prompt and output verifications.
Eliminating Quadratic Bottlenecks in Multi-Session Thread Reconstruction
As an engineering memory store expanded to 1,293 indexed session threads, pointer resolution latency degraded severely due to regex backtracking and quadratic scanning loops during thread table construction. By replacing unmerged regex passes with linear segment parsers, adding service-level open-items memoization, and caching thread vectors on write, we reduced table construction from 19.0 seconds to 1.5 seconds with byte-identical output, accelerating warm pointer lookups from 8.5 seconds to 0.62 seconds.
Sub-Second Agent Memory Recall via Asynchronous Speculative Prefetch
Retrieving historical engineering context in interactive agent sessions typically requires 15 to 25 seconds for query planning, database filtering, and response synthesis. By implementing an asynchronous, stdlib-only prompt interception hook that spawns a detached background prefetch worker with local vector embeddings, we reduced memory recall latency to 0.29 seconds on past-work queries with zero cloud model spend.
A Code-Retrieval Index That Scored 0.94 In-Sample Scored 0.45 Out of Sample
We tried to replace an agent's grep-and-read code search with deterministic retrieval rules mined from that agent's own traces. On the 50 questions the rules were built from, symbol recall@5 was 0.94. On 20 out-of-sample questions written by an independent pass, it fell to 0.45, while the agentic baseline scored 0.975. We had pre-registered an overfit gap of at least 0.10; the observed gap was 0.49.
Semantic Memory Living Paper: An Empirical Synthesis Across Session Memory Evaluations
This living paper (v1) synthesizes eight session-memory question sets, diagnostic reanalyses and planner reruns. Data collected September 28 to October 3, 2026; synthesized October 2026. On sealed Bank Slice 2, H scored 46/50 versus B0 35/50 (exact McNemar test, p = 0.0034). B0 was a notes, handoff-log and memory-file grep baseline without raw transcripts. The arms had different source access, so observed gains include corpus availability. The October 2 run recorded 152,239 planner tokens versus zero, with both arms scoring 135/180. The October 3 unlimited-planner rerun scored 141/180 versus 142/180 and measured pointer p50 latency of 8.8s versus 0.67s. On these reused question sets, deterministic routing reduced pointer latency, with no detected accuracy difference.
Developer Tooling
8 of 8Mechanical verification gates, slop prevention tooling, and lightweight runtime footprints.
Working Is Not Organized: Measuring How Data Access Spread in an AI-Built Application
AI coding agents are usually judged task by task: does the change work, do the tests pass, is the code clean? We looked at what those changes add up to in one application our agents built over several months. Operations on one settings table are named in 29 files through 14 differently named calls. A rule that a visit with no service delivered cannot be paid exists as two separate implementations that return different errors. Of 145 record types named by literal operation strings in the code, 77 are referenced from more than one file. This is a different problem from code slop. Slop is dead code, swallowed errors, invented functions and fake tests, and each instance is visible in the diff that adds it. Scatter is hard to see in any one diff: a new read path can look reasonable where it is written. It shows up when you count across the codebase. The application's slop gate has 42 numbered checks, and none of them counts how many files reach the same table. We then measured one consolidation lane that moved data access into per-table owner modules: settings references went from 29 files to 1, and user profile references from 19 files to 2. That lane is unmerged and its validation is blocked, so it describes what the rule does to the code, not a released outcome. Data from October 22 and 23, 2026 (code as of those commits), analyzed October 23, 2026.
Find the Bug Class, Not the Bug
During a database cutover, AI agent lanes fixed bugs as they surfaced, one log entry per bug. The flow for creating a new worker account and signing that worker in appears in five related entries across two fix rounds, each with a different immediate cause. One access-policy synchronization defect recorded as repaired in the first round was still blocking new worker sign-ins in the second: the first entry had already noted that stored data was not migrated, and the repair did not reach it. Storage gaps turned up the same way: three record types broke one at a time, then a single sweep listed 11 more with no home in the new database. We then had agents count whole classes with a syntax-aware code-search and census tool we built. It found 708 direct legacy data calls in 155 files, 995 path-related candidate sites, 901 test files that are candidates for testing against a fake of the platform API, and permission definitions in at least six separately maintained places. Counting was the easy part. A guarded rewrite tool changed 4 of the 777 data calls automatically and left 385 for reviewed manual changes, 207 of them because the module those calls should move into did not exist yet. Data from October 21 to 22, 2026, analyzed October 23, 2026. This is a tentative case study of one codebase during one migration. We did not measure whether census-first triage reduced repeat fixes afterwards.
From Recurring AI Checks to Scripts: What Actually Ran?
An AI engineering team can spend part of each session checking facts that ordinary code could calculate. Our team mined its own session logs and operating instructions for recurring checks, then had selected checks written as deterministic scripts. The intention was to let scheduled software perform the routine work and bring exceptions back for attention. The operational result was incomplete. The deployment note reports that two of four selected jobs had disabled timers and never ran; the retained journal contains no executions for those jobs. The active baseline comparison repeatedly finished successfully. The daily audit freshness check finished with alarms, but its notifications remained pending at the recorded attempts. Having a script, running it, detecting a condition, and delivering a notification were distinct outcomes. Data from October 2026, analyzed October 2026. This is a tentative case study of one node with roughly four days of retained evidence, without a measured token-savings baseline.
Patching on Top: An AI Fix Series That Was Merged, Then Reverted
AI coding agents can complete a sequence of repairs without resolving the design that keeps producing repair work. In this case, a fix series labeled as rounds 1 through 5 reached a merge and was then reverted. A later engineering directive described the affected read operation as slow and called for consolidation, removal of duplicate logic, and deletion of the old path. This is a retrospective about the work assigned to agents and what the resulting artifacts establish. It is not a comparison of model capabilities. The record supports a narrow finding: repeated remediation and integration did not make this repair series durable. It does not establish that the agents lacked the ability to refactor, or that a different prompt would have prevented the outcome. Data from October 2026, analyzed October 2026. This extends our published work on repeated review findings by examining what happened after a repair series reached integration.
Record What the Agent Did Before Asking a Model to Summarize It
A session summary can help explain work, but it should not be the only record that work happened. We examined a handoff system that stored structured session state and git events through deterministic code, then offered an optional model summary. Its demonstration showed a large latency difference between raw retrieval and summarization, with important differences in what each output provided.
When the Edit Harness Makes a Valid Model Patch Fail
An edit-format sweep initially favored replacing whole functions over asking a local model for a unified diff. A later harness defect complicated the story: the tool applying replacements could corrupt valid generated code. The result extends our published local-coder study by separating representation failures, application failures, and code that applies but fails its tests.
Preventing Code Slop and Enforcing Task Completion with Mechanical Gates
Autonomous agents introduce two subtle forms of codebase rot: generating duplicate helper functions instead of reusing existing ones ('code slop'), and declaring tasks complete while omitting steps requested in the prompt. Required items completed on skip-told briefs: Hook-only runs completed 56/58 items on Claude (96.6%) and 29/29 on Gemini (100%). Ungated skip-told runs completed 39/65 and 21/29 items respectively. Total measured slop fell from 4 events in 12 Claude OFF runs to zero in 12 Claude ON and 6 Gemini ON runs.
Running Agent Memory in Under 500 MB RAM on ARM64
Can production-grade agent memory and code search run unobtrusively on developer laptops and low-power ARM64 single-board computers? Across cross-platform profiling on Windows 11, WSL x86_64, and Linux ARM64, we verified that full session memory and symbol search run comfortably in 440 MB of RAM with sub-50ms process cleanup.
Local Inference & Systems
6 of 6Workstation model deployment benchmarks, hardware offloading profiles, and serving latency.
The Hidden Infrastructure Tax in Multi-Agent Coding: An Estimated Replay of 39,526 Calls and an RTX 5090 Lab
Updated October 16, 2026: revised after peer review. When multi-agent engineering sessions across Claude Code, Codex, and Gemini feel slow, the default assumption is that the frontier reasoning model is the bottleneck. We audited 39,526 raw telemetry records (121.82 hours of cumulative summed call duration across concurrent agent lanes and background workers over 14 days) and ran a 6-arm laboratory study (our RTX 5090 infrastructure study v3_optimized) on an NVIDIA GeForce RTX 5090 (32 GB) and AMD Ryzen 7 9800X3D workstation. The frontier model was not the main bottleneck: per-turn and per-tool hook spawns, serial test gates, cold interpreter imports, and a 31.9-hour/week background cloud-LLM prefetch storm were. Tagging every saving by the smallest hardware tier it needs, a CPU-only laptop (Tier 1) captures 96.5% of the named-fix savings in the 14-day replay and 97.7% in the 7-day projection; a 4 GB-class GPU running qwen3:4b and nomic-embed-text (2.77 GB of weights, a fit we project from footprint but did not benchmark) adds 3.61 hours; the 32 GB RTX 5090 adds accuracy and multi-model co-residency, not logged time. A further 16.0% of the replayed reduction is a generic cap on calls with no named fix and is not claimed as a saving. With a 4% Claude Haiku fallback priced in, the replayed 14-day total is 45.36 to 53.30 minutes. In-sample, checklist extraction reached 99.46% Micro F1 and a two-stage local PHI cascade matched 100/100 on a synthetic set; out of sample, the deterministic checklist extractor scored 97.04% on 14 briefs written after the rules froze, while the qwen3:14b hybrid and all PHI arms still await a holdout run. The CEO approved the local PHI cascade on 2026-10-15; on its first production day it answered 620 checks locally at a 217 ms median and fell back to Haiku 29 times, all at service start.
Measured: Local MoE Inference on a Single GPU
Can an 80B-parameter Mixture-of-Experts (MoE) coding model run locally on a single consumer GPU? Naive layer-based offloading across PCIe choked at 0.9 tokens per second. By offloading by tensor role rather than layer number, we measured a 31x speedup on identical hardware.
Local Model Serving Traps: IPv6 Stalls and Context Ceiling Reloads
When running local language models on developer workstations, mysterious multi-second latency spikes are rarely caused by GPU compute bottlenecks. In our production profiling, we uncovered two architectural traps: a 2,043 ms Windows IPv6 socket connect stall and a 12.6 s PCIe weight reload caused by context ceiling mismatches in Ollama.
Falsifying Transformer Activation Cache Reuse for Code Edits
Can language models avoid recomputing activations after small code edits by injecting difference vectors into cached hidden states? In an empirical study across Qwen2.5 models on 24 code contexts, we falsified activation delta caching: hidden state divergence crossed 90% by layers 12 to 15, yielding only 3-4% net compute savings while corrupting downstream token agreement.
Local Neural 3D Mesh Generation: GPU Flow vs the CPU Topology Bottleneck
Neural 3D mesh generation models can synthesize complex polygonal geometry from single reference images. However, profiling local execution on an NVIDIA RTX 5090 revealed a massive pipeline imbalance: while neural GPU flow completed in under 3.5 minutes, single-threaded CPU topology cleanup (hole-filling and welding) created a 17.5-minute bottleneck.
Multi-Lane Memory Saturation: Measuring the Limits of Parallel Local Agents
Autonomous AI coding workflows encourage high concurrency: an orchestrator dispatches several subagents in parallel to draft features, run tests, and synthesize assets. However, when local agents execute native compilers, 3D engines, and neural generators, memory contention shifts to workstation hardware constraints. Across three incident reports on a workstation with approximately 62 GB host RAM, concurrent agent dispatches triggered out-of-memory failures: three simultaneous lanes saturated host RAM, and a larger trial with 11 parallel lanes resulted in 9 processes being reaped mid-task. The initial incident involved two engine-build lanes and one Blender/rendering lane, with Blender validation and measurement steps estimated at about 5 GB each. Some detached generation jobs survived shell termination and produced reusable outputs. Adopted rules capped lanes at 2 to 3, required every Blender process to take the heavy lock, and required checking at least 30 GB free RAM before adding a lane. These rules were adopted after the incidents. The cited evidence does not measure post-intervention stability.
Graphics & Engine
13 of 13Real-time rendering pipeline optimization, presentation budgets, and automated soak testing.
Frame Budget at 120 FPS: What We Cut and What It Bought
Most modern game pipelines rely on temporal upscalers and dynamic resolution scaling to mask render budget deficits. In Atlas Engine, our standing mandate targets 120 fps (8.33 ms frame budget) at 3840x2160 native with zero upscaling. Here is the telemetry on how we profiled our compute and draw queues to hold 2.51 ms of honest GPU/staging work.
Culling at Scale: Eliminating Edge Pop and Shadow Desync
In dense crowd rendering, standard frustum culling frequently introduces edge pop and shadow desynchronization. In Atlas Engine crowd benchmarks across 1,024 animated instances, hoisting light-fit calculations reduced whole-frame time by 64% (31.07 ms to 11.20 ms) and shadow recording by 12.5x, while phase-independent union bounding volumes eliminated cull-pop artifacts entirely.
Why We Optimize Natively: Hardware Block Compression Over Neural Upscaling
Neural texture compression (NTC) and temporal upscaling promise massive memory reductions, but introduce proprietary SDK dependencies and neural reconstruction overhead. In direct empirical bake-offs on 2048² PBR maps, standard GPU hardware block compression achieved a 6.4x VRAM reduction while outperforming neural codecs on normal and height fidelity.
Automated 30-Minute Engine Soak Testing: Measuring Frame Latency and Memory Drift
Short 30-second rendering benchmarks often mask slow-burn memory leaks, shader cache thrashing, and asset streaming degradation. In Atlas Engine, we deployed autonomous playtest agents to execute continuous 30-minute traversals (9.5+ km across 53 streaming regions), measuring steady sub-8ms worst-frame p50 latencies and zero private byte accumulation.
Reaching 120 FPS at Native 4K: The 3.5 ms That Wasn't Where We Looked
At a fixed 4K camera anchor the frame was CPU-bound: a 10.50 ms wall against a 2.41 ms GPU envelope. Sub-bracketing the CPU prep showed one region, visibility prep, owned 80.0% of it. Four candidate fixes measured neutral or near-neutral before the true owner was found: twelve wandering NPCs whose bounds were re-derived from every vertex each frame. Replacing that with skeleton bone bounds cut visibility prep from 3.23 ms to 0.107 ms.
Our Determinism Hash Was Wrong on 74.3% of Steps
The engine's per-step state hash, which replicas and replays compare against, was served from a digest cache whose guard keyed on a buffer's pointer and size rather than its content. Over a 30,000-step run, 22,290 steps (74.3%) carried a wrong hash, starting at step 7,710. A content-version counter fixed it to 0 of 30,000, and no committed golden moved.
A Stale Safety Comment Parked a 10 ms Win for Three Days
A code comment barred switching character skinning to the GPU until an A/B showed the body rendered correctly. It was written when that was true, then outlived two fixes by three days. Re-running the A/B it demanded showed 1 differing pixel in 3,686,400 and a frame-time drop from 15.520 ms to 5.064 ms.
Fixed Cost vs Pixel Cost: A Resolution Ladder With R^2 = 0.99993
We measured frame time at six render extents from 1080p to native 4K and fit T = F + k·P. On the three native rungs, p50 frame time fit with R^2 = 0.999931, and three resampled rungs landed on the same line within 0.04 ms. Dropping from 4K to 1080p cut frame time by 48%, not the roughly 20% an earlier spot measurement had suggested.
Asset Decode Was 83% of Boot. Parallelizing Textures Bought 5.6%.
A phase-stamped boot trace of the game found asset decode and registration took 2.333 s of a 2.805 s boot (83.2%). We planned a parallel decode with strictly serial registration to keep handle numbering and logs byte-identical. The first version warmed texture decode only and cut that phase by 5.6%, because model loaders turned out to be about 66% of it.
Our Culling Number Was Right and Still Misleading
We published an internal verification annex showing the CPU frustum culler handles 100,000 bounding boxes in 0.104 ms. The same day, a residency stress test showed the per-frame bounds-maintenance pass that must run alongside the cull costs 47.9x the cull on the live scene and 64.4x at 100,000 objects. Every number was correctly measured; the error was that the timer boundary was not part of the record.
39.7 Million Random Pairs Found Zero Bugs. 144 Targeted Pairs Found Two.
We sped up a per-vertex world-bounds routine 7.91x under a hard bit-identity constraint. The obvious fast min/max instructions differ from the library functions on signed zero, a case that a 39.7-million-pair random float fuzz never hit and a 144-pair targeted table hit twice. The shipped version corrects the zero sign once after the loop and stays bit-identical.
Five Optimizations That Measured Flat
Over three weeks, five engine optimizations each passed their correctness gates and then measured flat or near-flat on whole-frame time. In two cases the change itself worked as designed, but the cost it targeted had been mis-attributed, once by about 1000x.
First Vulkan Wins: Four Fixes Worth 2x to 900x
In the first week the Atlas Engine rendered its playable beach scene through Vulkan, four separate fixes moved the frame or startup time by large factors. The biggest was moving a scene off a single-core CPU compositor onto the GPU, cutting per-frame cost from about 1650 ms to about 1.8 ms. Each fix was checked for bit-identical or byte-identical output before it shipped.
Interested in our empirical benchmarks or invariant methodology?
We collaborate with engineering teams and research labs studying multi-agent systems, verification runtimes, and automated governance.