Published by Sean Carr
1,610 Commits in 5 Days, Then an Untouched 27k-Line Monolith: Three-Tier Verification for Multi-Agent Refactors
Across a 5-day internal overhaul (1,610 commits, 3,090 files, 4,924 permission checks collapsed into 1 pure function) and an external replication on FlaskBB (27,039 lines, 800 tests, 17 oversized modules reduced to 0 across 72 split files), a three-tier verification plane caught 10 classes of refactor defects—including three untested error-branch NameErrors and three missing ORM equality decorators that all 800 unit tests missed.
At a Glance
5-Day Internal Overhaul
1,610 commits, 3,090 files
Collapsed 4,924 scattered permission checks into 1 pure function (43,005 verified access cells) and cut p50 auth latency by 51.9% (128.7 ms to 61.9 ms).
External Monolith Split (FlaskBB)
17 -> 0 oversized files
All 17 oversized modules in FlaskBB (9,322 code lines / 13,262 physical lines) decomposed into 72 files (+55 new; 144 -> 199 total modules; peak size cut 82.2%).
Upstream Test Suite Parity
800 passed, 0 failed
100% behavioral parity across all 800 upstream FlaskBB unit and end-to-end tests in 12.80s, with 16/17 runtime equivalence PASS and 1 procedural migration split.
Pre-Merge Defect Classes Caught
10 catches across 3 tiers
Caught compiler directive drift, pytz.UTC identity swaps, monkeypatch breaks, 3 untested error-branch NameErrors, and 3 missing @make_comparable ORM decorators.
| Telemetry Metric | Measured Value | Context & Observability |
|---|---|---|
| Internal Sprint Throughput | 1,610 commits, 3,090 files | Executed between October 29 and November 3, 2026 across Core and application repositories, backed by 2,360+ formal verification receipts. |
| Authorization Consolidation | 4,924 -> 1 pure function | Collapsed 5 disagreeing permission layers into 1 pure Core decision function governing 883 declarative operations across 43,005 verified access and CRUD cells. |
| Authorization Latency Reduction | -51.9% p50 (128.7 ms -> 61.9 ms) | Cut Core RPCs per auth resolution from 2 to 1 (-50%), document reads from 170 to 86 (-49.4%), SQL SELECTs from 9 to 4 (-55.6%), and p95 latency from 195.8 ms to 154.0 ms. |
| External Monolith Census (FlaskBB) | 17 -> 0 oversized files | Decomposed all 17 oversized modules (9,322 code lines / 13,262 physical lines) into 72 files (+55 new, taking flaskbb/ from 144 to 199 modules) with peak module size cut from 1,391 to 247 code lines (-82.2%). |
| External Test & Equivalence Parity | 800 passed, 0 failed (16/17 PASS) | Full 800-test upstream FlaskBB suite passed in 12.80s; 16 of 17 splits passed full runtime equivalence and 1 single-function Alembic migration passed procedural + 19-test migration verification. |
| Pre-Merge Defect Classes Caught | 10 catches across 3 tiers | Caught across Tier-1 equivalence verification, Tier-2 full-suite pytest integration, and Tier-3 cross-model review (with Catch #9 promoted into Tier 1, uncovering Catch #10). |
Enforce a strict per-module size ceiling so parallel agents can own disjoint files in a single worktree without merge collisions.
Run parallel coding agents against 1,000+ line monoliths where every task collides on the same file.
Verify export equivalence, public contract parity, and scope integrity deterministically in Tier 1.
Claim a green test suite proves zero defects: 800 passing FlaskBB tests missed 3 missing imports in unexercised error branches and 3 dropped model comparison contracts.
When review detects a defect pattern that slipped past standard checks, harden the deterministic verification gate with a regression rule permanently.
Rely solely on non-deterministic model review for mechanical errors like missing imports or dropped class contracts.
Published by Sean Carr
View & print executive brief version →Executive Summary & Core Finding
Over five days (October 29 to November 3, 2026), we executed 1,610 verified commits across 3,090 files in our own multi-tenant production software: decomposing a 9,790-line monolith into bounded domain submodules so parallel agents could work in one repository without file-lock contention, and collapsing 5 disagreeing permission layers (4,924 scattered decision points) into 1 pure authorization function backed by 883 declarative operation rows and 43,005 verified access cells, cutting p50 authorization latency by 51.9% (128.7 ms to 61.9 ms). To test whether the same mechanical refactor pipeline held up on unfamiliar outside code, we pointed it at an untouched open-source Python application (FlaskBB at commit 737d76f1: 27,039 application lines across 144 Python modules, 800 passing tests, and 17 oversized modules totaling 9,322 code lines and 13,262 physical lines). Up to eight concurrent single-file agent workers decomposed all 17 oversized modules into 72 cohesive files (17 modified entrypoints + 55 new files, expanding the application tree from 144 to 199 modules and cutting peak module size by 82.2%, from 1,391 to 247 code lines) while preserving 100% test parity (800 passed, 0 failed in 12.80s). Across the FlaskBB run, our three-tier verification plane caught and resolved 10 distinct classes of agent-introduced defects before sealing the branch at commit 274427b. Most importantly, after all 800 upstream unit and end-to-end tests passed, cross-model adversarial review caught 3 latent NameError missing imports hiding inside untested error and permission-denial branches (including a lambda wrapped inside a third-party decorator), and our deterministic contract verifier caught 3 SQLAlchemy models where equality comparison contracts had been silently dropped—defects that both the 800-test suite and standard static linters had missed.
Method
Part I (Internal Production Overhaul, October 29 to November 3, 2026): We audited 5 days of continuous multi-agent execution across our core governance engine and healthcare staffing application (1,610 git commits across 3,090 unique files, 1,809 Phase-1 checklist entries across 20 workstream ledgers, and 498 authorization consolidation receipts). We measured authorization latency (p50, p95, RPC count, document reads, SQL queries) before and after replacing 5 disagreeing permission layers (4,924 decision points) with a single pure decision function.
Part II (External Open-Source Replication on FlaskBB): We selected FlaskBB (pinned at upstream commit 737d76f1), a mature BSD-licensed Flask, SQLAlchemy, Click, and Alembic forum codebase with 27,039 application lines across 144 Python modules and 800 passing tests. A structural census identified 17 modules exceeding our per-module code-line ceiling (totaling 9,322 code lines and 13,262 physical lines).
Up to 8 concurrent single-file worker agents ran in the same repository worktree under strict single-writer-per-file ownership. Each worker decomposed its assigned target into cohesive submodules while re-exporting public symbols on the original entrypoint.
Every split was gated by a three-tier verification plane: (1) a deterministic pre-merge structural verifier that validates public API surface parity, symbol origins, contract adherence, and lexical scope integrity between the baseline commit and head; (2) parallel automated test execution across covering and full regression suites; and (3) an independent cross-model adversarial review across all modified and created files.
What We Measured
Internal 5-day consolidation telemetry (1,610 commits) and external FlaskBB decomposition results (17/17 monoliths split into 72 files, 800/800 tests passing, 10 defect classes caught).
Internal Sprint Throughput
1,610 commits, 3,090 files
Executed between October 29 and November 3, 2026 across Core and application repositories, backed by 2,360+ formal verification receipts.
Authorization Consolidation
4,924 -> 1 pure function
Collapsed 5 disagreeing permission layers into 1 pure Core decision function governing 883 declarative operations across 43,005 verified access and CRUD cells.
Authorization Latency Reduction
-51.9% p50 (128.7 ms -> 61.9 ms)
Cut Core RPCs per auth resolution from 2 to 1 (-50%), document reads from 170 to 86 (-49.4%), SQL SELECTs from 9 to 4 (-55.6%), and p95 latency from 195.8 ms to 154.0 ms.
External Monolith Census (FlaskBB)
17 -> 0 oversized files
Decomposed all 17 oversized modules (9,322 code lines / 13,262 physical lines) into 72 files (+55 new, taking flaskbb/ from 144 to 199 modules) with peak module size cut from 1,391 to 247 code lines (-82.2%).
External Test & Equivalence Parity
800 passed, 0 failed (16/17 PASS)
Full 800-test upstream FlaskBB suite passed in 12.80s; 16 of 17 splits passed full runtime equivalence and 1 single-function Alembic migration passed procedural + 19-test migration verification.
Pre-Merge Defect Classes Caught
10 catches across 3 tiers
Caught across Tier-1 equivalence verification, Tier-2 full-suite pytest integration, and Tier-3 cross-model review (with Catch #9 promoted into Tier 1, uncovering Catch #10).
Experimental Notes & Observability Details
- Tier 1 (Deterministic Equivalence & Symbol Verifier) caught 6 initial classes of subtle refactor defects that standard linters allow: (1) an unrequested future-annotations compiler directive in a split database helper that altered compiled function evaluation flags; (2) hoisting a lazy import out of count_online_users() to module top-level, altering its global dependency bindings; (3) importing date formatters from a wrapper package instead of babel.dates and swapping pytz.UTC for datetime.UTC across 101 helper exports; (4) exposing unaliased helper dictionaries and an unrequested __all__ as new public module attributes; (5) class __module__ annotation repr drift on split SQLAlchemy user models; and (6) static detection of test monkeypatch targets pointing at parent view modules.
- Tier 2 (Parallel Test Execution) caught 2 integration hazards: (1) placing shared migration helper modules in the top-level Alembic migrations directory, which broke Alembic's revision scanner until moved into a private _helpers/ package; and (2) cross-module monkeypatch bindings where tests patched configuration and migration helpers on parent view modules while extracted view classes bound their own module globals.
- Tier 3 (Cross-Model Adversarial Review) -> Hardened Gate (Catches #9 and #10): After all 800 upstream pytest tests passed (800 passed, 0 failed), independent model review caught 3 missing imports in untested error branches (3b922d6). We immediately encoded deterministic scope and contract validation rules into Tier 1, which deterministically flagged those imports and uncovered Catch #10: 3 SQLAlchemy models (Report, Attachment, Category) in flaskbb/forum/models.py that had lost class equality comparison contracts when Row 17 initially ran only a static export check—a defect missed by both the 800 unit tests and model review. Restoring full contract parity sealed the branch at commit 274427b with 16/17 full runtime PASS and 800/800 tests passing in 12.80s.
What We Think It Suggests
- **Multi-agent velocity requires bounded files and single-file ownership.** Decomposing 1,000+ line monoliths into cohesive modules allowed 8 concurrent worker agents to refactor the same worktree simultaneously by pathspec with zero merge conflicts or lock contention.
- **One can only count defects that were caught—and passing tests have blind spots.** In FlaskBB, 800 passing tests missed a compiler flag mutation, a pytz.UTC to datetime.UTC identity swap, 3 missing imports in unexercised error branches, and 3 dropped @make_comparable ORM decorators. Combining deterministic equivalence checks with cross-model review—and promoting review catches into deterministic Tier-1 rules—closed those blind spots.
- **Consolidating scattered policy checks into one declarative table pays both security and latency dividends.** Replacing 4,924 scattered permission checks across 5 layers with 1 pure function and 883 declarative rows eliminated cross-layer policy disagreement while cutting p50 auth latency in half.
Limits
- **Language Generalizability & The Hardest Frontier.** Dynamic languages like Python represent the most difficult testing ground for automated multi-agent refactoring because of runtime monkeypatching and dynamic attribute lookup. In statically typed and compiled languages (TypeScript, Go, Rust), language compilers (tsc, go vet, cargo check) already enforce type, interface, and symbol parity natively, making the three-tier verification architecture even more direct to adopt across multi-language enterprise codebases.
- **Procedural Single-Function Refactoring.** Massive procedural functions (such as monolithic database schema migrations) cannot be decomposed without delegating statements to structured helpers. In these workflows, the verification plane transitions from raw lexical parity to module contract validation and dedicated migration integration suites (19/19 passing).
- **Dynamic monkeypatch delegation adds thin wrapper indirection.** When upstream tests monkeypatch attributes directly on a monolith entrypoint rather than the defining module, preserving 100% test compatibility without editing upstream tests requires thin runtime lookup wrappers in the extracted submodules.
Detailed Analysis & Field Observations
Part I: The 5-Day Internal Platform Overhaul (1,610 Commits)
| Architectural Metric | Baseline (Oct 29, 2026) | Post-Refactor (Nov 3, 2026) | Measured Delta |
|---|---|---|---|
| Authorization Decision Layers | 5 disagreeing layers (4,924 checks) | 1 pure function (883 declarative rows) | -99.98% decision points |
| Verified Permission & CRUD Matrix Cells | Ad-hoc route tests | 43,005 verified cells (invariant + admission + CRUD) | 100% matrix coverage |
| Auth Resolution Latency (p50 / p95) | 128.7 ms / 195.8 ms | 61.9 ms / 154.0 ms | -51.9% p50 / -21.3% p95 |
| Core RPCs per Auth Check | 2 RPCs (170 doc reads, 9 SQL SELECTs) | 1 RPC (86 doc reads, 4 SQL SELECTs) | -50.0% RPCs / -49.4% reads |
| Shared Multi-Tenant Domain Services | Embedded in single application | 22 isolated platform services | Multi-tenant ready |
Part II: Reconciled 17/17 FlaskBB Decomposition Table (72 Files, 9,322 Code Lines)
| # | Target Module in FlaskBB | Before Code Lines (Raw) | After Max Code Lines | Files (New) | Resulting Modules (Code Lines Each) | Tier-1 Equivalence Gate | Covering & Full Pytest |
|---|---|---|---|---|---|---|---|
| 1 | flaskbb/utils/database.py | 255 (370) | 153 | 2 (+1) | database (153), database_ops (122) | PASS (29 names, 9 probes) | 33 passed (2.59s) |
| 2 | flaskbb/user/views.py | 271 (350) | 169 | 2 (+1) | views (145), views_settings (169) | PASS (62 names, 7 probes) | 40 passed (4.04s) |
| 3 | flaskbb/markup.py | 284 (373) | 185 | 2 (+1) | markup (185), markup_quote (118) | PASS (67 names, 7 probes) | 41 passed (2.62s) |
| 4 | flaskbb/settings/fixture.py | 324 (349) | 191 | 2 (+1) | fixture (191), fixture_groups (150) | PASS (16 names, 0 probes) | 21 passed (2.64s) |
| 5 | flaskbb/cli/plugins.py | 322 (402) | 186 | 3 (+2) | plugins (132), plugins_install (186), plugins_new (32) | PASS (27 names, 5 probes) | 24 passed (2.93s) |
| 6 | flaskbb/auth/views.py | 396 (508) | 247 | 2 (+1) | views (247), views_recovery (180) | PASS (65 names, 3 probes) | 40 passed (3.07s) |
| 7 | flaskbb/management/forms.py | 402 (560) | 176 | 4 (+3) | forms (75), forms_user (176), forms_forum (146), forms_group (110) | PASS (74 names, 10 probes) | 23 passed (3.06s) |
| 8 | flaskbb/user/models.py | 421 (646) | 246 | 2 (+1) | models (246), models_group (204) | PASS (52 names, 9 probes) | 91 passed (3.68s) |
| 9 | flaskbb/utils/helpers.py | 424 (851) | 240 | 4 (+3) | helpers (240), helpers_datetime (99), helpers_forum (94), helpers_proxy (75) | PASS (101 names, 39 probes) | 39 passed (2.85s) |
| 10 | flaskbb/app.py | 511 (747) | 207 | 6 (+5) | app (207), app_hooks (166), app_plugins (99), app_extensions (67), app_logging (56), app_config (48) | PASS (125 names, 5 probes) | 46 passed (3.10s) |
| 11 | flaskbb/cli/main.py | 574 (722) | 229 | 4 (+3) | main (131), main_config (229), main_run (142), main_db (127) | PASS (53 names, 1 probe) | 74 passed (3.17s) |
| 12 | flaskbb/migrations/20180228...emoji.py | 904 (944) | 219 | 6 (+5) | migration (41), _helpers/__init__ (0), _helpers/emoji_1..4 (219, 219, 219, 217) | PASS (9 names, 0 probes) | 32 passed (3.15s) |
| 13 | flaskbb/migrations/20180202...cascades.py | 286 (350) | 211 | 2 (+1) | migration (211), _helpers/cascades_ops (78) | PASS (9 names, 0 probes) | 19 passed (7.16s) |
| 14 | flaskbb/migrations/20150108...init.py | 295 (315) | 180 | 2 (+1) | migration (180), _helpers/init_schema (120) | Procedural Split (7 names) | 19 passed (2.85s) |
| 15 | flaskbb/forum/views.py | 1,011 (1,237) | 207 | 8 (+7) | views (206), views_moderation (207), views_forum (170), views_tracker (169), views_topic (167), views_post (158), views_users (73), views_helpers (27) | PASS (105 names, 18 probes) | 58 passed (3.20s) + 800/800 |
| 16 | flaskbb/management/views.py | 1,391 (1,751) | 231 | 10 (+9) | views (213), views_plugins (231), views_users (230), views_forums (185), views_attachments (177), views_bans (136), views_reports (119), views_groups (116), views_overview (72), views_settings (71) | PASS (125 names, 25 probes) | 46 passed (3.40s) + 800/800 |
| 17 | flaskbb/forum/models.py | 1,251 (1,823) | 208 | 11 (+10) | models (95), models_topic (208), models_category (195), models_forum (182), models_forum_queries (180), models_post (141), models_topic_read (139), models_tables (139), models_post_ops (119), models_topic_ops (97), models_attachment (85) | PASS (56 names, 9 probes) | 43 passed (3.47s) + 800/800 |
| **TOTAL** | **17 Target Monoliths** | **9,322 (13,262)** | **247 max** | **72 (+55)** | **144 -> 199 Python modules in flaskbb/ (0 oversized remaining)** | **16/17 Runtime PASS + 1 Procedural** | **800 passed, 0 failed (12.80s)** |
Part III: 'Battle-Tested' in Production vs. 'Laboratory Proven' in Open Source
A common diligence question is whether automated multi-agent refactoring is merely a clean-room laboratory experiment. In MeshTen's case, the sequence was exactly the reverse: 1. **Born in Regulated Production:** MeshTen's three-tier verification plane was engineered to solve high-stakes production instability inside a multi-tenant healthcare staffing platform (mercyStaffing) operating under real HIPAA compliance, 24/7 hospital shift scheduling, and state-mandated credential tracking. The 1,610 verified commits in Part I were executed on live production infrastructure, collapsing 4,924 scattered authorization checks across 3,090 files into 1 pure function and cutting authorization p50 latency by 51.9%. 2. **The Open-Source Stress Test:** We pointed the exact same pipeline at FlaskBB (and subsequent external repositories) specifically to eliminate internal confirmation bias. We wanted an un-cherry-picked, unfamiliar codebase written by strangers with 10+ years of architectural drift. The 17 monolith splits and 800 passing tests prove that the system does not depend on internal tribal knowledge to deliver zero-regression migrations. 3. **Autonomous End-to-End Manufacturing:** In our subsequent benchmark (FACTORY-1103), the platform went further: autonomously planning, generating, and operating a 5-stage field service business application (fieldService) with intake, scheduling, dispatch, and real transaction invoicing ($880.21) while proving that all 2,234 platform core and service files remained 100% frozen (0 SHA-256 byte changes).
Part IV: The Language Frontier — Why Python Was the Hardest Mountain, Not a 'Cage'
Another critical architectural question: *Does reliance on runtime structural inspection mean this verification architecture is confined to Python?* In computer science, dynamically typed languages with runtime monkeypatching (Python, JavaScript/Ruby) represent the **most difficult problem** in automated refactoring. In Python, an unimported symbol or dropped decorator can hide in an unexercised exception branch without failing compilation until triggered in production. Solving Python required building deterministic contract and scope verifiers that catch what static linters and unit test suites miss. In statically typed, compiled enterprise languages (Java, C#, Go, Rust, TypeScript): * **The Compiler IS Tier 1:** The language's native compiler and Language Server Protocol toolchains (javac, Roslyn, go build/go vet, cargo check, tsc) natively enforce types, method signatures, interface implementations, and symbol resolutions at compile time in sub-second intervals. * **Transferable Architecture, Simpler Implementation:** The three-tier paradigm (Deterministic Contract Gate -> Regression Test Execution -> Adversarial Model Review) maps directly to Java or Go—except implementing Tier 1 is vastly simpler and faster because the native compiler provides the mathematical proof of symbol resolution and type soundness out of the box. * **Enterprise Breadth:** Far from a limitation, mastering the dynamic complexity of Python—the world's most dominant language for AI and data platforms—demonstrates that the architecture can safely govern even the most dynamic codebases before accelerating into statically compiled environments.
The 10 Pre-Merge Catches Across Our Three-Tier Verification Plane
| # | Verification Tier | Defect Class Introduced During Agent Split | How the Gate Caught & Resolved It Before Branch Lock |
|---|---|---|---|
| 1 | Tier 1: Equivalence Verifier | Unrequested `from __future__ import annotations` mutated compiled function evaluation flags on moved database helpers. | Function comparator rejected the flag mismatch on `try_commit`, `drop_all`, and `database_exists`; future import removed (`f7357fe`). |
| 2 | Tier 1 + Tier 2: Mock Target | Moving `DeleteAvatar` broke `monkeypatch.setattr` targeting `flaskbb.user.views.delete_avatar_file`. | Flagged by static test-patch scanner and caught by unit test; class kept in entrypoint module (`b0e137f`). |
| 3 | Tier 2: Alembic Loader | Helper `.py` file in top-level `migrations/` directory broke Alembic's revision file scanner. | Migration test suite failed collection; shared helpers moved into `migrations/_helpers/` subpackage. |
| 4 | Tier 1: Equivalence Verifier | Hoisted lazy import (`User`) out of `count_online_users()` to module top-level, altering global dependency bindings. | Function comparator rejected the dependency change; lazy import restored inside function body (`0bdbd2a`). |
| 5 | Tier 1: Symbol Identity | Imported date formatters from wrapper package instead of `babel.dates` and swapped `pytz.UTC` for `datetime.UTC`. | Symbol-table verifier detected origin and object identity mismatch across 101 exports; exact upstream origins restored (`0bdbd2a`). |
| 6 | Tier 2 + Tier 1: Unbound Symbol | Extracted `format_quote` into `helpers_forum.py` without importing `lazy_gettext as _`. | Caught by `test_helpers.py` and static unbound-global analysis; import restored (`0bdbd2a`). |
| 7 | Tier 1: Public Namespace | Unaliased helper dictionary imports and an unrequested `__all__` exposed new public attributes on entrypoints. | Public symbol diff flagged added public names; imports aliased (`_P1.._P4`) and `__all__` removed (`0966d17`, `3084143`). |
| 8 | Tier 1 + Tier 2: Full-Suite Pytest | Tests patching `flaskbb_config` and plugin migration helpers on `views.py` bypassed extracted view submodules (`784 passed, 16 failed`). | Added dynamic attribute lookup against parent `views` module; full suite reached `800 passed, 0 failed` (`b2b041d`). |
| 9 | Tier 3 Review -> Promoted to Tier 1 | 3 missing imports (`abort`, `PersistenceError`, and `url_for` inside a decorator lambda) in untested error branches that all 800 tests missed. | Caught by Claude Sonnet 5.5 review (`3b922d6`); we then promoted recursive unbound-global checking across all nested code objects into Tier 1. |
| 10 | Upgraded Tier 1: Class Member Parity | Dropping `@make_comparable` (`__eq__`, `__ne__`, `__hash__`) on `Report`, `Attachment`, and `Category` when Row 17 initially ran only a static export check—missed by both 800 pytest tests and cross-model review. | Upgraded Tier-1 class-body comparator flagged the 3 missing methods on all 3 SQLAlchemy models; restored `@make_comparable` in `274427b` (`16/17 PASS`, `800 passed in 12.80s`). |