Field report · 2026-10-26

Software Factory Technical Report

Software development with an unattended agent loop.

Infected Research / FrackLabs

Preliminary: deployed 2026-10-27; live smoke checks passed; research analysis preliminary. Value and engineer-hours are AI estimates, not measured engineering effort.

848Mtokens (95% cached reads)
≈$493at list prices, all-in
132commits written
+3,761 / −905production lines, integrated
110commits integrated, tests green
77real bugs fixed (not minor)
≈98 hsenior-engineer time, est.
14 · 4.5 hparallel agents · wall clock

What we did

We pointed 14 Grok 4.7 coding agents at our production codebase — Twitch AI agents, a streaming studio, overlays, auth, games, a public API — each confined to one area, each on its own branch, looping find one real bug → fix it with a test → commit until the credits ran out. Instructions and policy prohibited deployment, secret access, and editing their own rules. Then Claude Opus 5.5 reviewed every commit, we corrected accepted changes, merged the approved work onto current master, and ran the full test suite.

01 / Agent passes186›02 / Commits written132›03 / Passed AI review117›04 / Integrated110›05 / Real fixes77Activity → review → integration → estimated value≈98 engineer-hours estimated
Figure 1. From agent activity to estimated value; stages count different units.

Focus changed everything

The first run was one agent with a broad prompt. It drifted into busywork: "tests" that read source files as text, plus helper code only those tests used. Splitting the work into narrow lanes with a stricter gate moved the keep rate from 30% to 89%.

0%25%50%75%100%One broad loop30%7 kept / 23 reviewed14 focused lanes89%97 kept / 109 reviewedKeep rate · observational comparison, not a controlled experiment
Figure 2. AI-review keep rate: broad loop versus scoped lanes; sample sizes shown.

What the value is

Every integrated commit was classified (security, user-facing bug, reliability, test, minor), with a rough estimate of how long a senior engineer would need to find and fix it. Against a blind spot-check by separate AI reviewers the classifier agreed on 10 of 11 classes and 9 of 11 "is this a real bug" calls, leaning slightly generous. The biggest wins were in the overlay manager and stream services: refunded redeems that still played, a chat listener that died on one bad message, speech silently skipped. Security-relevant fixes are counted here but described only by category until they are live.

ONE TILE = ONE COMMIT · AI-CLASSIFIEDSecurity or safety24User-facing bug35Reliability18Test of real behaviour10Minor9Not a real bug14Value estimates are model judgments, calibrated on a small blind sample.
Figure 3. Classification of 110 integrated commits; one tile per commit.

Where it went wrong

Throughput

186 passes; a commit-landing pass took about 8 minutes. 30M fresh and 629M cached tokens for writing (≈$391), ≈$19 for the AI review, ≈$82 for supervision, integration and this write-up. The agents wrote 15,982 lines; 11,339 survived into the integration branch, two-thirds of them tests.

CUMULATIVE COMMITS · EXACT EVENT TIMES · UTC04080120132 commits20:0021:0022:0023:0000:00Each step is a commit, not a validated fix. Review outcomes appear separately.
Figure 4. Cumulative factory commits over time; unsmoothed event-level data.

Verdicts per lane

Lane 0 is the first, broad run.

Lane 0 · legacy first run741223Lane 1 · Studio edge server55Lane 2 · auth and identity325Lane 3 · lab utilities1012Lane 4 · Vision1212Lane 5 · public API routers1112Lane 6 · Studio browser628Lane 7 · storage and data527Lane 8 · overlay manager78Lane 9 · games and audio66Lane 10 · social apps67Lane 11 · Studio lab side67Lane 12 · Twitch tracking data77Lane 13 · media pipeline66Lane 14 · content safety tests77keepneeds fixdrop
Figure 5. AI review verdicts per lane.

Method, briefly

Caveats

Production release and validation

The approved release is deployed. Production commit 9a5e6198f4f9; Studio edge release 20261027T020402Z, 27 October 2026. All 211 managed edge artifacts match the release. The landing-page asset was hash-verified separately. All 40 supervised lab services, plus clipper web and worker, restarted with new process IDs.

Validation: the final signed-off Python comparison found 181 baseline failures and 174 candidate failures, with zero new failures. All 1,123 Node tests passed; 18 focused tests passed after merging two concurrent master commits; the ship gate also passed 28 browser smoke tests. This is regression-relative validation, not a claim that every test passed.

Live checks completed:

Initial sign-off exclusions: the tips identity change and its partial performance fix; the malformed-duration guard; and the screamhole audio refactor. The headline totals describe that initial release.

Post-review remediation: seven content-filter findings originally delivered as expected-failure tests were fixed in 9b2e14b0 (378 related tests passed). The tips recipient defect was subsequently approved, fixed with an indexed lookup and ambiguous-identity rejection, and deployed in 2b77f12b (351 database/tips tests and 21 migration tests passed). The production index built in 36.2 seconds; three read-only lookup samples took 14–20 ms. The lab stack was restarted after deployment. These follow-up fixes are separate from the overnight accounting snapshot and initial-release value estimates.

Startup observation: username-history backfill held the SQLite writer lock briefly; completed; subsequent sampled service logs had no errors.

Verification limits:

The analysis remains preliminary: value and engineer-hours are AI judgments, and token/cost totals reflect the collected accounting snapshot rather than a continuously updated bill for subsequent report edits and rollout work.

Addendum: every change

All 147 commits on the lane branches, oldest first per lane. "Not factory output" marks commits made before the loop started or by hand; they are listed for completeness and excluded from every number above. Value labels are AI estimates (see "What the value is"). Excluded patches are not automatically unresolved defects: some were refactors, duplicates, or unsupported bug claims. Follow-up outcomes are identified separately; the original review verdicts remain unchanged.

LaneCommitChangeAI reviewOutcomeValue
03bd24f55Security/safety or high-risk change (category-only public record)fixincluded after fix · not factory output—
0ef335783nx: assert noindex on the editor and gallery responses (source grep could not see a wrong header)keepincluded · not factory output—
020445ccarelay: test the chat:relay-result payload (the source grep matched an unrelated clientId spread)keepincluded · not factory output—
023988442recap: render the stream recap card from a pure module (the test used to eval a slice of app.js)keepincluded · not factory output—
0b0afa352share: decide the share button and live flag in share-core (the test grepped app.js for IS_PWA)dropdropped in review · not factory output—
0dc7b1fd6reply: assert studio-reply-core load order on the script registry (index.html was grepped)keepincluded · not factory output—
04f2ab0dcauth: restore dashboard scroll from a tested module (the template text was the only check)keepreverted in integration (broke an admin page test) · not factory output—
00a45f1b4Security/safety or high-risk change (category-only public record)keepincluded · not factory output—
0d22d5771chat: plan bang-command side effects before any request (app.js was grepped for the rules)keepincluded · not factory output—
090a052bcaccount: paint auth identity before config from a tested choice (app.js call order was grepped)dropdropped in review · not factory output—
06fe7e57etests: lock overlay asset URLs, snapshot steps, and the dashboard scroll script (deploy coupling)dropdropped in review · not factory output—
0b2d1468dmoderation: cap deleted-message tombstones in the tested core (server dropped an untested half)keepincluded · not factory output—
02eacb93cwidgets: test that a throwing panel stays unmounted and hidden (it was untested)dropdropped in review · not factory output—
020283180Security/safety or high-risk change (category-only public record)fixincluded after fix · not factory output—
0b9fd7fddlanding: test the hero picker by running it (home.js was eval'd from a source slice)fixincluded after fixTest of real behaviour, low, not a real bug
03788dfafmoderation: decide surface wiring in the core (server.js was grepped for planDelete)dropdropped in review—
06ac7b25dfactory: serve the improvement board at /factory for admins (localhost was not reachable)fixexcluded—
0b41dad27quote-game: join the public base in withBase (app.py was grepped for the string)keepincludedTest of real behaviour, low, not a real bug
0a82627a3hot-or-not: join the public base in withBase (app.py was grepped for the string)fixincluded after fixTest of real behaviour, low, not a real bug
0ad6a7969Security/safety or high-risk change (category-only public record)fixremoved at human sign-off (no user-facing fix; touches the live OBS page)Test of real behaviour, low, not a real bug
0b0780d42boot: decide wiring markers in bootWiringFacts (app.js was grepped with match and indexOf)dropdropped in review—
0543bb756emotes: decide picker chrome in emotePickerChromeFacts (styles.css was grepped)dropdropped in review—
0de2f209dmonitor: decide stack-monitor install in stack_monitor_deployment (unit files were grepped)dropdropped in review—
060bbb204studio: never read config during the auth paint (boot could throw when auth/status beat bootstrap)keepincluded · not factory output—
0a10417a0Security/safety or high-risk change (category-only public record)dropdropped in review—
0746a8bdahot-or-not: decide nightly backup coverage in hot_or_not_nightly_backup (backup script was grepped)dropdropped in review—
0b55a3bcboutage-recovery: decide timer install in outage_recovery_deployment (unit files were grepped)dropdropped in review—
02d8da4beaudio: decide scenes HITL page wiring in scenes_hitl_page (templates were grepped)dropdropped in review—
0bc528706asr: decide test-webapp bind and text insertion in asr_test_page (source was grepped)dropdropped in review—
0a3ce2712Security/safety or high-risk change (category-only public record)dropdropped in review—
0e6f552b5soundboard: decide snapshot SEF upload in soundboard_snapshot_upload (backup script was grepped)dropdropped in review—
0374511d1Security/safety or high-risk change (category-only public record)dropdropped in review—
030021c94Security/safety or high-risk change (category-only public record)keepincludedSecurity or safety, low
0c7788803rate-limit: do not record a denied attempt (a retry kept the block open)keepincludedReliability, low
0cb5f655dcookies: keep parsing when one value is not percent-encoded (a bad cookie 500'd auth)keepincludedReliability, medium
029444a5aSecurity/safety or high-risk change (category-only public record)keepincludedSecurity or safety, high
02cc86180Security/safety or high-risk change (category-only public record)keepincludedMinor, medium, not a real bug
0e19c378dchat: drop clipped color codes and blank markup (a cut published hex garbage)keepincludedUser-facing bug, low
14d9f16c6chat: drop length-capped color codes (they leaked hex into the message)keepsuperseded—
1909c4e47chat: strip color codes before capping reply quotes (mobile showed the hex)keepsuperseded—
1d72f016dprivate rooms: stop counting expired invites as active (the card lied)keepincludedUser-facing bug, low
1d7dfa5e5private rooms: stop a max-length slug collision from hanging create (the suffix was sliced off)keepsuperseded—
17f48fdc0private rooms: reject an unparseable invite expiry instead of 500ingkeepincludedMinor, low, not a real bug
2a9c1cca0Security/safety or high-risk change (category-only public record)fixincluded after fixSecurity or safety, medium
2acbbab57Security/safety or high-risk change (category-only public record)fixincluded after fixSecurity or safety, medium
24140c66fSecurity/safety or high-risk change (category-only public record)keepincludedSecurity or safety, medium
2f09ef141Security/safety or high-risk change (category-only public record)keepincludedMinor, low, not a real bug
291e8ac71Security/safety or high-risk change (category-only public record)keepincludedSecurity or safety, medium
3dcfcd96aoffline queue: queue <@!id> mentions (bang form was dropped)keepincludedUser-facing bug, low
336cdfe39db retry: retry SQLITE_LOCKED table locks (message omits "database is locked")dropdropped in review—
3d9e21c90Security/safety or high-risk change (category-only public record)keepincludedReliability, medium
3870fa2c5traffic alerts: gate property cold-start floor (mature baselines paged forever)keepincludedReliability, medium
3e0eae208Security/safety or high-risk change (category-only public record)keepincludedReliability, medium
3c1053815Security/safety or high-risk change (category-only public record)keepincludedReliability, medium
3dff6bd66Security/safety or high-risk change (category-only public record)keepincludedSecurity or safety, low
3e2a4fc8aSecurity/safety or high-risk change (category-only public record)keepincludedSecurity or safety, low
3c98b40aaSecurity/safety or high-risk change (category-only public record)fixincluded after fixSecurity or safety, low
3c34e834dSecurity/safety or high-risk change (category-only public record)keepincludedMinor, low, not a real bug
3f1ee4a1cSecurity/safety or high-risk change (category-only public record)keepincludedSecurity or safety, low
33a09bd45Security/safety or high-risk change (category-only public record)keepincludedMinor, low, not a real bug
4eaeef7b0quote_game: reject malformed room settings and guesses (avoid 500s)keepincludedMinor, low, not a real bug
41b37c058quote_game: ignore unicode digits on practice routes (avoid 500s)keepincludedMinor, low, not a real bug
41278aa02quote_game: drop exclude ids SQLite cannot bind (avoid 500s)keepincludedMinor, low, not a real bug
4f89fc5d4soundboard: reject page offsets SQLite cannot bind (avoid 500s)keepincludedMinor, low, not a real bug
444f19c08tracker: reject list pages SQLite cannot bind (avoid 500s)keepincludedMinor, low, not a real bug
4d3fc3b64vision: let owners reopen archived collections (mine links 404'd)keepincludedUser-facing bug, medium
4b9b34c0fSecurity/safety or high-risk change (category-only public record)keepincludedSecurity or safety, medium
44fac3ad3Security/safety or high-risk change (category-only public record)keepincludedSecurity or safety, low
488a00ae8Security/safety or high-risk change (category-only public record)keepincludedSecurity or safety, low
4c0fac866Security/safety or high-risk change (category-only public record)keepincludedSecurity or safety, medium
4cc7c01e3vision: hide archived collections from photo uploads (replies confirmed the slug)keepincludedMinor, low
47912c0f2vision: hide archived collections from manage requests (403 confirmed the slug)keepincludedMinor, low
5cab51c3bthreads: keep fractional message page cursors (truncation skipped turns)keepincludedUser-facing bug, medium
55a126be1threads: stabilize user message ids across restarts (hash() was salted)keepincludedUser-facing bug, medium
5f9b69fb6Security/safety or high-risk change (category-only public record)keepincludedReliability, low
577106044nx: reject blank bundle names (whitespace was stored as untitled)keepincludedMinor, low
5b3e67052spum-router: accept scores timed in raw seconds (rounded millis rejected real runs)keepincludedUser-facing bug, medium
5ae9f618dSecurity/safety or high-risk change (category-only public record)keepincludedSecurity or safety, low
5ec41b773Security/safety or high-risk change (category-only public record)keepincludedReliability, medium
5a6f2b1a4nx: reject blank chat prompts (whitespace started a rewrite)keepincludedMinor, low
537ce2bb2threads: keep same-timestamp turns on a message page (exclusive cursor skipped them)keepincludedUser-facing bug, medium
53f97c164studio: return 400 for invalid agent-room proposals (429 retried them)dropdropped in review—
541df532cthreads: reject blank http messages (socket already refused them)keepincludedMinor, low
50a343c1creports: reject blank resolutions (whitespace closed the report)keepincludedMinor, low
640e7958bstudio chat: ignore events from a replaced socket (late close dropped the live connection)keepincludedUser-facing bug, medium
69f7b7de8Security/safety or high-risk change (category-only public record)fixincluded after fixSecurity or safety, low
68c00dfbcagent room: keep proposals single-flight (Enter repeat queued a second copy)fixincluded after fixUser-facing bug, low
65899ef37presence: keep the richer chatter when web and phone collapse (relay heartbeat wiped name and avatar)keepincludedUser-facing bug, low
6a5eead9aagent room: keep the throbber on the latest phase (elapsed tick restored the first snapshot)keepincludedUser-facing bug, low
6fd50af48Security/safety or high-risk change (category-only public record)keepincludedSecurity or safety, low
6a21f07b8share: keep emoji whole when a title is truncated (slice split the surrogate pair)keepincludedMinor, low
63cd88f33agent room: unlock a propose when the socket drops (ack never arrives, composer stayed disabled)keepincludedUser-facing bug, low
7ce0864cdstream-memory: hold the cursor below undigested speech (budget and LLM errors skipped rows)keepincludedReliability, medium
76c8fe3ffstream-memory: keep over-cap rows behind the cursor (tail trim skipped the head)fixincluded after fixReliability, medium
795f81374Security/safety or high-risk change (category-only public record)keepincludedUser-facing bug, low
7034c24d2Security/safety or high-risk change (category-only public record)keepincludedSecurity or safety, low
70e700d93live-watch: keep a poll error when no state row exists (first failure looked unchecked)keepincludedUser-facing bug, low
706365d18Tips recipient identity and lookup performancefixinitially held · indexed replacement deployedSecurity or safety, medium
734db1928Security/safety or high-risk change (category-only public record)keepincludedMinor, low
815b9ef51Security/safety or high-risk change (category-only public record)keepincludedUser-facing bug, low
86482b0b0clipper: follow replay-chat cursors (offset step skipped the rest of a busy second)fixincluded after fixReliability, medium
869a06130Security/safety or high-risk change (category-only public record)keepincludedUser-facing bug, medium
8a34be77dSecurity/safety or high-risk change (category-only public record)keepremoved at human sign-off (no real producer sends the bad input)Reliability, low
8981b9a38stream daemon: keep chat buffering after a bad inbox row (null username killed the listener)keepincludedReliability, medium
8e6d317d3Security/safety or high-risk change (category-only public record)keepincludedUser-facing bug, medium
87b62ec08Security/safety or high-risk change (category-only public record)keepincludedReliability, medium
89151514dSecurity/safety or high-risk change (category-only public record)keepincludedUser-facing bug, medium
9bc711bb7Security/safety or high-risk change (category-only public record)keepincludedSecurity or safety, low
97eff7348audio: do not resurrect a discarded generation (late mark_ready)keepincludedReliability, low
9da86f216hot_or_not: return skipped room photos to the pack (a skip is not a played round)keepincludedUser-facing bug, medium
98546101aSecurity/safety or high-risk change (category-only public record)keepincludedSecurity or safety, medium
9757a9b6dsoundboard: keep a taken-down upload's file when the last active copy is deleted (restore would 404)keepincludedReliability, low
922da1505hot_or_not: keep finished More Takes when a late failure arrives (slots 2 and 3 were deleted anyway)keepincludedReliability, low
1034056160messenger: refuse re-invite of a joined member (role rewrite)keepincludedMinor, low, not a real bug
1065757d89Security/safety or high-risk change (category-only public record)keepincludedSecurity or safety, low, not a real bug
10502bd15dSecurity/safety or high-risk change (category-only public record)keepincludedSecurity or safety, low
1098fc1472Security/safety or high-risk change (category-only public record)fixincluded after fixSecurity or safety, low, not a real bug
108aaa7014Security/safety or high-risk change (category-only public record)keepincludedSecurity or safety, medium
101a88e2e9Security/safety or high-risk change (category-only public record)keepincludedSecurity or safety, low
109224ca1aSecurity/safety or high-risk change (category-only public record)keepincludedSecurity or safety, low, not a real bug
11478f7508studio bridge: retry rate-limited Twitch relays on replaydropdropped in review—
11d774dd50studio db: reject older go-live noto claims (stale edge overwrote the live session)keepincludedReliability, low
1188ceaadfSecurity/safety or high-risk change (category-only public record)keepincludedSecurity or safety, medium
111b19fc3cstudio nx: drop hermes edits cancelled mid-run (cancel still published the revision)keepincludedUser-facing bug, low
11802cdf14studio db: ignore older chat reaction snapshots (replay rewound emote counts)keepincludedUser-facing bug, low
11cfd77296studio cli: list streamers by numeric twitch id (rename hid the channel)keepincludedMinor, low
11c4d092ecstudio db: scope chat replay to the current session (wrong clock key)keepincludedUser-facing bug, medium
12145331c4tracker: resolve reused Twitch logins to the latest channel (login is not identity)keepincludedUser-facing bug, low
126f807293community: resolve reused Twitch logins to the latest channel (login is not identity)keepincludedUser-facing bug, low
122be5c789tracker: keep numeric Twitch ids when a login is reused (canonical pages followed the alias)keepincludedUser-facing bug, low
1251a3aff8tracker: keep numeric Kick ids when a slug is reused (canonical pages followed the alias)keepincludedUser-facing bug, low
12eab238c6tracker: pin search hits to numeric ids when an alias is reused (login is not identity)keepincludedUser-facing bug, low
12138353actracker: page past a full rankings cache instead of skipping rows (lookahead is not the last page)keepincludedUser-facing bug, low
126208b680tracker: rank unknown followers behind a real zero (pages follow the cache)keepincludedUser-facing bug, low
1364cb5f5bsound_library: scale kHz note suffixes (2khz parsed as 2 Hz and bare hz crashed)keepincludedUser-facing bug, low
13981182eaytdownloader: parse Imgur ids after tag routes (prefix was used as the id)keepincludedUser-facing bug, low
1312fc12a2ytdownloader: download direct Imgur mp4 and gifv (image gate rejected them)keepincludedUser-facing bug, low
137cd5e02csound_library: parse A440 as concert pitch (octave 440 silenced the render)keepincludedUser-facing bug, low
13922321f4sound_library: parse timing unit suffixes (bare float crashed the render)keepincludedReliability, low
131b1cfd2cclipper: accept mobile Twitch VOD URLs (m.twitch.tv was rejected)keepincludedUser-facing bug, low
14cccb5054tos-filter: pin letters-only stem hits (spaces were joining ordinary words)keepincluded · findings fixed and deployed in follow-upTest of real behaviour, medium, not a real bug
14455ab45dtos-filter: pin l33t stems glued to the next word (digits never reach the stem scan)keepincluded · findings fixed and deployed in follow-upTest of real behaviour, medium, not a real bug
14877595d2tos-filter: pin slash form of the 14/88 hate code (regex only allows whitespace)keepincluded · findings fixed and deployed in follow-upTest of real behaviour, low, not a real bug
14d48f8533tos-filter: pin dot, underscore, and slash forms of preteen and underage (separator class is only hyphen or space)keepincluded · findings fixed and deployed in follow-upTest of real behaviour, medium, not a real bug
1448c97251tos-filter: pin hyphen and slash forms of numeric age phrases (age rules only accept whitespace)keepincluded · findings fixed and deployed in follow-upTest of real behaviour, medium, not a real bug
14f92788cdtos-filter: pin determiner forms of minor-plus-clothing phrases (in+garment allows no word between)keepincluded · findings fixed and deployed in follow-upTest of real behaviour, medium, not a real bug
14e0e629c1tos-filter: pin plural forms of sexualized-minor phrases (minor-word list is singular only)keepincluded · findings fixed and deployed in follow-upTest of real behaviour, medium, not a real bug