Reviewers Caught ~30 Dropped Names. A Script Found 1,703.

Nous Research let Hermes Agent refactor a million lines of its own Python with 1,393 subagents. The blog post tells the win. The team’s own post-mortem tells where the money and the misses went.

On September 2, Nous Research gave Hermes Agent one standing goal: shrink its own codebase by at least 30 percent. About 19 active hours and 1,393 subagents later, non-test Python went from 1,063,826 lines to 698,363. The pull request title promised zero behavior change, and the test suite agreed: 51 failures on the branch, the same 51 on untouched main.

Then people read it. Reviewers caught about 30 public names the agents had deleted. A script Nous wrote afterward counted 1,703, in 18 seconds. That figure sits in Nous’s own forensic post-mortem, not in the launch post, and none of the coverage we read picked it up.

This issue reads the run through those receipts. The short version: the model did the refactor, but the checks and the harness decided what it missed and what it cost.

The setup is a general contractor with 36 crews. The top agent measured the repo, cut it into 36 non-overlapping groups, and wrote each worker a brief: what to simplify, which interfaces to keep, which checks to pass before committing. Workers edited in separate git worktrees (independent checkouts of one repo) so they never overwrote each other, and some hired their own sub-crews, three levels deep.

The guardrails were real. A tool’s JSON schema had to stay identical, and a command’s help output was compared byte for byte. The worker briefs came partly from a Hermes skill the agent had built up from daily work with Teknium, who ran the job, including one habit that matters later: rerun a failing test on unchanged main to see whether you caused it.

On a building site, the punch list is what the inspector finds after the builder says the job is done. Here is Nous’s own inspection of the pull request as first opened, from the post-mortem.

The mechanism is simple. A worker saw a name with zero callers inside the repo and deleted it as dead code. But Hermes has external plugins, and their imports live in other people’s repos. Nothing inside the repo can see them, so nothing inside the repo, tests included, can object.

Review found other classes too. An automated rewrite around Python’s suppress() helper touched 1,580 sites, and a syntax-tree differential (comparing the parsed code before and after) found about 65 semantic mismatches: wider or narrower exception types, changed scope, one dropped log line. The first review round also found a packaged wheel missing 17 modules (two were already missing on main) and 3,272 lost comment blocks, all restored before merge.

Of the 72 fix commits the post-mortem classified, 16 restored dropped symbols or tests: exactly the class the new surface-diff script catches.

The pull request’s verification section reads: 44,859 passed, 51 failed, the identical 51 failing on main, regressions attributable to this change, zero. That is a sound method with one hidden assumption: that both runs used the same test suite.

They did not, at first. The same agents that wrote the code had dropped 126 test definitions and trimmed five suites around behavior that was still live. Parity against a suite the agent is editing is like a contractor inspecting his own wiring with a meter he rewired. Nous’s later benchmark shows test lines up 0.1% on the final head, but that is after reviewers put the tests back.

The post-mortem splits spend by bucket, using per-token rates that reproduce the run’s own recorded estimate. It says plainly that this checks the estimator, not a bill.

Cache writes are the cost of re-sending context. Every request carried the one-hour cache tier from the operator config, so a worker that lived 20 minutes paid for an hour of retention. On the same loop, one-hour markers measured 20 dollars per million written tokens against 12.51 for five-minute markers; the fix drops subagents to five minutes. Depth-2 workers alone were 67% of spend.

Then there is what the harness lost on its own:

To Nous’s credit, the post-mortem withdraws its own earlier cost-reduction headline and claims no total cost reduction. Each fix is reported as a before-and-after measurement, not a projected reduction.

The blog closes on “1% of the cost and 1% of the time” against a manual estimate of 150,000 to 1.8 million dollars, and it notes that human review time is excluded. The estimate range is 12 times wide, so the ratio depends on which end you use.

The size headline moved too. The pull request body reports 674,569 lines (−36.6%); the blog reports 698,363 (−34.4%). The 23,794-line gap is close to the size of the plugin compatibility layer added during review (25,958 lines, per Nous’s benchmark notes), which looks like the main cause, though Nous does not reconcile the two numbers.

That layer matters to anyone outside the repo. It keeps 2,091 old import paths working, including 290 public definitions the agents had deleted as unused and review restored verbatim. The manifest on main set the layer’s end at September 14, this past Monday: from that date an affected plugin is disabled unless the user sets plugins.allow_deprecated_imports: true. Plugin authors got ten days from merge.

The transparency is the strongest part of this story. Most teams would ship the blog. Nous also shipped a ledger of its own misses, with counts precise enough to argue with.

The coverage led with the cost ratio. The number worth carrying into your own pipeline is smaller: the least expensive check in the whole run took 18 seconds, and it was not a model.