|
researchaudio.io · field key no. 2610.28380 · oct 27 2026
Without the First Run, Google's Harness Router Drops 12.64 Points
Opus 4.8 writes subclasses for Gemini. On 49 unfamiliar tasks, the right call was to change nothing.
| specimen label |
| specimen | Object-Oriented Harness (OO-Harness) |
| collected | arXiv 2610.28380 · Oct 20 2026 |
| collectors | 12 authors: 10 Google Cloud AI Research, 2 University of Illinois Urbana-Champaign (lead author Gaotang Li, as an intern) |
| det. | evolution: Claude Opus 4.8 · routing: Claude Sonnet 5 · worker: gemini-3.5-flash, fixed |
| habitat | AutomationBench · Workspace-Bench · Toolathlon-GYM |
Google Cloud AI Research named its new harness paper after inheritance. The biggest swing in its ablations comes from somewhere else. On AutomationBench, one frozen set of evolved harnesses scored 64.88 when the router read the task description and metadata alone, and 77.52 when it also read the base agent's first attempt.
That 12.64-point gap is the paper's own ablation. For anyone shipping an agent, it is the most portable result in the paper. Below: how the class tree grows, what the scoreboard says with error bars back on, the cost the step column leaves out, and the 49 unfamiliar tasks where doing nothing was correct.
| couplet 01 · what shipped
keyed to: the full paper, appendices included |
A family of harnesses instead of one
A harness is the code around a model: prompts, tools, context rules, and the checks that decide when a run is finished. OO-Harness keeps the model fixed and evolves that code. Instead of one global harness, it grows a family, where every specialist is a Python subclass of the base or of an earlier specialist.
Three models fill three roles. Claude Opus 4.8 writes the subclasses, up to 40 evolution iterations per benchmark. Claude Sonnet 5 routes each test task in the main comparison. The work itself is done by gemini-3.5-flash, which never changes. So a Google paper uses Anthropic models as the engineers and Google's Flash model as the labor, and Opus 4.8 is also the evolution agent for every competing harness optimizer.
Each benchmark is split into discovery tasks (for evolution) and held-out tasks (for scoring): AutomationBench business workflows, 300 and 300; Workspace-Bench file-heavy workspace tasks, 288 and 100; Toolathlon-GYM multi-tool tasks, 251 and 252. The headline: best primary score on all three, up to 15.79 points over the base harness. The split matters. A July 2026 study, Rethinking the Evaluation of Harness Evolution, found automatic harness evolution did not beat plain test-time scaling on Terminal-Bench 2.1 and generalized poorly.
| couplet 02 · how the tree grows
keyed to: section 3, appendix A.4 and B |
Same as the parent, plus one step
Think of a recipe card that says "same as Grandma's, plus one step." A subclass inherits every method it does not touch. When it does override one, it calls super() first, so the parent's behavior still runs before the new check.
The loop: run the base on discovery tasks, then cluster those tasks by shared needs seen in those runs, with no fixed domain list. The evolution agent picks a cluster and a parent, writes one subclass around one hypothesis, and is told: do not hardcode task IDs, answers, or filenames. The candidate is scored on that cluster's tasks alone and kept if it beats the cluster's current score. Lessons from every attempt, kept or rejected, are frozen before testing.
| plate 1 · two real lineages from the appendix |
| automationbench · discovery-setup lineage |
| s9 | Completion check: if the task needs a write and no mutating API call happened, ask for another turn |
| ↳ s12 | Adds a reminder near the turn limit when no write has succeeded |
| ↳ s29 active | Adds hints for unrecoverable errors and repeated unproductive tool results, keeping both checks above |
| toolathlon-gym | manage_context | verify_before_finish |
| s7 | overrides: calls super(), then reminds the agent to write computed values, not formula strings, to .xlsx files | inherited from base |
| ↳ s30 | inherited from s7, untouched | overrides: calls super(), then checks percent-labelled cells in the written workbook |
Source: paper Figure 9 and Listing 1. Blue cells are overrides.
Scoring candidates on one cluster instead of the full discovery set is also why the search is faster. Mean recorded search time was 29.0 hours for OO-Harness against 109.1 for Meta-Harness (Figure 4).
| couplet 03 · the scoreboard, error bars on
keyed to: Table 1, three runs per cell, mean ± std |
Specialists carry most of it
| method | automation
mean | workspace
mean | toolathlon
strict pass |
| base harness | 61.62 ±0.15 | 25.45 ±2.98 | 31.15 ±0.28 |
| best-of-5 base runs | 60.24 ±1.07 | 37.71 ±0.45 | 32.14 ±0.56 |
| Meta-Harness global | 63.92 ±0.89 | 29.56 ±1.23 | 33.45 ±1.23 |
| Agentic Harness Eng. global | 64.18 ±0.91 | 38.43 ±4.32 | 36.23 ±1.19 |
| Adaptive Auto-Harness specialists, no shared code | 75.40 ±1.26 | 38.62 ±2.98 | 34.32 ±1.47 |
| OO-Harness | 77.41 ±1.37 | 39.48 ±3.21 | 37.45 ±0.98 |
Source: paper Table 1, primary metric per benchmark, scores in percent. Two other test-time scaling rows omitted.
Two readings. First, most of the AutomationBench jump comes from having specialists at all. Adaptive Auto-Harness, which keeps each specialist as a separate implementation with no inheritance, already gets 13.78 of OO-Harness's 15.79 points. The two global optimizers, Meta-Harness and Agentic Harness Engineering, get 2.30 and 2.56.
Second, on Workspace-Bench the top three sit within one standard deviation of each other: 39.48, 38.62 and 38.43, with spreads of 3 to 4 points. Five plain base runs with an Opus 4.8 picker reach 37.71. On Toolathlon-GYM the lead over the best global harness is 1.22 points. OO-Harness is first in every column; in two of them, first is a narrow margin.
| couplet 04 · the 12.64 points
keyed to: Table 2 routing ablation, single runs |
The router has to watch one attempt
Same frozen catalogue, same worker model, same test split. The one change is what the router sees. The task-text router gets the description and metadata, and has to report confidence of at least 0.7 to pick a specialist. The trace router also gets a digest of the base run. Neither sees a grade.
| benchmark | base | router reads
task text | router reads
text + base run | the run
adds |
| automation mean | 61.30 | 64.88 | 77.52 | +12.64 |
| toolathlon strict pass | 31.35 | 33.33 | 37.70 | +4.37 |
| workspace mean | 24.57 | 40.03 | 40.56 | +0.53 |
Source: paper Table 2. Single ablation runs, separate from the three-run averages above.
Read the AutomationBench row slowly. Routed on task text, the entire evolved family adds 3.58 points over base, about what one global harness manages. Routed on the trace, it adds 16.22. Roughly 78 percent of that gain depends on the router watching the base agent try once.
Why Workspace-Bench barely moves: there, the need is printed in the request. A task that asks for a .docx needs the harness that writes a real Word file, because the base file backend wrote plain encoded text under a .docx name. On AutomationBench the need appears after the fact: the run made zero writes, or HubSpot and Intercom answered 401, or a query failed to parse twice in a row. No prompt shows you that in advance.
| plate 2 · the router, drawn as a field key |
| 1a | Base run finished cleanly: three writes, no errors | keep base result |
| 1b | Base run shows trouble | go to 2 |
| 2a | The same query fails to parse, is revised, fails again | API error-normalizer |
| 2b | No writes; HubSpot and Intercom return 401; a Lead lookup returns 404; empty final answer | resource-manifest harness |
| 2c | No learned mechanism clearly fits | keep base result |
Illustrative arrangement of lessons and choices quoted in the paper (Figure 6c, Table 4, routing prompt). The real router is an LLM call, not a fixed tree.
The sorting works. In one AutomationBench run of 300 tasks, the router kept the base result on 134 and sent 166 to specialists. The kept tasks had scored 77.15 mean partial credit on their base run; the rerouted ones, 44.40. The router never saw those grades. It sorted strong runs from weak ones on visible signals.
The digest is small enough to copy. In the AutomationBench reference router: the query is cut at 600 characters, the digest covers at most 60 message entries and is capped at 5,000 characters, each routing lesson gets 180 characters and each harness description 120. One call retrieves the relevant rules, a second picks a harness and returns JSON with the choice, the rule, and one sentence of why.
| couplet 05 · what the step column leaves out
keyed to: Table 1 footnote, section 3.5 |
Fewest steps, with the scouting runs off the books
OO-Harness posts the lowest step count on AutomationBench: 27.99 against 31.27 for base. The footnote says Steps counts the finally picked trajectory and excludes other rollouts and auxiliary selection calls. Under OO-Harness, every test task runs the base harness first, because the router needs that trace.
| what runs, per test task | when | in Steps |
| Base harness, full run on gemini-3.5-flash | every task | when kept |
| Router: rule retrieval, then a pick (400 s and 1,200 s timeouts in the reference router) | every task | never |
| Specialist harness, second full run | 166 of 300 in the case study | yes |
Source: paper section 3.5, Table 1 footnote, appendix A.4.2. On rerouted tasks the base run is spent and not counted.
The paper's fair answer is already in the table. It includes test-time scaling rows that also spend a second run. On AutomationBench, best-of-2 scored 61.46, and a second run guided by Opus 4.8's notes on the first scored 59.29, both at or below the 61.62 base. A second attempt by itself adds nothing there. A second attempt with a targeted fix is worth about 16 points. The comparison holds; what is missing is a cost-per-task column.
| couplet 06 · the 49 unfamiliar tasks
keyed to: section 5.3, Figure 7, appendix A.7 |
The best specialist was no specialist
The team evolved harnesses on 254 Toolathlon-GYM tasks from Google Forms, Calendar and Sheets, HowToCook, Notion, Snowflake and WooCommerce. Then it tested on 49 original Toolathlon tasks from Amazon, Canvas, Git and GitHub, Kubernetes, arXiv and Scholar, MySQL, 12306, Redis, Yahoo Finance and YouTube. No app family appears on both sides.
29 / 49 base harness
59.18% strict pass |
22 / 49 Meta-Harness, evolved global
44.90%, 7 tasks lost |
29 / 49 OO-Harness
59.18%, kept base on all 49 |
Source: paper Figure 7. Task counts computed from the published percentages.
The evolved global harness turned 7 passing tasks into failures on apps it had never seen. OO-Harness matched base because its router declined every specialist. That is a real property, and it is the one global optimizers lack: a do-nothing branch. It also means this figure measures restraint, not specialization. EvoHarnessBench found the same hazard from another angle: adding tools, skills and specialist agents can break tasks an agent used to solve.
| couplet 07 · honest notes
keyed to: what the evidence carries, claim by claim |
The title idea has the thinnest evidence
| claim | what supports it | weight |
| Inheritance beats copying code | One benchmark, one run per variant: 40.56 vs 35.60 copied vs 29.39 from scratch. Run-to-run spread on that benchmark is ±3.21. | thin |
| Specialists beat one global harness | All three benchmarks; Adaptive Auto-Harness gets most of it without inheritance. | holds |
| Routing on the base run | +12.64 / +4.37 / +0.53, single runs. Large where failures hide in the trace. | holds |
| Best on Workspace-Bench | Top three within one std. Strict pass is 8.33%, so about 92 in 100 tasks still miss a rubric. | narrow |
| Fixes target the model | The workbook check multiplies percent-labelled values between 0 and 1 by 100. The paper notes it assumes the task wants 20, not 0.2. A spreadsheet cell formatted as a percent stores 0.2. | some learn the grader |
| Reproducible | The paper links code for its three evolved baselines. I found no public OO-Harness repository as of Oct 27. The limitations section covers outages, rate limits and credentials, not cost. | open |
Source: paper Figure 5, Tables 1 and 2, appendix A.8 and B.3.
One more pattern, which is my reading and not a test the paper ran. On Workspace-Bench, all six of OO-Harness's learned clusters line up with Adaptive Auto-Harness's task-type groups, and the two land 0.86 apart. On AutomationBench, Adaptive Auto-Harness groups by business domain while OO-Harness clusters by failure (a missing-write cluster, an arithmetic cluster spanning five domains), and the gap is 2.01. Grouping tasks by how they fail may matter more than how the code is shared.
| couplet 08 · for your own harness
keyed to: what the results suggest for builders |
Four things to copy
| 01 | Write a trace digest for every run. Write count, error classes, unfinished artifacts. On AutomationBench that digest carried most of the routing signal; the prompt carried little. |
| 02 | Make "change nothing" a first-class route. It is why OO-Harness lost nothing on 49 unfamiliar tasks while a global harness lost 7. |
| 03 | Test a fix on the slice it targets. Cluster-level scoring is the paper's explanation for 29.0 hours of mean search time against 109.1 for Meta-Harness. |
| 04 | Count steps per task, not per final run. Two full attempts plus the router calls is the real bill whenever a specialist runs. |
| The class tree gives the paper its name. The result that travels is smaller: watch one attempt before you change anything, and keep the option to change nothing. |
sources
Li et al., Object-Oriented Harness: Evolving Agent Harnesses through Object-Oriented Design, arXiv 2610.28380
Benchmarks: AutomationBench · Workspace-Bench · Toolathlon-GYM
Baselines and context are linked inline above.
|
