| researchaudio.io · change record · ref arXiv 2610.14224 | system: agent harness
model weights: no change | approver: a test run
answer key: none |
Same Weights, New Files: GLM-5 Scored 11.5 Points HigherChanges with no test run decreased a code benchmark result from 73.4 to 63.2. | chg 01 | The result state: approved |
GLM-5 scored 64.2 on WorkBuddy-Bench. Gemini 3.5 Flash scored 72.4 on the same benchmark. Both models used the same stock agent setup, OpenCode. Google Cloud AI Research then applied one step of Harness² to GLM-5. The model weights did not change. GLM-5 scored 75.7, which is higher than the Gemini score. 64.2 GLM-5
stock harness | 75.7 GLM-5
one Harness² step | 72.4 Gemini 3.5 Flash
stock harness |
WorkBuddy-Bench average of three domains (Code, Web, Office), OpenCode harness, one improvement step (k=1). Source: Table 1 of the paper. The step changed the files around the model: the system prompt, the guardrails, the skills, the scripts and other parts. The paper calls this set of files the harness. This issue explains how the method changes a harness with no answer key. It also shows where the method failed. | chg 02 | The problem: no answer key state: open |
An agent is a model plus a harness. The harness controls what the model reads, which tools it can use, and how it gives work to other agents. In Harness², the harness has eight parts. Each part is a file. Each change is a diff that you can read and test. | group | parts (each part is a file) |
|---|
| context | system prompt, guardrails, memory: in the context for each call. Skills: loaded when the agent needs them. | | tools | tool guidance: when and how to use each tool. Scripts: Python helpers that the agent runs through bash. | | delegation | subagents: workers with their own context window. They send back their result and nothing more. | | middleware | plugin: code in the tool loop. It enforces a behavior. It does not ask for it. |
The eight editable components. Source: Figure 2 and Table 5 of the paper. Most methods tune a harness offline. They test many versions against a grader on training tasks, then they ship the best version. Meta-Harness, one of the baselines, works this way. In production, a new task has no training data and no grader. The problem is clear: the system must decide if a change helped, but it has no answer key. | chg 03 | Propose, Probe, Compose state: mechanism |
Harness² uses a contrastive step with three parts. Think of a canary deployment. You do not send a config change to all servers. You send it to a small group first. You compare that group with the baseline on the same traffic. Then you keep the change if the data supports it. Reference run. First, the agent does the task with the current harness. Nobody grades this run. It shows how the harness behaves. Propose. Two editors read the reference run. The invariant editor changes how the agent works: plans, tool use, and checks of its own output. The domain editor adds what an expert knows: rules of the field, the shape of the deliverable, and expert checks. Each edit must point to a specific moment in the run. Probe. Each candidate harness does the same task, with the same model, in the same environment. The differences between the runs are the evidence. Compose. A composer reads all runs and all diffs. It can merge the edits, revise them, adopt one candidate, or keep the current harness. Each decision must point to an observation in a run. reference run current harness, not graded | → | propose invariant edit + domain edit | → | probe same task, model, environment | → | compose merge, revise, adopt or keep | → | reflect store harness and lessons | | never in the workspace: rubric · score · grader |
One improvement step. The reflect stage feeds the library for the next task (chg 06). Source: Figure 3 of the paper. No stage sees a rubric, a score or a grader. A guard does not give an editor any workspace that contains a verdict, a rubric or a score. In the main experiments, the editor is Claude Opus 4.8. The budget k controls the scale. Parallel recursion makes k invariant candidates and k domain candidates, then k compositions from the full pool. Sequential recursion links k steps, and each step starts from the last result. When k is more than 1, a selector compares the anonymous deliverables and picks one. | chg 04 | The change with no test run state: rolled back |
The team also tested a direct edit. An editor read the harness files and changed them. It had no reference run, no probes and no library. | domain | stock | direct edit | edit + k=1 library | edit + k=5 library |
|---|
| Code | 73.4 | 63.2 −10.2 | 73.8 +0.4 | 74.3 +0.9 | | Web | 69.9 | 65.0 −4.9 | 72.6 +2.7 | 74.3 +4.4 | | Office | 74.0 | 77.1 +3.1 | 77.5 +3.5 | 79.1 +5.1 |
Gemini 3.5 Flash, OpenCode, WorkBuddy-Bench. Change against the stock harness. The library columns use a library from earlier tasks with no new probes. Source: Figure 7 (left) of the paper. With Gemini 3.5 Flash, the direct edit decreased the Code score by 10.2 points and the Web score by 4.9 points. The Office score increased by 3.1 points. A library from earlier k=1 runs returned the scores to the stock level. A library from k=5 runs increased all three scores, with no new probes. This part is our analysis. The paper did not test it. The direct edit is like a request to a model to improve a prompt file, with no test run after the change. | chg 05 | What the approved changes did state: approved |
The appendix shows the edits that the composer kept. No edit made the model smarter. Each edit changed a procedure. | task | what the run showed | the edit | score |
|---|
| WorkBuddy Code | The model wrote tool calls as plain text, so no tool ran. | Tool guidance (1,023 bytes): never write a tool call as text. | 60.0 → 100 | | WorkBuddy Office | The agent grouped data at the wrong level. No row matched the key. | Skill: the exact group key and two derived fields. Guardrail: do not change the inputs; clean the data before you aggregate it. | 23.6 → 100 | | LAB arbitration | The markup stayed in tracked changes. As text, the markup disappeared. | Skill: deletions as strikethrough and insertions as bold, in the body text. | 44.4 → 94.5 | | JobBench production | The agent did not compare figures across files. It did not open its outputs again. | Guardrail, reconciliation skill, and a script that opens each output file again. | 43.1 → 77.8 | | JobBench research, Opus 5 edits itself | Research used the full budget. The output files stayed empty. | Plugin: after 25 tool calls with an empty output folder, it adds a note to each tool result. | 72.0 → 100 |
Base run score to re-run score. Source: Figures 9 and 15 to 19 of the paper. About half of all edits were in skills and scripts. After one step, the tool error rate decreased on all three benchmarks. On JobBench, it decreased from 14.4% to 9.1%. The plugin case gives a useful rule. A code comment in the paper gives the reason. The model can skip an instruction. It cannot skip a line inside its own tool result. | chg 06 | The second loop: a library of lessons state: approved |
After each task, a reflector examines the full record. It stores two items in a library: the adopted harness and an EXPERIENCE.md file. That file records the weakness, the conditions, and the evidence. If the evidence is weak, the reflector adds nothing. The next task does not start from the last harness. It starts from the stock harness, and the editors can read the library. A fact from an earlier task is a hypothesis until the editor examines it again. | change to the method | WorkBuddy | JobBench |
|---|
| full Harness², k=1 | 77.5 | 36.0 | | no library | 74.4 −3.1 | 34.2 −1.8 | | two generic editors in place of invariant and domain | 74.6 −2.9 | 32.3 −3.7 | | no compose step | 75.9 −1.6 | 34.1 −1.9 |
Ablation, Gemini 3.5 Flash under OpenCode. Source: Table 3 of the paper. The library also moves between benchmarks. A library from JobBench increased the one-step WorkBuddy scores by 0.6, 0.5 and 2.9 points. The editor does not have to be a larger model. The team used the same model as executor and editor at k=2. All five models improved on every benchmark. GLM-5 increased its LAB score by 13.7 points. Claude Opus 5 increased its scores by 3.3, 5.5 and 3.0 points. | chg 07 | Results and cost state: held: more compute |
| benchmark | model | stock | k=1 | k=5 |
|---|
| WorkBuddy-Bench | Gemini 3.5 Flash | 72.4 | 77.5 | 80.8 | | GLM-5 | 64.2 | 75.7 | 79.6 | | JobBench | Gemini 3.5 Flash | 31.5 | 36.0 | 39.0 | | GLM-5 | 20.6 | 24.2 | 28.4 | | LAB (partial) | Gemini 3.5 Flash | 79.9 | 81.9 | 84.5 | | GLM-5 | 59.6 | 71.4 | 78.1 |
OpenCode harness. k=5 uses parallel recursion. LAB partial is the per-criterion pass rate. Source: Tables 1 and 2 of the paper. The baselines had lower scores. ReasoningBank was as much as 6.6 points below the stock harness on Code tasks. Harness Scaling was below the stock harness in 7 of 12 pairs. Meta-Harness was the strongest baseline. Harness² was better on WorkBuddy-Bench and JobBench in all four configurations. On LAB, Meta-Harness was better for Gemini 3.5 Flash under Codex. The team also compared production harnesses with Gemini 3.5 Flash on WorkBuddy-Bench. Claude Code had the best stock score on Code and Office. Codex had the best stock score on Web. Harness² started from OpenCode and was 4.6, 5.8 and 4.9 points above the best stock harness in each domain. Each candidate is a full run of the task. Thus, the method uses more tokens. The paper counts all tokens before the deliverable, which includes the edit work. With Gemini 3.5 Flash, k=5 increased the gain over the stock harness. On JobBench, the gain went from 4.5 points at k=1 to 7.5 points. On WorkBuddy-Bench, it went from 5.1 to 8.4. Both configurations are on the Pareto frontier of cost and score. Meta-Harness used more compute than k=5 and got a lower score. The improved agent also used fewer steps on WorkBuddy-Bench: 21.9 per task, from 25.4. | chg 08 | Where the method failed state: rolled back |
The authors show nine failures in detail. Four of them are useful for anyone who builds a loop like this. | failure | what happened | score |
|---|
| the selector picked the wrong output | The candidates scored 95, 85, 85, 80 and 55. The blind selector picked the 55 as the simplest full implementation. | 96 → 55 | | one edit spread to all candidates | An edit said that keys must keep their exact names. The tests used lowercase keys. All four candidates kept the rule and scored 42. | 50 → 42 | | the editor checked its own work | The editor assumed that a cache keyed on object identity was enough. Its own checker script reported a pass three times. 4 of 11 tests needed a content-keyed cache. | 75 → 64 | | an assumption became a rule | A skill told the agent not to recommend a claim for an extra 106,153 dollars. The benchmark criterion gave credit for that recommendation. | 83 → 81 |
Reference score to final score. The pipeline did not see these scores. Source: Figure 14 of the paper. The paper gives the lesson in one line: “A checker written by the editor cannot catch the editor’s assumption.” Candidates from the same model share the same blind spots. When candidates agree, the agreement is a signal, not proof. | chg 09 | Post-change review for your own agent state: actions |
| [1] | Test each harness change with a run. A change that nobody runs is not tested. The direct edit decreased one score by 10.2 points. |
| [2] | Write two edits with different scopes. Make one edit for procedure and one edit for domain knowledge. When the team removed this split, the scores decreased by 2.9 and 3.7 points. |
| [3] | Put new instructions in on-demand parts first. Use this sequence from the editor prompts:
1. a skill for a procedure with a trigger
2. a script for a repeated mechanical step
3. a subagent for a large input with a small answer
4. a plugin for a rule that the model skips
5. always-on prompt text, last |
| [4] | Use a check that the editor did not write. The editor’s own checker reported a pass three times on a wrong assumption. |
The model weights did not change in any of these experiments. The authors name one open problem: the library. It grows with each task. The current version does not merge or remove old entries. | 
