|
Codex is an AI coding agent. In this test, it had to find 80 numbers in old Treasury reports. Then it had to calculate one result from them.
At step 22, Codex found the correct value for May 1942: 4400. At step 166, Codex wrote a note to itself: “Need use 4400.” But Codex did not use 4400. After 170 steps, it gave an answer that used the incorrect value, 4100.
A new paper, LongHarness, from Shanghai AI Lab and Fudan University, studies this problem. This issue explains the problem, the fix, and the test results.
| tag 01 |
What the team built |
| logged one model, four harnesses |
LongHarness is a harness. A harness is the software around an AI model. It gives the model tools, and it controls each step.
The test documents are too large for the model to read at one time. OfficeQA Pro V1 has 697 Treasury documents (approximately 100M tokens) and 133 questions. V2 has 1,435 documents and 90 harder questions. Each V2 question needs values from more than one source.
The team put the same model, GLM-5.3-Flash, in four harnesses: Codex, Claude Code, Recursive Language Models (RLM), and LongHarness. Coding agents can read long documents well (Cao et al.). Thus, they are good comparison harnesses.
| exhibit 1 · correct answers (%), same model in each row |
| harness | v1 · 133 questions | v2 · 90 questions |
| no harness | 15.8 | 4.4 |
| rlm | 45.1 | 38.9 |
| codex | 46.6 | 42.2 |
| claude code | 48.1 | 35.6 |
| longharness | 60.2 | 61.1 |
| nine models with no harness got 2.2 to 5.6 on v2. gpt-5.6-sol and kimi-k3 are in this group. |
| Source: Table 1. Model: GLM-5.3-Flash. Bars to scale, 0 to 100. |
On V2, the harness changed the result from 4 correct answers to 55. The model did not change. On V1, LongHarness got 12.0 points more than the next harness. On V2, it got 18.9 points more.
| tag 02 |
Two failures |
| retracted the agent had the correct value |
The paper identifies two failures. Each failure makes the other failure worse.
Failure 1, unbounded exploration. The agent continues to search, but it does not find the missing fact. Then it uses all of its steps.
Failure 2, evidence-state drift. The agent finds a fact or a correction. But that fact does not go into the final answer.
The Codex run shows the two failures. The question asks for the geometric mean of 80 monthly values (March 1942 to October 1948). At step 29, Codex had 74 values. It wrote: “Still 74; I’m missing values.” Then it read 103 times in sequence (steps 43 to 145).
| exhibit 2 · one question, two records of steps |
| codex · 170 steps | longharness · 10 steps |
| 1-22 · finds table 6: may 1942 = 4400 | 1-2 · dec 1949 report: may 1942 = 4400 |
| 43-145 · reads 103 times in sequence | 3 · records fact e1 = 4400 as active |
| 147 · calculates 4962.22 | 4 · feb 1950 report shows 4100: records an open question |
| 164 · reads feb 1950 report: 4100 | 5 · jan 1950 report (third source): 4400 |
| 165 · calculates with 4100: 4958.09 | 6 · keeps e1 = 4400 |
| 166 · total 57452, table 57751: “need use 4400” | 7-8 · corrects e2 (7199 → 7499) and e4 (8112 → 8142); old ids stop |
| 168 · apr 1950 report confirms 4400 | 9 · calculates from active facts; closes the question |
| answer · 4958.09 (incorrect) | 10 · answer 4962.46 (correct) |
| one incorrect value of 80 changes the result by 0.088%. exact match grades it incorrect. a 5% tolerance grades it correct. |
| Source: Figure 4 and Appendix E, steps in our words. We put 4400 into the 4958.09 calculation and got 4962.47 (the reference gives 4962.46). |
The value 4100 was not a random error. The February 1950 report had a different value for May 1942 than the reports before and after it. Codex saw this conflict.
At step 166, Codex identified 4400 as the correct value. At step 168, another report confirmed 4400. But Codex did not change its answer. The total was 299 too low, and 4400 minus 4100 is 300.
Note: the authors took this example at random from the Codex failures. It shows how the failure occurs. It does not show how frequently the failure occurs.
| tag 03 |
How LongHarness works |
| logged two records, the model sees them at each step |
LongHarness keeps two records outside the chat history. The model sees the two records at each step.
Record 1 is the EvidenceBank. It is a list of facts. Each fact must show its source: a passage that the agent read. A search result is not a source.
Each fact gets a permanent ID. Nobody can edit a fact. To correct a fact, the agent adds a new fact. Then the old fact gets the status “superseded.” To remove a fact, the agent “retracts” it. Compare this to an evidence locker: each item has a tag, and nobody changes an item after it goes in.
Record 2 is the Conclusion State. It has three lists: established facts, possible explanations (hypotheses), and open questions. The first rule in the prompt tells the agent what to search for: the open question that can change the answer most.
| exhibit 3 · the life of one fact |
read
shows the text and records a source id |
→ |
upsert
the harness rejects a fact with no recorded source |
→ |
active
the agent can cite it from the next step |
→ |
revise
superseded or retracted; the agent cannot cite it in a calculation or an answer |
| conclusion state, from the exhibit 2 run (in our words) |
| established | may 1942 = 4400 [e1] |
| hypotheses | possible explanations, each with active fact ids |
| open questions | check other reports for may 1942: opened at step 4, closed at step 9 |
|
| one action at each step · calculations and answers must cite active ids · the history stays complete |
| Source: Section 2, Appendix B, Tables 5 and 6. The hypotheses row gives the paper’s definition. The figure shows no hypothesis. |
If you copy this design, know two more details.
1. The history stays complete. LongHarness keeps the full chat history in each prompt. It does not make the history shorter. If a request is larger than the context window, the run stops, and the answer counts as incorrect. Thus, the two records are an index of the history. They do not replace it. Microsoft’s Less Context, Better Agents did a different thing: it removed old tool output.
2. The harness checks IDs, not facts. The calculation tool makes sure that each cited ID is active. It does not make sure that the numbers in the calculation agree with the source. When a fact becomes superseded, the harness flags the related conclusions. It does not change them. The model must change them.
Other work does part of this. LedgerMind checks that each claim has a source. ReContext shows a fact to the model again if the model found it but did not use it. LongHarness also records the open questions. A claim can have a correct source, but the answer can still need more facts.
| tag 04 |
The memory alone gave zero more answers |
| active percentages changed back to correct answers |
An ablation removes one part of a system. Then it measures the result. The authors did five ablations with GLM-5.3-Flash. The paper gives one score for each. We changed the percentages back to the number of correct answers.
| exhibit 4 · correct answers for each setup |
| setup | v1 / 133 | v2 / 90 |
| 1 · full longharness | 80 | 55 |
| 2 · no records, chat history alone | 74 | 48 |
| 3 · two records kept, search rules removed from the prompt | 74 | 48 |
| 4 · fact list plus a notes file with any text | 71 | 48 |
| 5 · fact list, no conclusion lists | 73 | 47* |
| 6 · conclusions kept, facts cannot be corrected | 75 | 46 |
| Source: Table 2. We calculated the counts from the percentages. *The paper gives 51.7%. No score out of 90 gives 51.7% (46 is 51.1%, 47 is 52.2%). |
Look at the two red rows. In row 3, the agent has the two records. But the prompt does not tell it to use the open questions to decide what to search. This agent got 74 and 48 correct answers.
The agent with no records (row 2) also got 74 and 48. Thus, the records alone gave zero more correct answers. When the authors put the search rules back, the agent got 6 more correct answers on V1 and 7 more on V2.
Row 4 is a setup that many teams use: a fact list plus a notes file. The agent can write any text in the notes file. On V2, this setup got the same score as no records. On V1, it got 3 fewer. One run is not sufficient to show if this difference is real.
Row 6 shows a second lesson. In this setup, the agent cannot correct or remove a fact. On V2, it got 46 correct answers, the lowest result. Thus, the ability to remove an old fact is as necessary as the ability to keep a fact.
All five ablations lost more on V2 than on V1. But one question on V2 is 1.1 points. Thus, the data cannot show which part is the most necessary.
| tag 05 |
Where the difference starts |
| active same facts found, different answers |
The authors used GPT-5.6-Luna to examine each failed run. It checked three stages:
Found: the agent found the necessary facts.
Kept: the agent kept the facts and used them.
Converted: the agent changed the facts into the exact answer.
The “kept” and “converted” rates use the runs that passed the stage before.
| exhibit 5 · pass rate at each stage (%), glm-5.3-flash |
| stage | v1 cc | v1 codex | v1 lh | v2 cc | v2 codex | v2 lh |
| found | 85.0 | 72.2 | 85.0 | 72.2 | 76.7 | 87.8 |
| kept | 85.8 | 83.3 | 87.6 | 84.6 | 78.3 | 87.3 |
| converted | 66.0 | 77.5 | 80.8 | 58.2 | 70.4 | 79.7 |
| median steps, v1 → v2 · longharness 22 → 30 · codex 19 → 43 · claude code 21 → 116 |
| Source: Table 3 and Section 3.4. cc = Claude Code, lh = LongHarness. One run for each version. An AI model did the grading. |
On V1, Claude Code and LongHarness found the necessary facts at the same rate: 85.0%. The difference starts at the last stage: 80.8% against 66.0%. Thus, Claude Code had the correct facts, but it gave incorrect answers more frequently. This is evidence-state drift.
The number of steps shows the search failure. From V1 to V2, the median steps for Claude Code increased from 21 to 116. Codex used all 200 steps on 21.1% of V2 questions (19 of 90). LongHarness increased from 22 steps to 30.
| tag 06 |
Limits of the results |
| superseded claims that need a limit |
| exhibit 6 · limits, checked against the paper and npm |
| models | Claude Code and Codex used GLM-5.3-Flash and Qwen3.8-Flash. They did not use Claude or GPT models. Thus, “better than Claude Code” means: better than the Claude Code harness with a model that it was not tuned for. |
| versions | The paper used Claude Code 2.1.216 (npm, July 20) and Codex 0.135.0 (May 28). Before the paper date (October 9), npm had 69 newer Claude Code versions and 53 newer Codex versions. |
| step limits | LongHarness: 200 steps, 60 searches, 1M read tokens, 16,384 tokens for each read. Claude Code and Codex: 200 turns and 200 tool calls. RLM: 30 iterations. |
| second model | With Qwen3.8-Flash, LongHarness is better than Codex on V2 by 51.1 to 50.0. That is 1 question. With a 5% tolerance, it is better than Claude Code on V1 by 3.0 points (4 questions). |
| three runs | Appendix D gives three runs for each harness. The means agree with the main claim. V1: 59.9 against 49.4 for Claude Code (10.5 points, not 12.0). V2: 59.6 against 41.1 for Codex (18.5 points). Claude Code changed from 27.8 to 35.6 between its V2 runs. |
| graders | On AA-LCR, GPT-5.6-Luna graded the harness runs. The scores for models with no harness come from Artificial Analysis. Thus, the two parts of that column have different graders. |
| code | Version 1 of the paper has no link to the code. |
These limits do not change the main result. In Table 1, LongHarness got the highest score of the four harnesses on all four benchmarks, with the two models. It also had the lowest standard deviation on V1 (0.43 points).
The limits change two things: the size of the difference with the second model, and the meaning of “better than Claude Code.”
| tag 07 |
Four rules for your agent |
| active each rule comes from a result above |
| e1 · use typed records, not a notes file. Keep the facts (each with a source ID) separate from your conclusions. Put the conclusions in three lists: established, hypotheses, open questions. The notes-file setup got the same V2 score as no records. |
| e2 · use the open questions to control each search. The records gave zero more correct answers until the prompt told the agent to use them. In LongHarness, the agent searches for the open question that can change the answer most. It stops when no open question can change the answer. |
| e3 · do not edit a fact. Give each fact a permanent ID. To correct a fact, add a new fact and mark the old fact as superseded. The final answer must cite active facts and no other facts. The setup that cannot correct facts got the lowest V2 score. |
| e4 · grade the exact answer. Do calculations in a tool that must cite fact IDs. Use exact match to grade. With a 5% tolerance, the incorrect Codex answer gets a pass. |
Codex had the correct number. It did not have a way to mark the incorrect number as old.
|
