RESEARCHAUDIO · PAPER NOTES

Why AI agents keep relearning the same lessons

A study turns software knowledge into tested instructions for research agents. The gains are substantial. Understanding where they come from takes a closer look.

Imagine watching an agent spend an afternoon on a machine-learning experiment. It finds a plausible method, writes the training code, and starts running it. Then the ordinary problems arrive: an unexpected data format, a package option that behaves differently than assumed, a validation setup that needs fixing.

Each mistake is recoverable. Together, they can consume the experiment.

That is the problem behind Repo-To-Skill: Distilling GitHub Repositories Into AI4AI Skills, a September 2 preprint by Jianlyu Chen and colleagues. The authors ask whether research agents could do better if the practical knowledge needed to use a method arrived in a form they could immediately work with.

Their system, DisCo, builds and verifies skills from software repositories, papers, and task exploration. A research agent can then consult those skills while working. The model itself does not need to be retrained.

The knowledge between an idea and a working run

The authors call this operational knowledge: how to choose an approach, configure it, check its outputs, and recover when something goes wrong.

Think about the difference between reading an API reference and sitting beside someone who has used the package for six months. Both can tell you what a function accepts. The experienced person can also tell you which assumption will quietly invalidate your result.

DisCo tries to package that practical guidance. Its repository skills have an entry document, SKILL.md, with instructions and applicability conditions. Supporting references hold deeper detail, and scripts can handle repeatable operations. The agent opens additional material as needed.

This selective reading matters. A large collection is useful only if the agent can find the right procedure without filling its context with unrelated instructions.

The public AREX-Skill collection contains more than 5,000 skills from 1,000 ML repositories, organized into 20 areas and 178 capability families. DisCo scopes a source, gathers evidence, constructs skills, and verifies them before admission. Verification includes practical checks and records unresolved gaps. It should not be read as a guarantee that every workflow works in every environment. Sections 3 and 4 describe the design.

What improved

The main evaluations compare Codex using GPT-5.5 at the same reasoning setting, with and without distilled skills. The downstream execution budgets are matched.

Source: paper, Tables 1–4. Each row uses a different metric. MLE-bench results average three runs across 75 competitions.

The MLE-bench result is the headline: a 41.78 percentage-point increase in medal rate, or a 134.3% relative improvement. Those are two descriptions of the same change. It is not a 134-point gain.

On PaperBench, skills improved 18 of 20 paper-reproduction tasks. The resulting average score, 39.59%, also shows how much work remains. Better replication is still a long way from consistently complete replication.

The preparation is part of the result

There is an easy misreading here: give an agent the public repository library and expect its medal rate to more than double.

The experiments used additional skill construction tailored to the evaluations. MLE-bench received a skill graph prepared for each competition. PaperBench used skills distilled from related papers and their repositories, with the target paper and its released artifacts excluded as skill sources. FrontierCS and PassNet each used a shared graph developed for that benchmark.

For MLE-bench, preparation allowed up to 24 GPU-hours per task, followed by a separate running allowance of up to 24 GPU-hours per task. Preparation included diagnostic experiments and refinement. The final skills contained descriptive guidance, not runnable training or inference scripts. Appendix A.2.1 explains the separation.

That makes the result evidence that prior preparation can improve a subsequent run under a fixed execution budget. It does not establish an equal-total-compute advantage over an agent allowed to spend that additional preparation budget during its own attempt.

Even matching time limits does not mean matching actual usage. In FrontierCS, average tokens per task increased from 2.46 million to 4.47 million with skills. The score improved, but the agent also did more work.

My reading is that the economic case depends on reuse. Preparing guidance once can be worthwhile when many later tasks benefit. For a one-off problem, construction belongs in the cost calculation. The paper's runtime gains alone cannot settle that calculation.

When good advice becomes a bad rule

A PassNet example in the appendix is especially revealing. A skill warned against replacing heavily optimized vendor operations with custom kernels. That is a sensible default: rewriting a fast library operation is often wasted effort.

But the warning was too absolute. In a special convolution case, the agent recognized a possible simplification and then abandoned it because the skill discouraged the rewrite. The authors revised the guidance to allow exceptions supported by the graph structure and evaluation evidence. Appendix A.2.4 gives the example.

This is the part I would keep in mind when building a skill system. Instructions influence which possibilities an agent explores. A useful shortcut can become a reason to stop thinking.

Testing whether instructions are readable and executable is only the beginning. You also need to see what they cause the agent to rule out.

What I would test next

For an engineering team, I would start with one recurring failure: a broken evaluation setup, an incorrect data transformation, or an expensive experiment launched before a basic check.

Write a small procedure with a clear trigger, verified steps, a success check, and conditions under which the agent should disregard the default advice. Then compare it against the existing workflow on new tasks. Track completion, failures, tokens, and the cost of maintaining the procedure.

That is a narrower experiment than importing thousands of skills, but it answers a useful question: does this particular piece of experience prevent this particular class of mistake?

Repo-To-Skill makes a credible case for treating practical knowledge as something an agent system can build, test, and reuse. The unanswered question is how well that knowledge travels to unfamiliar tasks and changing software. An agent that remembers yesterday's fix is useful. An agent that recognizes when yesterday's fix no longer applies is more useful still.

Research note based on Chen et al., Repo-To-Skill: Distilling GitHub Repositories Into AI4AI Skills, arXiv v1, September 2, 2026. Results are reported by the authors and have not been independently reproduced for this article. Author-linked code and skill collection.