The Best Skill Never Read a Reasoning Trace

Reasoning modes work, and they bill you for it on every single episode. In Microsoft's measurements, flipping GPT-5.4-mini into reasoning mode multiplies per-episode output tokens 3.0 to 5.1 times across four agentic benchmarks, and up to 6.2 times for Qwen3.6-27B. That premium never amortizes on its own, because reasoning tokens are generated fresh each run.

Read the traces and you can see where the money goes. Most of the deliberation is not instance-specific problem solving. It is the same domain procedure, re-derived again and again: the retail agent reasoning its way, once more, to "do not call the lookup tool until the customer actually gives an email."

A new Microsoft paper, Reason Wide, Not Deep (Singh, Gautam, Gupta, Mehrotra, Bakshi and Gulwani, Aug 8), asks the obvious follow-up: how much of that recurring bill can you pay once, offline. Their answer is a pipeline they call passive skill distillation, and the distiller is a tool you already have installed.

The recipe is three steps, and none of them touches weights, collects new rollouts, or runs per-instance search at deployment.

The rules the agent writes are not vibes. They are imperative, concrete, and cite corpus statistics the distiller computed itself. From the retail skill: never call an authentication tool with a guessed argument, because this exact bug appeared in 13 of 22 rollouts and produced 17 of the 18 observed tool errors.

From the ALFWorld skill: adjectives in the task are actions, not descriptions. "Cool tomato" means you must issue an explicit cool command with the fridge, not just relocate a tomato and hope. From the spreadsheet skill: finish inside the workbook, because explaining the right formula in chat and never entering it is the single most common way the task dies.

Here is the central table for GPT-5.4-mini, on held-out tasks the distiller never saw, averaged over three seeds. Watch the token column as closely as the score column.

Across all four benchmarks the skill recovers 55 percent to more than 100 percent of the reasoning gap, while emitting 2.9 to 4.5 times fewer output tokens and zero reasoning tokens. On ALFWorld and the spreadsheet benchmark, the skilled model even spends fewer tokens than the plain no-think baseline, because fewer flailing retries means shorter episodes: 21.8 turns per ALFWorld episode instead of 27.0.

Beating the teacher sounds paradoxical until you look at what a skill is. A rule aggregated across 50 training episodes is more reliable than a derivation the reasoning model must reproduce correctly every single time. The paper notes the reasoning model itself occasionally falls into the ALFWorld appliance-door loop that the skill forbids outright.

The Qwen3.6-27B columns are their own small story. Qwen's thinking mode is unreliable on these tasks and actually hurts on ALFWorld and SSB-Verified, yet a Qwen-specific skill pushes ALFWorld to 0.980, near ceiling, and matches thinking mode on telecom at 5.9 times fewer output tokens. The one regression is retail, minus 4.2 points, where the base model was already competent and added rules appear to over-constrain.

The obvious assumption is that you distill skills from reasoning traces. The model thought its way to the answer, so the thoughts must be the valuable part. Microsoft ablated exactly this: one skill distilled purely from non-reasoning rollouts, one from a paired corpus that adds think-mode trajectories on the same tasks.

The two sources are statistically close on ALFWorld and telecom, and reasoning traces do help on retail, where think-mode demonstrations model the authentication discipline that no-think rollouts keep violating. But the spreadsheet reversal is sharp, and the authors' hypothesis is the sentence worth keeping. Reasoning traces are verbose, stylized, and describe what the model believed. Environment feedback in failed trajectories records what was true.

On a workbook task, anchoring the distiller on the model's narrative instead of workbook-level evidence produces a worse rulebook. The practical consequence is bigger than the mechanism: the whole loop, deploy the fast model, collect logs, distill, redeploy, can run without ever invoking a reasoning model. The low-cost trajectories are sufficient fuel.

If this sounds like prompt optimization, that comparison is in the paper too. They ran GEPA, a state-of-the-art reflective prompt evolver, on both tau2 domains with a 120-metric-call budget and the same Sonnet 5 doing reflection.

The amortization math is short. On telecom the skill cuts 1,546 output tokens per episode against think mode, which is essentially the model's entire 1,572-token reasoning budget, while the skill itself sits in a cacheable input prefix. One coding-agent pass, then every episode is on the right side of the ledger. An active optimizer has to burn an evaluation budget before any of that starts, and cannot run at all where fresh rollouts are unavailable.

The paper's own framing tells you where this stops working. Reasoning is deep search inside one episode. Distillation is wide search across many episodes. They acquire the same commodity, procedural knowledge, with different cost structures, and width dominates strictly when the knowledge is episode-invariant.

Telecom and spreadsheets are where the residual gap survives, 0.450 versus 0.333 and 0.613 versus 0.560, and both for the same reason: instance-specific structure. Which line, which plan, what the user just toggled, what one-off logic this particular workbook encodes. No fixed prompt anticipates that, and per-instance deliberation is doing irreplaceable work there. The honest conclusion is a hybrid: a skill so you stop re-deriving the invariants, reasoning reserved for instances that need it.

Three more caveats before you quote this at your infra review. Each skill was distilled exactly once per model-domain pair, so distillation variance is unmeasured, and the Qwen retail regression suggests the process is not uniformly reliable. Skills are model-specific and cross-model transfer is untested. And the whole study covers two models and four benchmark domains, all with clean reward signals to learn from.

Anthropic has been shipping the "skills as markdown in the prompt" idea as a product for a while. What this paper adds is the measurement nobody had put in one table: skills framed as a substitute for a reasoning mode, scored by how much of the think gap they recover per token. That framing turns a prompt-engineering habit into an economic decision you can defend with numbers.

The 48-hour version for anyone running agents in production: you already store the trajectories. Pick one domain, take 35 to 50 logged episodes with their outcomes, open Claude Code in that directory, and ask it to contrast failures against successes and compile the recurring failure modes into explicit rules. Inject the output as a cached system-prompt prefix and A/B it against your current setup. The paper's entire evidence base was produced by that loop for about two dollars a domain, and the appendix shows exactly what the rules look like.

Reasoning is deep search paid again on every episode. A distilled skill is wide search paid once. On two of four benchmarks, the two-dollar markdown file beat the reasoning mode it was supposed to imitate.

tomato, cooled. The paper's favorite trajectory: without the skill, the agent puts an uncooled tomato in the microwave, then issues the look command twenty times in a row until the 40-step budget dies. With the skill it cools the tomato at step 19 and finishes at step 30. Missed-transform failures drop from 35.9 to 11.5 percent of transform tasks, stall loops from 28.7 to 5.3 percent.

elicitation, prompt-side. There is evidence that RL-trained reasoning mostly elicits capabilities already latent in the base model. This paper is the prompt-side counterpart: if the priors are already in the weights, a distilled description of where search reliably lands is a sufficient, and far less costly, elicitor.

lineage. The extract-once-reuse-forever idea runs through Voyager and Agent Workflow Memory. The new move is the yardstick: measure the skill against the reasoning mode's bill, and have an external coding agent write it rather than the acting model itself.

benchmark note. The four testbeds are tau2-bench telecom and retail, ALFWorld, and a verified subset of SpreadsheetBench run against live workbooks. Held-out splits throughout; the distiller never reads a test task.