occurrence: thinking switch set to off · reasoning passed the signal

GPT-5.4 Obeyed “No Reasoning” and Lost 28.5 Points

Thinking off still reasons in 99.69 percent of open answers. The fix is the question.

Six models answered 572,688 prompts across six settings. With thinking switched off, 99.69 percent of their open-ended answers still carried a visible reasoning step before the final answer. Then the authors added the sentence most of us have typed into a system prompt, “Give the final answer directly without any reasoning,” and the models that obeyed it paid in accuracy. GPT-5.4 returned a bare answer on 100 percent of GSM8K and dropped 22.7 points, then on 99.8 percent of MATH and dropped 28.5.

A bare answer, in this issue, means the model wrote the final answer and nothing else. The paper calls it the Empty-Thinking Rate. The register below records what happened at each signal.

aspect: steel · the paper, the code, the trajectories

A NeurIPS poster that measures the setting everyone flips and nobody checks

Thinking Inertia: LLMs Keep Thinking When Told Not To is by Dianqiao Lei (Tsinghua), Kevin Qinghong Lin and Philip Torr (Oxford), and Pan Lu and James Zou (Stanford). It was posted on October 8 2026, and the evaluation code (MIT) lists it as a NeurIPS 2026 poster. The full response set, with the pre-answer text and final answer split per record, is on Hugging Face.

The study runs six models (Qwen3-4B, Qwen3-32B, Qwen3-235B-A22B, DeepSeek-V4-Flash, GPT-5.4, Gemini 3 Flash) through six prompt modes on six benchmarks grouped into three answer spaces: yes/no (BoolQ, StrategyQA), multiple choice (MMLU, MMLU-Pro) and open-ended (GSM8K, MATH). Temperature 0, an 8,192-token cap for every mode, and 1,000 sampled items per benchmark for the API models.

aspect: steel · how no-thinking gets measured

Three proxies that every eval uses, and a three-level check that replaces them

Nobody had a definition of a no-thinking response. In practice it is inferred from one of three proxies: the thinking mode is off, the block is absent, or the output is short. Each fails in the same way. A model with its reasoning channel disabled can put the derivation in the answer channel instead, and a 20-word reply can still contain the one equation that solved the problem.

The authors split every response into pre-answer text and a final answer, then grade the pre-answer text at three levels. Level one is the bare-answer rate: the whole output is the answer and nothing else. Level two is relevance: cosine similarity between the question and the pre-answer text using Qwen3-Embedding-4B under an instruction asking for reasoning-relevant text. Level three is the explicit-inference rate: a blinded GPT-5.5 judge sees the question and the pre-answer text, nothing else, and labels it. A restatement or a relevant fact does not count. One deduction, computation, comparison or elimination step does, even if it is wrong.

Table 2 of the paper. Think-off is the provider switch (enable_thinking=false, reasoning effort none, DeepSeek thinking disabled). Gemini has no off, so its minimal setting is used and treated as near-off.

aspect: red · signal passed on open-ended tasks, every model

With thinking off, 99.69 percent of open-ended answers still reason out loud

The first result is the one in the title of the paper. Under the native think-off setting, the share of responses with a visible inference step is 5.15 percent on yes/no questions, 45.57 percent on multiple choice and 99.69 percent on open-ended math. The strict instruction cuts each number, but the staircase survives it: 0.12, 25.39 and 62.00 percent.

Explicit-inference rate over all outputs, GPT-5.5 judge, Table D.3. A second judge from another family (Claude Opus 4.8) reads 4.24 / 40.73 / 98.55 under think-off and 0.10 / 24.21 / 61.02 under strict.

The judge is not the weak link. Five human annotators labelled 600 sampled outputs blind and agreed with the GPT-5.5 label on 98.8 percent of them (96.7 percent with Opus 4.8). The human rates for the six think-off and strict cells were 4.0, 30.4 and 100.0 percent, then 0.0, 11.8 and 57.1 percent: the same staircase, the same drop under strict.

Per model, the switch does what it says on yes/no questions and stops doing it as the answer space opens. GPT-5.4 at reasoning effort none is the most obedient model in the set and still shows a visible inference step in 99.5 percent of open-ended answers.

Figure H.1 of the paper. Bare answer = Empty-Thinking Rate; reasoning = explicit-inference rate. DeepSeek-V4-Flash is 284B parameters; the two API models publish no size.

What the register shows is relocation, not removal. One Qwen3-4B record on MATH makes it concrete: thinking off, correct answer, and 84 words of working in the answer field that rewrite 4 as 2², set 2⁸ = 2²ˣ, and solve x = 4. You turned off the channel and paid for the tokens anyway.

aspect: amber · the models that stopped are the models that paid

Strict costs about 15 points on open-ended math, and the obedient models carry all of it

Across the five models with a true off switch, moving from think-off to the strict instruction raises the bare-answer rate on open-ended math from near zero to roughly 40 percent and lowers mean accuracy by about 15 points. The mean hides the shape. Read the per-model rows and the pattern is a straight trade: the higher a model’s compliance, the larger its loss.

Figure G.2 of the paper. Bare = share of strict-mode responses with no pre-answer text; accuracy = change in points from think-off to strict on the same items.

Qwen3-4B and Qwen3-235B-A22B essentially ignored the instruction on open-ended math and kept their scores. Qwen3-32B obeyed on 93.2 percent of GSM8K and fell from 93.4 to 44.4. GPT-5.4 obeyed on everything and lost 22.7 and 28.5. On yes/no questions the same instruction has no measured cost: accuracy stays near 85 percent across all four no-think modes, and on multiple choice the moves are modest.

The failure mode is specific. On one MATH item, Qwen3-4B under think-off wrote 61 words and answered −9, which is correct. Under strict it answered 9. The working it skipped was the part that checked the word “smallest.”

Difficulty is not the driver either. Holding the interface fixed at four-choice MMLU, STEM has the highest accuracy for all four models tested and the lowest bare-answer rate (0.3 to 36.4 percent), with explicit inference in 59.7 to 85.5 percent of responses. Humanities and social sciences compress more easily; a STEM question does not, however well the model scores on it.

aspect: red · forcing the template brings the reasoning back

“The answer is \boxed{” is not a stronger strict. It behaves like think-off.

If prefilling the answer template were a harder constraint than the strict sentence, the bare-answer rate would rise and the inference rate would fall. The opposite happens. Mean bare-answer rate drops from strict to prefix on multiple choice and open-ended tasks, and relevance and inference rebound in every answer family. The authors file prefix forcing as a separate branch, not a stronger suppression.

Table H.1 of the paper. The strict check scans the complete output, so working written after the box also fails the bare-answer test.

aspect: green · same questions, bare answers at the same score

Rewrite the interface and the same model stops reasoning without losing a point

The cleanest experiment in the paper keeps the questions fixed and changes how the answer is requested. Qwen3-32B, thinking off, on the numeric-answer items of MMLU and MMLU-Pro. As multiple choice, the model gave a bare answer 10.5 percent of the time at 77.8 accuracy. Rewritten as verification (here is a proposed value, yes or no), bare answers rose to 62.5 percent and accuracy held at 78.9. Rewritten as open generation, bare answers fell to zero and accuracy fell to 52.6.

Figure 6 of the paper. Verification is not the model agreeing with every proposal: in the broader MMLU verification rewrite (Table I.4), accuracy on true proposals was 81.1 and on false ones 81.2.

Shuffling the options or swapping letter labels for numbers moves the bare-answer rate by at most 8 points, so the effect is the answer space, not the surface form. The register reads the same way at every level: a yes/no check gives the model a candidate to confirm, multiple choice gives it candidates to compare, open generation gives it nothing, and the amount of visible reasoning tracks how much of the answer is already on the table.

aspect: steel · scope lines from the paper itself

Visible text, math benchmarks, one judge family, one anomaly

Visible, not latent. Every claim is about text you can read. The authors say so in their own discussion: visible compliance should not be equated with lower inference cost when concealed reasoning may remain active. Whether the computation disappears, moves to a channel you cannot see, or is simply left out of the output is handed to future internal-state work.

Open-ended means math. GSM8K and MATH have a checkable number at the end. Code, tool use, multi-turn, multilingual and sampled decoding are listed by the authors as untested.

Gemini. Its minimal setting is near-off, not off. Under strict it produced zero bare answers on every benchmark and its accuracy went up: plus 7.2 on MMLU, plus 22.0 on MMLU-Pro, plus 11.0 on GSM8K, plus 12.6 on MATH. The paper excludes it from the five-model trade-off mean and reports the numbers without discussing them.

The judge is a model. GPT-5.5 labels the pre-answer text. It was cross-checked with Opus 4.8 on all 572,645 usable outputs (97.3 percent binary agreement) and with 600 human-labelled outputs, but a rubric is still a rubric. The paper also cites Turpin et al. and Lanham on the point that a visible chain need not be the computation that produced the answer.

Provider switches are opaque. Reasoning effort none and thinking disabled are the settings the provider makes available. The paper records the outputs and makes no claim about what either vendor does behind the setting. Related work on transferring encrypted reasoning blocks is a reminder that the channel still exists even when you cannot read it.

aspect: green · what to change in a pipeline that uses think-off

Five changes, each tied to one number above

Treat thinking off as routing, not a budget. On open-ended work, measure output tokens with the switch on and off before you book the reduction. The 99.69 percent figure says the derivation moves into the answer field.

Do not send “no reasoning” to open generation without an accuracy A/B. The models that comply are the ones that lose 17 to 49 points. On yes/no and multiple-choice classification the cost is small: the yes/no mean stayed near 85 across all four no-think modes.

Drop answer prefill as a suppression trick. It returned Qwen3-32B to 0.0 percent bare answers on GSM8K, the same as no instruction at all.

Give the model a candidate. Anywhere the pipeline already holds a value (an upstream answer to verify, a label to confirm, a cached result to re-check), ask for verification. Same items, bare answers from 10.5 to 62.5 percent, accuracy 77.8 to 78.9.

Log the pre-answer text. The thinking flag is not evidence of anything. The trajectories dataset has a usable schema: input prompt, pre-answer text, final answer, is_answer_only, is_correct, mode. Record those fields on your own traffic and you can reproduce the staircase on your prompts in an afternoon.

The authors stop where the measurement stops, at the text. The switch labelled “thinking off” turned out to be a statement about what you get to read, and the sentence “answer without reasoning” turned out to be a trade you make blind. The dial that moved the number with no bill attached was the shape of the question, which was on your side of the API the whole time.