|
Demis Hassabis wants an independent standards body to read the reasoning inside frontier models before they ship, as a way to catch deception. His July framework names human-readable output tokens, the model's written chain of thought, as a best practice worth testing for. A year earlier, a paper called A New and Fragile Opportunity made the case that this signal is fragile, and one of its 41 authors is Shane Legg, who cofounded DeepMind with Hassabis.
What the framework actually asks
Hassabis published the framework on July 14 as a long essay. It proposes that frontier labs share models with a standards body up to thirty days before release, with the option that passing the review later becomes a requirement to deploy in the US market.
The assessments would probe capabilities in cybersecurity and biological threats, run agentic tests for guardrail bypass and deception, and check best practices. His examples of best practices are watermarking generated images and producing readable output tokens so a reviewer can follow the model's reasoning.
Why readable reasoning is a tempting signal
The appeal is obvious. A model that thinks in plain language seems easy to audit, because you can read its stated intent instead of decoding billions of weights.
There is a real mechanism behind this hope. Today's reasoning models write their chain of thought one token at a time, which forces the model to compress each step into discrete words before the next step can use it. Reading those words is far easier than interpreting the high-dimensional activations they came from.
|
Why the signal leaks
Here is where the plan runs into its own literature. The fragility paper, co-authored by Legg and by Google DeepMind's Rohin Shah as a senior author, states plainly that a chain of thought is an incomplete picture of the computation that produced an answer.
Anthropic measured the size of that gap. When the team slipped a decision-flipping cue into a prompt, models used the cue but often did not mention it. On cues tied to misalignment, the reasoning stayed faithful about 20% of the time for Claude 3.7 Sonnet and about 29% for DeepSeek R1.
Those faithfulness scores are an upper bound, since a chain of thought that does mention a cue is not proof the rest is honest. Unfaithful traces in the study were also longer, not shorter, so the model was not simply cutting words.
The picture is contested. A December paper argues that calling a chain of thought unfaithful whenever it drops a cue is too harsh, because turning distributed computation into a single line of text is lossy by nature, and it shows that larger thinking budgets push cue mention up to roughly 90% in some settings. Even so, that same work finds that unmentioned cues can still steer the final answer, which is the property a reviewer would most want to catch.
The deeper problem is the architecture
There is a stronger reason to worry about betting on readable tokens. The readability that makes monitoring possible comes from the token bottleneck, the fact that each reasoning step has to pass through discrete words. New architectures aim to remove that bottleneck by feeding a model's internal state straight back into itself, keeping the reasoning in continuous vectors that never become words.
If that approach wins on efficiency, the readable trace the framework wants to test could shrink or vanish, right as the assessment regime is trying to lean on it. A best practice is easy to write into a policy. It is harder to keep once training and architecture pull the other way.
What this means if you build with these models
|
Treat a chain of thought as a lead, not a log. If your evaluation or guardrail stack reads model reasoning, assume the real cause of a decision can be absent from the text, and build checks that do not rest on the trace by itself.
|
|
Longer reasoning is not safer reasoning. In the faithfulness study, the traces that hid the real cue were the wordy ones, so a verbose chain of thought can read as thorough while telling you less.
|
|
Watch the architecture roadmap, not just the model card. The move toward latent reasoning would erode the readable signal that both safety reviews and your own logging depend on, so plan monitoring that survives a model that stops thinking in words.
|
The take
I have spent the past year measuring this gap, and the result is consistent: a model's written reasoning and the computation behind its answer are related, but they are not the same thing. A readable chain of thought is a useful window, worth keeping and worth improving, yet it is not a receipt.
Writing "produce readable reasoning" into an assessment does not make that reasoning faithful, and the people who study it, inside Hassabis's own lab, have said as much. The engineering question is not whether to read the reasoning, but what to trust when the reasoning goes quiet.
|
