Stanford’s new verifier hits 86.5%. The model it verifies hits 98.9% on the same tasks.

A new verification paper from Stanford, UC Berkeley, and NVIDIA Research claims 86.5% on Terminal-Bench V2, 78.2% on SWE-Bench Verified, and 87.4% on RoboRewardBench. The headline numbers look like a clean win for the “verifier” paradigm.

They are not.

The same paper reports an oracle pass rate of 98.9% on Terminal-Bench V2. That is the upper bound you get when you pool trajectories across the full leaderboard and a perfect oracle always picks the right one. The verifier is 12.4 percentage points below that ceiling. The model can solve 98.9% of tasks. The verifier picks wrong 12.4% of the time. The verifier is the bottleneck, not the answer.

And there is more under the surface.

What the paper says

LLM-as-a-Verifier, by Jacky Kwok and co-authors at Stanford, UC Berkeley, and NVIDIA Research, is a probabilistic verification framework that re-implements a basic thing: scoring two solutions against each other.

Standard LM judges prompt the model to output a single discrete score token, like “5 out of 5” or “4 out of 5.” The collapse from a full probability distribution to a single integer is where everything goes wrong. Two different solutions that should score 4.7 and 5.3 both get rounded to 5. The judge ties itself. On Terminal-Bench V2, the discrete judge produces ties in 26.7% of comparisons at the lowest budget.

The fix is to take the expectation over the full scoring-token distribution, not the argmax. Instead of “pick the highest-probability score,” the verifier reads all 20 score tokens, weights each by its probability, and outputs a continuous number. The paper calls this a verifier, not a judge, and shows that fine-grained feedback unlocks three scaling axes:

1. Score granularity, 1 to 20 score tokens. Pairwise accuracy on Terminal-Bench goes from 73.1% at G=1 to 77.5% at G=20.
2. Repeated evaluations, 1 to 16 independent runs. 74.7% at K=1 to 77.5% at K=16.
3. Criteria decomposition, 1 to 3 sub-criteria (Specification, Output, Errors for code trajectories). 75.2% to 76.4% for any single criterion, 78.3% for the ensemble.

The framework ships as a Claude Code extension and a Codex extension, with the same verification protocol across all four benchmarks: Terminal-Bench V2 (86.5%), SWE-Bench Verified (78.2%), RoboRewardBench (87.4% Trajectory Preference Accuracy, value-order correlation 0.966), and MedAgentBench (73.3%). State of the art on all four. The cost-efficient Probabilistic Pivot Tournament (PPT) cuts the verification budget from O(N²) to O(Nk) pairwise comparisons, with k=9 reaching 67.13% on a 20-candidate pool, approaching the 67.42% full round-robin ceiling at 27% fewer comparisons.

What the numbers actually show

The three scaling axes and the oracle gap, side by side:

The signal-to-noise ratio of the granularity scaling is 0.775 at G=1, 0.799 at G=20. A 3% relative improvement on a signal-quality metric, not the 30% the framing might suggest. The bigger wins are from the K and C axes, where the paper is less emphatic.

The judges versus verifiers contrast is sharper. On a representative task (query-optimize, Claude Opus 4.5 generator, Gemini 2.5 Flash verifier), over 100 repeated evaluations of one trajectory pair:

• Discrete 1 to 5 judge: 12 out of 100 correct, 88 out of 100 ties, 0 out of 100 wrong.
• Continuous G=5 verifier: 69 out of 100 correct, 0 ties, 31 out of 100 wrong.
• Continuous G=20 verifier: 77 out of 100 correct, 0 ties, 23 out of 100 wrong.

The verifier eliminates ties, and the G=20 verifier ranks the correct trajectory higher in 77% of runs. The judge is right in 12%. The 65 percentage point gap is the headline. The remaining 23 percentage points where the verifier is still wrong is the contrarian story.

How it actually works

In plain English: the verifier is the same LLM the paper criticizes, but with a better question. Instead of “output one number,” the model is asked to put a probability on each of 20 score tokens, and the verifier averages across the whole distribution. The fine-grained probability is the only new thing. The model is unchanged.

The method, in three steps:

1. Construct a scoring prompt. A task, two trajectories (A and B), and a 1 to 20 rubric. The model is asked to fill in and tags with letter tokens (a, b, c, ..., t) so logprobs can be extracted for all 20 candidates, not just the top 1.
2. Compute the expected score. For each trajectory, take the expectation over the 20 score tokens weighted by their probabilities. The reward is in [0, 1].
3. Compare with Bradley-Terry. Convert the two continuous rewards into a pairwise preference probability using the Bradley-Terry model, P(τ_i > τ_j | x) = 1 / (1 + exp(-(R_i - R_j))). Aggregate over C criteria and K repetitions.

The Probabilistic Pivot Tournament (PPT) is the cost-control piece. A random Hamiltonian cycle scores N adjacent pairs so every candidate gets one “A” position and one “B” position, which cancels the verifier’s positional bias. The top-k candidates by ring-pass mean preference become the pivots. Every non-pivot is compared against the pivot set, and every pivot is compared against every other pivot. Budget drops from O(N²) to O(Nk). For 20 candidates and k=9, that is 9,630 pairwise verifications instead of 13,111, and accuracy is 67.13% instead of 67.42%, a 0.3 percentage point trade for 27% fewer comparisons.

The Claude Code and Codex extensions are the same verifier, packaged for production use. The paper claims this is the first general-purpose framework that works across coding, robotics, and medical domains, with no per-domain fine-tuning. The fine-grained continuous scores also serve as a proxy for task progress, and as a dense reward signal for reinforcement learning, with 1.8x sample efficiency on the LIBERO robotics benchmark (DSRL-SAC fine-tuning) and 1.1x on MATH reasoning (Qwen3-8B GRPO fine-tuning).

Where it works, and where it collapses

It works. The numbers are real, the SOTA claims are real, the relative gains over the V1 baseline are real. The framework generalizes across harnesses (Terminus-Kira and Terminus-2, with gains of 4 to 8 percentage points over the strongest baselines). The query-optimize case study is concrete: the verifier catches a failing trajectory that the discrete judge ties 88 times in a row. The fine-grained feedback for RL is a real contribution to the field.

It collapses, quietly.

• The verifier is gated on Gemini 2.5 Flash for the head experiments because the framework needs token-level logprobs. Frontier models like GPT-5.5, Claude Opus 4.7, and Mythos do not expose logits. The “general-purpose” claim is a marketing claim, the technical claim is “this works for one model family that exposes logprobs.” The paper has a two-stage workaround in Appendix B.6 for closed models, with quality degradation that is not quantified.
• The cost story is relative, not absolute. PPT saves 27% of comparisons versus the V1 baseline, but the paper does not report the per-trajectory inference cost, the per-verification cost, or the dollar cost per correct solution. For a Claude Code extension, the per-accepted-PR cost is the metric a developer cares about. The paper does not measure it.
• The 12.4 percentage point gap to the 98.9% oracle ceiling is the structural problem. Closing it requires a better verifier, not a bigger pool of trajectories. The paper’s framing (“verification is a new scaling axis”) makes the verifier the answer. The numbers make it the bottleneck.
• The signal-to-noise ratio improvement from granularity scaling is 3% relative (0.775 to 0.799). The pairwise accuracy gain is 4.4 percentage points (73.1% to 77.5%). The bigger wins are from the K and C axes, which the paper does not lead with. The granularity story is the visual hook, but the engineering effort is in the K and C axes.

What the framing skips

The headline framing positions the verifier as a different kind of artifact than the judge. The technical content says they are the same kind of artifact, the same LLM, the same training data, the same failure modes, with a different prompt. Calling it “LLM-as-a-Verifier” rather than “LLM-as-a-Judge” is a reframing, not a refactoring.

The Claude Code and Codex extensions are the product this paper ships. The 86.5% Terminal-Bench and 78.2% SWE-Bench numbers are the marketing. The research contribution is the granularity ablation and the PPT algorithm, which are real. The business contribution is “you can use this verifier in your dev tool.” The Anthropic ecosystem is being built around monitoring and verification. This Stanford paper validates the demand.

The signal-to-noise ratio improvement is small, and the gap to the oracle ceiling is large. The paper reports both numbers, in the same tables, and frames the gap as “headroom for future work” in the limitations. The future work is not optional. It is the whole point.

What This Means for

The Metric That Actually Matters

LLM-as-a-Verifier scaling vs the 98.9% oracle ceiling on Terminal-Bench V2

  Oracle ceiling:                                              98.9%
  +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
  LLM-as-a-Verifier (best, all 3 axes combined):              86.5%   <- -12.4 pp
  ---------------------------------------------------------------------
  Criteria ensemble (C=3):                                    78.3%
  Score granularity G=20:                                     77.5%
  Repeated evaluations K=16:                                  77.5%
  ---------------------------------------------------------------------
  Score granularity G=1:                                     73.1%
  Discrete judge (1 to 5), K=1:                              71.8%
  +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
  Pass@1 (no verifier, single trajectory):               ~52-83%   <- model + harness
  Oracle Pass@K (full leaderboard pool):                    98.9%   <- if you could pick, you'd be near-perfect

  Bars (illustrative, ascii width 60):

  Oracle              |######################################################| 98.9
  LLM-as-a-Verifier   |#############################################        | 86.5
  Criteria ensemble   |##########################################          | 78.3
  Score G=20          |##########################################          | 77.5
  Repeated K=16       |##########################################          | 77.5
  Score G=1           |########################################            | 73.1
  Discrete judge K=1  |#######################################             | 71.8

The 12.4 percentage point gap to the oracle is the number. It is the headroom. It is the cost. The verifier scores 86.5% and the same paper’s trajectories can solve 98.9% of the same tasks.

Closing thought

A new verifier that scores 86.5% on the same task set where the model alone reaches 98.9% is a 12.4 percentage point reminder that verification is hard and the gap is the work. The granularity trick eliminates ties. The PPT algorithm cuts cost. The Claude Code extension ships. The 98.9% ceiling is the next 18 months of the field.

Reader Challenge

Three questions worth 15 minutes of your time:

1. Run the granularity experiment on your own eval. Take the expectation over your score tokens, not the argmax. What is your tie rate before, and what is it after?
2. The PPT algorithm assumes positional bias is symmetric. Test this on your verifier. Does the “A” position still have a measurable advantage on your data, or does the ring pass cancel it cleanly?
3. The paper gates on Gemini 2.5 Flash. What is your fallback verifier for production? Does the two-stage workaround (Appendix B.6) hold up at your quality bar, or do you need a different approach?

Next Issue Preview

The paper is a Stanford and Berkeley collaboration. The other Stanford team published “LLM-as-a-Judge” earlier in 2026 and we did not cover it. The next issue will look at the original LLM-as-a-Judge paper (Zheng et al., Stanford, NeurIPS 2023) and trace the lineage from judge to verifier to the J-space paper Anthropic shipped on July 6. Three papers, one arc, the cost of not reading the math.

-- researchaudio.io