| researchaudio.io |
no. 217 / 05 aug 2026 |
|
Qwen Scored 86.6. Vals Scored It 67.4.
Same Terminal-Bench, different timeouts. What that means for your own evals.
|
|
[ 01 ]
Two numbers, one model
Alibaba's launch table puts Qwen3.8-Max at 86.6 on Terminal-Bench 2.1, ahead of Claude Opus 4.8 and Claude Fable 5 at 84.6. Vals AI ran the same model on the same benchmark and measured 67.4. Neither figure is fabricated.
Vals published the reason alongside the score: Alibaba's reported Terminal-Bench results modify the benchmark timeouts, and Vals preserved the original ones. Nineteen points of difference came from a harness setting, not from the weights.
|
| Terminal-Bench 2.1 · same model, two harnesses |
Reported by Alibaba modified timeouts |
|
86.6 |
Measured by Vals AI original timeouts |
|
67.4 |
Qwen3.7-Max, Vals original timeouts |
|
61.0 |
|
| Spread attributable to harness config: |
19.2 pts |
Sources: Alibaba launch table via MarkTechPost; Vals AI eval notes, 3 Aug 2026.
|
|
|
[ 02 ]
What actually shipped on 3 August
Alibaba previewed the model on 19 July at the World AI Conference in Shanghai and moved it to general availability on 3 August under the API id qwen3.8-max. It is a sparse mixture-of-experts model: 2.4 trillion total parameters, of which a small fraction fires per token. It accepts text, image and video, and returns text.
One spec is missing. Alibaba has not published the activated-parameter count. The figure circulating as roughly 95B active, about 4 percent of total, comes from secondary summaries rather than an Alibaba spec sheet. Until it is confirmed, serving cost cannot be modeled from the outside.
|
| Panel A · envelope · Panel B · token price |
|
2.4T
total params
|
~95B
active, reported
|
991K
max input tokens
|
131K
max output tokens
|
|
|
Price per 1M tokens
| Fresh input |
|
2.00 |
| Output |
|
6.00 |
| Implicit cache read |
|
0.25 |
| Explicit cache read |
|
0.17 |
Explicit cache creation costs 2.50. Cached input reads land eight times below fresh input. Source: Qwen Cloud model page, via MarkTechPost.
|
|
|
Rate limits sit at 2M tokens per minute and 15K requests per minute. The maximum reasoning budget is 262K tokens, and the input ceiling drops to 983K when thinking is enabled. The endpoint is OpenAI- and DashScope-compatible, so adopting it is a base-URL and model-id change. Five built-in tools ship on the Responses API: code_interpreter, web_search, web_extractor, t2i_search and i2i_search.
One caveat on price. Several trackers describe the launch rates as introductory, with standard rates higher after August. Alibaba's launch materials list the figures above, so treat the introductory framing as reported rather than confirmed, and budget for a revision.
|
|
[ 03 ]
Why a timeout moves a score nineteen points
Terminal-Bench scores an agent on tasks it has to finish inside a real shell. Think of it as a timed practical exam where the candidate is allowed to check their own work, and the proctor decides how long the exam runs.
Each task is a loop: issue a command, read the output, notice the failure, correct, run again. A wall-clock cap truncates that loop wherever it happens to be. Models trained with reinforcement learning on verification behaviour spend a large share of their tokens re-running tests, re-reading diffs and re-checking builds, which means a generous cap converts near-misses into passes without changing a single weight. Cline reported roughly 20 percent gains on open-weight models from harness changes alone, which is the same lever pointed in the useful direction.
So a published agentic score is conditional on a configuration you almost certainly do not share. The vendor number describes the model paired with a generous harness. The independent number describes the model paired with the default one. Your number depends on yours.
|
|
Insight 01Put the harness config in the comparison table, next to the score. Timeout, step cap and retry policy are as load-bearing as the model id, and a leaderboard row without them is a number without units.
|
|
|
[ 04 ]
Reading the rest of the table
Alibaba published a broad benchmark table, including rows it loses. That is worth crediting. It also included two things that should slow down anyone quoting the generational delta.
|
| Where it leads, where it trails |
| Benchmark |
Qwen3.8-Max |
Claude Fable 5 |
| PaperBench |
93.0 |
88.8 |
| Terminal-Bench 2.1 |
86.6 |
84.6 |
| SWE-bench Pro |
67.7 |
80.0 |
| FrontierSWE |
73.5 |
88.8 |
| Generation over generation, coding rows |
| DeepSWE 1.1 |
21.6 → 56.6 |
| FrontierSWE |
40.7 → 73.5 |
| JobBench |
31.3 → 53.4 |
| GPQA Diamond |
92.4 → 92.6 |
|
|
Vendor-reported figures from Alibaba's launch table, compiled by MarkTechPost. Terminal-Bench row reflects modified timeouts.
|
|
|
The pattern is clear once the rows are sorted. Agentic and multimodal work moved sharply. Reasoning did not: GPQA Diamond went from 92.4 to 92.6 across a full generation. Long-horizon coding is where the money went.
Two caveats belong in any honest read. The multimodal comparison rows benchmark against Qwen3.7-Plus rather than Qwen3.7-Max, which flatters the generational delta against a weaker sibling. And Alibaba's own reinforcement-learning scaling curve peaks at 0.725 near 4,000 training environments, then falls back to 0.719 and 0.689. The vendor published its own diminishing return.
Independent placements land in a similar band. The model debuted at #4 on Frontend Code Arena with 1,668 Elo, behind Claude Opus 5 [Max] at 1,705 and Kimi K3 [Max] at 1,676, and effectively level with Claude Opus 5 [High] at 1,669. It took #2 on Vision Arena at 1,305, thirteen points behind Claude Fable 5 [High]. On the Vals Index it scored 66.1, second among open-weight models and tenth of 43 overall, matching Claude Opus 4.7 at about 2.3 times lower cost per test, 2.68 dollars against 6.17.
|
|
Insight 02Check the baseline before quoting a jump. A generational delta measured against the Plus tier of the previous generation is a different claim from one measured against the Max tier, and the row label rarely says which.
|
|
|
[ 05 ]
The word "open" is doing a lot of work
This is the first Max-class Qwen model slated for open weights, which is the genuinely notable part of the release. Alibaba said the checkpoint arrives the week after launch, alongside a much smaller Qwen3.8-27B. As of this writing, neither the weights nor the weight license has been published, so region and commercial terms are unknown.
Even with weights in hand, a 2.4T sparse model is a multi-node datacenter artifact, not something that lands on a workstation. For comparison, Kimi K3 at 2.8 trillion parameters needs more than a terabyte of memory to load weights alone. The 27B checkpoint is the one that fits ordinary on-premise hardware, and it may end up mattering more to most teams than the flagship does.
|
| Shipped / promised / undisclosed |
| [shipped] |
Hosted API, general availability 3 Aug. OpenAI and DashScope compatible. Function calling, structured outputs, batches, prefix completion, fine-tuning. |
| [shipped] |
Published price, published context envelope, published benchmark table. |
| [promised] |
Open weights for Qwen3.8-Max and Qwen3.8-27B, stated for the week after launch. No checkpoint at time of writing. |
| [missing] |
Weight license. Region terms, commercial terms and redistribution rights are all unknown until it lands. |
| [missing] |
Confirmed activated-parameter count, which is the figure that drives serving cost and latency. |
|
|
|
Insight 03Prefix stability is the cost lever, not prompt length. At eight times below fresh input, a cached read means an agent replaying a stable codebase and tool trace pays a fraction of what an agent that reshuffles its prompt pays for identical work.
|
|
The most quotable figure in this release, 86.6, is a property of a harness as much as of a model. The version that belongs in your roadmap is the one you measure with your own timeouts.
|
|
Primary sources
Qwen3.8-Max launch post ·
Alibaba announcement thread ·
Model page and limits ·
Vals AI timeout note ·
Vals Index placement ·
Frontend Code Arena debut ·
Cline on harness gains ·
MarkTechPost specs writeup ·
AINews launch roundup
|
|
researchaudio.io
Measurement notes for engineers building on frontier models. Vendor-reported figures are labelled as such throughout.
|
|
|