A 27B model that fits in 3.9GB and runs on a phone.

The 645-point Hacker News thread noticed the actual CPU throughput is 6 tokens per second.

Finding 1 of 5

The "1-bit" is 1.125 effective bits per weight.

Most "1-bit" and "ternary" claims in industry don't survive scrutiny. The whitepaper is the only place this is honestly disclosed:

• Binary Bonsai 27B: 1.125 effective bits per weight (log2 2 + 16/128 from the FP16 group-wise scale)
• Ternary Bonsai 27B: 1.71 effective bits per weight (log2 3 + 16/128)
• Q4_K_XL (marketed as 4-bit): actually 5.2 bits per weight
• IQ2_XXS (marketed as 2-bit): actually 2.8 bits per weight

The Bonsai whitepaper is one of the only technical reports in the field to call this out: the conventional "4-bit" and "2-bit" quantization labels are routinely wrong. The metric that matters is true bits per weight, not the marketed name.

Finding 2 of 5

The agentic drop is 14 points. The blog says "a few."

The blog headline is "agentic workflows stay coherent across many steps." The actual numbers (Table 11 of the whitepaper):

The agentic category drops 14pp on the 1-bit variant. The blog post says "tool calling stays within a few points of full precision." A 14pp drop on the headline use case is not "a few points." For an agentic coder (the most-cited use case for local LLMs in 2026), this is the regression that matters.

Finding 3 of 5

The iPhone sustains 11 tok/s. The laptop claims 26.

The blog headline says "up to 26 tok/s on laptop." The laptop is a $4,000 M5 Max. The iPhone 17 Pro Max (the headline target device) sustains 10.82 tok/s under thermal throttle. The CPU inference is even worse: 6 tok/s for binary, 0.7 tok/s for ternary on a Ryzen 7 5700X desktop (per @networked's HN benchmark).

11 tok/s on a phone is interactive. 6 tok/s on a desktop CPU is not. The marketing claims 26, 44, 66, 87, 163 tok/s on laptop, M4 Pro, M5 Pro, M5 Max, RTX 5090. The actual on-device number is 11.

Finding 4 of 5

The IQ2_XXS collapse is the real validation.

The conventional "2-bit" quantization (IQ2_XXS) on the same base model (Qwen3.6-27B):

• AIME25: 93.29 to 66.67 (-26.62)
• AIME26: 93.33 to 57.50 (-35.83)
• LiveCodeBench: 87.77 to 56.40 (-31.37)

The Bonsai ternary model on the same benchmarks: AIME25 at 90.84 (only -2.45 from FP16), AIME26 at 87.50 (only -5.83), LiveCodeBench at 82.75 (only -5.02). This is the contrarian case PrismML is making: conventional sub-4-bit methods collapse on chain-of-thought reasoning. The Bonsai Caltech IP specifically avoids the collapse by preserving the FP16 group-wise scaling through the quantization. The collapse-versus-retention contrast is the entire technical story.

Finding 5 of 5

The on-device number is 4-5x more energy-efficient than any GPU.

The whitepaper measures energy per token (Table 15). The on-device operating point is 0.275 mWh per token on the M5 Pro. The H100, RTX 5090, L40S, and A100 all sit between 0.627 and 1.322 mWh per token. That is 2.3x to 4.8x more energy per token than the local-laptop number.

On the iPhone 17 Pro Max specifically: 672 tokens per 1% of battery. A full charge projects to roughly 67,000 tokens of sustained on-device generation, with the sustained decode rate (10.82 tok/s) close to the batch-1 tg128 figure and a mild thermal throttle emerging over a 5.2-minute run.

The "data center is the cheap way to run AI" narrative is structurally wrong for token-economics at small batches. The 0.275 mWh per token on the M5 Pro is the single most under-reported number in the launch.

What This Means For

Community Reaction (645 pts / 229 cm)

SKEPTICAL / TECHNICAL CRITIQUE

TECHNICAL DEEP-DIVE

OPTIMISTIC / UTILITY

The Metric That Actually Matters

Intelligence density (1 per GB). Higher is better. The 1-bit Bonsai 27B is 2.7x the densest conventional build and 10x the FP16 baseline. This is the number that makes the case for the next 2 years of edge-AI deployment.

Intelligence Density (per GB)

1-bit Bonsai 27B     | 0.530
Ternary Bonsai 27B    | 0.400
Qwen3.6-27B IQ2_XXS   | 0.199
Gemma-4-31B Q2_K_XL   | 0.162
Qwen3.6-27B Q4_K_XL   | 0.155
Gemma-4-31B QAT       | 0.111
Qwen3.6-27B FP16      | 0.051
Gemma-4-31B FP16      | 0.044

Source: PrismML Bonsai 27B whitepaper, Table 12

The 4 Things That Didn't Make The Marketing

1. The tool-calling regression is 14pp on the 1-bit variant. Not "a few points." The blog post understates this.
2. The CPU throughput is 6 tok/s for binary, 0.7 tok/s for ternary. Marketing claims 26, 44, 66, 87, 163 tok/s. The actual desktop CPU number is 10-100x lower.
3. The 1-bit variant does not actually fit on most iPhones. It requires the 17 Pro Max (12GB). The 17 Pro (8GB) doesn't have the 6GB per-app budget the model needs.
4. The "4-bit" and "2-bit" quants PrismML compares against are mislabeled. The whitepaper is the first place anyone in the industry has called this out, and the comparison is favorable to PrismML only because they use the true bits per weight.

Reader Challenge

1. If you ran a 27B model on your phone, what would you build? The 0.275 mWh/tok number is the unlock.
2. What does it mean that the industry has been mislabeling "4-bit" and "2-bit" quants for years? If Q4_K_XL is really 5.2 bpw, what does that mean for the open-source quantization community?
3. The agentic drop is 14pp. What does that mean for the next 12 months of local-agent products?

Every other AI lab is racing to scale up. PrismML is the contrarian: scale down, not up. The 3.9GB Bonsai 27B is a fundamentally different bet on the future of AI deployment. The bet: intelligence density matters more than raw capability, on-device inference beats cloud, and the bottleneck for the next 2 years is not "how big is your model" but "how small can you make the same intelligence." The data, for the first time, supports the bet.

researchaudio.io