The largest open model ever runs on 3.7 percent of itself

Kimi K3 just went public: 2.8 trillion parameters, 896 experts with only 16 awake per token, and a million token context made possible by deleting most of its attention. The size makes the headline. The accounting is the story.

7 minute read

On Saturday morning, Moonshot AI pushed 1.56 terabytes of weights to Hugging Face and ended a long running argument about how big an open model can get. Kimi K3 has 2.8 trillion parameters. That is close to triple the size of Kimi K2, it reads images natively, it holds a million tokens of context, and anyone with the bandwidth can download it right now.

What makes K3 worth your time is not the number. It is that nearly every design decision in this model attacks the same enemy, memory bandwidth, and the result is a trillion scale system that behaves at inference time like something a tenth its size. Here is how they did it, piece by piece.

Three of every four attention layers are not attention

The expensive part of serving long context is not compute. It is the KV cache, the transcript of keys and values that every full attention layer keeps for every token it has ever seen. At a million tokens, that transcript becomes the whole GPU bill.

K3's answer is Kimi Delta Attention, or KDA, which Moonshot first tested in its Kimi Linear research last year and has now trusted with a flagship. KDA is linear attention. Instead of keeping the full transcript, each layer maintains a fixed size memory that it updates as tokens stream through. The delta part is the interesting bit: the layer does not just pour new information in, it decides what to overwrite, channel by channel, so stale facts get evicted instead of endlessly diluted.

Fixed memory means a KDA layer costs the same at token 1,000,000 as at token 100. The catch is that compressed memory is lossy, and sometimes a model genuinely needs the exact wording of something from 400,000 tokens back. So roughly every fourth layer is Gated MLA, real full attention with a gate on its output, and those 24 layers carry the burden of precise recall.

An economy of 896 experts

The 2.8 trillion parameters live almost entirely in the mixture of experts layers. For every token, a router picks 16 experts from a pool of 896, and 2 shared experts run no matter what. Add it up and you get 104 billion active parameters doing all the work for any given token. That is 3.7 percent of the model. The other 96.3 percent sits in memory, cold, waiting for a token that needs it.

Sparsity this aggressive usually gets unstable. Routers collapse, popular experts get overtrained, unpopular ones die. Moonshot's fix is a framework it calls Stable LatentMoE. Experts talk to the network through a latent channel 3,584 wide, half the size of the 7,168 residual stream, which cuts the cost of shuttling activations between them. A scheme called Quantile Balancing keeps traffic spread evenly across the pool instead of letting a few favorites absorb everything.

Moonshot credits this stack, together with the attention changes, for a claimed 2.5x gain in scaling efficiency over K2. In plain terms, the same training budget now buys two and a half times more model.

How 2.8 trillion parameters fit in 1.5 terabytes

At standard 16 bit precision, K3's weights would need about 5.6 TB of storage. The files on Hugging Face total 1.56 TB. The trick is that Moonshot did not compress the model after training and hope for the best. From the supervised fine tuning stage onward, K3 was trained quantization aware, with MXFP4 weights and MXFP8 activations. The model learned to live inside 4 bit weights, compensating for rounding error during training instead of absorbing the damage afterward.

MXFP4 is a 4 bit floating point format with per block scale factors, and both NVIDIA's Blackwell chips and AMD's MI400 line run it natively. That alignment with current hardware is not a coincidence, and it is a big part of why seven providers were serving K3 on OpenRouter within a day of the weights landing.

Be clear about what this does not mean: you will not run K3 at home. Practical serving still wants a node cluster with 64 or more accelerators and roughly 5 TB of pooled memory, and today's inference frameworks need patches before they can schedule routing across 896 experts. The 4 bit weights change who can afford to host this model, not whether it fits on your workstation.

The quiet training wins

My favorite details are the unglamorous ones. Attention Residuals, or AttnRes, let deeper layers reach back and reuse representations computed earlier in the stack instead of rebuilding them, which Moonshot says bought roughly 25 percent training efficiency for under 2 percent extra cost. The optimizer is Per-Head Muon, a sharper cut of the Muon variant that trained K2, now working per attention head. Even the activation function is homemade, a gated unit built on something called the Sigmoid Tanh Unit. None of this makes a keynote slide. Together, it is why the training run finished.

The scoreboard, with footnotes

The numbers are strong. 93.5 on GPQA Diamond. 88.3 on Terminal-Bench 2.1, effectively tied with GPT-5.6 Sol at 88.8. 91.2 on BrowseComp, the best reported by anyone. 97.8 on MathVision, using the built in 401 million parameter vision encoder. Artificial Analysis ranks K3 third among all model families, the highest any open model has ever placed.

Now the footnotes. K3 burned about 1.9x more output tokens than Sol to earn those scores, so the listed prices, $3 per million input and $15 per million output, overstate the savings somewhat. Early testers also flag two behavioral quirks: it takes actions without asking more often than it should, and it is unusually sensitive to whether you preserve its reasoning trace between turns. If you build agents, both of those will find you.

Read the license before you build

K2 shipped under a lightly modified MIT license. K3 does not. It ships under a custom open weight license with two tripwires. Cross 100 million monthly users or $20 million in monthly revenue, and you owe Moonshot prominent attribution. Sell K3 access as a service and clear $20 million across any 12 month stretch, and you need a separate agreement with Moonshot to keep going.

Simon Willison gave Moonshot credit for one thing here, and I agree with him. They call this open weight rather than pretending it is open source, and that honesty is rarer than it should be.

That's the issue. If a section didn't land, hit reply and tell me which one. I read everything.

Until next week,
ResearchAudio