More Compute Made the Model Worse. Here Is Why.

ResearchAudio · 6-minute read

Imagine this. Your team gets 10 times more GPUs. You use the best published recipe. You train the model.

The model gets worse.

This is not a bug in your code. It is a bug in the recipe. A new paper from Q Labs, NYU, and Princeton found it, explained it, and fixed it.

The fix has two parts. Together, they cut perplexity by 14% on the same data. And they make one old idea important again.

Part 1

The data wall is here

For years, the rule was simple. Get more data, train a bigger model, and train for one pass over the data. Data seemed to have no limit.

That rule is breaking. One estimate says LLM training sets will equal all public human text in 2026. Compute continues to grow. Good data does not.

So labs now repeat the same data for many epochs. Think of a student who reads the same textbook 64 times. At some point, the student stops learning and starts to memorize. That is overfitting.

A 2026 recipe (Lovelace et al.) added an overfitting penalty to the classic Chinchilla scaling law. It looked correct. But the authors of this paper ran that recipe out to larger budgets.

On a dataset of 100M tokens, the recipe reaches its best loss at about 10¹⁸ to 10¹⁹ FLOPs. After that, more compute makes the loss go up. Earlier papers did not see this, because they stopped testing too early.

Why does this happen? Every extra FLOP must go somewhere. You can train a bigger model, or you can train for more epochs. In the old recipe, both choices make the overfitting penalty larger. At some point the penalty grows faster than the gains.

Push the old recipe to very large budgets, and it gives up. It says: go back to one epoch, and only make the model bigger. In other words, it throws away the main benefit of repeating your data.

Why have big labs not hit this wall yet? Dataset size. By the old recipe's own math, with 2B tokens, the same recipe still improves smoothly. With 500M tokens, it already shows a dip. With 100M tokens, it clearly turns back up. As compute grows faster than data, more teams will move into the danger zone.

Part 2

Fix one: stop freezing your weight decay

Here is the hidden cause. The old recipe kept weight decay constant, at 0.1 or 1.0, for every model size and every epoch count.

Weight decay is the force that keeps a model simple. A big model that reads the same data 32 times needs much more of it. With a fixed value, those models had too little regularization. They memorized.

So the authors measured the best settings directly. They did grid searches at 110 configurations of data size, model size, and epoch count. Then they fit power laws to the results.

The search was wide. Weight decay ranged from 0.0017 to 5.8, a range of more than 3,000 times. At each configuration, they fit a smooth bowl-shaped curve to the loss and took its lowest point. That found the best point more precisely than the grid alone.

Two things stood out. First, the two settings matter at different times. The learning rate matters most when you train for few epochs. Weight decay matters most when you train for many. Second, both settings become more sensitive when the data is small or the model is big. That is exactly where each tuning run is most expensive.

The result in plain words:

• More data: use less weight decay (roughly 1 divided by data size).
• Bigger model: use more weight decay (roughly model size to the power 0.4).
• More epochs: use much more weight decay. This is the strongest effect.

Does it work? The authors fit the laws on small runs only, then predicted the best settings for a run with 16 times more compute. The laws beat every popular heuristic. On 2x extrapolation tests, they landed within 0.01 nats of the true optimum.

The heuristics they beat are ones many teams use today. Keep weight decay fixed. Scale it as 1 divided by data size. Scale it as 1 divided by the square root of data times model size. Scale it as 1 divided by model width. Each one works in some cases. None works in all of them. These rules also disagree with each other, and none of them says what to do as the number of epochs grows.

Part 3

Fix two: the old trick that is back

Fix one makes things better. But even with good weight decay, the loss starts to climb again at higher budgets. A model can only grow so far on a fixed dataset.

So where should the extra compute go? The answer is a classic from the pre-LLM era: ensembles.

Train several models with different random seeds. Average their predictions. Think of it as asking five students instead of one. Each student makes different mistakes, so the average is more correct.

This is not obvious. Some earlier work even doubted that ensembles are ever a good use of compute. The reason it works here is variance. When you average M independent guesses, the noise drops in proportion to 1 − 1/M. The authors let the data choose the exact shape. The fitted exponent came out at 1.12, very close to the textbook value of 1.

Here is the important part. A bigger model and more ensemble members both add parameters. But they act in opposite ways. A bigger model overfits more. More members overfit less. So you cannot treat an ensemble as one big model.

The authors added one new term to the scaling law for this effect. Now the law chooses three things together: model size, epochs, and the number of models. It also predicts ensemble sizes that it never saw, and it fits data from a separate research group better than the alternatives.

Two details make the ensembles cheap:

• Free members from one run. In the second half of the training, the learning rate cycles once each epoch. The run saves a snapshot at the end of each cycle. These snapshots join the ensemble at no extra training cost. As a bonus, the cycles allow a learning rate about 2 times higher, and they give a lower loss than a constant rate.
• Average probabilities, not weights. This gave better results than an average of logits or of weights.
• Let a small set choose. A 2.1M-token calibration set decides which snapshots to keep and how much weight each one gets. This happens after training, so it costs almost nothing.

And one test surprised me. Adding a second model cut the loss by 0.040 nats. Re-tuning the hyperparameters for that ensemble cut it by only 0.006 nats, which is close to seed noise. The second model was worth almost 7 times more than the tuning.

The scoreboard

Same 100M tokens of FineWeb. Same compute. The only change is the recipe.

The bigger story is the shape of the curve. With both fixes, the loss continues to go down as compute goes up. The old recipe stops at its minimum and turns back up.

Each fix helps on its own. But only the pair keeps improving. With better weight decay alone, the loss still turns back up eventually, just at a higher budget. The model is forced to grow larger or train longer than anything it saw in the sweep. Ensembles give the extra compute somewhere safe to go.

Watch where the compute goes. Model size and epochs grow fast at first, then level off. After that point, almost every extra FLOP goes to more ensemble members.

Part 5

Three things I did not expect

Who should care

If your dataset is fixed but your compute is not, this is for you. That includes domain models for code, law, biology, and languages with little text online.

Three habits from the single-pass era are now wrong for you: fixed weight decay, fixed learning rate, and ensembles seen as only an inference trick.

There is a cost. An ensemble of M models costs M times more to serve. The paper measures training loss per FLOP, not serving cost. A natural next step is to distill the ensemble into one model. The paper does not test this.

The fine print

• Small scale: 25M to 400M tokens, 30M to 398M parameters, with tests up to about 1.2B.
• One metric: next-token loss on FineWeb. No downstream benchmarks.
• The exact numbers fit this Muon-based setup. The authors say other setups need their own sweeps.

Your turn: do you train on a fixed dataset? Have you seen your loss get worse with more compute? Hit reply and tell me. I read every answer.

The paper

Know someone who trains models on limited data? Forward this issue to them.