A 56-Layer Network Lost to a 20-Layer One on Its Own Training Data

ResNet’s answer was a plus sign that teaches every layer how to do nothing. A decade later, every LLM you call still runs on it.

In 2015, four researchers at Microsoft Research trained two plain networks on CIFAR-10. One had 20 layers, one had 56. The deeper one lost. Not on the test set, where you would expect a bigger model to overfit. On the training set, the 50,000 images it had already seen, pass after pass.

That is a strange way to fail. A network with more capacity that cannot even memorize its homework is not overfitting. Something about depth itself was broken, and the fix this paper found now sits inside every transformer block of every model you call.

examiner’s note: adding 16 layers made the score worse on data the model had already seen.

Fig. 1 of the paper shows the CIFAR-10 curves: the 56-layer plain net sits above the 20-layer one on both training and test error for the entire run. The ImageNet version is Table 2. Two plain nets, 18 and 34 layers, then the same two shapes with shortcuts added and nothing else changed. Same depth, same width, same parameter count.

Source: Table 2, top-1 error, 10-crop testing on ImageNet validation. The shortcut nets use zero-padding where channels double, so they add zero parameters.

Read the red row against the green row. Same 34 layers. One cannot beat its shallower sibling, the other beats it by 2.8 points. The plain 34-layer net had higher training error through the entire run, even though every function the 18-layer net can represent, the 34-layer net can too. And at 18 layers, Fig. 4 shows both nets landing at the same error with the residual one getting there faster.

examiner’s note: three easy answers are wrong. the right one is still a conjecture.

The paper is careful here, and it is the part most summaries skip. It takes the obvious explanations one at a time. On the first, the verdict is blunt: degradation is “not caused by overfitting.”

Keep that last row in mind. The most cited architecture paper in deep learning ships with an honest admission: the authors could describe the disease but not explain it. The 34-layer plain net still reaches competitive accuracy, so the solver works. It just works worse the deeper you go.

examiner’s note: a correct answer is in the textbook. the student cannot find the page.

Here is the argument that makes the result so odd. You can build a deep net that is exactly as good as the shallow one, by hand, in two steps.

So a good answer exists. SGD does not land on it. The authors’ reading: for a stack of layers with ReLUs between them, “do nothing” is surprisingly hard to learn. To pass an input through untouched, the weights have to line up into an exact identity, and nothing in training pulls them there.

Think of a relay team where every runner must hand the baton forward without dropping it. Add runners and the odds of a fumble go up, even if the new runners add nothing. The fix is not better runners. It is a second lane where the baton rolls forward on its own.

Worth noticing: footnote 2 flags the assumption under the whole field, that stacked nonlinear layers can approximate any complicated function, as “still an open question.” The paper builds on it anyway and says so.

examiner’s note: change what the layers are asked to learn, not how they learn it.

Call the mapping you want from a block H(x). A plain block learns H(x) directly. A residual block learns the difference, F(x) = H(x) − x, and adds the input back at the end. The block’s output is y = F(x) + x.

Now watch what happens when the weights are pushed toward zero, which is where weight decay pulls them.

FIG C · redrawn from Fig. 2 of the paper. Biases omitted, as in the paper.

That is the whole trick. In a plain block, zero weights mean “destroy the signal.” In a residual block, zero weights mean “leave it alone.” Identity stops being a hard target and becomes the resting state. All the layers have to learn is the edit.

The shortcut costs nothing: no parameters, and the extra compute is one element-wise addition. Where channels double, the paper tests three ways to match shapes: zero-padding, 1x1 projections at those points, and projections everywhere. ResNet-34 lands at 25.03, 24.52 and 24.19. The authors read the small gap as proof that projections are not what fixes degradation. The identity path is.

One contrast the paper draws with Highway networks, published around the same time: Highway had a shortcut too, with a learned gate on it. A gate can close. ResNet’s shortcut has no gate, so the signal always gets through, and Highway nets had not shown gains past 100 layers.

examiner’s note: 152 layers, and still fewer FLOPs than 16-layer VGG.

To go past 34 layers without the compute bill exploding, the paper swaps the two-layer block for a bottleneck. A 1x1 conv shrinks 256 channels to 64, a 3x3 conv does the work in the narrow space, and a 1x1 conv expands back to 256. Drawn to scale, the block is shaped like its name.

FIG D · redrawn from Fig. 5 right. The paper notes that replacing this identity shortcut with a projection would double the block’s parameters and compute.

Source: Tables 1 and 3, ImageNet validation, 10-crop testing. VGG-19 is 19.6 GFLOPs.

Every row down the ResNet column is deeper and better. No degradation anywhere, and the 152-layer net still costs fewer FLOPs than VGG-16.

The receipts: a single ResNet-152 scored 4.49% top-5 on validation, beating every previous ensemble, and an ensemble of six ResNets hit 3.57% on the test set for 1st place in ILSVRC 2015. Swapping VGG-16 for ResNet-101 inside the same Faster R-CNN detector lifted COCO mAP@[.5, .95] from 21.2 to 27.2, a 28% relative gain from the backbone alone, and the same nets took 1st in four more detection and localization tracks.

examiner’s note: three loose ends. the authors tied the big one off themselves a few months later.

Loose end 1: the highway still has a toll booth. Look at FIG C again. The second ReLU is applied after the addition, so the shortcut is not a clean identity across the network. Every block’s output passes through a nonlinearity before the next block sees it.

He’s team fixed this in Identity Mappings in Deep Residual Networks (2016). Batch norm and ReLU moved inside the residual branch, before the weights, so nothing sits on the shortcut. With that “pre-activation” block a 1001-layer ResNet reached 4.62% error on CIFAR-10. Their one-line finding: keep a clean information path and optimization gets easier.

Loose end 2: depth hits a different wall. The paper trained a 1202-layer net on CIFAR-10. It optimized fine, with training error under 0.1%. Its test error was 7.93%, worse than the 110-layer net’s 6.43%. The authors blame overfitting: 19.4M parameters on 50,000 images, with no dropout. Depth stopped being an optimization problem and became a regular one.

Source: Table 6, plus the Fig. 6 caption for plain-110, whose error the paper says is higher than 60% and left off the chart. ResNet-110 is the best of 5 runs, mean 6.61.

Loose end 3: even ResNet needed a hand at 110 layers. Footnote 5: a starting learning rate of 0.1 was slightly too large for the 110-layer net to begin converging, so they warmed up at 0.01 until training error fell below 80%, about 400 iterations, then switched back. The identity path makes depth trainable. It does not make the first few hundred steps automatic.

There is also a quiet finding in Fig. 7 that mattered more than the authors could know. They measured the standard deviation of each layer’s response. Residual layers respond less than plain layers, and the deeper the ResNet, the smaller each layer’s response. The paper’s reading: with more layers, each one tends to modify the signal less. Many small edits on one shared stream, instead of a chain of full rewrites. Hold that picture.

examiner’s note: the image paper is the plumbing of every language model.

The Transformer paper cites this one for its residual connections. Then language models repeated ResNet’s journey step for step, toll booth included.

Pre-norm became the common default for decoder LLMs after GPT-2.

One decoder block, drawn with the same rules as FIG C. It is two residual blocks stacked, with attention and an MLP playing F.

If you build on frontier models, three habits fall out of this paper.

1. Read a model as one stream with edits. Every block in a decoder LLM reads a shared vector and adds a small change back. Interpretability researchers call it the residual stream, and it is why you can probe, patch or steer an intermediate layer and have the effect carry forward. It is Fig. 7 at scale: many layers, each nudging.

2. When you bolt something on, start it at “do nothing.” ResNet applied to fine-tuning. LoRA initializes one of its two adapter matrices to zero, so on step zero the adapter adds nothing and the base model is untouched. GPT-2’s 1/√N residual init is the same instinct: keep each edit small so the stream survives depth.

3. Keep the shortcut clean. When a deep custom model trains badly, look at what sits on the identity path. A norm, an activation, a learned scale or a gate on the shortcut is something every gradient must cross. The 2016 follow-up measured this: scaling, gating and 1x1 convs on the skip path all trained worse than a bare identity.