Layers, Residuals, and Depth

Why models stack dozens of identical blocks, and the two small tricks that make deep stacks trainable at all.

A transformer is the same block repeated — attention, then feedforward — dozens of times. A large model might stack 80 of them.

Two questions follow. Why does depth help? And why was depth impossible before a couple of specific tricks?

What depth buys

Each layer refines the representation it receives. Inspect a trained model and a rough progression appears:

Early layers work locally. Which sense of bank is this? What part of speech? Which adjacent tokens group together? Mostly syntax and disambiguation.

Middle layers build relationships. Who does she refer to? What is the subject of this verb? Which earlier clause does this depend on? This is where most entity tracking happens.

Late layers converge on the immediate task: assembling whatever is needed to predict the next token.

The useful framing: early layers determine what each word is, middle layers determine how words relate, late layers determine what comes next.

Depth also enables composition. A single attention pass can connect two positions. Two stacked passes can connect A to B to C — information routed through an intermediate. Multi-step reasoning appears to require this, which is part of why capability tracks depth rather than width alone.

Why deep stacks used to fail

Stack 50 layers naively and training breaks. The reason is the vanishing gradient problem.

Training works backward from the error, computing how each parameter contributed. That signal passes through every layer on its way back. Each passage multiplies it by something. Multiply 50 numbers slightly below 1 and the result approaches zero — early layers receive almost no signal and never learn. Multiply numbers slightly above 1 and it explodes instead.

Either way, depth becomes unusable. Two mechanisms fix this.

Residual connections

The fix is almost too simple. Instead of passing a layer’s output forward, pass input + output.

output = x + Layer(x)

The addition creates a path where the signal reaches earlier layers undiminished — gradients flow straight through the x term without being multiplied by the layer’s transformation. Depth stops degrading the training signal.

This reframes what a layer does. It no longer computes a new representation from scratch; it computes a modification to what it received. Each layer writes an adjustment onto a running representation.

That running representation is often called the residual stream, and it is a genuinely useful mental model: a wide channel of information flowing from input to output, with every attention and feedforward block reading from it and writing back. Layers communicate by leaving things in this shared channel rather than by strict hand-off.

It also explains why layers can be surprisingly redundant. Since each writes an increment, removing one from a trained model often degrades output only mildly — the others were not depending on its exact contribution.

Normalization

The second mechanism controls scale. Without it, values drift as they pass through layers — growing or shrinking until they hit numerical trouble.

Layer normalization rescales each token’s vector to a consistent statistical profile, then applies learned scale and shift parameters. Applied at every block, it keeps the whole stack numerically stable.

One placement detail with real consequences. Normalizing after the residual addition (post-norm) was the original design and trains poorly at depth. Normalizing before the layer (pre-norm) leaves the residual path completely clean, and it is what current models use. That change is a substantial part of why very deep transformers became trainable.

Depth versus width

Given a parameter budget, you can spend it on more layers or on wider layers.

Deeper favors composition and multi-step reasoning. Wider favors capacity per step and parallelizes better on real hardware. Depth also costs latency during inference, since layers must run in sequence — a 100-layer model cannot start layer 50 before layer 49 finishes.

Practical architectures settle in a middle range, tuned empirically rather than derived. There is no clean formula.

What to remember

• Layers progress from local syntax to relationships to next-token assembly; depth also enables multi-step composition.
• Naive deep stacks fail because gradients vanish or explode over many multiplications.
• Residual connections (x + Layer(x)) give gradients a clean path and turn each layer into an increment on a shared residual stream.
• Normalization keeps values stable; placing it before the layer rather than after is what made great depth practical.
• Depth aids reasoning but costs sequential latency.

Next: What Is a KV Cache?