Normalization Layers: LayerNorm and RMSNorm

Why every transformer block rescales its activations, how LayerNorm and RMSNorm do it, and why this unglamorous step is what makes deep models trainable at all.

Between every attention and feed-forward sub-layer sits a step that does no reasoning and holds almost no knowledge: normalization. It rescales the numbers flowing through the model. It sounds like plumbing, and it is — but it is the plumbing that makes stacking dozens of transformer layers possible at all. Without it, deep models simply do not train.

The problem it solves

As a vector passes through layer after layer, each one adds to it. Left unchecked, the magnitudes of those numbers drift — some dimensions blow up into the thousands, others collapse toward zero. This drift compounds with depth.

Two failures follow. Exploding values make gradients huge and training unstable, with the loss jumping around or diverging. Vanishing values make gradients tiny, so early layers barely update and effectively stop learning. Either way, a deep stack becomes untrainable. The deeper the model, the worse the drift, which is exactly why normalization became non-negotiable as models grew.

Normalization fixes this by resetting the scale of each token’s vector at every block, keeping the numbers in a stable range no matter how deep the stack goes.

What LayerNorm does

Layer normalization operates on one token’s vector at a time. For that vector it:

1. Computes the mean and the standard deviation across the vector’s dimensions.
2. Subtracts the mean and divides by the standard deviation — recentering to mean 0 and rescaling to variance 1.
3. Applies two learned parameters, a scale and a shift, so the model can undo or adjust the normalization if that helps.

The key detail is what it normalizes over. LayerNorm computes its statistics across the features of a single token, independently of every other token in the sequence. That independence is what makes it fit transformers: it behaves identically whether the sequence is 10 tokens or 10,000, and whether it is running one token during generation or a full batch during training. No cross-token or cross-batch dependence means no surprises between training and inference.

What RMSNorm changes

RMSNorm (root mean square normalization) is a stripped-down variant that has become common in large models. It asks a simple question: is the mean-subtraction step actually necessary?

RMSNorm drops it. Instead of centering and then scaling, it only rescales — dividing the vector by its root-mean-square magnitude and applying a learned scale. No mean is computed, no shift parameter, no recentering.

The payoff is speed. It removes a reduction operation and some parameters, which matters when the normalization runs twice per layer across dozens of layers and billions of tokens. The finding that justified the change was empirical: for these models, the recentering step contributed little, and the rescaling was doing nearly all the useful work. Same stabilizing effect, less compute. This is why many recent architectures use RMSNorm, though the choice is an efficiency refinement, not a change in what normalization is for.

Where it goes: pre-norm vs post-norm

Normalization’s placement around the block turned out to matter as much as the normalization itself.

The original transformer applied normalization after each sub-layer and its residual addition — post-norm. This works for shallow stacks but gets unstable as depth grows, often needing careful warmup schedules to train at all.

Nearly all large models moved to pre-norm: normalize the input before it enters attention or the feed-forward network, and leave the residual path clean. The reason connects to residual connections. In pre-norm, the residual highway runs from input to output without normalization sitting on it, so gradients flow straight back through the whole stack unimpeded. That clean path is what lets very deep models train stably. The switch from post-norm to pre-norm was one of the quiet changes that made scaling to many layers routine.

Why this “boring” layer matters

It is tempting to treat normalization as a footnote next to attention. But the causal chain is direct: no stable normalization, no deep stacks; no deep stacks, no capable models. Every capability discussed elsewhere assumes dozens of layers cooperating, and that cooperation only holds because normalization keeps the numbers in range at every step.

It is also cheap insurance. The layer adds negligible parameters and does no representational heavy lifting, yet removing it breaks training entirely. That asymmetry — tiny cost, load-bearing role — is why it is present in every block of every modern transformer.

What to remember

• Normalization rescales each token’s vector at every block, preventing the value drift that otherwise makes deep stacks untrainable.
• LayerNorm centers and scales across a single token’s features, independent of other tokens and of batch — which is why it fits transformers cleanly.
• RMSNorm drops the centering step and only rescales, giving the same stability with less compute.
• Pre-norm (normalize before the sub-layer) keeps the residual path clean and is what makes very deep models train stably; it replaced the original post-norm.
• The layer holds almost no knowledge but is load-bearing: without it, none of the depth that produces capability would train.

Next: The Output Projection — how the final normalized vector becomes vocabulary scores.