Layer Normalization

Layer normalization stabilizes training by normalizing each sample across features so mean and variance stay consistent.

For each sample on its own, layer norm computes mean and variance across all features, scales to zero mean and unit variance, then applies learned scale and shift parameters. Batch normalization instead normalizes across the batch. Layer norm does not use batch statistics.

That independence matters for transformers: batch norm fails on variable-length sequences and the small batch sizes common in language modeling.

Layer norm keeps activations in a consistent range as they pass through many layers. Without it, deep networks go unstable. Activations drift. Gradients become unreliable. Training diverges. Transformers apply layer normalization before or after attention and feedforward blocks. Pre-LN, which places it before those blocks, is more stable for very deep models.

RMSNorm is a simpler variant that uses only the root mean square and skips mean centering. Layer normalization is cheap. It adds little compute and is now standard in transformer architectures.

Scale and shift, often called gamma and beta, let the network undo normalization when a layer needs a different range. Pre-LN versus Post-LN is not cosmetic. Post-LN can still train, but very deep stacks tend to need warmup tricks that Pre-LN can skip. Layer norm (2016) normalizes across features in one example. Transformers use it instead of batch norm.