Residual Connection

A residual connection, also called a skip connection, adds a layer's input directly to its output so information and gradients can skip the transformation.

If a layer computes f(x), the residual output is x + f(x). The layer learns only the residual: how much to change the input, not the full output. That change lets networks hundreds of layers deep train. Without skip connections, backpropagation sends gradients through every layer. Those gradients multiply together and often decay (vanishing gradients) or grow (exploding gradients).

Residual connections give a direct gradient path from output back to input, so gradients do not have to survive the full multiplicative chain.

ResNet introduced residual connections in 2015. Image classifiers jumped from tens of layers to hundreds, and accuracy rose. Every transformer block uses them: attention output is added to its input, then feedforward output is added to its input. GPT-3 has 96 layers and still trains stably because of this pattern. The identity mapping view is that residual nets learn small refinements.

Each layer adjusts the representation instead of rewriting it from scratch.

In a transformer block the residual around attention keeps token identity available after mixing. The residual around the feedforward net keeps the mixed representation available after the nonlinear expand-and-contract. Remove either skip and a 96-layer model like GPT-3 becomes much harder to train. ResNet made the same point for conv nets in 2015: depth only helped once the skip was there.

Kaiming He's ResNet paper (2015) added skip connections so 100-layer nets could train. Transformers use the same trick.