Transformer Architecture

The Transformer, introduced in 2017, processes sequences with attention instead of recurrent step-by-step connections.

An encoder stack turns raw inputs into representations. A decoder stack writes target sequences from those representations. Self-attention is the main change. Older recurrent models processed tokens one at a time and passed a hidden state forward, which became a bottleneck. Here each token examines all other tokens in parallel. Long-range dependencies in text are easier to capture.

Training is faster because work parallelizes across GPUs.

Transformers sit under almost all modern language AI: translation, chat assistants, search that ranks by contextual meaning, and code generation. The same design works on images in place of convolutional layers, on protein folding, and on video. One pretrained Transformer can be fine-tuned for many downstream tasks, which saves the compute of training a new architecture from scratch for each job.

The 2017 paper dropped recurrence. Self-attention builds a weighted mix of all tokens for each token, in parallel, which GPUs like. The encoder stack contextualizes inputs. The decoder stack generates, usually with causal masking. Long-range links no longer have to hop through a chain of hidden states. Translation, chat, contextual search ranking, and code tools all use this backbone.

Vision replaces conv layers with patch tokens. Protein folding and video models reuse the same block. Pretrain once, fine-tune many times, instead of a new architecture per task. The 2017 paper replaced recurrence with self-attention so models could train in parallel on long sequences.