Causal Masking

Causal masking blocks each position from attending to later tokens so generation can only use left context.

In a sequence of N tokens, position i may attend to positions 0 through i, never i+1 through N-1. The implementation adds negative infinity to attention scores for future positions before softmax, which zeros those weights. Without the mask, a model trained to predict the next token could look at the token it is supposed to predict. Training would be easy.

Generation would fail, because future tokens do not exist at inference.

On a training string like "The cat sat," the model predicts "cat" from "The" and "sat" from "The cat" in the same pass. Causal masking keeps each of those predictions on legitimate context only. The attention pattern is a lower triangular matrix: the last position sees everything, earlier positions see less. GPT and other decoder-only language models use causal masking everywhere.

BERT does not, because it is built for understanding, not left-to-right generation. Prefix-tuning and some multimodal models use prefix causal masking: full bidirectional attention inside a prefix, then causal attention for generation.

The lower triangular mask is applied every layer, not once. If any layer leaked future tokens, later layers could still cheat. Prefix causal masking keeps a bidirectional prefix, useful when a prompt or image tokens should all see each other, then switches to causal mode for generated tokens. GPT-style models mask future tokens so position i cannot see i+1.

That is teacher-forced next-token training.