Prefix Tuning

Prefix tuning adapts a frozen language model by learning continuous vectors prepended to the input.

Instead of training billions of parameters, it trains only prefix embeddings, typically a few hundred vectors per layer, under 1% of model parameters. Those vectors act as soft prompts. Attention sees the prefix before the real tokens, so later computation is conditioned on task-specific context. Gradients flow through the frozen model but only the prefix parameters update.

Memory drops because frozen weights need no optimizer states. Multi-task setups keep one base model and swap prefixes per task.

The method appeared in 2021 as a middle path between full fine-tuning and prompt engineering: more flexible than discrete prompts, cheaper than updating every weight. Related methods are prompt tuning, which learns prefixes only at the input layer; adapter layers, small trainable modules inserted in the net; and LoRA, low-rank weight updates.

Transformers are sensitive to context, so learned prefix context can steer behavior as effectively as changing weights.

A prefix at every layer is richer than a prefix only at the embeddings, because each block can condition on its own learned vectors. That is the 2021 design. Prompt tuning is the shallower cousin. Adapters insert bottlenecks. LoRA edits weights with low-rank matrices. Prefix tuning still freezes the original weights, so one GPU can hold several task prefixes and swap them.

Attention over the prefix is ordinary attention; the trick is that those keys and values are learned, not coming from real tokens. Prefix-tuning (2021) keeps the model frozen and learns a small prefix of virtual tokens per task.