Low-Rank Adaptation (LoRA)

Low-Rank Adaptation fine-tunes large nets cheaply by training two small matrices instead of every weight.

Those matrices have much lower rank than the full weight matrix, so they hold far fewer parameters. The original model stays frozen. Only the low-rank additions train. Memory drops and training can fit on a single GPU. A company can adapt a pretrained language model to its customer-support data without a server farm. Trainable parameters fall from billions to a few million.

Teams can train many domain adapters, compare them, and drop failures while the base model stays untouched. One base model can serve several tasks by swapping adapters. That modularity is why smaller organizations can customize large models without paying for full-model fine-tuning.

Freeze W. Learn A and B with rank much smaller than the hidden size. Train on a single GPU. Parameter count drops from billions to a few million, which is enough to adapt a language model to customer-support transcripts without a server farm. Keep adapters separate to A/B test domains. Swap adapters to serve many tasks from one base. Discard a bad adapter without touching the backbone.

That is the path for teams that cannot pay for full-model fine-tuning. The same LoRA paper: rank is tiny compared with the full matrix, so fine-tunes stay small enough to swap per task.