LoRA

LoRA, Low-Rank Adaptation, adds small trainable low-rank matrices to frozen weights for cheap task specialization.

Weight updates during fine-tuning tend to have low intrinsic rank, so the useful change fits in much smaller matrices. LoRA writes the update as two matrices A (d by r) and B (r by d), with rank r typically 8-64 and d the weight dimension. The product A times B has the original weight's shape but far fewer parameters.

At inference the frozen weight W still runs, plus the LoRA path: output = W(x) + B(A(x)). Extra latency is small. Adapters can be merged into W for zero extra cost, or kept separate so tasks can switch without reloading the base model.

A 7B model might need 14GB for full fine-tuning and under 100MB for LoRA adapters. Several adapters can sit on one base model and swap at inference. That is why LoRA is the usual way to customize large language models on consumer hardware. QLoRA stacks LoRA on quantization for still lower memory.

Rank r is the main knob. r=8 is small and cheap. r=64 is closer to full fine-tuning quality on some tasks and still tiny versus updating all of W. Merging computes W' = W + BA once, then inference is a single matmul. Keeping A and B separate lets you hot-swap tasks.

QLoRA keeps W in 4-bit and trains LoRA in higher precision, which is how 7B adapters under 100MB showed up on consumer GPUs instead of 14GB full fine-tunes. LoRA freezes the base weights and trains two small matrices whose product is added back. Microsoft published it in 2021.