Instruction Tuning

Instruction tuning fine-tunes language models on many instruction-response pairs so the model follows natural language instructions across task types.

Each example has an instruction, optional input context, and the desired output. Tasks include question answering, summarization, translation, code generation, creative writing, analysis, and more. Training on thousands of instruction types teaches instruction following as a skill, not just a handful of tasks.

Models then generalize to instructions they never saw, because humans already speak in commands. That is the step that turns a base autocomplete model into an assistant.

FLAN (Fine-tuned Language Net), InstructGPT, Alpaca, and Vicuna are landmark instruction-tuned models. Diversity of instructions improves generalization. Quality of responses improves answers. Self-instruct methods use the model to write synthetic instruction data, so the set can grow past what people hand-write.

Instruction tuning often comes before RLHF: first teach following instructions, then shape behavior with human preferences. Without it, you spend more effort prompting a base model into the right behavior.

Format is usually: instruction, optional input, target output. Mix thousands of those templates so the model does not overfit one task wording. FLAN, InstructGPT, Alpaca, and Vicuna differ in data mix and base model, but they share that recipe. Self-instruct grows the mix when human writing is too slow. RLHF still comes after if you need preference ranking, not just instruction following.

A base model without this stage will autocomplete; it will not reliably treat a user sentence as a command. Google's FLAN paper showed that fine-tuning on many instruction tasks makes a model better at new instructions it was not trained on.