Direct Preference Optimization

Direct Preference Optimization fine-tunes a language model on preference pairs without training a separate reward model. You still need humans, or a judge, to say which of two answers is better. You skip the RL loop.

Standard RLHF has three stages: collect preferences, train a reward model, then use reinforcement learning to push the language model toward high reward. DPO folds that into one supervised step. From the same preferred and rejected answers, it uses a loss that raises the chance of the preferred reply relative to the rejected one.

The math is derived from the RLHF objective, so you are aiming at the same target without the unstable RL stage and without reward-model error stacking on itself.

Training is calmer. Fewer hyperparameters. You can use ordinary supervised-learning tools. You still need paired data: two responses to the same prompt, with a winner. That is the bottleneck, not the optimizer.

DPO is popular in alignment work because it is easier to implement and debug than full RLHF. Variants such as IPO and KTO change the loss. The idea is the same: treat alignment as supervised learning with a loss built from preferences, not as a separate RL problem.

You still need good preference data. Garbage pairs produce a polite model that is wrong. DPO is simpler RLHF, not a substitute for labeling. DPO (2023) skips training a separate reward model. It fits the policy directly to preferred versus rejected answers.