Glossary
DPO (direct preference optimization)
A fine-tuning method that trains a model directly on preferred-versus-rejected response pairs, without building a separate reward model or running reinforcement learning.
DPO consumes the same preference pairs as RLHF but replaces the reward-model-plus-RL loop with a single classification-style training objective. That makes preference tuning simpler, cheaper, and more stable to run, and it has become a standard alignment recipe, especially for open-weight models. The human preference pairs remain the input that must be sourced.
Why it matters
DPO lowered the engineering cost of using preference data, which broadened demand for it.