Glossary
RLHF
Reinforcement learning from human feedback — training a model using human preference judgments to align its outputs.
In the standard recipe, humans compare pairs of model responses; a reward model is trained on those judgments; the base model is then optimized against the reward model with reinforcement learning. It is the alignment step behind modern chat assistants. The scarce ingredient is the human preference data itself, which must be collected deliberately.
Why it matters
RLHF is a large part of why labs pay for carefully collected human preference judgments at scale.