Glossary
Preference data
Model outputs annotated with human judgments of which response is better, gathered as pairs or rankings. The raw material for alignment methods such as RLHF.
The common format is a prompt, two candidate responses, and a label for which one a rater preferred, sometimes extended to rankings or scored rubrics. Inter-annotator agreement is tracked, because a reward model learns the raters’ inconsistencies along with their judgment. It is the direct input to RLHF and DPO.
Why it matters
Each new model generation needs fresh judgments on its own outputs, which makes preference data recurring demand rather than a one-off purchase.