Glossary
Pretraining
The first, large-scale stage of training, in which a model learns general patterns from broad data before any task-specific tuning. Pretraining consumes the bulk of a training corpus.
Modern pretraining runs are measured in tokens — frontier models see trillions — and in the compute spent processing them. The objective is usually self-supervised, such as next-token prediction, so no labels are needed: volume, breadth, and quality do the work. Scaling research such as the Chinchilla paper tied model size to how much data training should consume.
Why it matters
Pretraining’s appetite for tokens is the root demand behind large-scale data licensing.