Glossary
Benchmark contamination
The leak of test or benchmark material into training data, inflating a model’s measured performance. Data that never appeared in a public crawl carries far lower contamination risk.
Contamination happens quietly: a test set is posted publicly, crawled, and swept into the next pretraining run. Labs probe for it by checking overlap between training corpora and benchmark text, and publish contamination analyses alongside results. Once a benchmark leaks, scores on it stop being comparable and fresh held-out material is needed.
Why it matters
Contamination is why never-published is a checkable, valuable property of data.