Glossary
Data deduplication
Removing exact and near-duplicate items from a dataset before training. Duplicates waste compute and can cause a model to memorize repeated content.
Pipelines drop exact duplicates by hashing and near-duplicates by fuzzy matching; audio and video use perceptual fingerprints for the same job. Web-scale corpora shrink substantially when deduplicated, which is one measure of how repetitive public crawls are. Repetition also raises the risk that a model memorizes and regurgitates specific content.
Why it matters
Original archives deduplicate cleanly against the public web — evidence they add new material rather than repeats.