Data deduplication

Removing exact and near-duplicate items from a dataset before training. Duplicates waste compute and can cause a model to memorize repeated content.

See also

← All terms