Deduplication for pretraining

Also known as Corpus deduplication. This is the canonical page; those names redirect here.

Pairings in the atlas

Suffix-array exact substring removalspecialistPretraining data selection llm-training-alignment

Rivals: other methods for the same problems

Where it sits

llm-training-alignment · Machine Learning & AI