Data Processing: Inspection, Deduplication, and Cleaning
~45 min read
The technical pipeline for turning raw text into a clean, deduped, formatted training corpus.
Data Processing: Inspection, Deduplication, and Cleaning is a Pro topic
Sign in, then upgrade to Pro or Power to unlock this topic and the full AI Engineering curriculum.
Key points
- •Perplexity filtering (keep examples in the 'middle' perplexity range) removes both gibberish (very high) and repetitive boilerplate (very low)
- •MinHash LSH near-deduplication at Jaccard 0.8 threshold is the industry standard for large-scale deduplication (used in The Pile, RedPajama)
- •Packing short examples end-to-end to fill the context window is essential for training efficiency — naive batching can waste 50%+ of GPU compute on padding