Data Processing: Inspection, Deduplication, and Cleaning

Processing converts raw collected data into a training-ready corpus.

Step 1: Inspection (profiling)

• Length distribution: flag examples < 50 tokens (too short) or > 2× median (outliers)
• Language detection: fastText LangDetect on every example, drop non-target languages
• Perplexity filter: compute perplexity on a small reference LM; drop very high (gibberish/noise) and very low (repetitive/templated) examples. Used in CCNet/The Pile.
• Topic/intent clustering: BERTopic or K-Means to see domain distribution

Step 2: Deduplication

• Exact dedup: hash each example with MD5 or SHA-256; drop hash collisions. O(n), very fast.
• Near-dedup (MinHash LSH): 1. Tokenize to character n-grams (n=5) or word n-grams (n=13) 2. Compute MinHash signatures (128 hash functions) 3. Index into LSH bands; query for approximate nearest neighbors 4. Drop examples with Jaccard similarity > threshold (0.8 is standard)
  • Lee et al. (2022): deduped C4/Wikipedia → 13% fewer params needed for same perplexity

Step 3: Cleaning

• Strip HTML, markdown artifacts, boilerplate (headers, footers, cookie notices)
• Normalize unicode (NFKC normalization)
• Fix encoding issues (latin-1 decoded as utf-8)
• Remove toxic/NSFW content (classifier-based: Perspective API, fastText toxicity)
• Benchmark contamination: 13-gram overlap detection vs. eval sets

Step 4: Formatting

• Construct instruction-tuning triples (system, user, assistant)
• Tokenize with target model's tokenizer
• Packing: concatenate short examples to fill context window (reduces padding waste)
  • Use attention masks to prevent cross-example attention
• Shuffle with fixed random seed for reproducibility