Data Augmentation and Synthetic Generation

When real labeled data is scarce, augmentation and synthesis extend your training set.

Augmentation techniques (transform existing examples):

1. Back-translation: translate to language X, then back to English → paraphrase

   • Preserves semantics, changes surface form
   • Works well for classification and summarization tasks
   • Limitation: poor with proper nouns and technical terms

2. EDA (Easy Data Augmentation, Wei & Zou 2019):

   • Synonym replacement: swap k random non-stopword tokens with WordNet synonyms
   • Random insertion: insert synonym of a random word at random position
   • Random deletion: delete each token with probability p
   • Random swap: swap positions of two random tokens
   • Effective for < 500 examples; diminishing returns above 10,000

3. Paraphrase generation: prompt GPT/T5 to rephrase user instructions

Synthetic generation (create new examples from scratch):

Self-Instruct (Wang et al., 2022):

1. Start with 175 human-written seed tasks
2. Prompt LLM to generate new instruction + instance pairs
3. Filter: remove duplicates (ROUGE-L ≥ 0.7), invalid, too long
4. Add to pool; repeat → Used to generate Alpaca-52K for $500

Risks of synthetic data:

• Model collapse: student can't exceed teacher model quality
• Error amplification: teacher model errors become training signal
• Style homogenization: all examples sound like GPT-4, reducing diversity

Mitigation: human-audit 5-10% of synthetic examples; cap synthetic share at 30-40% of total training data; prefer synthesis to fill gaps, not replace real data.