Data Augmentation and Synthetic Generation
~40 min read
Expanding limited training data with paraphrases, back-translation, and LLM-generated synthetic examples.
Data Augmentation and Synthetic Generation is a Pro topic
Sign in, then upgrade to Pro or Power to unlock this topic and the full AI Engineering curriculum.
Key points
- •Self-Instruct pattern can generate thousands of instruction-tuning examples from ~175 seeds for ~$500 in API costs
- •Model collapse: a model fine-tuned only on synthetic data cannot exceed the quality of the generating model
- •Cap synthetic data at 30-40% of your total training set; use it to fill gaps, not replace real human-generated data