Data Curation: Quality, Coverage, and Quantity
~45 min read
How to define what data you need, audit what you have, and fill coverage gaps before training.
Curation is the upstream phase of dataset engineering: figuring out what data you need and ensuring what you have meets the bar.
The four quality axes (Chip Huyen, AI Engineering Ch.6):
| Axis | Definition | How to measure | Common failure |
|---|---|---|---|
| Accuracy | Labels/answers are factually correct | Human audit 200-sample | Crowdsource labels on hard tasks |
| Consistency | Same input → same label across annotators | Cohen's kappa ≥ 0.7 | Multiple annotators, no shared rubric |
| Completeness | All domains/skills in target distribution covered | Topic distribution analysis | Single-source corpus |
| Timeliness | Data reflects the current world state | Date distribution of sources | Training on 2021 data for 2024 deployment |
Coverage analysis workflow:
- Sample 1,000 examples from your production query log
- Cluster by topic/intent (BERTopic, K-Means on embeddings)
- Compare cluster distribution vs. your training set
- Topics under-represented in training by > 2x their production frequency → fill gap
Quantity rules of thumb:
- Fine-tuning a chat model on instruction-following: 1,000-10,000 high-quality examples often sufficient
- Domain adaptation (PEFT/LoRA): 500-5,000 domain examples
- Pre-training from scratch: billions of tokens minimum
- Key insight: quality > quantity past the minimum threshold. Removing the worst 5% of data often helps more than adding 50% more mediocre data.
💬 Deep Dive with AI
Key points
- •Measure accuracy, consistency, completeness, and timeliness independently — a dataset can pass three axes and fail one
- •Coverage analysis requires comparing your training distribution to your production query distribution, not just counting examples
- •For PEFT fine-tuning, 500-5,000 high-quality domain examples are often enough — quality over quantity