Dataset Engineering for LLMs
The end-to-end discipline of building training data for LLMs: understanding what makes data high-quality, how to acquire and annotate it, how to augment it when real examples are scarce, and how to process raw data into clean, deduped, formatted training sets.
Fine-Tuning Controls:
▶📚 Prerequisites(2)
🎓 Learning objectives
- •Evaluate a dataset across the four quality axes: accuracy, consistency, completeness, and timeliness
- •Design a data acquisition strategy that balances cost, quality, and coverage
- •Apply data augmentation and synthetic generation techniques to expand small datasets
- •Run a deduplication pipeline to remove exact and near-duplicate examples from a training corpus
- •Format a raw dataset into instruction-tuning pairs suitable for supervised fine-tuning
What is it?
Dataset engineering is the systematic process of building high-quality training data for LLMs. It spans four phases: (1) curation — deciding what data you need and why; (2) acquisition — collecting or generating that data from real sources or via synthesis; (3) augmentation — expanding limited data with paraphrases, back-translations, or LLM-generated variants; and (4) processing — inspecting, deduplicating, cleaning, and formatting the raw corpus into something a model can actually train on.
Chip Huyen (AI Engineering, Ch.6) frames dataset engineering as the single highest-leverage activity in an LLM project: 'A model trained on garbage data will produce garbage outputs, regardless of how sophisticated the architecture is.' The quality of your training data determines the quality ceiling of your fine-tuned model — no amount of architectural improvement can overcome fundamentally bad data.
Why it exists
LLMs learn patterns from data, not from hand-coded rules. If the training data has the wrong distribution, the model learns the wrong distribution. If the data is inconsistent (the same question answered differently in 20% of examples), the model learns inconsistency. If the data is too narrow (all examples from one domain), the model overfit to that domain.
Dataset engineering exists because raw data is almost never suitable for training. Web crawls contain boilerplate, spam, near-duplicate content, and toxic text. Human-annotated data has labeler disagreement and coverage gaps. Even 'clean' public datasets often contain benchmark contamination (test data leaked into training sets). The discipline of dataset engineering is the set of practices for taking raw data and making it something a model can actually learn well from.
Problem it solves
- My fine-tuned model still hallucinates in my domain — how do I fix the training data?
- I only have 200 labeled examples but need 10,000 — how do I augment or synthesize more?
- My training corpus has millions of entries from a web crawl — how do I clean and deduplicate it?
- How do I format raw Q&A pairs into instruction-tuning format for supervised fine-tuning?
- How do I know if my dataset has good coverage of the production distribution before I train?
Intuition
Dataset engineering is like being a chef rather than a farmer.
A farmer grows raw ingredients. But before you cook, you need to: select the best produce (curation), buy from additional suppliers when you don't have enough of an ingredient (acquisition), substitute or extend scarce ingredients where you can (augmentation), and then wash, peel, dice, and prep everything before it goes in the pan (processing).
Just as a chef can't make a great meal with rotten tomatoes no matter how skilled they are, a machine learning engineer can't produce a great model with corrupt, inconsistent, or narrowly distributed training data — no matter how clever the training setup is.
The key insight from Chip Huyen (Ch.6): the relationship between data quality and model quality is not linear. Doubling the amount of mediocre data often helps less than removing 5% of the worst data. Quality beats quantity once you have sufficient volume.
Analogy
A training dataset is like a textbook for a student.
A good textbook: covers all the topics in the curriculum (coverage), has accurate information (accuracy), uses consistent terminology and style (consistency), is up to date (timeliness), and progresses from simple to complex examples (formatting). A bad textbook: has chapters duplicated almost verbatim (near-duplicates), contradicts itself across chapters (inconsistency), covers some topics exhaustively and misses others entirely (coverage gap), or uses examples from one narrow domain as if they represent everything (distribution shift).
The same applies to LLM training data. Near-duplicate examples are wasted capacity — the model 'rereads' the same chapter many times instead of learning something new. Inconsistent labels teach the model to be uncertain when it should be confident. Narrow coverage produces a model that's great at examples it's seen and terrible at the real distribution.
Technical explanation
DATA QUALITY AXES (Chip Huyen, Ch.6):
-
Accuracy: are the labels/answers correct?
- Measure: human audit of sample (target ≥ 95% correct)
- Common failure: crowdsourced labels on hard tasks
-
Consistency: are similar inputs labeled/answered similarly?
- Measure: inter-annotator agreement (IAA, Cohen's kappa ≥ 0.7)
- Common failure: different annotators using different rubrics
-
Completeness: are all required fields present? Does the data cover the full intended domain?
- Measure: coverage analysis (are all topic categories represented?), missing-value rate
- Common failure: data collected from one source over-represents its domain
-
Timeliness: is the data current? Does it reflect the world state the model should know?
- Measure: date distribution of data sources
- Common failure: training on 2021 web data for a model deployed in 2024
DATA ACQUISITION STRATEGIES:
- Web scraping: highest volume, lowest label quality. Needs heavy filtering.
- Licensed datasets: quality varies. Check license for fine-tuning rights.
- Human annotation: most accurate, most expensive. $0.02-$2/example. • Crowdsource (MTurk, Scale AI): fast, cheap, quality controlled via gold questions • Expert annotation: slow, expensive, highest quality • Active learning: annotate examples the model is most uncertain about — 3-10x more efficient
- Synthetic generation: using LLMs to generate training data. • Self-Instruct (Wang et al., 2022): generate instructions from a small seed set • Alpaca pattern: GPT-4 generates (instruction, input, output) triples • Risk: model distillation — the student model can't exceed the teacher model
DATA AUGMENTATION:
- Back-translation: translate to another language and back → paraphrase
- Synonym replacement: replace k random non-stopword tokens with WordNet synonyms (EDA, Wei & Zou 2019)
- Paraphrase generation: use T5/GPT to rephrase instructions while preserving meaning
- Mixup (Classifier tasks): interpolate between two training examples in embedding space Rule: augmentation helps most when training set is < 1,000 examples; at > 10,000 examples the marginal benefit shrinks and quality filtering matters more.
DEDUPLICATION:
- Exact dedup: hash each example (MD5/SHA-256), drop duplicates. Cheap and fast.
- Near-dedup: MinHash + Locality-Sensitive Hashing (LSH) to find examples with Jaccard similarity > 0.8 at n-gram level. Used in The Pile, RedPajama. Empirical result (Lee et al., 2022): 13% fewer parameters needed when training on deduplicated data to match the same perplexity.
- Semantic dedup: embed examples, cluster, drop near-neighbors in embedding space.
INSTRUCTION-TUNING FORMAT: Each training example becomes a (system_prompt, user_instruction, assistant_response) triple: { 'messages': [ {'role': 'system', 'content': 'You are a helpful assistant.'}, {'role': 'user', 'content': ''}, {'role': 'assistant', 'content': ''} ] } Packing: concatenate multiple short examples end-to-end to fill the context window, separated by EOS tokens, to avoid wasted GPU compute on short sequences.
BENCHMARK CONTAMINATION: Test data (MMLU, HumanEval, GSM8K) leaked into training sets artificially inflates benchmark scores. Mitigation: run n-gram overlap detection (13-gram match between training corpus and benchmark examples) before training.
Architecture
Production Data Pipeline (Chip Huyen Ch.6 pattern):
┌──────────────────────────────────────────────────────────────┐ │ INGESTION LAYER │ │ ├── Web crawl (CommonCrawl C4 subset) │ │ ├── Licensed text corpora (books, papers, code) │ │ ├── Human annotation pipeline (task queue → annotators) │ │ └── Synthetic generation (LLM batch API with validation) │ └──────────────────────────┬───────────────────────────────────┘ │ ▼ ┌──────────────────────────────────────────────────────────────┐ │ PROCESSING LAYER (streaming, embarrassingly parallel) │ │ │ │ [Inspect] → length histogram, language detection, │ │ perplexity-based quality filter │ │ │ │ │ [Filter] → drop: short (<50 tokens), repetitive, │ │ toxic (classifier), non-target language │ │ │ │ │ [Dedup] → exact hash → MinHash LSH near-dedup │ │ (13-gram, Jaccard threshold 0.8) │ │ │ │ │ [Clean] → normalize unicode, strip HTML/boilerplate, │ │ fix encoding, standardize whitespace │ │ │ │ │ [Format] → instruction-tuning pair construction, │ │ tokenize, pack to context length │ └──────────────────────────┬───────────────────────────────────┘ │ ▼ ┌──────────────────────────────────────────────────────────────┐ │ VALIDATION LAYER │ │ ├── Benchmark contamination check (n-gram overlap) │ │ ├── Coverage analysis (topic/domain distribution) │ │ ├── Human spot-audit (sample 200, measure accuracy) │ │ └── Train a small model on sample → eval vs. held-out │ └──────────────────────────┬───────────────────────────────────┘ │ ▼ Dataset Registry (versioned, hash-committed)
Workflow
-
DEFINE quality criteria for YOUR task:
- What does 'accurate' mean for your domain?
- What topics/skills must be represented (coverage)?
- Minimum training set size for your model and task?
-
AUDIT existing data first:
- Sample 200 examples, manually check accuracy
- Run length/language/topic distribution profiling
- Measure IAA on labeled data (kappa ≥ 0.7 target)
-
FILL coverage gaps:
- Identify under-represented topics or skills
- Acquire targeted examples (annotation, synthesis, scraping)
- Avoid augmenting low-quality seed data — garbage × 10 = more garbage
-
PROCESS the full corpus: a. Exact dedup (hash) b. Perplexity filter (remove very high/low perplexity — outliers) c. Near-dedup (MinHash LSH at Jaccard 0.8) d. Quality filtering (classifier, rule-based) e. Unicode normalization, HTML stripping, whitespace normalization f. Contamination check vs. your eval sets g. Format into instruction-tuning triples h. Tokenize and pack
-
VALIDATE before training:
- Re-audit sample after processing (did cleaning introduce errors?)
- Train a tiny model on 10% of data, eval on held-out set
- Compare distribution vs. production queries
-
VERSION the dataset:
- Commit a hash of the training set alongside each model checkpoint
- Reproducibility requires knowing exactly what data was used
Example
# Near-deduplication with MinHash + LSH # Uses datasketch library (pip install datasketch) from datasketch import MinHash, MinHashLSH import re def tokenize_ngrams(text: str, n: int = 13) -> list[str]: tokens = re.findall(r'\w+', text.lower()) return [' '.join(tokens[i:i+n]) for i in range(len(tokens)-n+1)] def build_minhash(text: str, num_perm: int = 128) -> MinHash: m = MinHash(num_perm=num_perm) for ngram in tokenize_ngrams(text): m.update(ngram.encode('utf8')) return m def deduplicate_corpus( examples: list[dict], text_key: str = 'text', threshold: float = 0.8, ) -> list[dict]: lsh = MinHashLSH(threshold=threshold, num_perm=128) kept, seen = [], set() for i, ex in enumerate(examples): mh = build_minhash(ex[text_key]) result = lsh.query(mh) if not result: # no near-duplicates found lsh.insert(str(i), mh) kept.append(ex) # else: near-duplicate of an already-kept example — skip return kept # Format into instruction-tuning pairs def to_chat_format( instruction: str, response: str, system: str = 'You are a helpful assistant.', ) -> dict: return { 'messages': [ {'role': 'system', 'content': system}, {'role': 'user', 'content': instruction}, {'role': 'assistant', 'content': response}, ] } # Synthetic data generation via Self-Instruct pattern from anthropic import Anthropic client = Anthropic() SEED_INSTRUCTIONS = [ 'Summarize the following article in one sentence.', 'Translate this English text to French.', 'Classify the sentiment of the review as positive or negative.', ] def generate_synthetic_instructions( seed_instructions: list[str], n: int = 20 ) -> list[str]: '''Generate new instructions from seed examples (Self-Instruct pattern).''' seeds = '\n'.join(f'{i+1}. {s}' for i, s in enumerate(seed_instructions)) msg = client.messages.create( model='claude-sonnet-4-6', max_tokens=1024, messages=[{ 'role': 'user', 'content': ( f'Here are {len(seed_instructions)} example task instructions:\n' f'{seeds}\n\n' f'Generate {n} new diverse task instructions in the same style.' f' Output only a numbered list.' ) }] ) lines = msg.content[0].text.strip().split('\n') return [re.sub(r'^\d+\.\s*', '', l).strip() for l in lines if l.strip()]
Real-world usage
-
The Pile (EleutherAI, 2020): 825 GB text corpus built using exactly this pipeline: 22 diverse sources, quality filtered, near-deduplicated. Became the training dataset for GPT-NeoX and GPT-J. Dedup alone reduced size by ~30% while improving downstream eval.
-
RedPajama (Together AI, 2023): open replication of LLaMA training data using CommonCrawl with aggressive quality filtering (CCNet pipeline: language detection, perplexity filter, near-dedup with MinHash). Demonstrates the full pipeline at scale.
-
Alpaca (Stanford, 2023): 52,000 instruction-following examples generated from text-davinci-003 using the Self-Instruct method with 175 seed tasks. Cost: $500 via OpenAI API. Became the proof-of-concept for synthetic fine-tuning data at low cost.
-
Chip Huyen (AI Engineering, Ch.6): 'Data quality beats data quantity once you have minimum sufficient volume. Removing the worst 5% of training data often improves model quality more than doubling the size of the dataset.'
-
Lee et al. (2022, 'Deduplicating Training Data Makes Language Models Better'): removing near-duplicates from C4 and Wikipedia corpora reduced the parameters needed to reach the same perplexity by 13% and reduced memorization of training data.
Trade-offs
Quality vs. quantity: more training examples help, but only up to a point. Doubling a 100,000-example dataset often helps less than removing the 5% worst examples from the existing 100,000. Prioritize quality filtering before expanding volume.
Real data vs. synthetic: real human-generated data has higher quality ceiling but is expensive to collect. Synthetic data is cheap but risks model collapse (student can't exceed teacher). Best practice: use synthetic to fill gaps, not replace real.
Dedup aggressiveness: aggressive dedup (Jaccard 0.7) removes more near-duplicates but risks removing legitimately similar examples (e.g., two different summaries of the same event). Conservative dedup (Jaccard 0.9) removes less but keeps more wasted capacity in the training set. Tune threshold based on your domain's natural repetition.
Annotation coverage vs. depth: for a fixed annotation budget, you can annotate more examples at lower quality (crowdsource) or fewer examples at higher quality (expert). Use crowdsource for high-volume easy labels, expert for low-volume hard labels.
Visual explanation
Dataset Engineering Pipeline:
Raw Sources ├── Web crawl (CommonCrawl, custom scrape) ├── Licensed datasets (books, papers, code) ├── Human annotation (crowdsource, expert) └── Synthetic generation (LLM-generated) │ ▼ ┌─────────────────────────────────────────────────────┐ │ Phase 1: CURATION │ │ Define quality criteria: accuracy, consistency, │ │ completeness, timeliness, diversity │ │ Identify coverage gaps in current corpus │ └──────────────────────┬──────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────┐ │ Phase 2: ACQUISITION & ANNOTATION │ │ Collect from chosen sources │ │ Label with crowdsourcing or expert annotators │ │ Measure inter-annotator agreement (kappa ≥ 0.7) │ └──────────────────────┬──────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────┐ │ Phase 3: AUGMENTATION & SYNTHESIS (if data-scarce) │ │ Back-translation, paraphrase, synonym replace │ │ LLM-generated synthetic examples (Self-Instruct) │ │ Validate synthetic data quality before mixing │ └──────────────────────┬──────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────┐ │ Phase 4: PROCESSING │ │ Inspect: profiling, length distribution, anomalies │ │ Deduplicate: exact hash → near-dup (MinHash/SimHash│ │ Clean: normalize, filter toxic/low-quality content │ │ Format: instruction-tuning pairs, tokenize, pack │ └──────────────────────┬──────────────────────────────┘ │ ▼ Clean Training Dataset (ready for fine-tuning)
Data Quality Four-Axis Model:
Accuracy
▲
│
Timeliness ◄──┼──► Consistency │ ▼ Completeness
All four axes must be healthy; weakness in any one propagates directly into model behavior.
Advantages
- —
High-quality, well-curated data is the single highest-leverage intervention in an LLM project — more impactful than architecture changes
- —
Near-deduplication reduces training compute waste and reduces verbatim memorization of training data
- —
Synthetic data generation (Self-Instruct pattern) can produce thousands of training examples for ~$500 in API costs
- —
Versioning datasets alongside model checkpoints enables reproducible ML experiments and root-cause analysis of model regressions
- —
Coverage analysis catches distribution gaps before training, avoiding expensive 'why does it fail on X?' debugging after the fact
Disadvantages
- —
Dataset engineering is time-intensive — building a high-quality 10,000-example fine-tuning set takes weeks of curation, annotation, and processing
- —
Synthetic data has a ceiling: models trained primarily on LLM-generated data can't exceed the quality of the generating model (model collapse risk)
- —
Near-deduplication at web scale is compute-intensive — MinHash LSH over billions of examples requires specialized infrastructure
- —
Human annotation quality varies significantly — crowdsourced labels require gold-question quality control and can still have 10-15% error rates on hard tasks
- —
Benchmark contamination is hard to detect exhaustively — n-gram overlap catches exact copies but misses paraphrased test data
Common mistakes
- —
Augmenting low-quality data. Applying back-translation or paraphrase augmentation to already-poor examples multiplies the problem. Always clean and filter before augmenting. 'Garbage × 10 = more garbage.'
- —
Skipping deduplication. Training on near-duplicate data wastes compute (the model sees the same content many times) and inflates benchmark scores via memorization — the model is 'recalling' training examples rather than generalizing. Deduplicate every corpus before fine-tuning.
- —
Using evaluation benchmark data in training. Even a small overlap between training data and eval benchmarks (MMLU, HumanEval) invalidates the benchmark score. Run a 13-gram overlap check between your training corpus and all evaluation sets before training. Treat contamination as a critical bug.
- —
Narrow annotation coverage. If your 10,000 training examples all come from one source (e.g., Wikipedia articles), the model will be great at Wikipedia-style outputs and poor at everything else. Deliberately sample across all the use cases the model will face in production. Check topic distribution with BERTopic or LDA before training.
- —
Treating synthetic data as ground truth. LLM-generated training examples can contain subtle errors that are hard to catch at scale. Always human-audit a 5-10% sample of synthetic data before mixing it into training. If error rate > 5%, the synthetic data may hurt more than it helps.
🎤 Interview questions
Walk through the four data quality axes. For each, give a concrete failure mode and how you'd detect it in a training dataset.
You have 500 real labeled examples and need 5,000 for fine-tuning. What are your options for getting to 5,000? What are the trade-offs of each approach?
Why does near-deduplication improve model quality, not just training efficiency? Reference the Lee et al. 2022 finding in your answer.
📂 Subtopics
Data Curation: Quality, Coverage, and Quantity
How to define what data you need, audit what you have, and fill coverage gaps before training.
~45 min
Data Augmentation and Synthetic Generation
Expanding limited training data with paraphrases, back-translation, and LLM-generated synthetic examples.
~40 min
Data Processing: Inspection, Deduplication, and Cleaning
The technical pipeline for turning raw text into a clean, deduped, formatted training corpus.
~45 min