Evaluation Pipeline Design
How to design a complete LLM evaluation pipeline: decomposing complex tasks into per-component evaluations, writing evaluation guidelines, choosing annotation methods and data, and wiring the pipeline into a CI/CD deployment gate.
G-Eval Rubric Weights:
▶📚 Prerequisites(2)
🎓 Learning objectives
- •Decompose a complex LLM application into individually testable components
- •Write evaluation guidelines that produce high inter-annotator agreement
- •Choose between human annotation, AI-as-Judge, and automated metrics for each evaluation dimension
- •Design a test set that is representative, uncontaminated, and maintainable
- •Integrate eval into a CI/CD pipeline as a deployment quality gate
What is it?
An evaluation pipeline is the systematic process of measuring whether an LLM system meets its quality bar before and after changes. Rather than relying on a single aggregate score, a well-designed pipeline decomposes the system into its components (retrieval, generation, safety filtering, etc.), evaluates each independently with the right method (automated metrics, AI-as-Judge, or human annotation), and aggregates the results into deployment gates. Chip Huyen (AI Engineering, Ch.4) frames this as the 'evaluation stack': functional correctness at the bottom, quality metrics in the middle, and A/B tests at the top — each layer catching different failure modes.
Why it exists
LLM systems fail in subtle, non-uniform ways: a model can ace headline benchmarks while failing on the exact queries your users send. Without a pipeline, teams rely on anecdotes ('it felt better'), one-off manual reviews, or leaking test data into training. A proper eval pipeline catches regressions before production, gives engineers an objective signal for comparing model versions, and creates a feedback loop so that every deployment either provably maintains or provably improves the quality bar. It converts 'is this model good?' — a vague, expensive question — into 'did the eval suite pass?' — a cheap, automated, repeatable check.
Problem it solves
- How do we catch quality regressions before a bad model ships to users?
- How do we evaluate a system that has many components (retrieval + generation + safety), each of which can fail independently?
- How do we write annotation guidelines that produce consistent results across multiple annotators?
- How do we build a test set that actually represents production traffic, not just the easy cases we thought of first?
- How do we integrate evaluation into CI/CD so it runs automatically on every model or prompt change?
Intuition
Think of an eval pipeline like a car assembly line QA process. You don't just test the fully assembled car — you check each component at its station: brake pads before assembly, engine torque after installation, headlight alignment before final QA. If the brake pads fail their spec, you don't need to finish building the car to know it's not safe to ship.
LLM evaluation follows the same logic. A RAG system has at least three testable components: retrieval (did we get the right docs?), generation (did we answer accurately given the docs?), and safety (is the output appropriate?). Testing only the final output means a retrieval failure and a generation failure look identical from the outside — you can't fix what you can't locate.
Chip Huyen (AI Engineering, Ch.4): 'Evaluation is not a single step at the end. It is a set of experiments run at multiple stages of the development cycle.'
Analogy
An eval pipeline is like a medical diagnostic protocol.
A doctor doesn't run a single test and declare a patient healthy — they run a panel of targeted tests, each designed for a specific organ system: blood work for metabolic health, an ECG for cardiac function, imaging for structural issues. Each test uses the method best suited to what it measures (you can't use an ECG to check cholesterol). If any test fails its reference range, the patient doesn't get discharge clearance.
An LLM eval pipeline works the same way: per-component tests (retrieval recall, generation faithfulness, safety classifier), each using the right evaluation method (IR metrics for retrieval, AI-as-Judge for faithfulness, automated classifier for safety), all aggregated into a deployment gate. A failure in any component blocks the 'discharge' — i.e., the production deploy.
Technical explanation
COMPONENT DECOMPOSITION: Break the system into stages. For each stage, define: (a) what inputs it receives, (b) what outputs it should produce, (c) what 'correct' means.
RAG example stages:
- Query understanding: did we parse intent correctly?
- Retrieval: did we surface the relevant documents? (Recall@K, MRR)
- Context assembly: did we truncate/rank context optimally?
- Generation: is the answer faithful to context? (AI-as-Judge faithfulness)
- Safety: is the answer appropriate? (classifier FPR/FNR)
TEST SET DESIGN: Chip Huyen (Ch.4) distinguishes three test set sources:
- Curated: hand-crafted to cover known failure modes and edge cases
- Sampled: random sample of real production queries (representative distribution)
- Adversarial: deliberately crafted to trigger failure (jailbreaks, edge cases) Rule: your curated set tells you if the model is getting better at known things; your sampled set tells you if it's getting better overall; your adversarial set tells you where the ceiling is.
ANNOTATION GUIDELINES: Good guidelines specify: (1) the task precisely, (2) what each rating level means with concrete examples, (3) how to handle ambiguous cases, (4) what NOT to rate on. Measure inter-annotator agreement (IAA) before going to scale:
- Cohen's kappa ≥ 0.7: good agreement, proceed
- 0.4-0.7: guidelines unclear, refine before scaling
- < 0.4: task is subjective or guidelines are broken, redesign
ANNOTATION METHODS (ranked by cost):
- Automated metrics (BLEU, ROUGE, exact match) — $0, poor quality for generative tasks
- AI-as-Judge with rubric — $0.001-0.03/eval, good for instruction-following
- Expert human annotators — $15-100/hour, high quality, slow
- Crowdsourcing (MTurk, Scale AI) — $0.02-2.00/eval, variable quality, needs QC
DEPLOYMENT GATE LOGIC:
- Regression threshold: new eval score ≥ (old score - δ), where δ depends on metric
- Absolute threshold: score must exceed minimum quality floor on each dimension
- Per-dimension gates: safety must be 100%, helpfulness must be ≥ 4/5
- Trend gate: if 7-day moving average drops > 2σ, block deploy and alert
Architecture
Full Evaluation Pipeline Architecture:
┌────────────────────────────────────────────────────────────┐ │ DATA LAYER │ │ ├── Eval set registry (versioned, hash-checked) │ │ │ ├── curated/ (hand-crafted cases) │ │ │ ├── sampled/ (from prod, PII-scrubbed) │ │ │ └── adversarial/ (attack scenarios) │ │ └── Ground truth labels (maintained, version-controlled) │ └─────────────────────────┬──────────────────────────────────┘ │ ▼ ┌────────────────────────────────────────────────────────────┐ │ EVALUATION RUNNERS (parallel where independent) │ │ ├── retrieval_eval.py → Recall@10, MRR, NDCG │ │ ├── generation_eval.py → AI-judge faithfulness, helpful │ │ ├── safety_eval.py → FPR/FNR on adversarial set │ │ └── latency_eval.py → p50/p95 latency, cost/query │ └─────────────────────────┬──────────────────────────────────┘ │ ▼ ┌────────────────────────────────────────────────────────────┐ │ AGGREGATION AND GATING │ │ ├── Compare vs. baseline (previous prod model) │ │ ├── Apply per-dimension thresholds │ │ ├── Emit: PASS / FAIL / REGRESSION with delta details │ │ └── Write results to eval DB (historical tracking) │ └─────────────────────────┬──────────────────────────────────┘ │ ┌──────────────┴──────────────┐ ▼ ▼ [PASS → deploy] [FAIL → block + alert] + diff report + worst-case examples
Workflow
-
DECOMPOSE the system: Map your LLM application's stages. For each stage define:
- Input type
- Expected output
- Failure modes (what can go wrong?)
-
CHOOSE an eval method per component:
- Retrieval: automated (Recall@K, MRR) — objective, cheap
- Generation quality: AI-as-Judge (faithfulness, helpfulness)
- Safety: automated classifier + human audit sample
- Latency/cost: automated benchmark
-
BUILD the test set:
- Start with 50-200 manually curated cases per component
- Add a random sample from production logs (100-500 queries)
- Add adversarial cases for known failure modes
- Version-control the eval set; treat it like production code
-
WRITE annotation guidelines (if using humans or AI-as-Judge):
- Define each rubric dimension with concrete anchor examples
- Pilot on 20 examples, measure IAA, refine until kappa ≥ 0.7
-
BASELINE the current model:
- Run the full pipeline on the current production model
- Record baseline scores per component per metric
- Store in eval DB as the reference point
-
WIRE INTO CI/CD:
- Trigger eval suite on: model weight change, prompt change, config change
- Gate: block deploy if score drops > threshold vs. baseline
- Alert on regression; auto-approve if all gates pass
-
MAINTAIN:
- Add new cases when new failure modes are discovered in production
- Re-baseline when a model upgrade clearly supersedes the old baseline
- Run human eval panel quarterly to validate AI-as-Judge calibration
Example
# Minimal eval pipeline for a RAG system import asyncio, json from anthropic import Anthropic client = Anthropic() FAITHFULNESS_JUDGE = ''' You evaluate whether an AI response is faithful to the provided source documents. Score 1-5: 1=contains claims unsupported by docs, 5=every claim is grounded. Think step by step, then output ONLY JSON: {"faithfulness": N, "reasoning": "..."} ''' def judge_faithfulness(query: str, context: str, response: str) -> dict: user_msg = ( f'Query: {query}\n\n' f'Source documents:\n{context}\n\n' f'AI response: {response}' ) result = client.messages.create( model='claude-sonnet-4-6', max_tokens=200, temperature=0, system=FAITHFULNESS_JUDGE, messages=[{'role': 'user', 'content': user_msg}], ) return json.loads(result.content[0].text) def run_eval_suite(eval_cases: list[dict]) -> dict: '''Run full pipeline eval. Returns pass/fail with per-case details.''' scores = [] for case in eval_cases: score = judge_faithfulness( case['query'], case['context'], case['response'] ) scores.append(score['faithfulness']) avg = sum(scores) / len(scores) baseline = 4.2 # previously recorded production score passed = avg >= baseline - 0.15 # allow 0.15 regression tolerance return { 'avg_faithfulness': avg, 'baseline': baseline, 'delta': avg - baseline, 'passed': passed, 'gate': 'PASS' if passed else 'FAIL — regression detected', }
Real-world usage
-
Stripe: runs automated eval suites on every change to their LLM-powered support assistant. The eval set is sampled from real support tickets with human-verified labels. A score drop > 2% blocks deployment automatically.
-
Anthropic (internal): uses a multi-layer eval stack — automated tests at commit time, AI-as-Judge on a 5,000-question eval set at model release, human red-teaming before public launch. Each layer catches different failure modes.
-
Chip Huyen (AI Engineering, Ch.4): provides the canonical decomposition — 'evaluate each component independently before evaluating end-to-end, because end-to-end eval cannot isolate which component caused a failure.'
-
Hugging Face Open LLM Leaderboard: runs a fixed eval suite (ARC, HellaSwag, MMLU, TruthfulQA) on every submitted model checkpoint, using a standardized pipeline to ensure fair comparison across model families.
-
LangSmith / Braintrust / HoneyHive: commercial eval pipeline tools that provide eval set versioning, AI-as-Judge integration, and CI/CD hooks — codifying this pipeline into a product because teams consistently rebuilt it.
Trade-offs
Speed vs. coverage: a fast eval suite (50 curated cases, 2 minutes) can be run on every PR but misses tail risks. A thorough suite (2,000 cases, 30 minutes) gives better coverage but creates friction in fast-moving dev cycles. Solution: tiered pipeline — fast suite on every PR, full suite before release.
Automated vs. human: automated evals are cheap and repeatable but miss nuance; human evals are expensive but catch what automation misses. Use automated for regression detection (high volume, low cost), human for release qualification (low volume, high quality).
Curated vs. sampled test sets: curated sets test known failure modes but have selection bias (you pick cases you think matter). Sampled sets are representative but may include low-difficulty cases that never fail. Use both.
Per-component vs. end-to-end: per-component is essential for debugging but misses system-level emergence (two components each within spec can still produce a bad end-to-end output). Always run both.
Visual explanation
Evaluation Stack (Chip Huyen, AI Engineering Ch.4):
Layer 3 (slowest, most expensive, highest signal) ┌──────────────────────────────────────────────────────┐ │ A/B TEST (live traffic split) │ │ • Real users, real queries, engagement metrics │ │ • Gold standard but takes days/weeks │ │ • Run for major model swaps only │ └──────────────────────────────────────────────────────┘ ▲ Layer 2 (hours, moderate cost) ┌──────────────────────────────────────────────────────┐ │ HUMAN EVALUATION (panel of annotators) │ │ • Representative sample of production queries │ │ • Inter-annotator agreement measured (kappa ≥ 0.7) │ │ • Run before major releases │ └──────────────────────────────────────────────────────┘ ▲ Layer 1 (minutes, low cost, automated) ┌──────────────────────────────────────────────────────┐ │ CI/CD EVAL SUITE │ │ ├── Component evals (retrieval Recall@K, BLEU) │ │ ├── AI-as-Judge on curated eval set │ │ ├── Safety classifier on adversarial prompts │ │ └── Latency / cost regression check │ │ Runs on every PR / every model update │ └──────────────────────────────────────────────────────┘
Per-component decomposition for a RAG system:
User Query │ ▼ [RETRIEVAL]───────Eval: Recall@K, MRR, NDCG │ (ground truth: human-labeled relevant docs) ▼ [GENERATION]──────Eval: faithfulness (AI-as-Judge vs. retrieved context) │ helpfulness (AI-as-Judge vs. user need) ▼ [SAFETY FILTER]───Eval: FPR/FNR on adversarial prompt set │ ▼ Final Output
Advantages
- —
Catches regressions automatically before they reach production users
- —
Per-component eval isolates WHERE a failure occurred, not just that it occurred
- —
Versioned eval sets create a reproducible benchmark for comparing model updates over time
- —
Integrates into CI/CD — same workflow engineers use for code quality
- —
Separates the signal (did quality improve?) from the noise (my subjective impression)
Disadvantages
- —
Expensive to build well — writing eval guidelines and curating a representative test set takes weeks
- —
Eval sets go stale: as the application evolves, test cases that cover old functionality may miss new failure modes
- —
Metric Goodhart: teams optimize for eval scores rather than actual user value, especially when eval gates promotion
- —
Automated metrics and AI-as-Judge both have blind spots — a pipeline that passes all automated checks can still fail in production
- —
Requires ongoing maintenance: new model capabilities surface new failure modes not covered by existing test cases
Common mistakes
- —
Using the same data for both training/fine-tuning and evaluation. If the model has seen the eval set, scores are inflated and the pipeline gives false confidence. Strictly separate train and eval sets; treat contamination as a critical bug.
- —
Building an eval pipeline only after a production incident. Eval infrastructure built under pressure is underspecified and misses failure modes. Build it before the first production deploy, even if it starts small (50 curated cases is better than zero).
- —
Setting gates too tight too early. If your baseline score is 3.8/5 and you set the gate at 4.0, you'll never ship anything. Calibrate gates to catch meaningful regressions (> 0.2 point drop) not noise. Tighten gates as the system matures.
- —
Evaluating only the 'happy path'. Real users send ambiguous queries, queries in multiple languages, queries that mix topics. If your eval set is 100 well-formed English questions, your pipeline will not catch failures on real user traffic. Include adversarial and edge-case examples from day one.
- —
Treating AI-as-Judge scores as absolute truth without periodic human calibration. AI judge behavior drifts as judge model versions change. Validate judge-human agreement (Spearman ≥ 0.7) quarterly. A pipeline that always passes because the judge has drifted toward leniency is worse than no pipeline.
🎤 Interview questions
Why should you evaluate each component of an LLM pipeline separately, rather than just evaluating the end-to-end output? Give a concrete example where end-to-end eval would miss a real failure.
What is inter-annotator agreement, how do you measure it, and what threshold indicates your annotation guidelines are usable?