Per-Component Evaluation
~40 min read
How to decompose LLM systems and evaluate each component independently with the right metric.
Every LLM system has multiple components that can fail independently. Evaluating only the final output creates an attribution problem: you know something is wrong but not which stage caused it.
RAG pipeline decomposition:
| Stage | What fails | Right metric |
|---|---|---|
| Query understanding | Wrong intent parsed | Accuracy vs. human labels |
| Retrieval | Wrong docs surfaced | Recall@K, MRR, NDCG |
| Context assembly | Key info truncated | Coverage rate |
| Generation | Hallucination, unfaithful | AI-Judge faithfulness |
| Safety filter | FP (blocks good) / FN (allows bad) | FPR + FNR |
Retrieval metrics:
- Recall@K: fraction of relevant docs appearing in top-K results
- Precision@K: fraction of top-K results that are relevant
- MRR (Mean Reciprocal Rank): 1/rank of first relevant result; good for fact-lookup
- NDCG: normalized discounted cumulative gain; rewards ranking relevant docs higher
Isolation technique: to test generation independently of retrieval, inject perfect (human-verified) context rather than retrieved context. If generation fails with perfect context, the model is the problem. If it fails only with retrieved context, retrieval is the problem.
💬 Deep Dive with AI
Key points
- •Each component can fail independently; end-to-end eval cannot isolate which stage caused the failure
- •Use automated metrics (Recall@K, MRR) for retrieval; AI-as-Judge for generation quality
- •Inject perfect context to test generation in isolation from retrieval errors