Human Evaluation: When Automation Isn't Enough
~12 min read
Human evaluation is the gold standard for subjective, high-stakes, or novel quality dimensions. Its value depends entirely on clear annotation guidelines and measuring inter-rater agreement.
Human Evaluation: When Automation Isn't Enough is a Pro topic
Sign in, then upgrade to Pro or Power to unlock this topic and the full AI Engineering curriculum.
Key points
- •Human evaluation is the gold standard for subjective, high-stakes, or novel qualities — and the ground truth used to calibrate automated metrics and LLM judges
- •Annotation guidelines turn 'is this good?' into a repeatable task: concrete level definitions, examples, edge-case rules, and separated dimensions
- •Inter-rater agreement (Cohen's/Fleiss' kappa) measures whether guidelines produce consistent judgments, correcting for chance agreement
- •Kappa below ~0.6 signals ambiguous guidelines or a genuinely hard task — fix and re-measure before trusting the labels
- •Use human eval strategically on a representative sample; run it as the periodic ground-truth check atop cheaper automated layers (an eval pyramid)