Evaluation Guidelines and Annotation Design
~35 min read
Writing evaluation guidelines that produce consistent, high-quality labels at scale.
Evaluation guidelines are instructions given to human annotators (or an AI judge system prompt) that specify precisely what to rate, on what scale, and how to handle ambiguous cases.
Structure of good guidelines:
- Task description: what is being evaluated and why
- Rating scale with concrete anchor examples (don't just say '3/5 is OK')
- Rubric per dimension (factuality, helpfulness, safety, etc.)
- Decision rules for ambiguous cases ('if the response is helpful but slightly inaccurate, rate helpfulness 4 and factuality 2')
- What NOT to penalize (typos in source, length preference)
Inter-Annotator Agreement (IAA):
- Run the same 20 examples through 2+ annotators before going to scale
- Cohen's kappa: κ = (Po - Pe) / (1 - Pe) where Po = observed agreement, Pe = expected by chance
- κ ≥ 0.7: guidelines are clear, proceed to scale
- κ 0.4-0.7: significant ambiguity, refine guidelines
- κ < 0.4: task is inherently subjective OR guidelines are broken; redesign
Test set construction (Chip Huyen, Ch.4):
- Curated (50-200): covers known failure modes and critical paths
- Sampled (100-500): random production queries; represents real distribution
- Adversarial: jailbreaks, ambiguous input, multi-hop reasoning, rare languages
- Size rule: you need enough cases that a 5% improvement is statistically significant (typically 200+ cases per dimension for a standard eval)
💬 Deep Dive with AI
Key points
- •Anchor examples per rating level are the most important part of annotation guidelines — abstract descriptions alone produce low IAA
- •Target Cohen's kappa ≥ 0.7 before scaling annotation; refine guidelines if below this
- •Test sets need three source types: curated (known failures), sampled (real distribution), adversarial (edge cases)