Human Evaluation: When Automation Isn't Enough

Automated metrics and LLM judges are proxies for what you actually care about: whether real people find the output good. For subjective qualities (tone, helpfulness, safety), high-stakes decisions (medical, legal, hiring), or brand-new capabilities where you don't yet trust any automated proxy, human evaluation remains the gold standard — and it's also what you use to CALIBRATE your automated metrics and LLM judges in the first place.

But 'ask some humans' is not a method. The quality of human evaluation is only as good as its process, and the two pillars are annotation guidelines and inter-rater agreement.

Annotation guidelines turn a vague question ('is this good?') into a repeatable task. A strong guideline defines each rating level concretely ('a 3 on faithfulness means every claim is supported by the source; a 2 means one minor unsupported detail...'), gives positive and negative examples for each level, specifies how to handle edge cases (empty answers, refusals, partial correctness), and separates distinct dimensions (correctness vs. fluency vs. safety) so raters don't collapse them into one gut feeling. Without this, two annotators interpret the scale differently and your data is noise.

Inter-rater agreement measures whether your guidelines actually produce consistent judgments. You have multiple annotators label the same items and compute an agreement statistic — Cohen's kappa (two raters) or Fleiss' kappa (more than two), which correct for agreement that would happen by chance. Kappa runs from 0 (chance) to 1 (perfect); roughly, 0.4-0.6 is moderate, 0.6-0.8 is substantial, and above 0.8 is strong. Low agreement is a signal that your guidelines are ambiguous or the task is genuinely hard — either way, you fix the guidelines and re-measure before trusting the labels.

Practical realities: human eval is slow and expensive, so you use it strategically — on a curated, representative sample rather than everything; on the dimensions automated metrics can't capture; and as the periodic ground-truth check that keeps your cheaper automated evaluation honest. A mature evaluation stack is a pyramid: cheap automated metrics run constantly, LLM judges run frequently, and human evaluation runs periodically to validate and recalibrate the layers beneath it.

# Computing Cohen's kappa: inter-rater agreement corrected for chance.
def cohens_kappa(rater_a: list, rater_b: list) -> float:
    """Agreement between two annotators on categorical labels,
    corrected for the agreement expected by chance."""
    assert len(rater_a) == len(rater_b)
    n = len(rater_a)
    labels = set(rater_a) | set(rater_b)

    observed = sum(1 for a, b in zip(rater_a, rater_b) if a == b) / n

    # expected agreement by chance, from each rater's label frequencies
    expected = 0.0
    for lbl in labels:
        pa = rater_a.count(lbl) / n
        pb = rater_b.count(lbl) / n
        expected += pa * pb

    if expected == 1.0:
        return 1.0
    return (observed - expected) / (1 - expected)

alice = ["good", "good", "bad", "good", "bad"]
bob   = ["good", "bad",  "bad", "good", "bad"]
k = cohens_kappa(alice, bob)
print(f"Cohen's kappa: {k:.2f}")  # ~0.58 -> moderate; guidelines may need tightening