Calibration Metrics: Expected Calibration Error and Brier Score

The previous three subtopics covered entropy-derived metrics (perplexity, cross-entropy) as measures of raw predictive fit; this subtopic covers a related but genuinely distinct evaluation dimension that also builds directly on probability theory: calibration — whether a model's stated CONFIDENCE in its own predictions actually matches how often it's really correct.

A model is well-calibrated if, among all the times it says 'I'm 80% confident in this answer,' it's actually correct roughly 80% of the time — not 95% (dramatically underconfident) and not 50% (dramatically overconfident). This matters enormously in practice: a model's raw accuracy number alone doesn't tell you whether you can TRUST its confidence signal to decide when to double-check an answer or escalate to a human — two models with identical 85% accuracy can behave completely differently in this respect, one reliably flagging its own uncertain cases with lower confidence scores, the other expressing near-100% confidence on everything, correct or not.

Expected Calibration Error (ECE) quantifies this directly: bucket a model's predictions by their stated confidence (e.g. 0-10% confident, 10-20% confident, and so on), and for each bucket, compare the AVERAGE stated confidence in that bucket against the ACTUAL fraction of predictions in that bucket that were correct. ECE is the weighted average gap between stated confidence and actual accuracy across all buckets — 0 means perfect calibration (confidence always matches reality), and larger values mean the model's confidence signal is systematically unreliable, either over- or under-stating how sure it should be.

Brier score takes a related but more direct approach, computing the mean squared error between a model's predicted probability and the actual outcome (1 if correct, 0 if not) across every individual prediction, rather than bucketing first. A Brier score of 0 is perfect (the model's stated probability always exactly matches whether it turned out correct); higher values indicate worse calibration, and — usefully — the score decomposes into both a calibration component (is the confidence trustworthy) and a resolution/sharpness component (does the model actually distinguish confident-correct cases from confident-wrong ones, rather than just hedging everything toward 50%).

The practical payoff for AI engineering specifically: calibration metrics are what justify (or invalidate) confidence-based automation decisions — for instance, 'auto-approve this model's output if its stated confidence exceeds 90%, otherwise route to human review,' which is directly the pattern this curriculum's ai-as-judge companion topic uses for confidence-based escalation in an automated judge pipeline. That kind of threshold only makes sense — only actually reduces human workload without silently letting more errors through — if the model's confidence signal is genuinely well-calibrated in the first place, which is exactly what ECE and Brier score are built to verify before you rely on it.

# Computing Expected Calibration Error (ECE) and Brier score from a
# set of (predicted_confidence, was_correct) pairs.

def expected_calibration_error(predictions: list[tuple[float, bool]], num_bins: int = 5) -> float:
    """Bucket predictions by confidence, compare average stated
    confidence vs actual accuracy within each bucket."""
    bins = [[] for _ in range(num_bins)]
    for confidence, correct in predictions:
        bin_idx = min(int(confidence * num_bins), num_bins - 1)
        bins[bin_idx].append((confidence, correct))

    total = len(predictions)
    ece = 0.0
    for bucket in bins:
        if not bucket:
            continue
        avg_confidence = sum(c for c, _ in bucket) / len(bucket)
        actual_accuracy = sum(1 for _, correct in bucket if correct) / len(bucket)
        weight = len(bucket) / total
        ece += weight * abs(avg_confidence - actual_accuracy)
    return ece

def brier_score(predictions: list[tuple[float, bool]]) -> float:
    """Mean squared error between predicted probability and actual outcome."""
    return sum((conf - (1.0 if correct else 0.0)) ** 2 for conf, correct in predictions) / len(predictions)

# Well-calibrated model: when it says 80% confident, it's right ~80% of the time
well_calibrated = [
    (0.9, True), (0.9, True), (0.9, False), (0.9, True), (0.9, True),   # ~80% right at 90% conf
    (0.5, True), (0.5, False), (0.5, True), (0.5, False),                # ~50% right at 50% conf
]

# Overconfident model: ALWAYS says 95% confident, but only right ~60% of the time
overconfident = [(0.95, True), (0.95, True), (0.95, False), (0.95, False),
                 (0.95, True), (0.95, False), (0.95, True), (0.95, False)]

print(f"Well-calibrated model -- ECE: {expected_calibration_error(well_calibrated):.3f}, "
      f"Brier: {brier_score(well_calibrated):.3f}")
print(f"Overconfident model   -- ECE: {expected_calibration_error(overconfident):.3f}, "
      f"Brier: {brier_score(overconfident):.3f}")
print("\n-> The overconfident model has much higher ECE and Brier score,")
print("   flagging that its stated confidence CANNOT be trusted for automation")