Calibration Metrics: Expected Calibration Error and Brier Score
~13 min read
A model can be accurate but overconfident, or humble but underconfident — calibration metrics (ECE, Brier score) measure whether a model's STATED confidence actually matches its real-world accuracy.
Calibration Metrics: Expected Calibration Error and Brier Score is a Pro topic
Sign in, then upgrade to Pro or Power to unlock this topic and the full AI Engineering curriculum.
Key points
- •A model is well-calibrated if its stated confidence matches its actual accuracy — 80% confidence should mean correct roughly 80% of the time
- •Two models with identical accuracy can have very different calibration — one reliably flags uncertain cases, another is confidently wrong just as often as confidently right
- •Expected Calibration Error (ECE) buckets predictions by stated confidence and measures the gap between average confidence and actual accuracy per bucket
- •Brier score computes mean squared error between predicted probability and actual outcome per prediction, decomposing into calibration and resolution/sharpness components
- •Calibration metrics justify confidence-based automation (e.g. 'auto-approve above 90% confidence, else escalate to human') — a threshold only works if confidence is genuinely trustworthy