Calibration, Cold Start, and Production Serving
~40 min read
Why calibration is a first-class requirement for ad CTR models, how to detect and fix miscalibration, how to handle new ads with no history, and the serving architecture that meets a < 10ms auction SLA.
Calibration, Cold Start, and Production Serving is a Pro topic
Sign in, then upgrade to Pro or Power to unlock this topic and the full AI Engineering curriculum.
Key points
- •Calibration is a first-class requirement for CTR prediction: predicted P(click) must match actual click rates in absolute terms because eCPM = CTR × bid — a miscalibrated model breaks auction pricing
- •Negative downsampling (keeping only 10% of non-clicks to handle class imbalance) systematically overestimates CTR — correct with p_corrected = p* / (p* + (1-p*)/q); or apply Platt scaling post-hoc
- •Cold start for new ads: content-based CTR prior (find similar historical ads via embedding similarity) + exploration injection (show to random users to collect data) are the two standard approaches