Calibration, Cold Start, and Production Serving

The Calibration Requirement — Why Ads Are Different

In a standard classification or ranking task, only the relative ordering of predictions matters. In ad auctions, absolute probabilities matter:

eCPM = predicted_CTR × bid_price × 1000

If predicted_CTR is systematically wrong:

• Overestimated (model predicts 5%, actual is 2%): the auction overvalues this ad, shows it more than justified, and charges more than earned → advertisers overpay and lose trust
• Underestimated (model predicts 1%, actual is 5%): the ad loses auctions it should win; advertiser gets fewer impressions than their bid warrants

Detecting Miscalibration

Reliability diagram (calibration plot):

1. Bucket all predictions into N groups by predicted CTR (e.g., [0%, 1%), [1%, 2%), ...)
2. For each bucket, compute avg(predicted CTR) vs. avg(actual click rate)
3. A perfectly calibrated model falls on the diagonal (y=x)
4. Above diagonal: overconfident (predicted higher than actual)
5. Below diagonal: underconfident (predicted lower than actual)

Expected Calibration Error (ECE): ECE = Σ_b (|B_b| / N) × |avg_confidence(B_b) - avg_accuracy(B_b)| Where B_b is the set of samples in bucket b. ECE = 0: perfect calibration. Monitor ECE in production as a model health metric.

Common Causes of Miscalibration

1. Negative sampling bias: With ~2% click rate, naive training uses all positives but only 10% of negatives (to avoid 98% of training time on non-clicks). This shifts the predicted probability distribution upward.

   Fix: p_corrected = p* / (p* + (1-p*)/q) where q = fraction of negatives kept (e.g., q=0.1 if kept 10%)

2. Temporal distribution shift: A model trained on July data may overestimate CTR in December (holiday season has different click patterns). Requires periodic recalibration.

3. New ad format or platform: If a new ad format launches, the model has no data for it. Its predictions may be systematically off.

Platt Scaling (General Fix): Train a small logistic regression on a held-out calibration set: calibrated_p = sigmoid(a × logit(p*) + b) Parameters a, b are fit on a small labeled calibration set. This is a post-hoc fix that doesn't require retraining the main model.

Cold Start for New Ads

A new ad has no impression history → no historical_ctr feature → CTR prediction degrades.

Strategy 1 — Content-based prior: Embed the new ad's title + image → find K most similar historical ads Use avg(CTR of K nearest historical ads) as the initial CTR prior This provides a reasonable starting point without any impression data

Strategy 2 — Exploration injection: Show the new ad to a random sample of users (ignore CTR prediction) After ~1000 impressions, compute actual CTR and update the model Bandit algorithms (UCB, Thompson sampling) balance exploration (show new ads) with exploitation (show ads with known good CTR)

Strategy 3 — Advertiser history: Even if this specific ad is new, the advertiser's past campaigns have history Use advertiser_id_historical_ctr as a proxy feature

Production Serving Architecture

Auction happens in < 10ms. The serving path must be extremely fast:

1. Feature assembly (< 3ms):

   • User features: Redis lookup by user_id (pre-computed, updated hourly)
   • Ad features: Redis lookup by ad_id (pre-computed at ad creation)
   • Context features: extracted from the request directly
   • Cross-features: assembled on-the-fly from user + ad features

2. Model inference (< 5ms):

   • Quantized model (fp16 or int8) deployed on CPU or GPU
   • Batch all N candidate ads together → single forward pass
   • Output: N predicted CTR values

3. Calibration correction + auction (< 1ms):

   • Apply calibration correction to each predicted CTR
   • Compute eCPM = corrected_CTR × bid
   • Select winner; log impression