Case Study: Ad Click-Through Rate Prediction System Design
End-to-end design of an ad click-through rate (CTR) prediction system — the foundational ML system for every ad-supported platform (Google, Meta, Twitter/X, TikTok). Covers feature engineering for ads, the Wide & Deep architecture, GBDT + logistic regression, probability calibration, cold start for new ads, and real-time serving requirements. Grounded in Khang Pham's ad click prediction case study and Google's published production systems.
▶📚 Prerequisites(3)
🎓 Learning objectives
- •Explain what CTR prediction is and why it is the foundational ML problem for ad-supported platforms
- •Identify the four feature categories used in CTR prediction (user, ad, context, interaction) and explain the outsized importance of cross-features
- •Describe the GBDT + logistic regression architecture and explain why it became the standard before deep learning
- •Explain the Wide & Deep architecture (Google, 2016) and describe what each component contributes
- •Explain the calibration problem in CTR prediction: why predicted probabilities must match actual click rates, and how Platt scaling fixes miscalibration
- •Describe the cold-start problem for new ads and the two standard mitigation strategies
- •Design an evaluation framework that distinguishes calibration quality from ranking quality
What is it?
Ad click-through rate (CTR) prediction is the ML problem of estimating P(click | user, ad, context) — the probability that a specific user will click on a specific ad shown in a specific context. This probability is the core input to an ad auction: the platform must predict CTR to calculate effective CPM (cost per thousand impressions) and decide which ad to show.
CTR prediction is arguably the highest-revenue ML system in the world. Google's $220B+ annual ad revenue and Meta's $100B+ are almost entirely determined by the quality of their CTR prediction models. A 0.1% improvement in prediction accuracy at Google's scale translates to billions of dollars.
Khang Pham's ML Primer presents CTR prediction as the canonical high-stakes ML system design case study because it requires every dimension of production ML simultaneously: massive scale (trillions of training examples), extreme feature engineering complexity (billions of sparse features), tight latency requirements (< 10ms per prediction), and precise probability calibration (the predicted probability must actually match the true click rate).
Key distinction from recommendation systems: recommendation systems optimize for user engagement (maximize probability of watch/like/share). Ad CTR prediction must also optimize for revenue (the predicted CTR × bid price determines auction outcome) and must produce calibrated probabilities (not just rankings).
Why it exists
Ad CTR prediction enables three critical decisions in the ad serving pipeline:
-
Ad auction pricing: in a cost-per-click (CPC) auction, the platform charges per click. To set a fair CPM (cost per 1000 impressions), it must predict how often the ad will actually be clicked. eCPM = predicted_CTR × bid_price × 1000.
-
Ad ranking: when multiple ads compete for the same impression slot, the platform must decide which ad to show. In a Vickrey-Clarke-Groves (second-price) auction: rank by bid × CTR (eCPM). The highest-eCPM ad wins.
-
Advertiser feedback: advertisers need accurate CTR predictions to set bids. If predicted CTR is systematically wrong (e.g., overestimates for certain categories), advertisers overpay and eventually lose trust in the platform.
Without CTR prediction: all ads would have equal predicted value regardless of their relevance to the user, leading to showing irrelevant ads that users don't click, destroying advertiser ROI and user experience simultaneously.
Problem it solves
- Auction design: platforms need P(click) to compute eCPM and run a fair auction between multiple competing ads
- Budget efficiency: advertisers set target CPA (cost per acquisition) — accurate CTR prediction prevents wasted spend on low-converting impressions
- User experience: showing ads predicted to be clicked is better user experience than showing random or purely highest-bid ads
- Sparsity: most (user, ad) pairs are never observed — the model must generalize to unseen combinations from feature interactions
- Scale: platforms serve billions of ad impressions per day — the model must make predictions in under 10ms per impression
- Calibration: predicted CTR = 0.5% should mean approximately 0.5% of users actually click — miscalibration breaks the auction economics
Intuition
CTR prediction is like an experienced gambler estimating the probability that a specific bet will pay off, based on everything they know about the context.
The 'bet' is: show this specific ad to this specific user right now. The payoff is: the user clicks on the ad.
The naive gambler (historical average CTR): 'This ad campaign has a 2% CTR on average. I'll predict 2% for every impression.' This ignores context completely — the same ad might have 8% CTR for users who searched for the product and 0.1% CTR for completely unrelated users.
The smart gambler (CTR prediction model): considers every signal available:
- Who is this user? (demographics, past clicks, purchase history)
- What is the ad? (category, advertiser, creative, landing page)
- What is the context? (platform, time of day, device, current page)
- What is the history of this user seeing this category of ad?
- Have similar users clicked on similar ads in similar contexts?
The calibration requirement (unique to ads): the smart gambler's estimate must be accurate in absolute terms, not just relative. If they say '5% click probability' for 100 impressions, approximately 5 users must actually click. A model that's good at ranking (ad A should rank above ad B) but bad at calibration (estimates 5% but actual rate is 2%) breaks the auction — advertisers who bid based on predicted CTR will overpay.
Analogy
CTR prediction is like an insurance actuary estimating the probability that a specific policyholder will file a claim.
An actuary doesn't just use 'average claim rate for this coverage type.' They consider: age, health history, location, type of coverage, claims history, and interactions between these factors (young + sports car + city = higher risk).
Wide features (like rule-based insurance factors): 'Young driver in NYC — automatically apply the high-risk city premium.' Direct memorization of known high-risk combinations.
Deep features (like actuarial pattern discovery): 'We've discovered that small business owners who work from home and own hybrid vehicles have unexpectedly low claim rates — even though no individual feature predicted this.' Generalizing from observed patterns to unseen combinations.
Calibration requirement: the actuary's probability estimate must be right in absolute terms. If they say '3% claim probability' and price the premium accordingly, but actual claims are 8%, the insurance company loses money. Just being able to rank customers by risk (higher vs. lower) is not enough — the actual probability must be accurate for pricing to work.
The scale challenge: insurance companies have millions of policies. Ad platforms have trillions of impressions per year. The prediction model must handle this scale while maintaining per-user, per-ad, per-context accuracy.
Technical explanation
STEP 1 — PROBLEM FRAMING:
CTR prediction is binary classification at massive scale: Input: (user, ad, context) feature vector Output: P(click) ∈ [0, 1] Label: 1 if user clicked, 0 otherwise Training data: impression logs (trillions per year at major platforms)
Scale assumptions: 10M daily active users, 100 ad impressions per user per day, = 1B impressions/day, < 10ms prediction latency per impression.
Two evaluation goals (distinct):
- Ranking quality: does P(click | ad_A) > P(click | ad_B) when ad_A is more relevant? Metric: AUC-ROC
- Calibration quality: if P(click) = 0.02 for 1000 impressions, ~20 should click Metric: expected calibration error (ECE), reliability diagrams
Both are necessary. A perfectly ranked model with wrong absolute probabilities breaks the auction. A well-calibrated model with poor ranking shows the wrong ads.
STEP 2 — FEATURE ENGINEERING:
CTR prediction relies on extremely sparse features — billions of possible (user, ad) combinations, most never seen in training.
User features:
- Demographic bucket: age_group, gender, country, device_type
- Behavioral history: past 7/30/90-day CTR by category, past ad formats clicked
- User_id embedding (dense): learned representation of user's ad preferences
Ad features:
- Advertiser_id, ad_category, ad_format (banner, video, native)
- Ad creative features: title embeddings, image embeddings
- Historical global CTR for this ad (average across all past impressions)
- Ad_id embedding (dense): learned per-ad representation
Context features:
- Platform: web, iOS app, Android app
- Page/app category: what is the user doing right now?
- Time: hour_of_day, day_of_week (CTR varies by time)
- Slot position: first ad slot has higher CTR than third
Cross-features (highest signal, unique to CTR): The interaction between features often carries more signal than individual features. A user who has installed fitness apps AND sees a fitness equipment ad (cross: user_app_category × ad_category = fitness × fitness) is much more likely to click than either feature alone predicts.
Cross-feature examples:
- user_country × ad_language (English ad to French user = low CTR)
- user_device × ad_format (video ad on slow mobile connection = low CTR)
- page_category × ad_category (sports page + sports equipment ad = high CTR)
- user_age_bucket × ad_category (teen + video games = high CTR)
Sparse cross-features (Wide component): user_app_history × ad_id (which specific apps did this user install vs. which specific ad is shown?). This is a huge sparse feature — billions of possible combinations — but it captures direct memorization of 'user type A clicks ad B.'
STEP 3 — MODEL ARCHITECTURE:
Historical evolution: Phase 1 (pre-2010): Logistic regression on hand-crafted features. Fast, interpretable, but can only learn linear patterns.
Phase 2 (2010-2016): GBDT + logistic regression (Facebook, 2014): Step 1: train a GBDT (gradient boosted decision tree) to learn feature interactions and transformations Step 2: use the leaf node outputs of the GBDT as new features Step 3: train logistic regression on these transformed features Why: GBDT automatically discovers important feature interactions and non-linear transformations that hand-crafted features miss.
Phase 3 (2016-present): Wide & Deep (Google, 2016): Wide: logistic regression on sparse cross-features → memorizes seen (user, ad) patterns Deep: dense layers on embeddings of categorical features → generalizes to unseen feature combinations Joint training: W&D trains both components simultaneously output = sigmoid(wide_output + deep_output)
Phase 4 (2018+): Deep neural networks with attention: DIN (Deep Interest Network): attention over user history, weighting past interactions by their relevance to the current ad DIEN (Deep Interest Evolution Network): models temporal evolution of interests DCN (Deep & Cross Network): learns explicit feature cross products automatically
STEP 4 — CALIBRATION:
Why calibration matters: in ad auctions, eCPM = CTR × bid. If CTR is systematically overestimated (model predicts 5% but actual is 2%), the platform shows more of this advertiser's ads than is economically justified, and advertisers who bid based on predicted CTR overpay.
Calibration check: divide all impressions into 10 buckets by predicted CTR. For each bucket, plot avg(predicted CTR) vs. avg(actual click rate). A perfectly calibrated model falls on the y=x diagonal.
Common miscalibration causes:
- Label distribution shift: model trained on last week's data, but CTR seasonally varies (e.g., Black Friday CTR ≠ average weekday CTR)
- Negative sampling: training on a random 10% of non-click impressions (to handle the class imbalance: 98% non-clicks) shifts predicted probabilities
Fix for negative sampling calibration error: If you train on fraction q of negative examples (downsample negatives), the predicted probability p* is biased upward. Correction: p_calibrated = p* / (p* + (1-p*)/q)
Platt scaling (general calibration fix): Train a small logistic regression on a held-out set: calibrated_p = sigmoid(a × logit(p*) + b) Parameters a, b learned on calibration set.
STEP 5 — COLD START FOR NEW ADS:
A brand new ad has no impression or click history → no historical CTR feature.
Mitigation 1 — Content-based prior: Use ad_embedding (title + image features) to find similar past ads. Initialize the new ad's historical CTR prior as the avg CTR of the K most similar historical ads.
Mitigation 2 — Exploration injection: Show the new ad to a random sample of users (regardless of predicted CTR) to collect initial click data. After ~1000 impressions, switch to model-based prediction.
Mitigation 3 — Advertiser-level features: Even if this specific ad is new, the advertiser's past campaigns may have strong history. Use advertiser_id historical CTR as a fallback feature.
Architecture
Wide & Deep Model Architecture (Google, 2016):
Categorical features (sparse): user_id, ad_id, device_type, country, ad_category ↓ ┌──────────────────────────────────────────────────────────┐ │ EMBEDDING LAYER │ │ Map high-cardinality categoricals to dense vectors: │ │ user_id → 256d embedding │ │ ad_id → 128d embedding │ │ country → 32d embedding │ │ (frozen or jointly trained) │ └──────────────────────────────────────────────────────────┘ ↓ ↓ ┌────────────────────┐ ┌────────────────────────┐ │ WIDE COMPONENT │ │ DEEP COMPONENT │ │ │ │ │ │ Input: sparse │ │ Input: concatenated │ │ cross-features │ │ embeddings (dense) │ │ │ │ │ │ e.g. cross( │ │ Dense(512, ReLU) │ │ user_app_history, │ │ Dense(256, ReLU) │ │ ad_category) │ │ Dense(128, ReLU) │ │ │ │ │ │ Logistic │ │ Learns generalizable │ │ Regression │ │ feature interactions │ │ │ │ │ │ Memorizes specific │ │ Generalizes to unseen │ │ known patterns │ │ combinations │ └─────────┬──────────┘ └─────────┬──────────────┘ │ │ └──────────────┬───────────────────┘ │ (add outputs) ▼ sigmoid(wide + deep) ↓ P(click) ∈ [0, 1] ↓ Calibration correction ↓ Calibrated P(click) (used in auction)
Workflow
How to design this system in a 45-minute ML system design interview:
Minutes 1-5 — Requirements clarification: Q: What is the ad format? (search ads, display, video, sponsored) Q: Scale? (daily impressions, users, latency SLA) Q: Objective? (maximize CTR? revenue? balanced?) Q: How mature is the ad inventory? (many new ads needing cold start?) Establish: display ads, 1B impressions/day, < 10ms SLA, maximize eCPM.
Minutes 5-10 — High-level design: Explain the ad auction: eCPM = CTR × bid → show highest eCPM ad. Name the two evaluation requirements: ranking quality AND calibration. Sketch the feature assembly → CTR model → calibration → auction pipeline.
Minutes 10-20 — Feature engineering: Walk through four feature categories (user, ad, context, cross). Emphasize cross-features as the highest-signal category. Explain sparse cross-features (user_app_history × ad_id) for memorization.
Minutes 20-30 — Model architecture: Historical evolution: LR → GBDT+LR → Wide & Deep → DIN. Explain Wide & Deep in detail: what wide memorizes, what deep generalizes. Name the training objective: binary cross-entropy on click/no-click labels.
Minutes 30-40 — Calibration and cold start: Explain why calibration matters for auction pricing. Describe the negative sampling calibration fix. Describe Platt scaling as the general calibration correction. Address cold start: content-based prior + exploration injection.
Minutes 40-45 — Scale and failure modes: Feature store for < 5ms feature serving. Model serving: online inference with quantized model. What happens if CTR model is down? (fall back to historical CTR averages) What happens if calibration drifts? (monitor ECE, trigger Platt recalibration)
Example
# Wide & Deep CTR prediction model (PyTorch) import torch import torch.nn as nn from typing import dict class WideAndDeep(nn.Module): def __init__( self, vocab_sizes: dict[str, int], # {feature_name: num_unique_values} embedding_dims: dict[str, int], # {feature_name: embedding_dim} wide_input_dim: int, # dimension of sparse cross-feature input deep_hidden: list[int] = [512, 256, 128], ): super().__init__() # Embeddings for high-cardinality categorical features self.embeddings = nn.ModuleDict({ name: nn.Embedding(vocab_sizes[name], embedding_dims[name]) for name in embedding_dims }) # Wide component: logistic regression on sparse cross-features self.wide = nn.Linear(wide_input_dim, 1, bias=True) # Deep component: dense layers on concatenated embeddings deep_input_dim = sum(embedding_dims.values()) layers = [] in_dim = deep_input_dim for h in deep_hidden: layers += [nn.Linear(in_dim, h), nn.ReLU(), nn.Dropout(0.1)] in_dim = h layers.append(nn.Linear(in_dim, 1)) self.deep = nn.Sequential(*layers) def forward( self, cat_features: dict[str, torch.Tensor], # {name: [batch_size]} wide_features: torch.Tensor, # [batch_size, wide_input_dim] ) -> torch.Tensor: # [batch_size] # Embed and concatenate for deep component embs = [self.embeddings[name](cat_features[name]) for name in self.embeddings] deep_input = torch.cat(embs, dim=-1) # [B, sum(emb_dims)] wide_out = self.wide(wide_features) # [B, 1] deep_out = self.deep(deep_input) # [B, 1] return torch.sigmoid(wide_out + deep_out).squeeze(-1) # [B] # Training loop sketch def train_step(model, optimizer, batch): cat_feats, wide_feats, labels = batch optimizer.zero_grad() preds = model(cat_feats, wide_feats) loss = nn.functional.binary_cross_entropy(preds, labels.float()) loss.backward() optimizer.step() return loss.item() # Calibration check + Platt scaling correction import numpy as np from sklearn.linear_model import LogisticRegression from sklearn.calibration import calibration_curve def check_calibration(y_true, y_pred_prob, n_bins=10): prob_true, prob_pred = calibration_curve(y_true, y_pred_prob, n_bins=n_bins) ece = np.mean(np.abs(prob_true - prob_pred)) # Expected Calibration Error return {'ece': ece, 'prob_true': prob_true, 'prob_pred': prob_pred} def platt_scale(y_true_cal, y_pred_cal, y_pred_test): 'Fit Platt scaling on calibration set, apply to test predictions.' lr = LogisticRegression(C=1e10) # near-zero regularization # Reshape to 2D: logits of predictions logits = np.log(y_pred_cal / (1 - y_pred_cal + 1e-9)).reshape(-1, 1) lr.fit(logits, y_true_cal) test_logits = np.log(y_pred_test / (1 - y_pred_test + 1e-9)).reshape(-1, 1) return lr.predict_proba(test_logits)[:, 1] # Negative sampling calibration correction def correct_negative_sampling(p_star: float, q: float) -> float: 'Correct P(click) estimate when negatives are downsampled by fraction q.' # q = fraction of negatives kept in training (e.g., 0.1 = 10% of non-clicks) # p_star = model's predicted probability (biased upward due to sampling) return p_star / (p_star + (1 - p_star) / q)
Real-world usage
-
Google Wide & Deep (Cheng et al., 2016): the foundational paper that defined the Wide & Deep architecture for recommendation and CTR prediction. Deployed in Google Play app recommendations. Key contribution: the formal separation of 'memorization' (wide component) and 'generalization' (deep component) as distinct ML capabilities.
-
Facebook/Meta GBDT + LR (He et al., 2014): 'Practical Lessons from Predicting Clicks on Ads at Facebook.' Showed that using GBDT leaf nodes as features for logistic regression significantly outperforms either method alone. Highly influential because it revealed that feature transformation (the GBDT step) was more important than the final classifier.
-
Alibaba DIN (Zhou et al., 2018): Deep Interest Network. Key insight: a user's interest in a specific ad is not captured by their entire historical behavior equally — only the subset of past interactions relevant to the current ad matters. DIN uses attention over user history weighted by relevance to the target ad. Became the standard for industrial CTR prediction after this paper.
-
Twitter/X Ads (2023, open-source): Twitter's open-sourced code reveals use of a feature-crossing approach similar to DCN (Deep & Cross Network) with heavy emphasis on user engagement features (how the user has interacted with content from the same advertiser category).
-
Khang Pham (ML Primer): presents CTR prediction as the ML system design problem most likely to appear in FAANG interviews, emphasizing the calibration requirement ('ranking quality is necessary but not sufficient — the auction requires calibrated probabilities') and the sparse cross-feature design as the key distinguishing factor from standard recommendation system design.
Trade-offs
AUC (ranking) vs. ECE (calibration): AUC measures whether the model correctly ranks ad A above ad B when A is more likely to be clicked. ECE measures whether predicted probabilities are accurate in absolute terms. A model can have excellent AUC and terrible calibration — it correctly ranks ads but its predicted 5% CTR is actually 2% CTR. In ad auctions, both matter: bad ranking means showing the wrong ads; bad calibration means overcharging or undercharging advertisers. You must measure and optimize for both.
Wide vs. Deep only: Wide (logistic regression on cross-features) memorizes known patterns but fails to generalize to unseen (user, ad) combinations. Deep (neural network) generalizes well but struggles to memorize specific high-confidence patterns (the deep component 'averages out' rare but important cross-features). W&D was specifically designed to capture both simultaneously through joint training.
Real-time vs. near-real-time vs. batch training: CTR prediction benefits from real-time feature updates (click 10 minutes ago changes user features) and near-real-time model updates. However, real-time training requires careful engineering (streaming data pipelines, online learning) and risks instability. The production standard: batch model retraining (daily or every few hours) with real-time feature serving (feature store captures recent user behavior).
Exploration vs. exploitation for cold-start ads: showing a new ad to random users (exploration) collects data but wastes impressions on non-targeted users. Showing it only to users predicted to click (exploitation with content-based prior) may reinforce a biased initial estimate. Thompson sampling or UCB-based bandit algorithms provide a principled exploration-exploitation balance for new ads.
Visual explanation
Ad CTR Prediction System — Architecture:
AD IMPRESSION EVENT (user visits page that has an ad slot) │ ▼ ┌────────────────────────────────────────────────────────────┐ │ AD AUCTION TRIGGER │ │ • ad_slot_id: where is this impression? │ │ • user_id: who is seeing it? │ │ • page_context: what page is it on? │ │ • N candidate ads: which advertisers bid for this slot? │ └──────────────────────────────┬─────────────────────────────┘ │ ▼ ┌────────────────────────────────────────────────────────────┐ │ FEATURE ASSEMBLY (< 5ms, all from cache) │ │ │ │ User features (from user feature store): │ │ user_age_bucket, device_type, country, │ │ user_ctr_history (past 7d), ad_category_affinity │ │ │ │ Ad features (from ad metadata store): │ │ advertiser_id, ad_category, ad_format, │ │ historical_ctr (global CTR for this ad) │ │ ad_embedding (title + image features) │ │ │ │ Context features (from request): │ │ hour_of_day, day_of_week, platform (web/mobile/app) │ │ page_category (what is the user currently viewing?) │ │ │ │ Cross features (pre-computed or assembled on-the-fly): │ │ user_category_affinity × ad_category │ │ user_device × ad_format │ │ page_category × ad_category (context-ad relevance) │ └──────────────────────────────┬─────────────────────────────┘ │ ▼ ┌────────────────────────────────────────────────────────────┐ │ CTR PREDICTION MODEL │ │ │ │ Wide component (memorization): │ │ Logistic regression on sparse cross-features │ │ Input: cross(user_app_history, ad_id) → one-hot │ │ Learns: 'user who installed fitness apps → fitness ads' │ │ │ │ Deep component (generalization): │ │ Embedding → dense layers → sigmoid output │ │ Input: embeddings of categorical features │ │ Learns: unseen feature combinations from embeddings │ │ │ │ Output: P(click) ∈ [0, 1] │ │ │ │ Then: calibration correction (Platt scaling or isotonic) │ └──────────────────────────────┬─────────────────────────────┘ │ ▼ ┌────────────────────────────────────────────────────────────┐ │ AD AUCTION │ │ For each candidate ad: │ │ eCPM = predicted_CTR × bid_price × 1000 │ │ Rank ads by eCPM → show the winner │ │ Charge winning advertiser: second_price × (eCPM_2 / CTR) │ └──────────────────────────────┬─────────────────────────────┘ │ ▼ AD SHOWN TO USER → log click/no-click
Offline Training Loop:
Impression + Click Logs └─► Label: clicked=1, not_clicked=0 └─► Feature join from feature store └─► Train Wide & Deep (daily batch or near-real-time) └─► Calibration check → Platt scaling if needed └─► Shadow test → A/B test → progressive rollout
Advantages
- —
Wide & Deep jointly trains memorization (wide) and generalization (deep) in a single model, addressing a fundamental tension in CTR prediction that earlier approaches (LR alone or NN alone) couldn't resolve
- —
The auction framework (eCPM = CTR × bid) creates a natural economic incentive alignment: advertisers with more relevant ads win impressions at lower effective cost, improving both platform and advertiser outcomes
- —
Calibration correction (Platt scaling, negative sampling correction) is a post-hoc fix that doesn't require retraining the model when calibration drifts — this is operationally critical in production
- —
The feature store approach (pre-compute and cache user/ad/context features) reduces online serving latency from seconds to milliseconds, enabling the tight < 10ms auction SLA
- —
AUC-ROC for ranking quality and ECE for calibration quality are complementary metrics that together provide a complete picture of model quality — neither alone is sufficient
Disadvantages
- —
Sparse cross-features (user_app_history × ad_id) create an extremely high-dimensional input space — billions of possible feature combinations that are mostly zero (sparse). Memory and computation cost scales with feature space, requiring careful feature selection
- —
CTR prediction models require retraining extremely frequently (daily or near-real-time) because click patterns shift rapidly with news events, seasonal patterns, and new ad creatives. Stale models degrade quickly
- —
The calibration requirement adds a post-hoc step that can break if the calibration set distribution differs from production distribution — requires ongoing monitoring
- —
Cold start for new advertisers and new ad formats requires heuristic approaches (content-based priors, exploration injection) that are suboptimal and require ongoing maintenance
- —
Negative sampling (downsampling non-click impressions to handle class imbalance) introduces systematic calibration bias that must be explicitly corrected — a source of bugs in production implementations
Common mistakes
- —
Ignoring calibration. Many engineers treat CTR prediction as a pure ranking problem and optimize only for AUC. In ad systems, calibration is equally critical — uncalibrated predictions break the auction pricing. A complete design always addresses: (1) how to detect calibration drift (reliability diagrams, ECE monitoring), (2) how to fix miscalibration (Platt scaling, negative sampling correction).
- —
Not addressing the negative sampling bias. Training data for CTR prediction has massive class imbalance: ~2% clicks, ~98% non-clicks. To handle this, practitioners often downsample non-click impressions. But this creates a calibration bias: the model predicts higher CTR than the actual rate because negative examples are underrepresented. A complete design explicitly corrects for this using the formula: p_corrected = p* / (p* + (1-p*)/q), where q is the sampling fraction.
- —
Conflating CTR prediction with recommendation. Recommendation optimizes for user engagement (maximize probability of watch/click/like). CTR prediction additionally requires calibrated probabilities for auction pricing. The evaluation metrics are different: recommendation uses NDCG/recall@K; CTR prediction uses AUC (ranking) + ECE (calibration). Treating them as identical leads to missing the calibration requirement.
- —
Not addressing cross-features. CTR prediction models that only use independent user features and ad features miss the most important signal category. The interaction between features (user_app_history × ad_category, page_context × ad_format) often carries more predictive signal than any individual feature. The Wide component in W&D exists specifically to capture sparse but high-signal cross-features.
- —
Forgetting the serving latency constraint. CTR prediction must happen within an ad auction in < 10ms (sometimes < 5ms). A complex model that achieves better AUC but runs in 100ms is unusable in production. Feature serving (cached in Redis), model quantization (int8 or fp16), and batched prediction across the N candidate ads in an auction are all necessary design considerations.
🎤 Interview questions
Design a click-through rate prediction system for a display advertising platform. Explain what CTR prediction is, why it's necessary for ad auctions, and walk me through the complete system design.
Explain the Wide & Deep architecture. What does the wide component memorize? What does the deep component generalize? Why is joint training better than training them separately?
What is the calibration problem in CTR prediction, and why does it matter for ad auctions? How would you detect and fix miscalibration in a production system?
How does CTR prediction differ from a recommendation system? Name at least two design differences that arise from the distinction.
📂 Subtopics
Problem Framing and Feature Engineering
What CTR prediction is, why it's necessary for ad auctions, and the four feature categories that power a CTR model — with emphasis on cross-features as the highest-signal input type.
~40 min
Wide & Deep Architecture and Model Evolution
The evolution from logistic regression to GBDT+LR to Wide & Deep to attention-based models, with a deep dive on what each component contributes and why joint training matters.
~45 min
Calibration, Cold Start, and Production Serving
Why calibration is a first-class requirement for ad CTR models, how to detect and fix miscalibration, how to handle new ads with no history, and the serving architecture that meets a < 10ms auction SLA.
~40 min