Wide & Deep Architecture and Model Evolution

Why Simple Logistic Regression Isn't Enough

Logistic regression (LR) on sparse features was the baseline from 2008-2014:

• Fast inference (dot product + sigmoid)
• Interpretable weights
• Handles sparse features naturally
• Limitation: can only learn linear relationships — cannot capture the interaction between user_app_history and ad_category unless that cross-feature is manually engineered

GBDT + LR (Facebook, 2014)

The key insight: gradient boosted decision trees are excellent at discovering feature transformations and interactions automatically.

Algorithm:

1. Train a GBDT on (user, ad, context) features → learns decision trees
2. For each training example, record which leaf node it falls into for each tree (a 500-tree forest → 500 leaf node IDs per example → one-hot encoding)
3. Use these 500 leaf node one-hot encodings as new features
4. Train logistic regression on the leaf node features

Why it works: GBDT automatically discovers non-linear transformations (e.g., 'user_age < 25 AND ad_category = gaming') as decision tree paths. The LR then learns a linear combination of these discovered patterns.

Wide & Deep Learning (Google, 2016)

The paper formalized the key tension in CTR prediction:

• Memorization: learning to remember specific seen patterns (user X always clicks ads from advertiser Y in context Z)
• Generalization: learning to predict for unseen (user, ad, context) combinations by generalizing from similar observed combinations

Neither LR nor neural networks alone solve both:

• LR: good at memorization (sparse features), poor at generalization (linear)
• NN: good at generalization (embeddings + dense layers), poor at memorizing specific rare but high-signal patterns (embeddings smooth them out)

Wide & Deep solves this with two jointly trained components:

Wide component (memorization):

Input: cross(user_app_history, ad_id) → very sparse, high-dimensional
Model: linear model (logistic regression)
What it learns: 'users who installed this app click this ad'
Strength: directly memorizes specific (user type, ad) patterns
Weakness: can only generalize linearly; misses unseen combinations

Deep component (generalization):

Input: embeddings of user_id, ad_id, country, category (dense vectors)
Model: 3-layer MLP (512 → 256 → 128 → 1)
What it learns: 'users with this type of history prefer this type of ad'
Strength: generalizes to unseen combinations via embedding similarity
Weakness: may miss rare but highly predictive specific patterns

Joint training: both components are trained simultaneously, sharing the same binary cross-entropy loss on click labels. The gradient flows through both paths, allowing each to specialize on what it does best while the combined output benefits from both.

Beyond W&D: Attention-Based Approaches

DIN (Deep Interest Network, Alibaba 2018): Key insight: not all of a user's history is equally relevant to the current ad. A user who bought running shoes 2 years ago and a gaming keyboard last week — for a sports drink ad, the running shoes are more relevant than the keyboard.

DIN adds an attention mechanism over user history:

attention_weight(past_item, target_ad) = softmax(f(past_item_emb, ad_emb))
user_rep = Σ attention_weight × past_item_emb  ← weighted sum

This produces a dynamic user representation that focuses on the subset of history most relevant to the current ad, rather than averaging over all history.