Case Study: Content Recommendation System Design
End-to-end design of a large-scale video/content recommendation system — the pattern behind YouTube, Netflix, and TikTok. Covers scoping and metric selection, two-tower retrieval architecture, deep ranking with multi-objective optimization, cold-start handling, and the full evaluation + feedback loop. Grounded in Khang Pham's video recommendation case study.
▶📚 Prerequisites(3)
🎓 Learning objectives
- •Scope a content recommendation system: define the problem, scale assumptions, and success metrics before touching model design
- •Explain why watch time or long-term engagement is a better optimization target than raw click-through rate for video recommendation
- •Design the two-stage retrieval-then-ranking pipeline and explain the role of each stage at the scale of millions of items
- •Describe the cold-start problem for both new users and new content, and explain at least two mitigation strategies for each
- •Identify the key feature categories used in a video ranking model and explain why cross-features between user and item matter
- •Explain the filter bubble problem and describe how diversity and exploration mechanisms counteract it
- •Design an A/B testing framework for a recommendation system, including what metrics to track and for how long
What is it?
A content recommendation system automatically selects which items from a large catalog to surface to each user, personalized to their interests. The goal is to connect the right content to the right person at the right time — increasing engagement, retention, and satisfaction.
Khang Pham's video recommendation case study (ML Primer) describes the archetypal large-scale system: a catalog of millions of videos, hundreds of millions of users, and a requirement to return a ranked list in under 200ms. This same pattern applies to Netflix (movies), Spotify (songs), Amazon (products), and TikTok (short videos).
The system has three interacting concerns:
- Retrieval — find hundreds of plausible candidates from millions in milliseconds
- Ranking — score those candidates precisely using rich user-item features
- Serving — apply final adjustments (diversity, freshness, exploration) and return a list
Each concern has different scale requirements, failure modes, and machine learning sub-problems that together form a complete ML system design interview answer.
Why it exists
Without recommendation, users face the discovery problem: a catalog of 10 million videos is worse than useless if you can't find what you want. The system turns abundance into value by filtering to relevance.
The scale argument: scoring 10M videos × 100M users at query time is 10^15 operations — impossible in real time. The recommendation system makes this tractable through the retrieval funnel (narrow to hundreds before scoring).
The business argument: recommendation drives disproportionate engagement. YouTube's recommendation engine drives over 70% of watch time. Netflix reports that 80% of content discovered comes through recommendations, not search. A 1% improvement in recommendation quality translates directly to retention metrics.
The technical argument: recommendation systems are the canonical large-scale ML system design problem. They require every component of a mature ML platform: feature stores, embedding services, model serving, A/B testing, and feedback loops.
Problem it solves
- Discovery: the catalog has 10M videos but users only see a homepage — how do you decide what to show?
- Personalization: two users with different tastes should see completely different homepages.
- Scale: 100M users each getting a fresh recommendation in < 200ms means you can't recompute from scratch per request.
- Cold start: new users have no history; new videos have no engagement data — both need recommendations anyway.
- Feedback loop: the popular gets more popular. Without intervention, the system creates a filter bubble where users only see variants of what they've already watched.
- Metric gaming: if you optimize directly for CTR (clicks), users click clickbait titles and then immediately leave — improving CTR while degrading real experience.
Intuition
Designing a recommendation system is like running a very sophisticated matchmaking service at internet scale.
Stage 1 — Candidate generation is the matchmaker's Rolodex lookup: 'Given what I know about this person, which 200 of our 10 million members might they connect with?' Fast, approximate, high recall. You'd rather include a few bad matches than miss a good one.
Stage 2 — Ranking is the matchmaker carefully evaluating those 200 candidates: checking compatibility across many dimensions (interests, context, timing), scoring each pair, and producing a sorted list. Slower, more precise, done only on the short list.
Stage 3 — Serving is the matchmaker's final judgment: 'I'll recommend these 10 people, but let me make sure there's variety (not all engineers), include one promising newcomer, and avoid recommending the same person three times in a row.'
The cold-start problem is like getting a new client with no dating history — you fall back on demographics and stated preferences until behavioral data accumulates.
The filter bubble is like a matchmaker who only introduces you to people similar to your last relationship — you never discover new types of people you might love.
Analogy
A video recommendation system is like a personal librarian working at internet scale.
A library has 10 million books, and the librarian's job is to have 10 books ready for you when you walk in, personalized to your reading history and current mood.
The bad librarian (pure popularity): 'Everyone's reading these 10 bestsellers. Here you go.' Fast, but completely ignores who you are.
The good librarian (personalized recommendation): consults your reading history, knows you love mystery novels but not horror, notices you've been reading travel books lately, and has a shortlist of 10 books waiting before you finish asking.
The excellent librarian (full system): does all of the above, but also: includes one book from an author you've never read (exploration), makes sure not all 10 books are mystery (diversity), prioritizes recently published books for topics you're currently interested in (freshness), and uses your reactions to past recommendations to improve future ones (feedback loop).
The challenge: this librarian serves 100 million readers simultaneously, and has 3 seconds to prepare your personalized list.
Technical explanation
STEP 1 — PROBLEM FRAMING:
Before any model design, clarify:
- What is being recommended? Videos (long-form, short-form?), articles, products, songs?
- Context? Homepage, post-watch autoplay, search results, notification?
- Scale? 100M users, 10M items, 200ms SLA?
- Success metric? This is the most important design decision.
Metric selection is critical:
- CTR (click-through rate): easy to measure but gameable — clickbait maximizes it
- Watch time: better proxy for value — users signal approval by continuing to watch
- Completion rate: % of video watched — good for quality signal
- Long-term retention: do users come back tomorrow? — best business metric, hardest to optimize directly
- Satisfaction survey: ask users to rate recommendations — low volume but high signal
In practice: train on watch time (primary), likes/shares (positive signals), early abandonment (negative signal). A/B test against long-term retention as the guardrail metric that watch-time optimization doesn't degrade.
STEP 2 — CANDIDATE GENERATION:
Two-tower retrieval (primary source): User tower: user_id emb + demographic features + watch history aggregate emb Item tower: video_id emb + category + creator + title emb + duration Offline: train with in-batch negatives (sampled unwatch videos as negatives) Serving: pre-compute all video embeddings → FAISS HNSW index At query time: compute user_emb → ANN query → top-200 candidates
Co-watch collaborative filtering: 'Users who watched video A within the same session also watched video B' Builds item-item co-occurrence graph from session logs Given user's recent watches, retrieve the top-N co-watched items Strength: captures immediate 'what to watch next' intent
Trending + fresh content source: Separate retrieval path for recently uploaded or rapidly trending content Without this, new videos starve — they can't appear in two-tower results until they have engagement history for the item tower
STEP 3 — RANKING MODEL:
Architecture: multi-task deep neural network Input: user features + item features + context features + cross-features Shared bottom: dense layers learning joint representation Task-specific heads: Head 1: P(watch_time > threshold) — regression or classification Head 2: P(like | watched) Head 3: P(share | watched) Head 4: P(early_abandon | started) — penalty
Combined score = w1 × predicted_watch_time + w2 × P(like) + w3 × P(share) - w4 × P(early_abandon)
Weights {w1..w4} are tuned based on A/B tests and business goals.
Key features: User features: demographics, device, time-of-day, historical category affinity Item features: video category, duration, creator subscribe count, global avg watch time Cross-features: user_age × video_category, user_location × video_language, user_recent_category × video_category (session intent) Context features: user's last 5 watched videos (session context)
Position bias: items shown first are clicked more regardless of quality.
Fix: include position as a training feature, set position=1 for all at inference.
STEP 4 — COLD START:
New user cold start: Onboarding flow: ask user to select 3-5 topics or follow a few creators Use demographic embedding (age, location) as a proxy user embedding Show popular content in selected topics until behavioral data accumulates Transition: as user watches 5+ videos, gradually shift to personalized retrieval
New item cold start: New videos can't appear in two-tower results until they have an embedding Fallback: use content-based embedding (title + description → sentence transformer) Exploration injection: randomly surface new items to a small fraction of users Fast embedding update: run item tower on new video immediately on upload
STEP 5 — EVALUATION AND FEEDBACK LOOP:
Offline metrics: AUC on held-out (user, video, engaged/not) pairs Online metrics (A/B test): watch time, CTR, 7-day retention, diversity score
A/B testing design: Treatment: new recommendation model Control: current production model Traffic split: 5% treatment to limit exposure if model is worse Duration: minimum 2 weeks (capture day-of-week effects) Primary metric: average watch time per session Guardrail metrics: 7-day retention (don't sacrifice long-term for short-term), creator diversity (don't over-concentrate on a few creators)
Feedback loop closure: Log: impression_id, user_id, video_id, position, watch_seconds, liked, shared Daily batch job: ingest new logs → retrain two-tower (or fine-tune) → update FAISS index Near-real-time: session clicks update user embedding within the session
Architecture
Data and Model Infrastructure:
┌──────────────────────────────────────────────────────────────┐ │ OFFLINE SYSTEMS │ │ │ │ ┌─────────────────┐ ┌──────────────────────────────────┐ │ │ │ Event Stream │ │ Feature Engineering │ │ │ │ (Kafka/Flink) │───►│ • User watch history aggregates │ │ │ │ watch, like, │ │ • Video global engagement stats │ │ │ │ skip, upload │ │ • Cross-feature precomputation │ │ │ └─────────────────┘ └────────────────┬─────────────────┘ │ │ │ │ │ ┌───────────────────────────────────────▼─────────────────┐ │ │ │ Feature Store (Redis + offline batch store) │ │ │ │ • user_id → {age, location, category_affinity_vec} │ │ │ │ • video_id → {category, creator_id, duration, emb} │ │ │ └───────────────────────────────────────┬─────────────────┘ │ │ │ │ │ ┌────────────────┐ ┌────────────────▼──────────────────┐ │ │ │ Model Training │ │ Model Registry + FAISS Index │ │ │ │ (daily batch) │───►│ • two-tower weights │ │ │ │ Two-tower │ │ • ranking model weights │ │ │ │ Ranking model │ │ • pre-built ANN index (video embs)│ │ │ └────────────────┘ └───────────────────────────────────┘ │ └──────────────────────────────────────────────────────────────┘ │ (daily update deployed) ┌──────────────────────────────▼───────────────────────────────┐ │ ONLINE SERVING (< 200ms per request) │ │ │ │ Request → Feature Assembly (feature store lookup) │ │ → Candidate Generation (parallel: ANN + CF + fresh) │ │ → Ranking (deep model scoring on ~300 candidates) │ │ → Re-ranking (diversity, freshness, exploration) │ │ → Response (top 20 videos) │ └──────────────────────────────────────────────────────────────┘ │ (log: impression, position, user, video) ┌────────▼─────────────────────────────────────────────────────┐ │ FEEDBACK LOOP (async) │ │ Watch events → training data → updated models (next day) │ └──────────────────────────────────────────────────────────────┘
Workflow
How to design this system in a 45-minute interview:
Minutes 1-5 — Requirements clarification: Q: What type of content? (videos, articles, products) Q: What is the context? (homepage, post-watch, notifications) Q: Scale? Users, items, QPS? Q: What is the primary success metric? (watch time, retention, CTR) Establish: I'll design for 100M users, 10M videos, optimize watch time.
Minutes 5-10 — High-level design: Sketch the three-stage pipeline on the whiteboard. Name the components: candidate generation, ranking, re-ranking, serving. Explain the funnel: 10M → 300 → 50 → 20.
Minutes 10-20 — Candidate generation deep dive: Two-tower model architecture and training. How item embeddings are pre-computed and stored in FAISS. Additional sources: co-watch CF, trending/fresh. Cold start for new users and new videos.
Minutes 20-30 — Ranking model deep dive: Feature categories: user, item, cross, context. Multi-task head: predict watch time + like + share - skip. Position bias and how to correct it. Why watch time is preferred over CTR.
Minutes 30-40 — Serving, evaluation, and tradeoffs: Re-ranking: diversity, freshness, exploration. A/B testing design (metrics, guardrails, duration). Feedback loop: how impressions + watch events retrain models. Filter bubble: what it is and how exploration addresses it.
Minutes 40-45 — Scaling and failure modes: Feature store for low-latency feature serving. FAISS HNSW for approximate nearest neighbor at scale. What happens if the ranking model is down? (fall back to two-tower scores) What happens if the FAISS index is stale? (serve from last known good)
Example
# Illustrative multi-task ranking model (PyTorch) import torch import torch.nn as nn class VideoRankingModel(nn.Module): def __init__(self, user_feature_dim: int, item_feature_dim: int, hidden_dim: int = 256): super().__init__() input_dim = user_feature_dim + item_feature_dim # Shared bottom: learns joint user-item representation self.shared = nn.Sequential( nn.Linear(input_dim, hidden_dim), nn.ReLU(), nn.Linear(hidden_dim, hidden_dim), nn.ReLU(), ) # Task-specific heads self.watch_time_head = nn.Linear(hidden_dim, 1) # regression self.like_head = nn.Linear(hidden_dim, 1) # binary self.share_head = nn.Linear(hidden_dim, 1) # binary self.abandon_head = nn.Linear(hidden_dim, 1) # binary (penalty) def forward(self, user_features: torch.Tensor, item_features: torch.Tensor): x = torch.cat([user_features, item_features], dim=-1) h = self.shared(x) return { 'watch_time': self.watch_time_head(h).squeeze(-1), 'like_prob': torch.sigmoid(self.like_head(h)).squeeze(-1), 'share_prob': torch.sigmoid(self.share_head(h)).squeeze(-1), 'abandon_prob': torch.sigmoid(self.abandon_head(h)).squeeze(-1), } def combined_score(self, user_features, item_features, w_watch=1.0, w_like=0.3, w_share=0.2, w_abandon=0.5): preds = self.forward(user_features, item_features) return ( w_watch * preds['watch_time'] + w_like * preds['like_prob'] + w_share * preds['share_prob'] - w_abandon * preds['abandon_prob'] ) # Simplified cold-start fallback def get_candidates(user_id: str, user_has_history: bool, two_tower_index, cf_index, trending_pool): candidates = set() if user_has_history: # Primary: personalized two-tower retrieval user_emb = compute_user_embedding(user_id) candidates.update(two_tower_index.search(user_emb, k=200)) # Secondary: co-watch collaborative filtering recent_watches = get_recent_watches(user_id, n=5) for video_id in recent_watches: candidates.update(cf_index.get_similar(video_id, k=20)) else: # Cold start: topic-based popularity user_topics = get_onboarding_topics(user_id) # from signup flow candidates.update(trending_pool.get_by_topics(user_topics, k=100)) # Always include some fresh content candidates.update(trending_pool.get_recent(k=50)) return list(candidates)
Real-world usage
-
YouTube (Covington et al., 2016): the paper that defined the modern recommendation pipeline. Two-stage architecture: candidate generation via deep collaborative filtering network → ranking via second DNN. Optimizes for expected watch time rather than clicks. Key insight: 'clicks are an extremely noisy signal for video quality; watch time is a much better proxy.'
-
TikTok (For You Page): famously efficient cold start. New users get recommendations immediately without history. The system uses geographic trending, device language, and the first few interactions to bootstrap a profile rapidly. New videos are probed to small audiences and rapidly promoted if engagement metrics exceed thresholds — addressing the new-item cold start via explicit exploration.
-
Netflix: well-documented use of multi-objective optimization balancing short-term engagement (predicted watch) with long-term retention. Netflix famously does NOT optimize for raw watch time alone — they added 'hours of joy' as a metric, adjusting for content that users feel good about having watched vs. content they regret binge-watching.
-
Spotify Discover Weekly: uses collaborative filtering on listening sessions to create user taste profiles, then finds unseen tracks in similar users' libraries. Delivers recommendations weekly to build anticipation — a deliberate product choice that constrains the recommendation frequency.
-
Khang Pham (ML Primer): describes the canonical design pattern with emphasis on metric selection ('the choice of optimization target is the single most impactful decision in system design') and the multi-source candidate generation approach as the reliable baseline for new systems.
Trade-offs
CTR vs. watch time vs. long-term retention: CTR is the easiest to measure but most gameable metric — clickbait maximizes it while degrading satisfaction. Watch time correlates better with user intent. Long-term retention (7-day return rate) is the best business metric but too delayed for direct optimization. The standard approach: train on watch time, use retention as an A/B test guardrail metric.
Exploration vs. exploitation: exploration slots (show new/uncertain items) reduce short-term CTR and watch time but are necessary to avoid the popularity feedback loop. Netflix, YouTube, and Spotify all reserve explicit exploration capacity. The amount is tunable; 5-10% exploration is typical.
Retrieval recall vs. ranking cost: more candidates means better coverage but more ranking compute. Ranking 1,000 candidates is 10x more expensive than ranking 100. The practical target is 200-500 candidates — enough for high recall, manageable for real-time ranking.
Real-time vs. batch features: session-context features (what the user watched in the last 10 minutes) dramatically improve recommendation relevance but require real-time feature serving infrastructure. Batch features (all-time user profile) are cheaper but lag behind current intent. Production systems use both: batch features from the feature store + real-time session features from the request.
Visual explanation
Content Recommendation System — Full Architecture:
USER REQUEST (homepage load) │ ▼ ┌────────────────────────────────────────────────────────────┐ │ ONLINE FEATURE ASSEMBLY │ │ • Fetch user embedding from cache │ │ • Fetch user profile features (age, location, device) │ │ • Fetch session context (last 5 watched, time-of-day) │ │ • Fetch user's watch history embedding (from feature store)│ └──────────────────────────────┬─────────────────────────────┘ │ ▼ ┌────────────────────────────────────────────────────────────┐ │ CANDIDATE GENERATION (parallel, multiple sources) │ │ │ │ Source A: Two-Tower ANN 200 candidates │ │ user_emb → FAISS query (personalized) │ │ │ │ Source B: Co-watch CF 100 candidates │ │ 'users who watched X also watched...' │ │ │ │ Source C: Content-based 50 candidates │ │ similar to recently watched items │ │ │ │ Source D: Trending / Fresh 50 candidates │ │ viral videos + newly uploaded in user's topics │ │ │ │ ────► MERGE + DEDUPLICATE ◄──── ~300-400 candidates total│ └──────────────────────────────┬─────────────────────────────┘ │ ▼ ┌────────────────────────────────────────────────────────────┐ │ DEEP RANKING MODEL │ │ │ │ For each candidate, compute: │ │ • User features × item features (cross-features) │ │ • Predicted watch time (primary) │ │ • Predicted like/share probability (secondary) │ │ • Predicted skip/abandon probability (penalty) │ │ │ │ Combined score = w1×watch_time + w2×like - w3×skip │ │ │ │ ────► TOP 50 candidates ranked by score │ └──────────────────────────────┬─────────────────────────────┘ │ ▼ ┌────────────────────────────────────────────────────────────┐ │ RE-RANKING + SERVING POLICY │ │ • Diversity: max 2 videos per creator │ │ • Freshness boost: favor videos < 48h old │ │ • Exploration slot: 1 of 20 positions reserved │ │ • Position bias correction │ │ • Safety/policy filter (remove violating content) │ └──────────────────────────────┬─────────────────────────────┘ │ ▼ FINAL LIST (20 videos) → User
Offline Training Loop:
Impressions + Watch Events └─► Feature Engineering ─► Two-Tower Training ─► Updated Embeddings └─► Ranking Model Training ─► Updated Weights (batch update: daily or near-real-time)
Advantages
- —
Multi-source candidate generation (two-tower + CF + content-based + trending) ensures broad coverage — no single retrieval mechanism misses all good candidates
- —
Multi-task ranking with separate heads for different engagement signals (watch time, like, share, abandon) allows fine-tuning the combined score to match business goals without retraining the model
- —
Exploration slots in the final serving layer prevent filter bubbles without requiring algorithmic changes to the retrieval or ranking models
- —
The separation of retrieval (optimize recall) and ranking (optimize precision) allows each stage to be improved, scaled, and A/B tested independently
- —
Feature store architecture makes user and item features available to both online serving (< 5ms lookup) and offline training (batch refresh)
Disadvantages
- —
Two-tower retrieval is an approximate nearest neighbor search — it can miss items that a cross-encoder would rank highly, creating a retrieval recall ceiling that ranking cannot fix
- —
Multi-task ranking requires careful weight tuning: if w_abandon (penalty) is too high, the model avoids challenging or novel content that has higher abandon rates but is genuinely valuable
- —
Cold start remains unsolved for truly new users and truly new content — onboarding heuristics and exploration injection are patches, not solutions
- —
The feedback loop is biased: only impressions are logged, creating systematic blind spots for content the system never surfaces
- —
Watch time as an optimization target can inadvertently reward addictive or emotionally engaging content (outrage, anxiety) over genuinely informative or enriching content
Common mistakes
- —
Optimizing for CTR and shipping the result. CTR is a proxy metric, not the goal. A recommendation system that maximizes CTR will produce a feed full of clickbait thumbnails and misleading titles. Always define the metric hierarchy: primary (watch time or satisfaction), secondary (CTR, likes), guardrail (7-day retention, diversity). Never ship based on CTR improvement alone.
- —
Ignoring cold start in the design. A common interview mistake is to design the two-tower → ranking pipeline and declare it done. Cold start is a first-class problem: you have no embeddings for new users and no engagement data for new videos. A complete design explicitly addresses both (onboarding flow + popularity fallback for users; content-based embedding + exploration injection for videos).
- —
Building a single retrieval source. A two-tower model alone misses trending content, co-watch intent signals, and fresh uploads. Production systems use 4-6 candidate sources merged before ranking. Diversity of retrieval sources is as important as retrieval accuracy within a single source.
- —
Forgetting to account for position bias in training. If your training data comes from production impressions where position-1 items have 5x the click rate, your ranking model will learn to score position-1 items higher regardless of quality. Fix: add position as a training feature, hold it constant at inference time.
- —
Treating the system as static after launch. The recommendation model degrades as user behavior shifts, content distribution changes, and trends emerge. Without a feedback loop (periodic retraining on new interaction logs), model quality decays. Production systems retrain daily or even hourly for fast-moving domains.
🎤 Interview questions
Design a video recommendation system for a platform with 100M users and 10M videos. Walk me through every component from the user request to the final list of recommended videos.
Why is watch time a better optimization target than click-through rate for video recommendations? What are the risks of optimizing for watch time?
How would you handle the cold-start problem for both new users and new videos?
Describe how you would run an A/B test on a new recommendation model. What metrics would you track and what would a successful result look like?
📂 Subtopics
Problem Framing, Metrics, and Scale
How to scope a recommendation system: defining the problem, choosing the right success metric, establishing scale assumptions, and identifying the core challenges before touching any model design.
~40 min
Two-Tower Retrieval and Deep Ranking
Designing the candidate generation and ranking stages: two-tower architecture, multi-source retrieval, multi-task ranking model, and feature engineering.
~50 min
Cold Start, Evaluation, and System Tradeoffs
Solving cold start for new users and new videos, designing the A/B testing framework, closing the feedback loop, and navigating the key system tradeoffs.
~40 min