Problem Framing, Metrics, and Scale

The most common mistake in recommendation system design is jumping straight to model architecture before clarifying what problem you're solving.

Step 1: Define the recommendation context

Context determines the design:

• Homepage: user is browsing — want diverse, high-engagement content
• Post-watch autoplay: user finished a video — want continuation intent, high relevance
• Search results reranking: user has explicit intent — relevance > personalization
• Email/push notification: user is off-platform — high bar for relevance

Step 2: Establish scale

Typical assumptions to state explicitly:

• Users: 100M active users
• Items: 10M videos (growing at 500K/day with uploads)
• QPS: homepage loads = ~50K/second at peak
• Latency SLA: 200ms end-to-end

Scale determines architecture: at 10M videos you can't score everything at query time.

Step 3: Choose the primary success metric

This is the highest-leverage design decision:

Metric	Pros	Cons
CTR	Easy to measure, fast feedback	Gameable — clickbait maximizes it
Watch time	Reflects actual engagement	Rewards long videos, addictive content
Completion rate	Quality signal	Penalizes long-form content
Like/share rate	Explicit positive signal	Selection bias (only engaged users rate)
7-day retention	Best business proxy	Too delayed for training signal

Standard answer: optimize for watch time (primary training signal), A/B test against 7-day retention (guardrail — ensure watch time gain doesn't sacrifice long-term engagement). Track diversity score as a secondary guardrail.

Step 4: Identify the core challenges

1. Scale: can't score 10M videos per request
2. Personalization: 100M users need 100M different recommendations
3. Cold start: new users (no history) and new videos (no engagement data)
4. Metric gaming / filter bubble: over-optimizing for short-term engagement
5. Freshness: the model should surface newly uploaded content

Stating these explicitly in the first 5 minutes of a design interview shows architectural maturity — most engineers skip directly to model design.