Cold Start, Evaluation, and System Tradeoffs

Cold Start: New Users

A new user has no watch history → no two-tower embedding → no personalized retrieval.

Strategy:

1. Onboarding: ask user to select 3-5 interest categories at signup
2. Demographic proxy: use age + location + device as a cold-start user embedding
3. Topic-based popularity: show the most-watched recent videos in their selected topics
4. Fast transition: after 5-10 interactions, compute a real user embedding and switch

Cold Start: New Videos

A new video has no watch data → no engagement statistics → no item tower embedding.

Strategy:

1. Content-based embedding: embed title + description + tags using a sentence transformer → use this as the item embedding for FAISS retrieval (not trained on engagement)
2. Creator history: if the creator has high historical engagement, inherit that as a prior
3. Exploration injection: deliberately surface the new video to a random sample of likely-interested users to collect initial engagement data
4. Fast item tower update: run the trained item tower on new video features immediately at upload time to produce an engagement-model-compatible embedding

A/B Testing Design

A complete A/B test design for a recommendation model change:

Control:   current production model
Treatment: new model (e.g., added cross-features)
Traffic split: 95% control / 5% treatment (conservative, new model untested)
Duration: 14 days minimum (capture Mon-Sun variation twice)

Primary metric: avg watch time per session
  → must improve by > 0.5% with p < 0.05

Guardrail metrics (must not degrade):
  7-day return rate (long-term retention)
  Creator diversity (number of unique creators in recommendations)
  New content exposure rate (% recommendations that are < 7 days old)

Success criterion: primary metric improves, guardrail metrics not degraded
Failure criterion: guardrail metric drops by > 1% even if primary improves

Feedback Loop

The system improves only if training data is refreshed:

1. Log every impression (user_id, video_id, position, timestamp)
2. Log every engagement event (watch_seconds, liked, shared, abandoned)
3. Daily batch job: join impressions + engagement → training examples
4. Retrain two-tower on new (user, video, engaged) pairs
5. Rebuild FAISS index with updated video embeddings
6. Retrain ranking model on new feature + label pairs
7. Shadow test new models (run alongside production, compare offline)
8. Progressive rollout: 1% → 5% → 20% → 100% with monitoring at each step

Filter Bubble and Diversity

Without intervention, recommendation systems develop echo chambers:

• User watches 3 cooking videos → system shows 20 cooking videos
• User never discovers travel content they might love
• Popular cooking channels dominate; niche channels never get exposure

Mitigations:

• Diversity constraint in re-ranking: max 3 videos per category per page
• Exploration slots: 1-2 of 20 positions reserved for categories the user hasn't explored recently
• Serendipity metric: track what fraction of recommendations are outside the user's historical top-3 categories; add as an A/B test metric