Two-Tower Retrieval and Deep Ranking

Candidate Generation — The Scale Problem

At 10M videos × 100M users, you cannot run a deep ranking model on every video at query time. Candidate generation solves this with approximate, fast retrieval that reduces the search space to ~300 candidates.

Two-Tower Model Architecture

User Tower:                    Item Tower:
  user_id embedding (256d)       video_id embedding (256d)
  age, location (dense)          category one-hot
  watch_history_emb (avg pool)   creator subscribe count
  device, time-of-day            title/desc embedding (BERT)
        ↓                              ↓
  Dense layers                  Dense layers
        ↓                              ↓
  user_emb (128d)               video_emb (128d)
        └─── dot_product = relevance score ───┘

Training: in-batch negatives — for each (user, video_they_watched) positive pair, treat all other videos in the batch as negatives. Loss: softmax cross-entropy.

Serving: pre-compute video_emb for all 10M videos offline → build FAISS HNSW index. Online: compute user_emb → HNSW query → top-200 videos in < 10ms.

Multi-Source Candidate Generation

A single two-tower model is not enough. Production systems use 4-6 sources:

Source	Signal	Candidate count
Two-tower ANN	Long-term user interest	200
Co-watch CF	Session intent	100
Content-based	Item similarity	50
Trending/fresh	New content exposure	50
Social graph	Friends' activity	30

Merge and deduplicate. Pass ~300-400 candidates to the ranking model.

Deep Ranking Model — Multi-Task Architecture

The ranking model scores each candidate with features too expensive to compute at retrieval time:

# Multi-task ranking: watch time + like + share - abandon
score = (1.0 × predicted_watch_time
       + 0.3 × P(like | watched)
       + 0.2 × P(share | watched)
       - 0.5 × P(early_abandon))

Feature engineering for ranking (the most impactful design choice):

User features:

• Historical category affinity vector (% of watch time per category)
• Device type, time-of-day, day-of-week
• Session context: last 5 watched videos

Item features:

• Video category, duration, language
• Creator subscribe count, creator avg CTR
• Global avg watch time ratio (for this video across all users)
• Video age (hours since upload)

Cross features (highest leverage):

• user_category_affinity[video_category] — does this user historically watch this category?
• user_location × video_language — language preference
• session_last_category × video_category — are they on a binge of this topic?

Position bias correction: Add position as a training input. At inference, set position=1 for all candidates. This makes the model learn quality-conditional scores, not position-conditional scores.