Evaluation, Well-Being Tension, and System Tradeoffs

Offline Evaluation

Offline evaluation for feed ranking uses historical logs:

Training data: (user, post, impression_timestamp, engagement_label) Where engagement_label = 1 if user liked/replied/shared, 0 if scrolled past.

Offline metrics:

• AUC per task (like AUC, reply AUC, report AUC) — measures ranking quality
• Calibration: do predicted probabilities match actual rates? (predicted P(like) = 0.1 should correspond to ~10% actual like rate)

Limitation: offline metrics can't capture the feedback loop effects of a new ranking model — only online A/B tests reveal true quality.

Online A/B Testing Design

Control:   current production ranking model
Treatment: new model (e.g., added engagement velocity feature)
Traffic:   5% of users to treatment (conservative)
Duration:  14 days minimum — capture weekly cycle twice

Primary metrics:
  • Sessions per user per day (users return more → better feed)
  • Engagement rate (likes + replies + shares per session)

Guardrail metrics (must not degrade):
  • Report rate per session (must not increase > 0.1%)
  • Unfollow rate (must not increase > 0.5%)
  • Satisfaction survey score (sampled 0.1% of users)
  • Author diversity (Herfindahl-Hirschman Index of impressions per author)

Success: primary metrics improve + guardrail metrics hold
Failure: any guardrail metric degrades significantly

The Engagement vs. Well-Being Tension

This is the defining challenge of social feed ranking:

Outrage-inducing content (divisive political posts, fear content, sensational claims) reliably generates high reply rates and high share rates — users engage even when the engagement is negative (arguing in replies).

A system that optimizes purely for engagement will amplify such content because it genuinely scores higher on the engagement metrics.

Mitigations (none are complete solutions):

1. Penalize report signal heavily: p_report penalty weight -10.0 in the combined score means a post with 1% report rate is severely demoted
2. Sentiment of replies: if the replies on a post are predominantly negative sentiment, this signals the post is generating angry engagement, not positive engagement
3. 'Not interested' signal: users who mute or click 'not interested' signal the post doesn't match their intent even if they engaged with it
4. Satisfaction surveys: periodically ask a sample of users how they feel after using the feed — this captures well-being effects that engagement metrics miss
5. Human review of top-promoted content: content in the top 0.01% of engagement scores should be reviewed by human reviewers for policy compliance

Key Architectural Tradeoffs

Recency vs. quality: Time-decay ensures freshness but penalizes evergreen quality content. Fix: classify posts as time-sensitive (news, live events) vs. evergreen (tutorials, art) and apply different decay rates. Time-sensitive content decays in hours; evergreen content decays over days.

Network-only vs. algorithmic expansion: Pure network feed: predictable, user-controlled, but boring for small-network users. Pure algorithmic feed: high engagement potential, but unpredictable and less trustworthy. Most platforms now offer both modes with user control ('Following' vs. 'For You' tabs).