Re-ranking, Multi-Objective Scoring, and Evaluation

Multi-Objective Re-ranking

The LTR model's relevance score is not the only consideration for the final ranked list. Production search systems apply a blended score:

final_score = alpha × ltr_relevance_score
           + beta  × quality_score        # avg rating, review count
           + gamma × business_score       # margin, strategic importance
           - delta × defect_penalty       # high return rate, complaints

The weights {alpha, beta, gamma, delta} are business policy decisions. They are tuned via A/B testing, not learned during model training. This separation is intentional: the LTR model learns relevance from user behavior; the weights reflect business strategy that may override pure relevance.

Freshness and cold-start for new documents: New products have no review count, no click history, no quality signals. Strategy: assign a prior based on similar products (same category, similar price) and include a freshness boost to give new listings a chance to accumulate engagement data.

Diversity constraints: Limit to max 2 results from the same seller or brand per page. Why: users prefer to see options from multiple sources, even if one seller dominates the ranking scores. This is especially important for marketplaces.

Evaluation Design

Offline evaluation: Dataset: (query, document, relevance_label) triples Labels: human-labeled (0=irrelevant, 1=poor, 2=good, 3=perfect) or click-based (binary: clicked vs. not clicked, IPS-corrected) Metric: NDCG@5, NDCG@10, MRR

NDCG formula: DCG@k = Σ (2^rel_i - 1) / log2(i+1) for i in 1..k NDCG@k = DCG@k / IDCG@k (normalize by ideal ordering) NDCG = 1.0 means the model produces the ideal ranked order.

MRR formula: MRR = mean over queries of (1 / rank of first relevant result) Good for navigational queries where there's one right answer.

Online A/B test: Primary: conversion rate after search (user buys within 30 min of search) Secondary: click-through rate on search results Guardrail: zero-result rate (fraction of queries with no results) Duration: 1 week minimum; 2 weeks if including weekend behavior

Feedback Loop

Click logs → IPS-corrected training examples → LTR retraining (weekly) Human labels → validation set (updated monthly or on model version change) Query expansion rules → updated from click-through rate analysis: 'queries where BM25 has high zero-result rate' → add expansion rules 'queries where semantic retrieval has high CTR' → expand semantic search weight

The Zero-Result Problem

A zero-result page is the worst user experience in search. Causes:

1. Spelling error not caught by spell correction
2. Very specific query with no matching catalog items
3. Query expansion too narrow

Mitigations:

1. Relaxed query: remove attribute filters progressively until results appear
2. Semantic fallback: if BM25 returns 0, use semantic retrieval only
3. Did you mean? suggestion: show spell-corrected alternative
4. Related categories: 'no results for X, but here are similar products in Y'