Learning-to-Rank: Features, Models, and Position Bias

Why Learning-to-Rank (Not Manual Scoring)

A hand-crafted scoring formula like 0.4×bm25 + 0.3×rating + 0.3×recency cannot capture complex interactions between features. A user searching 'budget laptop for college' should receive different relative weights than a user searching 'high performance gaming laptop' even though both are product searches. LTR learns these context-dependent weights from data.

Three LTR Paradigms

Pointwise: Training: predict a relevance label for each (query, document) pair Model: binary classifier or regression Loss: binary cross-entropy (clicked vs. not) or MSE (relevance grade 0-3) Limitation: treats each document independently, ignores list structure

Pairwise (LambdaRank, RankNet): Training: given two documents A and B, predict which should rank higher Model: neural network or gradient boosted trees Loss: log_loss(P(A > B)) across all pairs in a query LambdaRank improvement: weight the pairwise loss by |ΔNDCG| — pairs where swapping the order would change NDCG the most get higher weight Practical choice: LightGBM with objective='lambdarank' is the production baseline

Listwise (ListNet, LambdaLoss): Training: optimize a loss over the full ranked list as a unit Model: neural network Loss: cross-entropy between probability of ideal permutation and model's permutation Strength: most principled, directly optimizes list-level metrics Weakness: computationally expensive; rarely used in latency-constrained production

Feature Engineering — The Most Important Design Decision

Feature importance order (empirical, from search literature):

Tier 1 — Query-Document Interaction (highest NDCG contribution): bm25_title — BM25 score between query and product title bm25_description — BM25 score against product description semantic_sim — cosine similarity of query_emb and product_emb attr_match_frac — fraction of query attributes found in product attributes hist_ctr_exact — historical CTR for exact (query_hash, product_id) pair hist_ctr_category — historical CTR for (category, product_id) pair

Tier 2 — Document Quality: log_review_count — log(1 + review count) — log to reduce outlier effect avg_star_rating — 1.0-5.0, weighted by review recency return_rate — fraction of orders returned (negative quality signal) seller_defect_rate — fraction of orders with complaints is_prime_eligible — binary (Amazon example: availability signal)

Tier 3 — Query Context: query_popularity_rank — log rank of how often this query is issued query_intent_type — one-hot: transactional/informational/navigational query_category — predicted product category from query text

Tier 4 — User Context: user_price_percentile — where does the user's median past purchase fall? user_brand_affinity[brand_id] — has user bought from this brand before? session_category — what category has user been browsing this session?

Position Bias Correction

The problem: P(click | doc, position=1) >> P(click | doc, position=5) even for the same document and same query. Training on raw clicks makes the model prefer documents that were historically placed at high-visibility positions.

IPS (Inverse Propensity Scoring) solution:

1. Estimate examination probability P(exam | position) from randomization experiments (randomly insert documents at all positions and measure CTR per position)
2. Weight each training click: sample_weight = 1 / P(exam | impression_position)
3. Train the LTR model with these sample weights

Result: clicks at position 8 (low examination probability) count MORE in training than clicks at position 1 (high examination probability), because clicks at position 8 are harder to get and thus more informative.