Case Study: Search and Listing Ranking System Design
End-to-end design of a search and listing ranking system — the pattern behind e-commerce search (Amazon, eBay), marketplace search (Airbnb listings), and enterprise search. Covers query understanding, hybrid retrieval (BM25 + semantic), learning-to-rank, personalization, position bias, and multi-objective ranking that balances relevance, quality, and revenue. Grounded in Khang Pham's search/listing ranking case study.
▶📚 Prerequisites(3)
🎓 Learning objectives
- •Distinguish search ranking from recommendation: explain where they share patterns and where they diverge
- •Design a query understanding pipeline covering tokenization, spelling correction, query expansion, and intent classification
- •Explain BM25 retrieval and describe when semantic (embedding) retrieval outperforms it and vice versa
- •Describe the three learning-to-rank paradigms (pointwise, pairwise, listwise) and explain when to use each
- •Identify the key feature categories used in search ranking (query, document, query-document interaction, user context) and explain why interaction features carry the most signal
- •Explain position bias in search and describe two approaches to correcting it (IPS and examination model)
- •Design a multi-objective re-ranking policy that balances relevance with quality signals and business goals
What is it?
A search ranking system decides, given a user's query, which documents or listings from a large index to surface and in what order. Unlike recommendation (where there is no explicit query), search ranking must solve a bi-directional matching problem: match the user's query intent to a document's content.
Khang Pham's ML Primer describes the search/listing ranking pattern as the canonical ML system design case study that appears most frequently in FAANG/MAANG interviews — because virtually every large platform has a search surface: Airbnb searching listings, Amazon searching products, LinkedIn searching jobs and people, YouTube searching videos, and Google searching the web.
Despite product differences, all share the same ML architecture:
- Query understanding — parse, correct, expand, and classify the query
- Candidate retrieval — find hundreds of matching documents from millions
- Learning-to-rank (LTR) — score candidates precisely using rich features
- Re-ranking — apply multi-objective adjustments before the result is shown
The distinction from recommendation: in search, the user has stated an intent (the query). The system must respect that intent while also personalizing, balancing quality, and optimizing business objectives.
Why it exists
Without ranking, a search over 10M documents would return results in arbitrary order — the most relevant document might appear on page 100. Ranking converts retrieval into an ordered list where the most valuable item is at position 1.
The engineering argument: at scale, retrieval must be fast (< 50ms) which forces approximate methods that trade precision for speed. Ranking allows a fast-but-imprecise retrieval stage to be followed by a slow-but-precise scoring stage, combining speed and accuracy.
The business argument: search quality directly drives conversion. Amazon reports that search is the primary discovery path for most purchases. A 1% improvement in search NDCG at Amazon's scale translates to hundreds of millions in revenue.
The ML argument: the 'best' result is not a fixed property of a document — it depends on the query, the user's intent, the user's history, and context. A trained ranking model captures these multidimensional relevance signals far better than any hand-crafted scoring formula.
Problem it solves
- Scale: can't score 10M documents per query — need a fast retrieval stage
- Vocabulary mismatch: user types 'sneakers', documents say 'athletic shoes' — BM25 misses it; semantic retrieval catches it
- Relevance vs. quality: a highly relevant but low-quality document (spam, thin content) should not rank first
- Personalization: same query 'python books' means different things for a novice vs. an expert
- Multi-objective conflict: most relevant result may not be the highest-margin product or the most-clicked listing
- Position bias: users click position-1 results far more than position-5, regardless of quality — training data is biased
- Cold start for new documents: newly indexed documents have no click history
Intuition
A search ranking system is like a reference librarian with a photographic memory working at a library with 10 million books.
The user walks in and says: 'I need something on machine learning.' The librarian's job:
Query understanding: 'Did they mean machine learning as a field, or a specific technique? Beginner or advanced? For code or theory? Let me also consider spelling variants and synonyms (ML, artificial intelligence, neural networks...)'
Candidate retrieval: 'I know there are about 500 books that could match. Let me identify them quickly from the catalog.' (Not reading all 10M — too slow.)
Ranking: 'Now I have 500 candidates. Let me score each one carefully: how well does it match the query? How well-reviewed is this author? Have similar users liked this book after asking the same question?'
Re-ranking: 'I have my top 10. But I notice 8 of them are by the same author — let me add some variety. And this new book just arrived that looks highly relevant but has no reviews yet — let me include it.'
Position bias: if the librarian always puts the same book at eye level, users always check it first — this creates a feedback loop where popular = frequently placed = more checked = more 'popular'. The bias must be removed from training data.
Analogy
Search ranking is like the system a recruiting firm uses to match job candidates to job postings.
A recruiter has 100,000 candidates in their database. When a company posts a job, the recruiter's system must find and rank the top 10 candidates to present.
BM25 (keyword matching): matches candidates whose resumes contain the exact words in the job description. 'Python engineer, 5 years experience, distributed systems.' Strength: precise for exact matches. Weakness: misses the candidate whose resume says 'Golang' when the job says 'backend engineering' (same role, different word).
Semantic retrieval: matches candidates based on meaning, not keywords. A candidate who worked on 'large-scale data pipelines' matches a job posting about 'distributed systems infrastructure' even without shared words.
Learning-to-rank: rather than just word-matching, the system considers: past placement history (did this recruiter successfully place similar candidates?), candidate quality signals (interview scores, reference ratings), and the specific company's preferences.
Personalization: the same job posting means different things for different hiring managers. The system should learn that Manager A values startup experience while Manager B values large-company structure.
Position bias: if the recruiter always sends the first candidate in the ranking to the client, and clients tend to pick whoever they meet first, the 'popular' candidates keep accumulating placements not because they're better but because they happened to rank first.
Technical explanation
STEP 1 — PROBLEM FRAMING:
Key questions to establish before designing:
- Search type: keyword search? semantic search? faceted search with filters?
- Domain: e-commerce (product search), marketplace (listing search), enterprise (document search), general (web search)?
- Scale: number of indexed documents, QPS, catalog update frequency?
- Metrics: NDCG@10, MRR? Conversion rate? Revenue per search?
Standard scale: 10M products, 1K QPS, 100ms SLA.
Metric hierarchy: Primary (business): conversion rate after search, revenue per search Proxy (ML training): NDCG@10, MRR (using human relevance labels or click data) Guardrail: zero-result rate (fraction of queries returning no results)
STEP 2 — QUERY UNDERSTANDING:
The query is the user's stated intent — understanding it correctly is the highest-leverage component of a search system.
Pipeline stages:
-
Text normalization: lowercase, strip punctuation, normalize Unicode
-
Spell correction: noisy channel model or sequence-to-sequence 'haiking boots' → 'hiking boots'
-
Query expansion: add synonyms and related terms to the retrieval query 'athletic shoes' → also search 'sneakers', 'trainers', 'running shoes' Source: curated synonym dictionaries + learned expansion models
-
Intent classification: classify the query type
- Navigational: 'Amazon Nike Air Force 1' → user wants that specific product
- Informational: 'best hiking boots 2024' → user wants comparison
- Transactional: 'buy waterproof hiking boots under $150' → ready to purchase Different intents warrant different result types and ranking objectives
-
Query embedding: encode the full query into a dense vector for semantic retrieval Use a bi-encoder (same architecture as two-tower model) or BERT/sentence-transformer
STEP 3 — HYBRID RETRIEVAL:
BM25 (Best Match 25) — lexical retrieval: score(q, d) = Σ IDF(t) × TF(t,d) / (TF(t,d) + k × (1 - b + b × |d|/avgdl)) t ∈ query terms Where: IDF = inverse document frequency (rare terms weighted more) TF = term frequency in document k, b = tunable parameters (typically k=1.5, b=0.75)
BM25 strengths: exact keyword matching, fast (inverted index lookup), interpretable, works well for navigational and product ID queries. BM25 weaknesses: vocabulary mismatch, no understanding of synonyms or paraphrases.
Semantic retrieval — ANN over embedding index: Encode query → query_emb via bi-encoder Pre-compute product_emb for all products offline → FAISS HNSW index At query time: ANN search query_emb → top-K similar product embeddings
Semantic strengths: handles synonyms, paraphrases, and conceptual similarity. Semantic weaknesses: slower to build and query, may miss exact product SKU matches.
Hybrid: run both in parallel, merge results, pass to LTR. RRF (Reciprocal Rank Fusion): score_rrf = Σ 1/(k + rank_i) for each retrieval source
STEP 4 — LEARNING-TO-RANK (LTR):
Three paradigms:
Pointwise: train a regressor to predict relevance score for each (query, document) pair Loss: MSE or cross-entropy on relevance label (e.g., 0/1/2 scale) Pros: simple, easy to implement. Cons: doesn't model relative ordering directly.
Pairwise (LambdaRank, RankNet): train on (doc_A, doc_B) pairs where A should rank above B Loss: log loss on P(A should rank above B) Pros: directly optimizes pairwise ordering. Most common in production. LambdaRank: directly optimizes NDCG by weighting gradients by ΔNDCG.
Listwise (ListNet, LambdaLoss): treat the entire ranked list as the training unit Loss: cross-entropy over permutation probability (the ideal ranking vs. model ranking) Pros: most principled, directly optimizes list-level metrics. Cons: computationally expensive, complex to implement.
Feature categories (highest to lowest typical signal):
-
Query-document interaction (highest signal):
- BM25 score between query and product title
- Semantic similarity (cosine of query_emb · product_emb)
- Attribute match: fraction of query attributes found in product attributes
- Historical CTR for this exact (query_hash, product_id) pair
-
Document quality features:
- Review count, average rating
- Seller reputation score
- Return rate, complaint rate
- Days since listed (freshness)
-
Query features:
- Query length, query frequency (popular vs. rare query)
- Detected intent type (transactional vs. informational)
- Detected category from query text
-
User context features:
- User's price tier preference (past purchase price distribution)
- User's brand affinity
- User's session: what have they clicked on already in this session?
STEP 5 — POSITION BIAS AND CORRECTION:
Training data (click logs) is biased: users click position-1 items far more often than position-5 items, regardless of quality. A model trained on raw clicks will learn to rank popular items first (because they were shown first and thus clicked more), not the most relevant items.
Correction methods:
IPS (Inverse Propensity Scoring): Estimate P(click | relevant, position) = P(examine | position) × P(relevant) Weight each training example by 1 / P(examine | position) Downweights clicks from high-visibility positions in training
Examination model (two-stage): P(click) = P(examine | position) × P(relevant | query, document) Jointly train two models: examination model (learns position effect) and relevance model (learns true relevance) At inference: use only the relevance model score
Randomization experiment: occasionally insert results at random positions and collect clicks — this provides unbiased training data for calibrating the position effect.
Architecture
Search Ranking Data Infrastructure:
┌──────────────────────────────────────────────────────────────┐ │ OFFLINE SYSTEMS │ │ │ │ ┌─────────────────┐ ┌──────────────────────────────────┐ │ │ │ Document Index │ │ Feature Engineering │ │ │ │ Pipeline │───►│ • Product text/attribute embs │ │ │ │ (Spark/Flink) │ │ • Quality signals (ratings) │ │ │ │ daily refresh │ │ • Historical CTR by (q, item) │ │ │ └─────────────────┘ └────────────────┬─────────────────┘ │ │ │ │ │ ┌──────────────────────────────────────►▼─────────────────┐ │ │ │ Search Index + ANN Index │ │ │ │ • BM25 inverted index (Elasticsearch / Lucene) │ │ │ │ • FAISS HNSW index over product embeddings │ │ │ │ • Attribute filter index (structured filters) │ │ │ └──────────────────────────────────────┬─────────────────┘ │ │ │ │ │ ┌────────────────┐ ┌────────────────▼──────────────────┐ │ │ │ LTR Training │ │ Model + Feature Store │ │ │ │ (LambdaRank / │───►│ • LTR model weights │ │ │ │ LightGBM) │ │ • Query expansion rules │ │ │ │ weekly/daily │ │ • User feature cache (Redis) │ │ │ └────────────────┘ └───────────────────────────────────┘ │ └──────────────────────────────────────────────────────────────┘ ↑ fed by click logs + relevance labels ┌──────────────────────────────────────────────────────────────┐ │ ONLINE SERVING (< 100ms) │ │ │ │ Query → Query Understanding (NLP pipeline, ~5ms) │ │ → Hybrid Retrieval (BM25 + ANN + filter, ~20ms) │ │ → Feature Assembly (feature store + real-time, ~15ms) │ │ → LTR Scoring (~30ms for 500 candidates) │ │ → Re-ranking + ads injection (~5ms) │ │ → Response (top 10-20 results, ~5ms serialization) │ └──────────────────────────────────────────────────────────────┘ │ (log: query, position, product_id, click, conversion) ┌────────▼─────────────────────────────────────────────────────┐ │ FEEDBACK LOOP │ │ Click logs + relevance labels → LTR retraining (weekly) │ │ CTR trends → query expansion rule updates (weekly) │ └──────────────────────────────────────────────────────────────┘
Workflow
How to design this system in a 45-minute ML system design interview:
Minutes 1-5 — Requirements clarification: Q: Product domain? (e-commerce, marketplace, enterprise, web) Q: Scale? (catalog size, QPS, latency SLA) Q: Primary metric? (NDCG@10, conversion, revenue per search) Q: Personalization required? (user history, segment, context) Establish: 10M products, e-commerce, 1K QPS, 100ms SLA, optimize conversion.
Minutes 5-10 — High-level design: Sketch the four-stage pipeline on the whiteboard. Distinguish search ranking from content recommendation. Explain why a funnel (retrieval → ranking) is needed at scale.
Minutes 10-20 — Query understanding and retrieval: Walk through each stage of query understanding. Explain BM25 vs. semantic retrieval and when each wins. Name the hybrid approach and how to merge results (RRF).
Minutes 20-30 — Learning-to-rank deep dive: Name the three LTR paradigms; choose pairwise (LambdaRank) for your design. Walk through all four feature categories; emphasize interaction features. Explain position bias and your correction approach.
Minutes 30-40 — Re-ranking, evaluation, and feedback loop: Multi-objective re-ranking: relevance + quality + revenue. A/B testing: primary metric (NDCG@10 or conversion), guardrail (zero-result rate). Feedback loop: click logs → bias correction → LTR retraining. Cold start for new documents: feature fallback (quality signals only).
Minutes 40-45 — Scale and failure modes: Inverted index (Elasticsearch) + FAISS HNSW for the retrieval layer. What happens if LTR model is stale? (degrade gracefully to BM25 score) What happens if query understanding is down? (pass raw query to retrieval)
Example
# Learning-to-rank: LambdaRank simplified with LightGBM import lightgbm as lgb import numpy as np import pandas as pd # Feature schema: # Query features: query_length, query_freq_log # Document features: review_count_log, avg_rating, days_since_listed # Interaction features: bm25_score, semantic_sim, attr_match_frac, hist_ctr FEATURE_COLS = [ 'query_length', 'query_freq_log', 'review_count_log', 'avg_rating', 'days_since_listed', 'bm25_score', 'semantic_sim', 'attr_match_frac', 'hist_ctr', ] def build_ltr_dataset(df: pd.DataFrame) -> lgb.Dataset: # df must have: query_id, label (0/1/2/3), and all FEATURE_COLS # label: 0=irrelevant, 1=poor, 2=good, 3=perfect group = df.groupby('query_id')['query_id'].count().values # docs per query return lgb.Dataset( df[FEATURE_COLS], label=df['label'], group=group, free_raw_data=False, ) def train_ranker(train_df: pd.DataFrame, valid_df: pd.DataFrame) -> lgb.Booster: train_data = build_ltr_dataset(train_df) valid_data = build_ltr_dataset(valid_df) params = { 'objective': 'lambdarank', 'metric': 'ndcg', 'ndcg_eval_at': [5, 10], 'learning_rate': 0.05, 'num_leaves': 64, 'min_child_samples': 50, 'lambda_l2': 0.1, 'label_gain': [0, 1, 3, 7], # gain for labels 0/1/2/3 } model = lgb.train( params, train_data, num_boost_round=300, valid_sets=[valid_data], callbacks=[lgb.early_stopping(30), lgb.log_evaluation(50)], ) return model def rank_candidates(model: lgb.Booster, candidates: pd.DataFrame) -> pd.DataFrame: scores = model.predict(candidates[FEATURE_COLS]) return candidates.assign(ltr_score=scores).sort_values( 'ltr_score', ascending=False ) # --- Position bias correction (IPS weighting) --- EXAMINATION_PROB = { 1: 1.00, 2: 0.85, 3: 0.70, 4: 0.58, 5: 0.47, 6: 0.38, 7: 0.31, 8: 0.25, 9: 0.20, 10: 0.17 } def ips_weight(position: int) -> float: 'Inverse propensity weight: down-weight clicks from high-visibility positions.' return 1.0 / EXAMINATION_PROB.get(position, 0.10) # Apply IPS weight when constructing training labels from click logs: # weighted_label = click × ips_weight(impression_position) # This de-biases the click signal so clicks at position 1 count less # than clicks at position 8 (where exposure probability is much lower)
Real-world usage
-
Airbnb Search (Haldar et al., 2019): the most-cited real-world search ranking case study. Uses LambdaRank trained on (listing, query, booking/no-booking) pairs. Key insight: booking signal (stronger than click) is the ground truth label. Introduced listing embedding (neural embedding of listing features) as a key feature. Showed that personalization features (user price sensitivity, location preference) significantly improve NDCG despite adding complexity.
-
Amazon Product Search: uses a hybrid of keyword matching and semantic retrieval. Multi-objective ranking that explicitly optimizes for revenue per search (weighted by item margin) in addition to relevance. Deployed at massive scale (billions of queries per day). Well-documented use of query expansion to bridge vocabulary gaps between customer language and catalog language.
-
LinkedIn Talent Search (Shi et al., 2016): pairwise ranking on job-candidate matches. Uses a mix of member activity signals (profile views, applications) and recruiter feedback (positive/negative) as training labels. Demonstrates the bi-directional nature of marketplace search (both sides have preferences).
-
Etsy Search (Turczyn, 2019): small-team example of deploying semantic search on a handmade-goods marketplace where vocabulary is highly non-standard (buyers use creative terminology that sellers don't use). BM25 alone has poor coverage; semantic retrieval significantly improves zero-result rate.
-
Khang Pham (ML Primer): presents the search/listing ranking as the canonical ML system design problem with emphasis on hybrid retrieval and LTR feature categories. Recommends LambdaRank/LightGBM as the practical baseline before considering neural LTR approaches.
Trade-offs
BM25 vs. semantic retrieval: BM25 wins on navigational queries ('Nike Air Force 1 size 11'), exact product SKUs, and rare product categories where training data for embeddings is thin. Semantic retrieval wins on exploratory, natural-language queries where vocabulary mismatch is high. Production systems use both; the hybrid typically outperforms either alone. The engineering cost of maintaining a FAISS index (recomputing embeddings on catalog updates) is the primary argument for starting with BM25-only and adding semantic retrieval incrementally.
Pointwise vs. pairwise vs. listwise LTR: pointwise is the easiest to implement (standard regression/classification) but ignores relative ordering. Pairwise (LambdaRank) is the standard production choice — it directly models what ranking should be while being tractable to train. Listwise is the most principled but computationally expensive and rarely used in latency-sensitive production systems.
Click labels vs. human relevance labels: click data is free (implicit) but biased. Human labels are unbiased and use a calibrated scale (0-3 relevance) but are expensive. Production systems typically train on human labels and fine-tune with click data using IPS correction. The ratio between human labels and click data is a tunable hyperparameter.
Personalization depth: heavy personalization (per-user model) improves metrics for users with sufficient history but harms cold-start users, is expensive to compute, and makes debugging harder (why did this product rank here?). Segment-level personalization (group users by price sensitivity or category affinity) offers a good tradeoff.
Visual explanation
Search Ranking System — End-to-End Architecture:
USER QUERY: 'lightweight hiking boots waterproof' │ ▼ ┌────────────────────────────────────────────────────────────┐ │ QUERY UNDERSTANDING PIPELINE │ │ │ │ 1. Normalization: lowercase, trim, unicode normalize │ │ 2. Spell correction: 'waterproff' → 'waterproof' │ │ 3. Query expansion: │ │ synonyms: boots → {boots, footwear, shoes} │ │ related: waterproof → {water-resistant, DWR} │ │ 4. Intent classification: │ │ product_category=footwear, intent=transactional, │ │ attributes={lightweight, waterproof, hiking} │ │ 5. Query embedding: encode full query → 768d vector │ └──────────────────────────────┬─────────────────────────────┘ │ ▼ ┌────────────────────────────────────────────────────────────┐ │ HYBRID CANDIDATE RETRIEVAL (parallel) │ │ │ │ Path A: BM25 (lexical) 500 candidates │ │ 'waterproof hiking boots' term-matching in product │ │ title, description, category, attributes │ │ │ │ Path B: ANN semantic (embedding) 300 candidates │ │ query_emb → FAISS search over product_emb index │ │ catches 'waterproof trail footwear' ← same meaning │ │ │ │ Path C: Attribute filter retrieval 100 candidates │ │ filter on product_attributes: {waterproof=true, │ │ category=hiking, weight<500g} │ │ │ │ ────► MERGE + DEDUPLICATE ◄──── ~700-900 total │ └──────────────────────────────┬─────────────────────────────┘ │ ▼ ┌────────────────────────────────────────────────────────────┐ │ LEARNING-TO-RANK (LTR) MODEL │ │ │ │ For each candidate, compute features: │ │ • Query features: query length, intent type │ │ • Document features: review count, avg rating, recency │ │ • Query-document interaction features (highest signal): │ │ BM25 score, semantic similarity, attribute match % │ │ historical CTR for this query × product pair │ │ • User context features: session history, user segment │ │ │ │ Model: LambdaRank or LightGBM ranker │ │ Output: relevance score per candidate │ │ │ │ ────► TOP 50 candidates ranked │ └──────────────────────────────┬─────────────────────────────┘ │ ▼ ┌────────────────────────────────────────────────────────────┐ │ RE-RANKING + SERVING POLICY │ │ • Multi-objective blend: relevance + quality + revenue │ │ • Diversity: max 2 results from same seller/brand │ │ • Freshness: boost recently listed items (new inventory) │ │ • Promoted/paid slots injected at positions 1, 4 │ │ • Safety filter: remove recalled products, policy │ └──────────────────────────────┬─────────────────────────────┘ │ ▼ RESULTS PAGE (10 products) → User
Advantages
- —
Hybrid retrieval (BM25 + semantic) captures both exact-match intent (navigational queries) and semantic similarity (exploratory queries) that either method alone would miss
- —
LambdaRank directly optimizes NDCG by weighting gradients according to the improvement in list ranking — this produces better ranking models than pointwise approaches for the same amount of training data
- —
Query-document interaction features (BM25 score, semantic similarity, historical CTR for the specific query-product pair) carry significantly more signal than document quality features alone
- —
Position bias correction via IPS or examination model ensures the LTR model learns true relevance from click logs rather than position artifacts
- —
The four-stage funnel (understand → retrieve → rank → re-rank) allows each stage to be independently optimized, A/B tested, and scaled
Disadvantages
- —
BM25 retrieval requires a full inverted index rebuild when the catalog changes significantly — for a catalog with millions of daily updates, this creates infrastructure complexity
- —
LambdaRank requires relevance-labeled (query, document) pairs for training — human labeling is expensive, and using click data requires careful position bias correction that introduces its own approximation errors
- —
Semantic retrieval over FAISS requires product embeddings to be pre-computed and updated — new products have no embedding until the next offline index rebuild, creating a cold-start gap
- —
Query expansion can introduce false positives: expanding 'apple' to include 'apple computer' and 'apple fruit' and 'Apple Inc.' may retrieve many irrelevant results that increase retrieval latency and ranking noise
- —
Multi-objective re-ranking (relevance + revenue + quality) requires explicit weight calibration between objectives — these weights are policy decisions with no purely algorithmic correct answer
Common mistakes
- —
Using a single retrieval method. A BM25-only system fails on semantic matches; a semantic-only system fails on exact product IDs and rare terms. A complete design always uses hybrid retrieval. A commonly missed component: the attribute filter retrieval path — for e-commerce, structured filters (size=M, color=red, price<$50) are a critical retrieval source that pure text search misses.
- —
Not addressing position bias. If you train an LTR model on raw click logs without correcting for position bias, the model will learn to prefer products that were frequently shown at the top — not because they're most relevant, but because they got more clicks due to being in the most visible position. A complete design names IPS or the examination model as the bias correction approach.
- —
Treating query understanding as trivial. Many engineers jump straight to retrieval and ranking without a query understanding stage. In practice, query understanding (spell correction, expansion, intent classification) provides some of the highest ROI improvements. A design that skips this stage misses the most user-visible quality improvements.
- —
Using the same metric for training and evaluation. Training on click data (optimizing CTR-like signals) while evaluating on NDCG@10 (computed from relevance labels) is correct. But conflating them — training directly on NDCG from click data without bias correction — produces models that score well offline but don't improve conversion in A/B tests.
- —
Forgetting the zero-result rate. A search system with high NDCG@10 but a 15% zero-result rate is failing 15% of users completely. The zero-result rate (fraction of queries returning no results) is an essential guardrail metric. Query expansion, semantic fallback retrieval, and spelling correction all reduce zero-result rates and should be part of any complete design.
🎤 Interview questions
Design a search ranking system for an e-commerce platform with 10M products and 1K QPS. Walk me through every component from the user's query to the final ranked results page.
Explain the difference between BM25 and semantic (embedding-based) retrieval. When would you use each, and when would you use both?
What is position bias in search, and how do you correct for it when training a learning-to-rank model on click logs?
Compare pointwise, pairwise, and listwise learning-to-rank approaches. Which would you use in production and why?
📂 Subtopics
Query Understanding and Hybrid Retrieval
How to process a user's raw query into a structured signal for retrieval, and why hybrid retrieval (BM25 + semantic) outperforms either method alone.
~45 min
Learning-to-Rank: Features, Models, and Position Bias
The three learning-to-rank paradigms, feature engineering for search, why interaction features dominate, and how to correct position bias when training on click log data.
~45 min
Re-ranking, Multi-Objective Scoring, and Evaluation
Multi-objective re-ranking that blends relevance with quality and business signals, A/B testing design for search, and how to close the training feedback loop.
~40 min