Model Selection and Benchmarking
How to select the right LLM for your application: navigating public benchmarks, the build-vs-buy decision, and the proprietary vs. open-weight vs. open-source model framework covering cost, control, data privacy, and customization trade-offs.
G-Eval Rubric Weights:
▶📚 Prerequisites(2)
🎓 Learning objectives
- •Critique public LLM benchmarks (MMLU, HellaSwag, HumanEval) for benchmark gaming and leakage
- •Apply the build-vs-buy decision framework to LLM selection
- •Compare proprietary, open-weight, and open-source models across cost, control, privacy, and customization dimensions
- •Design a task-specific internal benchmark that predicts real application quality better than public leaderboards
- •Estimate total cost of ownership for hosted APIs vs. self-hosted open-weight models
What is it?
Model selection is the process of choosing which LLM to use in a production AI application. It involves three intertwined decisions: (1) which model TYPE to use (proprietary API, open-weight self-hosted, or open-source base), (2) which specific model to use within that type, and (3) whether to use an off-the-shelf model or build/fine-tune your own.
Benchmarking is the toolset: evaluating models on standardized tests (MMLU, HellaSwag, HumanEval) to compare capabilities, and building task-specific internal benchmarks that predict real application quality. Chip Huyen (AI Engineering, Ch.4) frames model selection as a decision with three axes — capability, control, cost — and warns that public benchmarks are necessary but insufficient for production selection.
Why it exists
Model selection matters because the choice of model is one of the highest-leverage decisions in an AI application: it determines quality ceiling, latency, cost per query, data privacy exposure, and how much you can customize. Using GPT-4 when Llama 3.1 8B would suffice costs 10-100x more; using a small model for complex reasoning causes constant hallucinations; using a proprietary API for medical data may violate HIPAA.
Public benchmarks exist to give a common vocabulary for comparing models, but they have serious limitations: gaming (training on benchmark data), saturation (GPT-4 scores 90%+ on MMLU; the ceiling tells you nothing), and poor correlation with task-specific performance. Engineers need both the benchmarks and internal evals to make good decisions.
Problem it solves
- Which model should I use for this specific task — and how do I decide without running a 3-week experiment?
- Should I use a hosted proprietary API (OpenAI, Anthropic) or self-host an open-weight model (LLaMA, Mistral)?
- How do I interpret public benchmarks without being misled by gaming or saturation?
- When should I build a custom model vs. using an existing one off the shelf?
- How do I estimate the true cost of different model deployment options?
Intuition
Model selection is like hiring a contractor for a construction project.
You don't just look at their overall reputation (public benchmark score). You check: can they do THIS specific job (task-specific eval)? What's their day rate (cost/token)? Do you own the work they produce (data ownership)? Can you adjust their approach mid-project (fine-tuning)? If the project is sensitive, can you have them work on-site rather than in their own office (self-hosted vs. API)?
A contractor with a Harvard MBA (GPT-4, top MMLU score) is overkill for framing a wall. A general contractor who can follow blueprints (smaller open-weight model, fine-tuned for your domain) is often faster, cheaper, and more controllable — as long as you're willing to do the work of training them.
Analogy
Choosing between proprietary APIs, open-weight, and open-source models is like choosing between SaaS, managed cloud, and on-premises software.
SaaS (proprietary API — GPT-4, Claude): pay per use, zero infrastructure work, constant updates, but vendor controls your data, pricing can change, and you can't peek inside or customize the internals. Great for fast starts and low-volume apps.
Managed cloud instance (open-weight via Bedrock/Vertex — LLaMA, Mistral): runs on your cloud tenant, data stays in your control, but you pay cloud markup and still can't modify the model weights without HuggingFace license constraints.
On-premises (true open-source — truly Apache/MIT licensed weights): full control, data never leaves your servers, can be fine-tuned any way you want — but YOU run the GPU cluster, YOU manage updates, and YOU pay all infrastructure costs. Right for regulated industries (healthcare, finance) and very high-volume workloads.
Technical explanation
PUBLIC BENCHMARKS — what they measure and their limits:
-
MMLU (Massive Multitask Language Understanding): 57 academic subjects, 4-choice MCQ. Measures breadth of world knowledge. Saturated at the top end — GPT-4 scores 87%, random = 25%. Limited to multiple-choice format; does not predict instruction-following or generative quality.
-
HellaSwag: commonsense reasoning, sentence completion. Human parity = 95.6%. GPT-4 = 95.3%. Fully saturated — top models are at human ceiling. No longer useful for differentiation.
-
HumanEval: Python function completion. pass@k metric. Heavily contaminated — many models trained on GitHub Copilot or similar data that overlaps eval. Models with high HumanEval scores may not generalize to your codebase.
-
LMSYS Chatbot Arena Elo: pairwise human preferences. Correlation with task quality ~0.8 (Zheng et al., 2023). Best public benchmark for general-purpose instruction-following. Limitation: dominated by English, chat-style queries — may not predict domain-specific tasks.
BENCHMARK GAMING (Chip Huyen, Ch.4): Three forms: (1) Data contamination — eval data in training corpus. (2) Overfitting — fine-tuned specifically on benchmark-style questions. (3) Selective reporting — only publish benchmarks where your model looks good. Mitigation: run internal task-specific benchmarks. If a model scores 90% on MMLU but poorly on your internal eval, trust your eval.
PROPRIETARY VS. OPEN-WEIGHT VS. OPEN-SOURCE:
Proprietary (OpenAI GPT-4, Anthropic Claude, Google Gemini):
- No access to weights — cannot inspect, fine-tune, or self-host
- Highest quality ceiling as of mid-2024
- Cost: $0.002-0.06 per 1K output tokens
- Data: your prompts are sent to their servers (GDPR/HIPAA implications)
- Vendor lock-in and pricing risk
Open-weight (Meta LLaMA, Mistral, Qwen):
- Weights available for download; can self-host or run via Bedrock/Vertex
- Can fine-tune (within license terms)
- Quality gap vs. proprietary: ~10-20% on complex reasoning (2024); closing fast
- Self-hosted cost: ~$0.0002-0.002/1K tokens at scale (GPU amortized)
- Data stays on your infrastructure when self-hosted
- LLaMA license allows commercial use but prohibits redistribution as a service for >700M MAU
Open-source (fully Apache/MIT licensed weights — GPT-2, BLOOM, Falcon):
- Unrestricted commercial use, redistribution, modification
- Generally smaller models (7B-40B); quality behind LLaMA 3.1 70B
- Requires substantial GPU infrastructure
TOTAL COST OF OWNERSHIP (TCO) MODEL: API cost: output_tokens/day × $X/1K tokens × 365 Self-host cost: GPU_hours × $GPU_rate + eng_ops_cost + infra_overhead Rule of thumb: self-hosting becomes cheaper than API at ~1M+ tokens/day for a mid-size model.
BUILD VS. BUY FRAMEWORK:
- Start with existing model + prompting. Most problems are solvable here.
- Fine-tune if: data is domain-specific, format consistency matters, task is well-defined.
- Pre-train from scratch if: data is proprietary + massive, domain vocabulary is unique, you have $5M+ compute budget. Rare for most companies.
Architecture
Model Selection Workflow:
┌──────────────────────────────────────────────────────────┐ │ STEP 1: Define the task precisely │ │ • What is the input format? │ │ • What does 'good output' mean? │ │ • What are the latency / cost / privacy constraints? │ └─────────────────────────┬────────────────────────────────┘ │ ▼ ┌──────────────────────────────────────────────────────────┐ │ STEP 2: Build an internal task benchmark (50-200 cases) │ │ • Use real or realistic production examples │ │ • Define 'correct' explicitly (reference answers) │ │ • Include hard cases from known failure modes │ └─────────────────────────┬────────────────────────────────┘ │ ▼ ┌──────────────────────────────────────────────────────────┐ │ STEP 3: Candidate short-list │ │ • Use public benchmarks for coarse filtering │ │ (eliminate clear underperformers) │ │ • 3-5 candidates: 1-2 proprietary, 2-3 open-weight │ └─────────────────────────┬────────────────────────────────┘ │ ▼ ┌──────────────────────────────────────────────────────────┐ │ STEP 4: Run internal benchmark on all candidates │ │ • Same eval set, same prompts, same AI judge │ │ • Record: quality score, p50/p95 latency, cost/query │ └─────────────────────────┬────────────────────────────────┘ │ ▼ ┌──────────────────────────────────────────────────────────┐ │ STEP 5: Apply constraints │ │ • Data privacy: HIPAA/GDPR → eliminate proprietary API │ │ • Cost ceiling: calculate TCO for top candidates │ │ • Latency SLA: eliminate models that miss p95 target │ └─────────────────────────┬────────────────────────────────┘ │ ▼ ┌──────────────────────────────────────────────────────────┐ │ STEP 6: Select and baseline │ │ • Choose the model that wins on quality within │ │ cost+latency+privacy constraints │ │ • Record baseline scores in eval DB │ │ • Set calendar reminder to re-evaluate in 6 months │ └──────────────────────────────────────────────────────────┘
Workflow
-
Define task constraints upfront:
- Latency SLA (p95 < 2s? < 200ms?)
- Cost budget ($ per 1K queries)
- Data sensitivity (PII? PHI? financial?)
- Customization need (domain-specific vocab? format?)
-
Build internal task benchmark BEFORE looking at models:
- 50-200 (query, expected_output) pairs from real or synthetic production data
- Use AI-as-Judge or exact match for automated scoring
-
Apply model type filter:
- Sensitive data → eliminate proprietary API; evaluate open-weight
- No GPU budget → evaluate managed open-weight or proprietary API
- Need heavy fine-tuning → eliminate proprietary API
-
Short-list 3-5 candidates using public benchmarks for coarse ranking:
- LMSYS Elo: best for instruction-following comparison
- MMLU + HellaSwag: general knowledge (both saturated, use as filter not ranker)
- HumanEval: for code tasks only
-
Run internal benchmark on all short-listed candidates:
- Same prompt template, same temperature, same eval judge
- Record quality, latency, cost per query
-
Calculate TCO:
- API: tokens/day × cost/token × 365 days
- Self-hosted: GPU_hours/day × $/GPU-hr + ops overhead
-
Select, document the decision, and schedule re-evaluation in 6 months (the model landscape changes faster than most teams track)
Example
# Task-specific benchmark runner for model selection import anthropic, openai, time, json # Load eval set with open('eval_set.json') as f: eval_cases = json.load(f) # [{query, expected_answer}, ...] JUDGE_PROMPT = ''' Compare the AI response to the expected answer. Score 1 (wrong) to 5 (correct and well-formed). Output ONLY JSON: {"score": N} ''' def score_response(query: str, response: str, expected: str) -> int: client = anthropic.Anthropic() result = client.messages.create( model='claude-haiku-4-5-20251001', # cheap judge max_tokens=50, temperature=0, system=JUDGE_PROMPT, messages=[{'role': 'user', 'content': f'Query: {query}\nExpected: {expected}\nResponse: {response}'}], ) return json.loads(result.content[0].text)['score'] def benchmark_model(model_name: str, client, eval_cases: list) -> dict: scores, latencies = [], [] for case in eval_cases: t0 = time.time() # Call model (adapter pattern — both clients support .chat.completions) response = client.chat.completions.create( model=model_name, messages=[{'role': 'user', 'content': case['query']}], max_tokens=512, ) latencies.append(time.time() - t0) text = response.choices[0].message.content scores.append(score_response(case['query'], text, case['expected_answer'])) return { 'model': model_name, 'avg_score': sum(scores) / len(scores), 'p95_latency_s': sorted(latencies)[int(len(latencies) * 0.95)], 'n_cases': len(eval_cases), } # Run on 3 candidates openai_client = openai.OpenAI() results = [ benchmark_model('gpt-4o-mini', openai_client, eval_cases), benchmark_model('gpt-4o', openai_client, eval_cases), ] # Compare: is gpt-4o worth 10x the price for this specific task? for r in sorted(results, key=lambda x: -x['avg_score']): print(f"{r['model']}: score={r['avg_score']:.2f}, p95={r['p95_latency_s']:.2f}s")
Real-world usage
-
Notion's AI team (2023): benchmarked GPT-4, Claude, and Llama 2 on their specific task (document summarization + Q&A) with an internal eval set. GPT-4 won on quality but the cost delta vs. GPT-3.5-turbo was only 3% quality improvement for 10x cost — they chose GPT-3.5-turbo. The decision was made on internal data, not MMLU scores.
-
Stripe (2023): moved from GPT-4 to fine-tuned LLaMA for support ticket classification after internal benchmarking showed the fine-tuned smaller model matched GPT-4 quality at 1/20th the cost on their specific task taxonomy.
-
Chip Huyen (AI Engineering, Ch.4): 'Public benchmarks are useful for a rough shortlist, not for a final decision. Build your own eval set and run every candidate model through it on your actual task.'
-
Healthcare startup pattern: cannot use GPT-4 API (PHI data leaves their cloud tenant). Run LLaMA 3.1 70B on HIPAA-compliant AWS infrastructure. Quality is within 8% of GPT-4 on their clinical note summarization task — acceptable tradeoff for regulatory compliance.
-
LMSYS Chatbot Arena (Stanford): runs 1M+ pairwise human votes to build the most credible public benchmark for instruction-following quality. Used as the industry standard for coarse model ranking.
Trade-offs
Quality vs. cost: GPT-4 is the quality ceiling but 10-100x more expensive than GPT-4o-mini or open-weight. For tasks where quality gap is <5%, the smaller model is almost always the right choice at production scale.
Privacy vs. quality: proprietary APIs have the highest quality ceiling but send data to external servers. Open-weight self-hosted models keep data on-premises but require GPU infrastructure. For regulated industries, privacy constraints often override quality in the selection decision.
Control vs. effort: open-source models give you full control (fine-tuning, customization, zero vendor risk) but require significant engineering effort. Proprietary APIs need only an API key but introduce pricing and terms-of-service risk.
Speed-to-market vs. optimization: starting with a proprietary API (OpenAI, Anthropic) is fastest for initial product validation. If the product scales, switching to a fine-tuned open-weight model can reduce costs 5-20x — plan for this migration from the start by keeping the model layer abstracted.
Visual explanation
Model Type Decision Matrix:
Proprietary API Open-Weight Open-Source
(GPT-4, Claude) (LLaMA, Qwen) (Apache/MIT)
───────────────────────────────────────────────────────────────── Quality ceiling ★★★★★ (highest) ★★★★ (close) ★★★ (varies) Cost/query ★☆☆☆☆ (expensive) ★★★☆☆ (medium) ★★★★★ (cheap) Data privacy ☆☆☆☆☆ (API sees) ★★★☆☆ (cloud) ★★★★★ (local) Customization ★☆☆☆☆ (prompt only) ★★★★☆ (finetune) ★★★★★ (full) Infra work ★★★★★ (none) ★★★☆☆ (some) ☆☆☆☆☆ (heavy) Latency ★★★☆☆ (network) ★★★★☆ (cloud) ★★★★★ (local)
Build vs. Buy Decision Tree:
Does a general model handle this task with good prompting? YES → Use API/open-weight off-the-shelf (buy) NO → Does fine-tuning a smaller model close the gap? YES → Fine-tune open-weight (buy + customize) NO → Do you have sufficient unique data + team + budget? YES → Pre-train from scratch (build) NO → Improve data quality and prompt engineering first
Public Benchmark Red Flags: ┌─────────────────────────────────────────────────────────────┐ │ MMLU saturated: top models 85-90%, random = 25% │ │ HellaSwag saturated: humans = 95%, GPT-4 = 95.3% │ │ HumanEval leaked: many models trained on GitHub Copilot │ │ Always check: does the benchmark data overlap training set?│ └─────────────────────────────────────────────────────────────┘
Advantages
- —
Task-specific internal benchmarks predict production quality far better than general public benchmarks
- —
Model selection framework prevents expensive mistakes — using over-powered models wastes 10-100x budget
- —
TCO analysis makes the self-host vs. API decision quantitative rather than opinion-based
- —
Understanding benchmark limitations prevents being misled by contaminated or gamed scores
- —
Re-evaluation cadence keeps the model selection current as the landscape changes every 3-6 months
Disadvantages
- —
Building an internal benchmark takes time — 2-4 weeks for a good 200-case eval set with labels
- —
The model landscape evolves so fast that a selection made 6 months ago may no longer be optimal
- —
Self-hosted open-weight models require GPU infrastructure expertise that most teams lack initially
- —
TCO calculations have many hidden costs: GPU memory overhead, serving framework ops, model update cycles
- —
Internal benchmarks can develop their own biases if the eval set isn't periodically refreshed with new production samples
Common mistakes
- —
Selecting a model based solely on public MMLU/HellaSwag scores. These benchmarks are saturated at the top end (multiple models within 2% of each other) and don't predict task-specific quality. Always run an internal task benchmark before committing to a model for production.
- —
Ignoring total cost of ownership when comparing API vs. self-hosted. Engineers compare API pricing (e.g., $0.03/1K tokens) against GPU cloud pricing ($X/hour) without accounting for: utilization rate, cold-start penalties, GPU memory over-provisioning, serving framework engineering, and model update cycles. These hidden costs often double the true self-hosting cost.
- —
Treating 'open-weight' and 'open-source' as synonymous. LLaMA 3.1 is open-weight (weights available) but NOT open-source (license restricts redistribution as a service for large-scale use). True open-source models (Apache/MIT license) are less common. Confirm license terms before building a product on a specific model.
- —
Picking the model once and never re-evaluating. The LLM landscape changes every 3-6 months — a model that was best-in-class in Q1 may be mediocre by Q3. Set a calendar reminder to re-run your internal benchmark against new model releases every 6 months.
- —
Benchmarking with a different prompt template than production. A model that looks best in a benchmark may be tuned for a specific prompt format. If your production prompt differs from the benchmark prompt, scores don't transfer. Always benchmark with the EXACT prompt template you'll use in production.
🎤 Interview questions
What is benchmark gaming in the context of LLM evaluation? Give two concrete mechanisms and explain how to detect them.
Walk through the decision framework for choosing between a proprietary API and a self-hosted open-weight model. What are the three most important factors, and how do they interact?
📂 Subtopics
Public Benchmarks: What They Measure and Where They Fail
Critical guide to MMLU, HellaSwag, HumanEval, and LMSYS Arena — saturation, contamination, and how to use them correctly.
~30 min
Proprietary vs. Open-Weight vs. Open-Source: Decision Framework
How to choose between GPT-4/Claude (proprietary), LLaMA/Mistral (open-weight), and truly open-source models.
~35 min