Public Benchmarks: What They Measure and Where They Fail

Public benchmarks are the industry's common language for comparing models, but every major benchmark has known failure modes you must understand.

MMLU (Massive Multitask Language Understanding):

• 57 subjects, 4-choice MCQ, 14,000 questions
• Measures breadth of academic world knowledge
• Random baseline: 25%. GPT-4: 87%. Claude 3 Opus: 86.8%
• Problem: fully saturated at the top. A 1% difference in MMLU (86% vs 87%) does NOT predict meaningful quality differences on generative tasks.
• Use for: coarse filter (eliminate models < 60%). Not for differentiating top models.

HellaSwag:

• Commonsense reasoning, sentence completion
• Human parity: 95.6%. GPT-4: 95.3%.
• Problem: fully saturated. Top models are at human ceiling. Useless for differentiating modern LLMs.

HumanEval:

• 164 Python coding problems, measured by pass@k
• Heavily contaminated: GitHub Copilot training data overlaps eval
• Use for: a rough signal on Python generation capability. Never use as sole criterion for a coding assistant.

LMSYS Chatbot Arena Elo:

• 1M+ pairwise human votes; Bradley-Terry Elo ratings
• Best public benchmark for instruction-following quality
• Limitation: English-dominant, chat-style format, may not predict structured-output tasks

Rule: if a model scores great on public benchmarks but poorly on your internal task eval, trust your task eval. The benchmark served its purpose — it got you to the short-list stage.