LLM-as-Judge: Using a Model to Grade Outputs (G-Eval, Arena)
~14 min read
An LLM-as-judge uses a strong model (GPT-4, Claude) to score or compare outputs against a rubric — flexible enough for open-ended tasks where reference metrics fail, but carrying real biases you must design around.
LLM-as-Judge: Using a Model to Grade Outputs (G-Eval, Arena) is a Pro topic
Sign in, then upgrade to Pro or Power to unlock this topic and the full AI Engineering curriculum.
Key points
- •LLM-as-judge uses a strong model + a rubric to evaluate open-ended outputs where reference metrics (BLEU/ROUGE) fail
- •Two shapes: direct scoring (G-Eval style, score one output against a rubric) and pairwise comparison (Arena style, pick the better of two)
- •Pairwise is usually more reliable than absolute scoring because 'which is better' is an easier, more consistent judgment for a model
- •Design around known biases: position bias (swap orders), verbosity bias (reward concision), self-preference (use a different-family judge)
- •Calibrate the judge against human labels and freeze/version the judge prompt — its scores are a correlated signal, not ground truth