Inside G-Eval: Auto-Generated Chain-of-Thought and Probability-Weighted Scoring
~13 min read
G-Eval isn't just 'ask an LLM to score with reasoning first' — its real mechanism auto-generates evaluation steps from your criteria, then computes a probability-weighted score across the model's output token probabilities, not just its single printed number.
Inside G-Eval: Auto-Generated Chain-of-Thought and Probability-Weighted Scoring is a Pro topic
Sign in, then upgrade to Pro or Power to unlock this topic and the full AI Engineering curriculum.
Key points
- •G-Eval's auto-CoT step has the LLM itself expand a high-level criterion (e.g. 'rate coherence') into detailed, task-specific evaluation steps — done once per criterion, reused across every example
- •Rather than reading off a single printed score, G-Eval looks at the model's output-token PROBABILITIES for every possible score value at the scoring position
- •The final score is a probability-weighted average across all candidate scores, not just whichever single token had the highest probability
- •This captures genuine model uncertainty on borderline cases (e.g. 4.2, between a 4 and a 5) as a smooth number, rather than forcing an arbitrary discrete choice
- •Probability-weighted scoring produces more fine-grained, less noisy aggregate statistics across many judged examples than averaging discrete integer scores would