advancedEvaluators — Testing & Safety
Why can't you just use normal unit-test assertions (`assertEquals`) to test LLM output quality?
LLM output is non-deterministic and phrased differently each time even when factually correct, so exact string matching almost always fails even for a 'correct' answer. Evaluators instead use techniques like using a second LLM call to judge whether the output meets criteria (relevance, factual grounding, absence of the actual answer inside the prompt/leakage), producing a score rather than a pass/fail on exact text.
Ready to master this question?
Generate a complete walkthrough — background, the full answer in plain language, a working code example explained line by line, a real-world scenario, common mistakes, and how this same question gets asked in different ways.
Sign in to generate a response