RAG-Specific Evaluation with RAGAS

A Retrieval-Augmented Generation system has two places it can go wrong, and a single end-to-end score can't tell them apart. It can retrieve the wrong documents (a retrieval failure), or retrieve the right documents but generate an answer that ignores or contradicts them (a generation failure). RAG-specific evaluation, popularized by the RAGAS framework, breaks the quality question into component metrics so you know WHICH half to fix. A clever property of several of these metrics is that they use an LLM to judge the relationships between question, retrieved context, and answer — so many of them need no human-written gold answer at all.

The four core RAGAS metrics map cleanly onto the two failure surfaces.

Generation-side:

• Faithfulness asks: is every claim in the answer actually supported by the retrieved context? It breaks the answer into individual claims and checks each against the context. Low faithfulness means the model is hallucinating beyond its sources — the classic RAG failure. This is often the single most important metric because it directly measures grounding.
• Answer relevancy asks: does the answer actually address the user's question (rather than being on-topic but evasive or padded)? It works by generating questions the answer WOULD be a good response to, then measuring how close those are to the real question.

Retrieval-side:

• Context precision asks: of the chunks we retrieved, how many are actually relevant, and are the relevant ones ranked near the top? Low precision means your retriever is dragging in noise.
• Context recall asks: did we retrieve ALL the information needed to answer? This one typically needs a reference answer to check whether every needed fact was present in the retrieved context. Low recall means the answer was doomed before generation — the facts simply weren't fetched.

Reading them together is what makes them powerful. High context recall but low faithfulness -> retrieval is fine, fix the prompt/model so it stops hallucinating past its sources. Low context recall -> the retriever/chunking is the problem, no amount of prompt tuning will help. High precision but low recall -> you're retrieving clean but incomplete context, so increase top-k or improve chunking. This componentized view is exactly why you evaluate RAG differently from a plain LLM: you're debugging a pipeline, not a single black box.

# Sketch of the RAGAS evaluation shape. Real RAGAS uses an LLM to
# score each metric; here we illustrate what each metric inspects.

# pip install ragas datasets
from dataclasses import dataclass

@dataclass
class RagSample:
    question: str
    contexts: list[str]   # retrieved chunks
    answer: str           # generated answer
    ground_truth: str     # reference (needed for context recall)

def faithfulness_claims_supported(answer_claims, contexts) -> float:
    """Generation-side: fraction of answer claims backed by context."""
    joined = " ".join(contexts).lower()
    supported = sum(1 for c in answer_claims if c.lower() in joined)
    return supported / len(answer_claims) if answer_claims else 0.0

sample = RagSample(
    question="When was the Eiffel Tower completed?",
    contexts=["The Eiffel Tower was completed in 1889 for the World's Fair."],
    answer="The Eiffel Tower was completed in 1889.",
    ground_truth="1889",
)
print("faithfulness:", faithfulness_claims_supported(
    ["The Eiffel Tower was completed in 1889"], sample.contexts))

# Real usage:
#   from ragas import evaluate
#   from ragas.metrics import faithfulness, answer_relevancy, \
#       context_precision, context_recall
#   result = evaluate(dataset, metrics=[faithfulness, answer_relevancy,
#                                       context_precision, context_recall])