Agentic RAG: Letting the LLM Decide When and What to Retrieve

Naive RAG's core weaknesses all trace back to the same root cause: there's no decision-making involved anywhere in the pipeline. It retrieves once, generates once, with no mechanism to modify its own strategy based on the specific problem at hand. If the retrieved context isn't enough, the LLM can't dynamically go search for more. If a query genuinely needs multiple retrieval steps to answer well, traditional RAG falls short because it was never designed to iterate.

Agentic RAG directly addresses this by using AI agents — with planning, reasoning (via patterns like ReAct or Chain-of-Thought), and memory — to orchestrate retrieval from multiple sources rather than running one fixed sequence. It's best suited for complex workflows that require tool use, external APIs, or combining multiple RAG techniques within a single query's handling.

A representative agentic RAG workflow looks like this: the incoming query first gets rewritten/cleaned up. A 'need more detail?' decision agent then determines whether retrieval is even necessary for this specific query, or whether the LLM can answer directly from its own knowledge. If retrieval is needed, a source-selection agent picks which source(s) to query — a vector database, external tools/APIs, or the open internet — rather than always hitting the same fixed vector store. Retrieval happens against the selected source(s), a response gets generated, and then a final relevance-checking agent evaluates whether that response actually answers the original query using the retrieved context appropriately. If it does, the answer is returned. If it doesn't, the system loops back to an earlier step (often the query-rewriting or source-selection stage) and tries again — this continues for a bounded number of iterations, until either the system produces a satisfactory answer or admits it cannot answer the query within the iteration budget.

This makes agentic RAG considerably more robust than naive RAG, since at every step, agentic decision-making ensures individual actions stay aligned with the final goal, rather than blindly executing a fixed sequence regardless of whether it's actually working for this particular query. The cost is real added complexity and latency — you're now running an agent loop with several LLM calls (decision, source-selection, relevance-check) per query instead of naive RAG's single retrieve-then-generate pass.

from openai import OpenAI

client = OpenAI()

def needs_retrieval(query: str) -> bool:
    resp = client.chat.completions.create(model="gpt-4.1", messages=[
        {"role": "user", "content": f"Does answering '{query}' require looking up external information? Answer yes or no."}
    ])
    return "yes" in resp.choices[0].message.content.lower()

def select_source(query: str) -> str:
    resp = client.chat.completions.create(model="gpt-4.1", messages=[
        {"role": "user", "content": f"For '{query}', which source is best: vector_db, web_search, or api? Answer with one word."}
    ])
    return resp.choices[0].message.content.strip().lower()

def is_relevant(query: str, answer: str) -> bool:
    resp = client.chat.completions.create(model="gpt-4.1", messages=[
        {"role": "user", "content": f"Does this answer '{answer}' properly address '{query}'? yes or no."}
    ])
    return "yes" in resp.choices[0].message.content.lower()

def agentic_rag(query: str, sources: dict, max_iterations: int = 3) -> str:
    for _ in range(max_iterations):
        if not needs_retrieval(query):
            return f"Direct answer (no retrieval needed) for: {query}"

        source_name = select_source(query)
        context = sources[source_name](query)
        answer = f"Answer using {source_name}: {context}"

        if is_relevant(query, answer):
            return answer
        # else: loop back and try again with a fresh source decision

    return "Could not produce a satisfactory answer within the iteration budget"