Iterative Retrieval: The Retrieve → Reason → Retrieve Again Loop

Naive RAG has no way to recover from a bad retrieval — whatever gets retrieved gets used, correct or not, end of story. The agentic RAG workflow's final and arguably most important addition over naive RAG is exactly this recovery mechanism: an explicit retrieve → reason → retrieve again loop, rather than a strictly one-shot pipeline.

In this course's 12-step workflow, this shows up as the final two steps: after a response is generated (from either the direct-answer path or the retrieval path), a final relevance-checking agent evaluates whether that response actually addresses the original query using the retrieved context appropriately. If the answer passes this check, it gets returned to the user. If it doesn't, the system goes back to step 1 — re-running query rewriting, the need-more-detail decision, source selection, and retrieval, potentially with a refined understanding of what went wrong the first time around.

Critically, this loop is explicitly bounded: it continues for a few iterations, not indefinitely, until the system either produces a satisfactory answer or admits it cannot answer the query within the iteration budget. This bound matters a great deal in practice — an unbounded retry loop risks burning unlimited compute and latency on a query that genuinely can't be answered well from the available sources, and a clean 'I cannot answer this confidently' fallback after a fixed number of attempts is far better UX than either an infinite hang or a confidently wrong answer.

This iterative loop is what this course means when it says agentic behavior makes RAG 'much more robust,' since at every step, agentic decision-making ensures individual outcomes stay aligned with the final goal — rather than a fixed pipeline blindly committing to whatever its first (possibly weak) retrieval attempt happened to surface.

from openai import OpenAI

client = OpenAI()

def is_answer_relevant(query: str, answer: str, context: str) -> bool:
    resp = client.chat.completions.create(model="gpt-4.1", messages=[
        {"role": "user", "content":
            f"Query: {query}\nContext used: {context}\nAnswer: {answer}\n\n"
            f"Does this answer properly address the query using the context? yes/no."}
    ])
    return "yes" in resp.choices[0].message.content.lower()

def iterative_agentic_rag(query: str, vector_store, max_iterations: int = 3) -> str:
    current_query = query

    for attempt in range(max_iterations):
        results = vector_store.similarity_search(current_query, k=3)
        context = "\n\n".join(r.text for r in results)

        answer = client.chat.completions.create(model="gpt-4.1", messages=[
            {"role": "user", "content": f"Context:\n{context}\n\nQuestion: {current_query}"}
        ]).choices[0].message.content

        if is_answer_relevant(query, answer, context):
            return answer  # good enough — stop the loop

        # Not good enough — refine the query for the next retrieval attempt
        current_query = client.chat.completions.create(model="gpt-4.1", messages=[
            {"role": "user", "content": f"Rewrite this query to retrieve better context: {current_query}"}
        ]).choices[0].message.content

    return "I cannot confidently answer this query with the available sources."