Agentic RAG: Architecture & 12-Step Workflow
The specific 12-step agentic RAG blueprint — query rewriting, a 'need more detail?' decision, source selection across vector DB/tools/internet, and a relevance-checking retry loop — that fixes traditional RAG's retrieve-once, reason-never limitations.
Semantic paragraph chunking
PDF textbooks are parsed and split into chunks with 100-character overlaps to keep semantic continuity.
Agentic RAG — 12-Step Workflow (distilled)
A query passes through specialist agents rather than a single fixed retrieval step. The Detail-Need Decision Agent can skip retrieval entirely; the Relevance-Checking Agent can loop the whole thing back for another attempt (bounded by a max iteration count) before falling back to 'cannot answer this query'.
Agentic RAG — 12-Step Workflow (distilled)
▶📚 Prerequisites(2)
🎓 Learning objectives
- •Name the 3 core limitations of traditional RAG that motivate Agentic RAG
- •Trace the full 12-step agentic RAG workflow from query input to final response
- •Explain the role of each agent in the workflow (query rewriter, detail-need decider, source selector, relevance checker)
- •Explain why the retry loop is bounded rather than infinite
What is it?
Agentic RAG introduces agentic behaviors at each stage of a RAG pipeline, rather than treating retrieval and generation as a single fixed, one-shot sequence. Where traditional RAG retrieves once and generates once, Agentic RAG threads multiple specialized agents through the workflow — a query rewriter, a detail-need decider, a source selector, and a relevance checker — so the system can actively think through a task: planning, adapting, and iterating until it arrives at the best solution, rather than blindly following a fixed set of instructions.
Why it exists
Traditional RAG has three specific, named limitations. First, these systems retrieve once and generate once — if the retrieved context isn't enough, the LLM cannot dynamically search for more information; it just generates from whatever was retrieved, insufficient or not. Second, RAG systems may provide relevant context but don't reason through complex queries — if a query genuinely requires multiple retrieval steps (find X, then use X to find Y), traditional RAG falls short, since it has no mechanism for sequential, dependent retrieval. Third, there's little adaptability — the LLM can't modify its retrieval or reasoning strategy based on the specific problem at hand; every query gets the same fixed pipeline regardless of its actual complexity. Agentic RAG exists specifically to fix these three limitations by introducing agents that can rewrite, decide, select, and verify at each stage.
Problem it solves
It solves the 'my RAG system gave an incomplete or wrong answer because the first retrieval pass wasn't enough' problem, by allowing the system to recognize insufficient context and go fetch more instead of generating from whatever it happened to retrieve the first time. It solves the 'my query needs multiple hops of reasoning/retrieval' problem, since traditional single-shot retrieval has no way to chain retrieval steps together. And it solves the 'my RAG system always does the same thing regardless of query difficulty' problem, since Agentic RAG's decision agents adapt the workflow's depth to what the specific query actually needs.
Intuition
Traditional RAG is like a student who reads exactly one page of a textbook, then writes their exam answer regardless of whether that page actually contained enough information — even if the answer they needed was on the next page, they don't go back and check. Agentic RAG is like a student who reads a page, pauses to ask themselves 'do I actually have enough to answer this?', and if not, goes and finds the specific additional page, chapter, or even a different book that's actually needed — repeating this process until they're confident in their answer, or until they've genuinely exhausted reasonable options and have to admit they can't fully answer.
Analogy
It's like the difference between a customer service rep who reads exactly one FAQ article and gives you whatever answer is on it, versus one who reads the FAQ, realizes it doesn't fully address your specific situation, checks two other systems (your account history, a knowledge base), verifies the combined answer actually addresses your question, and only then responds — going back to check more sources if their first combined answer still doesn't quite fit.
Technical explanation
The 12-step workflow, in detail: Steps 1-2 — the user inputs a query, and an agent rewrites it (removing spelling mistakes, simplifying it for embedding, etc.) to improve downstream retrieval quality. Step 3 — another agent decides whether the system needs more detail to answer the (rewritten) query at all, or whether it can proceed directly. Step 4 — if no additional detail is needed, the rewritten query is sent straight to the LLM as a prompt. Steps 5-8 — if more detail IS needed, another agent looks through the relevant sources it has access to (a vector database, tools & APIs, and the internet) and decides which source(s) should be useful for this specific query; the relevant context is retrieved from the chosen source(s) and sent to the LLM as part of the prompt. Step 9 — either path (the direct path from step 4, or the retrieval-augmented path from steps 5-8) produces a response from the LLM. Step 10 — a final agent checks whether the produced answer is actually relevant to the original query and consistent with the retrieved context. Step 11 — if relevant, the response is returned to the user. Step 12 — if not relevant, the system loops back to step 1 and repeats the whole process; this continues for a bounded number of iterations, and if the system still cannot produce a relevant answer after several attempts, it explicitly admits it cannot answer the query rather than looping forever or returning a low-confidence guess. This looped, multi-agent design makes RAG much more robust, since agentic behavior at every step ensures individual outcomes stay aligned with the final goal — at the cost of significantly more LLM calls and latency than single-shot retrieval.
Architecture
Four distinct agent roles compose the workflow: a Query Rewriter (cleans and simplifies the input), a Detail-Need Decider (a binary gate deciding whether retrieval is needed at all), a Source Selector (chooses among vector DB, tools/APIs, and internet search for the retrieval step), and a Relevance Checker (a final quality gate that can trigger the entire loop to restart). The loop-back edge from the Relevance Checker (step 12) back to the Query Rewriter (step 1) is what fundamentally distinguishes this from traditional single-shot RAG — it turns a linear pipeline into a bounded iterative refinement loop.
Workflow
- Implement the Query Rewriting Agent first — even without the rest of the pipeline, cleaning up spelling and phrasing before embedding is a cheap, high-value improvement.
- Implement the Detail-Need Decision Agent as a binary classifier: given the rewritten query (and, in later iterations, prior context), does this need retrieval at all, or can the LLM answer directly?
- Implement the Source-Selection Agent to choose among your available sources (vector DB, tools/APIs, internet) — this doesn't need to be sophisticated initially; even simple heuristics or a classification prompt work as a starting point.
- Wire the retrieval step to actually fetch from whichever source(s) were selected, and assemble that context into the generation prompt.
- Implement the Relevance-Checking Agent as a final gate: does the produced response actually address the original query using the retrieved context appropriately?
- Implement the loop-back with an explicit maximum iteration count (this continues 'for a few iterations' — not indefinitely) and a fallback 'cannot answer this query' response for when the bound is reached without success.
- Treat the specific diagram/step sequence as one of many valid blueprints — adapt the specific agents and their ordering to your actual use case rather than treating this exact 12-step sequence as mandatory.
Example
def rewrite_query(query: str) -> str: ... def needs_more_detail(query: str) -> bool: ... def select_source(query: str) -> str: # 'vector_db' | 'tools_apis' | 'internet' ... def retrieve(query: str, source: str) -> str: ... def generate(query: str, context: str = '') -> str: ... def is_relevant(query: str, response: str, context: str) -> bool: ...
def agentic_rag(original_query: str, max_iterations: int = 3) -> str: query = original_query for _ in range(max_iterations): query = rewrite_query(query) # Steps 1-2 if needs_more_detail(query): # Step 3 source = select_source(query) # Steps 5-6 context = retrieve(query, source) # Steps 7-8 response = generate(query, context) # Step 9 else: context = '' response = generate(query) # Step 4 -> 9 if is_relevant(query, response, context): # Step 10 return response # Step 11 # Step 12: not relevant -> loop back with the same (or refined) query return "I cannot fully answer this query with the available information."
Real-world usage
Perplexity's more advanced 'Pro Search' style modes implement a version of this iterative, source-selecting, relevance-checking loop rather than the single-shot retrieval of a basic search-and-summarize tool. Enterprise research and support copilots that need to combine internal knowledge bases, live APIs, and web search dynamically per query (rather than always hitting all three, or always hitting just one) rely on exactly this source-selection-agent pattern. Coding assistants that decide whether a question needs to search the codebase, check documentation, or can be answered from the model's own knowledge implement a version of the Detail-Need Decision Agent step, avoiding unnecessary retrieval latency on questions that don't actually need it.
Trade-offs
Agentic RAG's iterative, multi-agent design produces meaningfully more robust answers on complex, multi-hop, or ambiguous queries — but at real cost: multiple additional LLM calls per query (rewriter, decider, selector, checker, plus potentially multiple full loop iterations) compared to traditional RAG's single retrieve-then-generate call, translating directly into higher latency and API cost. For simple, single-hop factual queries where traditional RAG already succeeds reliably, this overhead is pure waste — Agentic RAG is worth its cost specifically for queries where traditional RAG's retrieve-once/generate-once/no-adaptability limitations are actually being hit in practice.
Visual explanation
A 12-step flowchart with a loop-back arrow. [1: User Query] → [2: Query Rewriting Agent (fixes spelling, simplifies for embedding)] → [3: Detail-Need Decision Agent] → branches: [4: No, sufficient] → straight to [9: LLM generates response] OR [5-8: Yes, need more] → [Source-Selection Agent chooses from: Vector DB / Tools & APIs / Internet] → retrieves relevant context → [9: LLM generates response] → [10: Relevance-Checking Agent verifies answer against query+context] → branches: [11: Relevant → Return response] OR [12: Not relevant → loop back to Step 1] — with a note that this loop continues for a bounded number of iterations until the system either succeeds or admits it cannot answer the query.
Advantages
- —
Directly fixes traditional RAG's three named limitations: retrieve-once/generate-once, inability to reason through multi-step queries, and lack of adaptability
- —
The relevance-checking loop catches and corrects bad answers before they reach the user, rather than returning a first-attempt answer regardless of quality
- —
Source selection lets the system choose the right retrieval source per query rather than always querying every available source
- —
The bounded retry loop provides graceful degradation — an explicit 'cannot answer' admission rather than an infinite loop or a confidently wrong low-quality answer
Disadvantages
- —
Significantly more LLM calls per query than traditional single-shot RAG, directly increasing latency and cost
- —
Added complexity: 4 distinct agent roles plus a loop-control mechanism, versus a simple linear retrieve-then-generate pipeline
- —
Non-determinism from multiple agent decision points makes behavior harder to predict and debug than a fixed pipeline
- —
Overkill for simple, high-volume, single-hop factual queries where traditional RAG already performs reliably
Common mistakes
- —
Applying the full 12-step agentic loop to every query regardless of complexity, paying its latency/cost overhead even on simple queries traditional RAG would have handled fine
- —
Implementing the loop-back without a maximum iteration bound, risking runaway cost on queries the system genuinely cannot answer
- —
Treating the specific 12-step diagram as a fixed, mandatory blueprint rather than adapting the agent roles and ordering to the actual use case
- —
Skipping the Relevance-Checking Agent step to save cost, losing the quality-gate that catches bad answers before they reach the user
- —
Not implementing an explicit 'cannot answer' fallback for when the retry bound is reached, leaving the system to either loop indefinitely or silently return a low-quality final attempt
📂 Subtopics
How Agentic RAG Differs from Naive RAG: The Agent Decides When and What to Retrieve
Naive RAG runs one fixed retrieve-then-generate sequence for every query. Agentic RAG threads decision-making agents through the pipeline instead, so the system itself decides whether retrieval is needed and where to get it from.
~15 min
Query Decomposition: Breaking Complex Questions into Sub-Queries
A single complex question often can't be answered by one retrieval pass. Query decomposition breaks it into smaller, independently-answerable sub-queries, retrieves for each, then synthesizes a final answer — a common complement to the core agentic RAG loop.
~15 min
Iterative Retrieval: The Retrieve → Reason → Retrieve Again Loop
The book's 12-step agentic RAG workflow includes an explicit retry loop: if a relevance-checking agent decides the answer isn't good enough, the whole process runs again — bounded by a maximum iteration count so it doesn't loop forever.
~15 min
Agentic RAG vs. Traditional RAG: A Decision Framework
Agentic RAG's extra robustness comes with real added latency, cost, and engineering complexity. A concrete decision framework for when that trade-off is actually worth making, versus when naive RAG is the better engineering call.
~15 min