Advanced RAG Architectures: HyDE, CAG, REFRAG
Explore Advanced RAG architectures: HyDE, Corrective RAG, Graph RAG, REFRAG, CAG, and Agentic RAG.
Semantic paragraph chunking
PDF textbooks are parsed and split into chunks with 100-character overlaps to keep semantic continuity.
▶📚 Prerequisites(1)
🎓 Learning objectives
- •Explain the difference between RAG and Context-Augmented Generation (CAG)
- •Detail HyDE query expansion steps
- •Describe Graph RAG entity extraction
What is it?
Advanced RAG architectures extend naive retrieve-then-read pipelines to handle three failure modes: (1) retrieval failure — wrong chunks fetched due to query/document mismatch; (2) context failure — right chunks fetched but the LLM ignores them or misreads them among noise; (3) staleness / global context failure — the answer requires cross-document reasoning that chunk-based retrieval cannot provide. Techniques: HyDE, Corrective RAG (CRAG), Graph RAG, REFRAG, Cache-Augmented Generation (CAG), Agentic RAG, and multi-hop retrieval.
Why it exists
Naive RAG pipelines retrieve irrelevant chunks due to query wording mismatch, and fail on global cross-document summary questions.
Problem it solves
Retrieval failures, dilute prompts, context misalignment, and slow vector search latency.
Intuition
RAG can be improved. Traditional search compares user queries directly with documents. "HyDE" fixes this: the AI first drafts a fake, hypothetical answer to the question, and searches for paragraphs that look like that fake answer (often matching document style better). "CAG" preloads the whole book directly in the AI's fast memory (KV Cache), avoiding search latency entirely. "Graph RAG" connects dots like a mind map.
Analogy
Traditional RAG is like looking up terms in the book index. HyDE is writing down what you think the answer should look like, then finding pages that match your draft. CAG is photocopy-pasting the entire handbook directly onto the exam paper.
Technical explanation
HyDE (Hypothetical Document Embeddings): instead of embedding the raw query, prompt an LLM to generate a hypothetical ideal document that would answer the query, then embed that document and use it as the retrieval vector. Bridges the query–document embedding gap: queries are short/interrogative; documents are long/declarative.
CRAG (Corrective RAG): adds a retrieval evaluator that scores each retrieved chunk as {correct, incorrect, ambiguous}. If all chunks are 'incorrect', CRAG triggers a web search fallback before generating. Prevents hallucination from irrelevant context.
Graph RAG (Microsoft): builds a knowledge graph of entities and relationships from the corpus at index time. At query time, traverses the graph to retrieve connected entity clusters rather than isolated chunks — enables global summarization and multi-hop reasoning.
CAG (Cache-Augmented Generation): preloads the entire knowledge base into the LLM's KV-cache once, then generates without any retrieval at query time. Works only when the corpus fits in context (≤200K tokens). Sub-millisecond retrieval latency.
REFRAG: decomposes multi-part queries into sub-questions, retrieves independently for each, then synthesizes. Similar to query decomposition / step-back prompting.
Architecture
Production Advanced RAG pipeline components (Building LLMs for Production Ch.9):
Index-time: Document chunker → Chunk embedder (bi-encoder) → Vector store (FAISS/Pinecone/Weaviate) Optional: entity extractor → Knowledge graph (Neo4j) for Graph RAG Optional: document preloader → KV-cache fill (CAG)
Query-time: Query analyzer → [HyDE generator if enabled] → Query embedder → ANN search (vector store) → Retrieval evaluator (CRAG) → [web fallback if needed] → Cross-encoder reranker (Cohere Rerank / BGE reranker) → Top-K chunk selector → Context assembler → LLM → Response
Key metrics: Retrieval precision@K, Answer faithfulness (RAGAS), Answer relevance (RAGAS), Context utilization.
Workflow
- Receive query -> 2. Run HyDE expansion -> 3. Retrieve database chunks -> 4. Apply cross-encoder rerank -> 5. Filter top 3 -> 6. Generate response.
Example
HyDE implementation
from openai import OpenAI import numpy as np
client = OpenAI()
def hyde_retrieve(query: str, vector_store, top_k: int = 5): # 1. Generate hypothetical document hyp_doc = client.chat.completions.create( model='gpt-4o-mini', messages=[{'role': 'user', 'content': f'Write a paragraph that directly answers: {query}'}] ).choices[0].message.content
# 2. Embed the hypothetical doc (not the query)
emb = client.embeddings.create(
input=hyp_doc, model='text-embedding-3-small'
).data[0].embedding
# 3. Retrieve using hypothetical doc embedding
return vector_store.search(np.array(emb), top_k=top_k)
CRAG relevance check
def evaluate_chunk(query: str, chunk: str) -> str: prompt = f'Is this chunk relevant to the query?\nQuery: {query}\nChunk: {chunk[:300]}' resp = client.chat.completions.create( model='gpt-4o-mini', messages=[{'role': 'user', 'content': prompt}] ).choices[0].message.content.lower() return 'correct' if 'yes' in resp else 'incorrect'
Real-world usage
Notion AI uses HyDE to improve retrieval over user notebooks — raw user queries are often vague keywords, while generated hypothetical documents match note-writing style. Microsoft's GraphRAG (open-source) is used for legal document analysis where answering 'what are all the obligations in these contracts?' requires cross-document entity aggregation that chunk-based RAG fails at. CAG is used in customer support bots with small, stable FAQ bases — the entire FAQ is preloaded into context, eliminating retrieval latency and chunk boundary errors. Cohere Rerank is used as the cross-encoder reranking step in production RAG at companies like Glean and Perplexity — reranking top-100 bi-encoder results to top-5 improves answer accuracy by 20–30% vs. using bi-encoder results directly.
Trade-offs
CAG saves latency but costs high VRAM, whereas RAG saves VRAM but incurs retrieval latency.
Visual explanation
HyDE vs. Cache-Augmented Generation (CAG): HyDE: [Query] ──> [LLM: Mock Doc] ──> [Embed] ──> [Vector Search] ──> [Retrieve Chunks] ──> [LLM Answer] CAG: [Knowledge Base] ──> [Pre-fill KV Cache] ──> [User Query] ──> [LLM (Immediate Generation)]
Advantages
- —
HyDE drastically improves retrieval accuracy
- —
CAG offers sub-second generation by skipping database queries
Disadvantages
- —
Graph RAG requires heavy entity extraction costs
- —
CAG consumes massive GPU memory for cached prompts
🎤 Interview questions
What is Graph RAG community summarization? How does it solve query synthesis over large archives?