RAG Pipeline & Vector Indexing
Learn document ingestion, chunking strategies, vector databases, and hybrid search pipelines.
Model Context Protocol creates a unified standard for AI models to query external databases. When combined with local fi
s. When combined with local filesystems, this allows developers to configure custom tools instantly.
instantly.
RAG Pipeline
This diagram traces a query through a complete Retrieval-Augmented Generation (RAG) system. The user's query is first embedded into a vector, which is used to search a Vector DB for semantically similar document chunks (top-K retrieval). Those retrieved chunks become the context injected alongside the original query into the LLM prompt. The LLM generates a response grounded in that retrieved context rather than relying solely on parametric memory. This is why RAG dramatically reduces hallucinations for domain-specific questions — the model cites retrieved evidence, not training-data guesses.
RAG Pipeline
▶📚 Prerequisites(2)
🎓 Learning objectives
- •Contrast different Chunking Strategies (Fixed, Semantic, Recursive)
- •Explain Vector DB Indexing mechanisms (HNSW vs. IVF)
- •Design a basic RAG pipeline with retrieval-grounded generation
What is it?
Retrieval-Augmented Generation (RAG) is an architecture that grounds LLM responses in a specific knowledge base by retrieving relevant documents at query time and injecting them into the prompt context. Instead of relying on parametric knowledge (memorized during training), the LLM reads retrieved text to formulate its answer — like an open-book exam instead of a closed-book one.
Why it exists
LLMs have two critical limitations: knowledge cutoffs (training data has an end date) and unreliable long-tail knowledge (the model may not have seen enough about your specific domain). RAG solves both by making retrieval — not memorization — the source of truth. Any document can become searchable context for any LLM without retraining.
Problem it solves
Without RAG, an LLM answering questions about your internal documentation, product catalog, or recent news will hallucinate — generating confident but wrong answers. RAG connects the LLM to a living knowledge base that can be updated without touching the model, and cites sources so users can verify answers.
Intuition
Imagine an extremely smart consultant who has amnesia about everything specific to your company. They know how to reason, write, and explain clearly — but they have never seen your internal docs. RAG is giving that consultant a search engine over your documents before they answer. They search, read the top results, then compose a response grounded in what they just read.
If you come from Java/Spring Boot: RAG is like a @Repository layer for your LLM service. The LLM is the @Service that composes the response; the vector database is the JPA repository that retrieves relevant "rows" (document chunks). The embedding is the query — instead of SQL WHERE clauses, you use semantic similarity.
If you come from React/Frontend: RAG is like fetching data before rendering. Without RAG, the LLM renders from its internal state only (unreliable). With RAG, you call an API (vector search) first, get the relevant documents as props, pass them into the LLM prompt, and it renders a grounded answer. The fetch() happens before every generation.
Analogy
RAG is like a researcher with a perfect memory of how to write, but who starts fresh on every question. Before answering "what is our refund policy?", they run to the filing cabinet (vector database), pull out the 3 most relevant documents about returns, read them carefully, then write a response based on what they just read — not what they vaguely remember from months ago.
Technical explanation
RAG has two phases:
-
INDEXING (offline): Documents → chunk into 256–512 token pieces with 10–15% overlap → embed each chunk with an embedding model (e.g. text-embedding-3-small, 1536-dim vectors) → store vectors + metadata in a vector database (Pinecone, Chroma, pgvector, Weaviate).
-
RETRIEVAL + GENERATION (online, per query): User query → embed with same model → cosine similarity search against stored vectors → retrieve top-k chunks (k=3–10) → assemble context → send to LLM with retrieved context in the prompt.
Critical: the embedding model used for indexing and querying MUST be identical — switching models invalidates the entire index.
Chunking strategy is the most impactful tuning decision: too small (64 tokens) = semantically incomplete chunks, poor retrieval. Too large (2048 tokens) = noisy embeddings, semantic dilution. 256–512 tokens with overlap is the standard starting point.
Reranking: a cross-encoder reranker (e.g. Cohere Rerank) can rescore retrieved chunks for relevance before generation. Adds ~100ms latency but improves answer quality significantly by filtering false positives from the approximate vector search.
Hybrid search combines dense (embedding) and sparse (BM25 keyword) retrieval. Dense catches semantic matches ("car" ↔ "automobile"); sparse catches exact matches and rare terms. Reciprocal Rank Fusion (RRF) merges the two result lists.
Architecture
INDEXING PIPELINE (run offline or on document update): Documents (PDF, HTML, MD, DOCX) → Document parser (LangChain loaders, LlamaIndex readers) → Text splitter (RecursiveCharacterTextSplitter, chunk=512, overlap=64) → Embedding model (OpenAI text-embedding-3-small or BAAI/bge-m3 locally) → Vector store upsert with metadata {source, page, timestamp}
QUERY PIPELINE (per user request): User query → Embed query (same model as indexing!) → Vector similarity search (top-k=5, cosine) → Optional: reranker (Cohere, cross-encoder) → Context assembly (concatenate chunks, cite sources) → LLM prompt: "Based on context: [chunks] Answer: [query]" → Stream response with citations
Workflow
- Document ingestion: load PDFs/HTML/Markdown using document loaders
- Chunking: split into 512-token pieces with 64-token overlap at sentence boundaries
- Embedding: embed each chunk → 1536-dim vector using text-embedding-3-small
- Index: upsert vectors + metadata (source filename, page number, timestamp) into vector DB
- Query arrives: embed the user question using the same embedding model
- Retrieve: cosine similarity search, return top-5 chunks above 0.75 similarity threshold
- Optional rerank: cross-encoder scores each chunk against the query, reorders results
- Augment: prepend retrieved chunks to prompt with source attribution
- Generate: LLM produces answer grounded in retrieved context
- Return answer + citations (source document names/URLs)
Example
from anthropic import Anthropic import chromadb
client = Anthropic() vdb = chromadb.Client() collection = vdb.get_collection("docs")
def rag_query(question: str) -> str: # 1. Retrieve relevant chunks results = collection.query( query_texts=[question], n_results=3 ) context = "\n\n".join(results["documents"][0])
# 2. Generate grounded answer
response = client.messages.create(
model="claude-opus-4-5",
max_tokens=512,
messages=[{"role": "user", "content":
f"Context:\n{context}\n\nQuestion: {question}\nAnswer based only on the context above."
}]
)
return response.content[0].text
Real-world usage
Notion AI answers questions about your workspace pages — RAG over your Notion documents. The vector index updates incrementally as you edit pages, keeping answers current without model retraining.
GitHub Copilot Chat uses local RAG over your open repository files. When you ask "how does the auth flow work?", it retrieves relevant source files before answering — that is why it gives repo-specific answers.
Intercom and Zendesk AI power customer support bots with RAG over help center articles and past ticket resolutions. Intercom reports 67% ticket deflection rate for customers using AI-powered RAG.
Trade-offs
Chunk size vs retrieval precision: smaller chunks (128 tokens) = precise retrieval but incomplete context for generation. Larger chunks (1024 tokens) = more context but semantically diluted embeddings. The 256–512 range balances both concerns.
Vanilla RAG vs fine-tuning: RAG is better when data changes frequently or needs citations. Fine-tuning is better when you need to change tone/style/format. Advanced teams combine both: fine-tune for style, RAG for facts.
Online vs offline indexing: real-time indexing lets the model answer about documents added seconds ago — but is expensive (every edit triggers re-embedding). Batch indexing (hourly/daily) is cheaper but introduces a freshness lag.
Visual explanation
RAG workflow: User Query ──> Embed Query ──> Query Vector DB ──> Retrieve Top-K chunks ──> Construct Grounded Prompt ──> LLM Output
Advantages
- —
Grounds answers in factual sources
- —
Highly dynamic (database updates are immediate)
- —
Low-cost compared to fine-tuning
- —
Supports citations and source attribution — critical for enterprise trust
Disadvantages
- —
Retrieval latency adds to response times
- —
Complex data pipeline parsing (handling tables is difficult)
- —
Quality degrades if retrieval fails — garbage in, garbage out
Common mistakes
- —
Using different embedding models for indexing vs querying. If you index with text-embedding-ada-002 and later switch to text-embedding-3-small, the vector spaces are incompatible — all retrieved results will be wrong. Always lock and document the exact embedding model version used for each index.
- —
Chunk size too large (>1024 tokens). A 2000-token chunk has an embedding that averages over all its content — the signal for any specific sentence gets diluted. The retriever may return the chunk but the LLM has to read 2000 tokens to find the 2 relevant sentences. Smaller, focused chunks retrieve more accurately.
- —
No overlap between chunks. If a key answer spans the boundary between chunk 3 and chunk 4, neither chunk alone contains the full answer. Use 10–15% overlap (64 tokens for 512-token chunks) to ensure boundary content appears in at least one complete chunk.
- —
Trusting retrieval without verifying faithfulness. The LLM can still hallucinate content that is not in the retrieved chunks — it blends parametric knowledge with retrieved text. Always instruct the model explicitly: "Answer ONLY from the provided context. If the answer is not present, say so."
- —
Not filtering low-similarity results. If you always return top-5 regardless of score, you will include irrelevant chunks when the user asks about something not in the index. Set a minimum similarity threshold (0.70–0.75 cosine) and tell the LLM "no relevant documents found" when nothing passes the threshold.
🎤 Interview questions
Explain the difference between HNSW and IVF vector indexes. What are the speed, memory, and recall tradeoffs?