RAG Pipeline Explained — Retrieval-Augmented Generation

Retrieval-Augmented Generation (RAG) is an architecture that grounds LLM responses in a specific knowledge base by retrieving relevant documents at query time and injecting them into the prompt context. Instead of relying on parametric knowledge (memorized during training), the LLM reads retrieved text to formulate its answer — like an open-book exam instead of a closed-book one.

RAG has two phases:

1. INDEXING (offline): Documents → chunk into 256–512 token pieces with 10–15% overlap → embed each chunk with an embedding model (e.g. text-embedding-3-small, 1536-dim vectors) → store vectors + metadata in a vector database (Pinecone, Chroma, pgvector, Weaviate).

2. RETRIEVAL + GENERATION (online, per query): User query → embed with same model → cosine similarity search against stored vectors → retrieve top-k chunks (k=3–10) → assemble context → send to LLM with retrieved context in the prompt.

Critical: the embedding model used for indexing and querying MUST be identical — switching models invalidates the entire index.

Reranking: a cross-encoder reranker (e.g. Cohere Rerank) can rescore retrieved chunks for relevance before generation. Adds ~100ms latency but improves answer quality significantly.

Hybrid search combines dense (embedding) and sparse (BM25 keyword) retrieval. Dense catches semantic matches ('car' ↔ 'automobile'); sparse catches exact matches and rare terms. Reciprocal Rank Fusion (RRF) merges the two result lists.

from anthropic import Anthropic
import chromadb

client = Anthropic()
vdb = chromadb.Client()
collection = vdb.get_collection("docs")

def rag_query(question: str) -> str:
    results = collection.query(query_texts=[question], n_results=3)
    context = "\n\n".join(results["documents"][0])
    response = client.messages.create(
        model="claude-opus-4-5", max_tokens=512,
        messages=[{"role": "user", "content":
            f"Context:\n{context}\n\nQuestion: {question}\nAnswer based only on the context above."}]
    )
    return response.content[0].text