RAG: When to Use It

RAG occupies a specific quadrant of the decision space: high need for external knowledge, but low need for behavioral adaptation. This course's takeaway is precise about this: use RAG to generate outputs based on a custom knowledge base IF the vocabulary and writing style of the LLM can remain the same.

This is a meaningfully narrower claim than 'use RAG whenever you need custom knowledge' — it specifically assumes the model's DEFAULT way of writing and reasoning is already appropriate for your task, and the only thing missing is access to information the model wasn't trained on. A support chatbot that needs to answer questions using your company's internal documentation is a clean fit: the model already knows how to write a helpful support response in natural, clear language — it just doesn't know your specific product details, policies, or troubleshooting steps, which is exactly what retrieval supplies.

RAG's other major structural advantage, beyond fitting this knowledge-without-behavior-change quadrant, is that it doesn't require retraining anything when the underlying information changes — update the documents in your vector store, and the very next query automatically reflects the change. This makes RAG particularly well suited to information that's dynamic or frequently updated (current pricing, recent policy changes, live inventory), where fine-tuning's baked-in knowledge would go stale the moment the underlying facts changed and require re-training to catch up.

Where RAG genuinely falls short of the requirement: if the task also needs the model to adopt unfamiliar vocabulary or a meaningfully different writing style — say, matching a very specific internal jargon or a particular tone that differs from the model's natural voice — RAG alone won't get you there, since RAG only changes what information is available to the model, not how the model fundamentally writes. That gap is exactly fine-tuning's territory, covered in the next subtopic.

from openai import OpenAI

client = OpenAI()

def embed(text: str) -> list[float]:
    return client.embeddings.create(model="text-embedding-3-small", input=text).data[0].embedding

def rag_is_the_fit(query: str, vector_store) -> str:
    """A support-bot query needing custom knowledge but NOT a
    behavior change — the model's natural writing style stays fine,
    it just needs access to information it wasn't trained on."""
    results = vector_store.similarity_search(query, k=3)
    context = "\n\n".join(r.text for r in results)

    resp = client.chat.completions.create(
        model="gpt-4.1",
        messages=[{"role": "user", "content":
            f"Using this documentation, answer the question naturally:\n\n"
            f"{context}\n\nQuestion: {query}"}],
    )
    return resp.choices[0].message.content

# Update the vector store's documents and the very next query reflects
# it — no retraining needed, exactly RAG's advantage for dynamic info