Prompting vs. RAG vs. Fine-tuning: A Decision Framework
A 2-axis decision matrix (external knowledge needed vs. behavior adaptation needed) for choosing between prompt engineering, RAG, fine-tuning, or a hybrid — plus a deeper 3-way comparison of full fine-tuning, LoRA, and RAG.
Semantic paragraph chunking
PDF textbooks are parsed and split into chunks with 100-character overlaps to keep semantic continuity.
Prompting vs. RAG vs. Fine-tuning — Decision Matrix
Plotted against two axes: how much external knowledge you need, and how much behavior/style adaptation you need.
| External knowledge needed | Behavior/style adaptation needed | Recommended approach | |
|---|---|---|---|
| Low / Low | Low | Low | Prompt engineering suffices |
| High / Low | High | Low | RAG |
| Low / High | Low | High | Fine-tuning |
| High / High | High | High | Hybrid (RAG + Fine-tuning) |
▶📚 Prerequisites(2)
🎓 Learning objectives
- •Apply the 2-axis decision framework (external knowledge vs. behavior adaptation) to pick the right approach for a new LLM application
- •Explain why RAG changes what a model knows while fine-tuning changes how it behaves
- •Compare full fine-tuning, LoRA, and RAG along the 'what gets modified' and 'what gets stored' dimensions
- •Identify RAG's specific structural limitations (question/answer mismatch, unsuitability for summarization)
What is it?
This is a practitioner decision framework for choosing among the 4 main ways to adapt an LLM to a real application: prompt engineering, fine-tuning, RAG, or a hybrid (RAG + fine-tuning). The decision hinges on two independent parameters: how much external knowledge the task requires, and how much adaptation (changing the model's behavior, vocabulary, or writing style) the task requires. The framework is paired with a deeper technical comparison of full fine-tuning, LoRA fine-tuning, and RAG specifically, since these three are the most commonly confused when the question is really 'how do I add knowledge or behavior to an existing model.'
Why it exists
If you are building a real-world LLM-based app, it's unlikely you can start using the model right away without adjustments — but 'which adjustment' is a genuinely confusing decision with real cost and complexity consequences if gotten wrong. This framework exists to replace ad hoc, guessed technique selection with two clear, checkable questions: does this task need external knowledge the model doesn't have? Does it need the model's behavior/vocabulary/style to change? The answers to those two questions alone are enough to point to the right technique.
Problem it solves
It solves the extremely common mistake of reaching for fine-tuning to fix a knowledge problem, or reaching for RAG to fix a behavior problem — neither works well for the other's job. For instance, an LLM might struggle to summarize company meeting transcripts because speakers use internal vocabulary in their discussions — this is fundamentally a behavior/vocabulary adaptation problem, and RAG (which only adds retrievable knowledge, not vocabulary/style change) would not fix it; fine-tuning would. Conversely, asking an LLM questions about a custom, evolving knowledge base is fundamentally a knowledge problem, and fine-tuning (baking static facts into weights, expensive to update) is the wrong tool; RAG is.
Intuition
Think of an LLM as a brilliant new hire. If they just need a reference manual to look things up in (facts about your company's products, current pricing, specific documents) — that's a knowledge problem, solved by handing them a well-organized binder they can consult (RAG). If they need to actually change how they write, talk, and think — adopting your company's specific jargon, tone, and conventions — that's a behavior problem, solved by actual training/coaching over time (fine-tuning). If they just need clearer verbal instructions for a one-off task and already have all the knowledge and skills needed — that's a prompting problem (prompt engineering). And if they need both a reference binder AND retraining in how they communicate — that's the hybrid case.
Analogy
It's like the difference between giving a new employee a company handbook to consult when needed (RAG — external knowledge, looked up on demand) versus sending them through months of on-the-job coaching to change how they actually write and speak (fine-tuning — behavior change, baked in permanently) versus just giving them clearer verbal instructions for today's specific task (prompt engineering — no lasting change, just better task framing).
Technical explanation
The 2-axis decision framework: use RAG to generate outputs grounded in a custom knowledge base if the vocabulary and writing style of the LLM can remain the same — RAG only adds retrievable facts, it doesn't retrain the model's behavior. Use fine-tuning to change the structure/behavior of the model rather than its knowledge — e.g., adapting to a specific internal vocabulary or writing convention that no amount of retrieved context alone would fix, since retrieved context doesn't change how the model itself writes. Prompt engineering is sufficient if you don't need a custom knowledge base and don't want to change model behavior — it's the cheapest, fastest option when the task is fundamentally about clearer instructions, not new knowledge or new behavior. If your application genuinely needs both a custom knowledge base AND a change in the model's behavior, use a hybrid RAG + fine-tuning approach.
The deeper 3-way technical comparison: (1) Full fine-tuning adjusts the weights of a pre-trained model on a new dataset for better performance — this has worked for a long time on smaller models, but for LLMs specifically it runs into serious problems due to model size, the cost of fine-tuning all weights, and the cost of storing/maintaining every resulting fine-tuned model copy. (2) LoRA fine-tuning addresses these limitations by decomposing some or all of the model's weight matrices into low-rank matrices and training only those, while freezing the original large model — the key insight is that neurons themselves don't determine memory footprint (they just illustrate dimensionality transformation between layers); it's the weight matrices (the connections between layers) that consume memory, and LoRA's added low-rank matrices have dramatically fewer connections than the original weight matrices they approximate, making LoRA far cheaper to train and store than full fine-tuning despite superficially 'adding' a network on top. (3) RAG augments a model with additional information without fine-tuning it at all — the 7-step process is: (1-2) take additional data and embed it into a vector database once (updating incrementally as data evolves, never needing to redo the whole corpus), (3) embed the user's query with the same embedding model, (4-5) find the nearest neighbors to the embedded query in the vector database, (6-7) provide the original query plus the retrieved documents as context to the LLM to generate a response. RAG's own name describes exactly this: Retrieval (accessing information from a knowledge source), Augmented (enriching the generation process with that additional context), Generation (producing the actual text).
RAG's specific structural limitations: RAG relies on similarity matching between the query vector and document vectors, but questions are structurally very different from their answers (a short interrogative query rarely closely resembles a long declarative answer in embedding space), which can cause relevant documents to be missed. Typical RAG systems are consequently well-suited only for lookup-based question-answering — RAG cannot be used to summarize the entirety of a knowledge base, since the LLM never sees all the documents in its prompt; the similarity-matching step only ever retrieves the top matches, not the full corpus.
Architecture
The decision framework has 2 independent axes producing 4 outcomes (Prompt Engineering, RAG, Fine-tuning, Hybrid), while the deeper 3-way comparison (Full FT, LoRA, RAG) differentiates along what actually changes: Full FT modifies all of the model's weights (expensive to train and store per variant); LoRA modifies a small set of additional low-rank weight matrices while the base model stays frozen (cheap to train and store, many variants can share one frozen base); RAG modifies nothing about the model at all — it only changes what's retrieved and placed in the prompt at inference time, leaving the model itself completely untouched.
Workflow
- For any new LLM application, ask the two framework questions explicitly: does this task need knowledge the model doesn't already have? Does this task need the model's behavior, vocabulary, or writing style to actually change?
- If neither: use prompt engineering — the cheapest, fastest starting point, and often sufficient on its own.
- If only external knowledge is needed (vocabulary/style can stay the same): use RAG.
- If only behavior/style adaptation is needed (no new external knowledge required): use fine-tuning — and specifically consider LoRA over full fine-tuning given its dramatically lower training and storage cost for LLM-scale models.
- If both are needed: use a hybrid RAG + fine-tuning approach.
- Before committing to RAG specifically, sanity-check the task against RAG's structural limitations: is this a lookup-style question-answering task (RAG-suited), or does it require summarizing/reasoning across the ENTIRE knowledge base (not RAG-suited, since only top-k matches ever reach the prompt)?
- If choosing fine-tuning over RAG for an LLM-scale model, default to LoRA rather than full fine-tuning unless you have a specific reason full fine-tuning is necessary, given the much lower cost of LoRA's low-rank matrix approach.
Example
def recommend_approach(needs_external_knowledge: bool, needs_behavior_adaptation: bool) -> str: if not needs_external_knowledge and not needs_behavior_adaptation: return 'Prompt Engineering' if needs_external_knowledge and not needs_behavior_adaptation: return 'RAG' if not needs_external_knowledge and needs_behavior_adaptation: return 'Fine-tuning (prefer LoRA over full fine-tuning at LLM scale)' return 'Hybrid: RAG + Fine-tuning'
Illustrative LoRA vs. full fine-tuning parameter-count comparison
def compare_trainable_params(hidden_dim: int, lora_rank: int = 8) -> dict: full_ft_params = hidden_dim * hidden_dim # one weight matrix, full update lora_params = hidden_dim * lora_rank * 2 # two low-rank matrices A, B return { 'full_finetuning_params': full_ft_params, 'lora_params': lora_params, 'reduction_factor': full_ft_params / lora_params, }
e.g. hidden_dim=4096, rank=8 -> LoRA trains ~256x fewer parameters for that matrix
RAG's 7-step process, illustrated
def rag_pipeline(query: str, vector_db, embed_fn, llm_complete) -> str: # Steps 1-2 (offline, done once, or incrementally as data evolves): # vector_db.upsert(embed_fn(doc) for doc in corpus) query_vec = embed_fn(query) # Step 3 docs = vector_db.nearest_neighbors(query_vec, k=5) # Steps 4-5 return llm_complete(query, context=docs) # Steps 6-7
Real-world usage
Customer support copilots that need to answer questions grounded in a constantly-updating product knowledge base (new features, changing policies) are a textbook RAG use case — vocabulary and tone stay standard, only the knowledge changes. Companies adapting an LLM to write in a specific brand voice or internal jargon (legal firms adapting to their own drafting conventions, companies with heavy internal acronym use) are a textbook fine-tuning use case — LoRA specifically, given the cost savings over full fine-tuning at LLM scale. Enterprise assistants needing both up-to-date internal knowledge AND a specific internal communication style (e.g., an internal HR assistant that must both know current policy AND write in the company's exact tone) are the hybrid RAG + fine-tuning case the framework's top-right quadrant describes. Simple, one-off content-generation or reformatting tasks (rewrite this email more concisely) where the model already knows everything needed and doesn't need behavior change are handled by prompt engineering alone, with neither RAG nor fine-tuning ever entering the picture.
Trade-offs
Prompt engineering is the cheapest and fastest but is fundamentally limited to what the model already knows and how it already behaves — no amount of clever prompting can inject genuinely new facts reliably or permanently change writing style. RAG avoids the compute cost of fine-tuning entirely and handles evolving knowledge gracefully (just keep adding to the vector database), but is structurally limited to lookup-style Q&A and cannot summarize or reason across an entire corpus at once. Fine-tuning (especially LoRA) can genuinely change model behavior and absorb domain vocabulary, but requires a training dataset and compute investment, and updating knowledge later means retraining or maintaining separate retrieval on top anyway. The hybrid approach gets both benefits but pays both costs — the added complexity is only worth it when the task genuinely needs both new knowledge and new behavior simultaneously.
Visual explanation
A 2x2 decision matrix. X-axis: 'Amount of external knowledge required' (low → high). Y-axis: 'Amount of behavior/style adaptation required' (low → high).
Bottom-left quadrant (low knowledge, low adaptation): Prompt Engineering suffices. Bottom-right quadrant (high knowledge, low adaptation): RAG. Top-left quadrant (low knowledge, high adaptation): Fine-tuning. Top-right quadrant (high knowledge, high adaptation): Hybrid (RAG + Fine-tuning).
A separate side panel shows the 3-way comparison of Full Fine-tuning, LoRA, and RAG along a 'what's actually happening' axis: Full FT (all weights updated, high storage cost per fine-tuned copy), LoRA (small low-rank matrices trained, base model frozen, low storage cost), RAG (nothing about the model changes at all — additional data is embedded and retrieved at query time from an external vector database).
Advantages
- —
Reduces technique selection to two clear, checkable questions instead of ad hoc guessing
- —
Directly prevents the common and costly mistake of using fine-tuning for a knowledge problem or RAG for a behavior problem
- —
The deeper Full-FT/LoRA/RAG comparison clarifies exactly what gets modified (weights vs. nothing) and what gets stored (fine-tuned copies vs. an external vector database)
- —
Explicitly flags RAG's structural limitations (question/answer mismatch, unsuitability for whole-corpus summarization) so teams don't discover them only after building
Disadvantages
- —
The 2-axis framework is a simplification — some real tasks don't cleanly fall into one quadrant and require judgment calls
- —
Doesn't account for other real-world constraints (data privacy requirements, latency budgets, team ML expertise) that can override the 'ideal' technique choice
- —
The hybrid quadrant is presented simply but combining RAG and fine-tuning well in practice is a genuinely harder engineering problem than either alone
- —
RAG's suitability boundary (lookup vs. summarization) isn't always obvious upfront — some tasks look like lookup problems but actually need broader corpus reasoning
Common mistakes
- —
Reaching for fine-tuning to fix a knowledge-freshness problem (e.g., trying to fine-tune in new product info weekly) when RAG's incrementally-updatable vector database would be far cheaper and simpler
- —
Reaching for RAG to fix a behavior/vocabulary problem (e.g., expecting retrieved context alone to make a model adopt a company's writing style) when fine-tuning is actually required
- —
Choosing full fine-tuning by default when LoRA would achieve comparable results at a small fraction of the training and storage cost for LLM-scale models
- —
Building a RAG system for a task that actually requires summarizing or reasoning across an entire knowledge base, not just answering lookup-style questions — a task RAG is structurally unsuited for
- —
Not considering the hybrid approach when a task genuinely needs both new knowledge and new behavior, instead forcing a single technique to do both jobs poorly
📂 Subtopics
Prompting: When It's Enough
If your task doesn't need a custom knowledge base and doesn't need to change the model's behavior or style, prompt engineering alone is sufficient — the cheapest, fastest option to reach for first.
~10 min
RAG: When to Use It
RAG is the right tool when you need the model to answer from a custom knowledge base, but its vocabulary and writing style don't need to change — knowledge injection without behavior change.
~10 min
Fine-Tuning: When to Use It
Fine-tuning is the right tool when you need to change the model's structure — its behavior, vocabulary, or writing style — rather than give it new knowledge. It changes HOW the model responds, not WHAT it knows.
~10 min
The Decision Flowchart: Combining Both Axes (Including the Hybrid Case)
Putting the two independent axes together into one concrete decision path — including the fourth quadrant, where both knowledge and behavior need to change, calling for a hybrid RAG + fine-tuning approach.
~12 min