RAG vs fine-tuning vs both: a decision tree

Every engagement has the same early conversation. The client describes their use case. We listen. Then we draw a decision tree on the whiteboard, and 90% of the time it leads to "RAG first, fine-tune later." The other 10% of the time it doesn't, and getting that judgment wrong wastes a month.

This is the tree.

What each tool actually does

RAG (Retrieval-Augmented Generation) injects facts into the model's context at inference time. The retriever finds relevant documents; the generator (the LLM) reads them and answers the question. The model's weights don't change. Knowledge updates are a database write.

Fine-tuning changes the model's weights. The new weights encode new patterns — domain vocabulary, output formats, reasoning steps, decision policies. Knowledge updates require re-training.

These tools answer different questions. RAG answers "what do I know about X?" Fine-tuning answers "how do I behave when I encounter X?"

The decision tree

Is the gap "knowledge" or "behavior"?
  Knowledge (facts the model doesn't know) → RAG
  Behavior (how the model responds) → fine-tuning
  Both → RAG + fine-tuning

Does the knowledge change frequently (daily, weekly)?
  Yes → RAG (re-training is too slow)
  No → either works

Is the knowledge proprietary / private?
  Yes, and we don't want it in weights → RAG
  Yes, and weights stay private → either

Is the behavior change small (output format, style)?
  Yes → prompt engineering first, fine-tune if needed
  No → fine-tune

Is the behavior change a different skill the base doesn't have?
  Yes → fine-tune, possibly with continued pre-training
  No → prompt + RAG might be enough

When RAG wins

Frequently updated knowledge. Customer support knowledge bases, product catalogs, news, technical documentation. Re-training a model to memorize today's inventory and tomorrow's is a non-starter; RAG handles it with a database update.

Hallucination must be auditable. RAG answers cite their sources. Fine-tuned answers do not. For legal, medical, and financial domains where auditability is a hard requirement, RAG is often the only acceptable answer regardless of other tradeoffs.

Knowledge is large. A model has finite weight capacity. A 70B model has billions of parameters but can't memorize a 10M-document corpus. RAG scales to any corpus size that fits in a vector database.

Knowledge changes by tenant. Multi-tenant SaaS where each customer has private knowledge — fine-tuning per customer is operationally expensive. RAG with per-tenant indices is much cleaner.

When fine-tuning wins

Output format / style. "Always answer in JSON with exactly these keys." "Match this specific tone of voice." RAG can put format instructions in the prompt, but enforcement is unreliable; fine-tuning bakes the format in.

Reasoning patterns. Domain-specific reasoning steps (legal analysis structure, medical differential diagnosis flow) that should happen every time. Hard to coach via prompt, easy to fine-tune.

Compression. A small fine-tuned model can match a much bigger general model on the trained task. Fine-tuned 7B beats general 70B on specific narrow tasks routinely. Cheaper to serve.

Latency. RAG adds retrieval time (50-500ms) and a longer prompt (more prefill). Fine-tuned models skip both.

Specialized vocabulary. Domains with vocabulary the base model rarely saw (rare scientific jargon, specific company internal terms). Fine-tuning gets the vocabulary into the weights so the model understands it as a first-class concept.

When you need both

A lot of production systems combine RAG and fine-tuning. Two common shapes:

RAG for knowledge, fine-tuning for behavior. The model is fine-tuned to follow a domain-specific reasoning structure (e.g., "for legal questions, identify jurisdiction → cite statutes → apply to facts → conclude"). RAG provides the actual statutes and facts. The fine-tune teaches the structure; the retriever fills in the content.

Fine-tuning for vocabulary, RAG for current facts. The model is fine-tuned (continued pre-training, more often) on a domain corpus so it understands the vocabulary as native. RAG handles up-to-date information that changes over time.

The trap is doing both when one would suffice — it's expensive to maintain two systems. Start with whichever is more clearly indicated, ship it, and only add the other when you have data that says you need to.

Common mistakes

Fine-tuning to memorize a corpus. This is the wrong tool. Fine-tuning is bad at memorizing specific facts; it's good at internalizing patterns. If you want the model to recall facts, use RAG. We've seen teams burn a month fine-tuning a model on a 10K-document knowledge base and getting worse performance than a basic RAG setup.

RAG when behavior matters. Putting "respond in JSON" in the prompt and hoping for compliance. It mostly works until it doesn't, and the failures are unpredictable. If the format matters, fine-tune.

Fine-tuning before measuring. "We're going to fine-tune to improve accuracy" without first measuring what the base model + prompt achieves. We've found that prompt engineering closes 30-50% of accuracy gaps for the cost of an afternoon. Fine-tune only what prompt engineering couldn't fix.

Skipping the evaluation harness. RAG and fine-tuning both require measurement to know if they helped. Without an eval set, you're shipping vibes. (We have a whole article on evaluation harnesses — read it before either decision.)

RAG done right

A "basic" RAG setup that works for 80% of engagements:

• Embedding model: BGE-Large-EN-v1.5 or E5-Mistral-7B. Open-weight, MIT-licensed.
• Vector database: Qdrant or Chroma for under 10M chunks. Move to Weaviate or pg_vector at scale.
• Chunking: 512-token chunks with 64-token overlap. Don't get fancy until you've measured.
• Top-k: Retrieve 8, re-rank to 4 with a cross-encoder (BGE-Reranker-Large is the default).
• Prompt: Static system prompt + retrieved chunks (in stable order) + user query. (Stable order matters for prompt cache; see the prompt cache article.)

This setup ships in 2 weeks and beats most custom attempts that started from scratch. Upgrade specific components based on measurement, not folklore.

Fine-tuning done right

A reasonable starting recipe:

• Method: LoRA with rank 16 on attention projections. Move to QLoRA if memory-constrained.
• Data: 500-5000 high-quality examples beat 50K mediocre ones every time.
• Hyperparameters: Learning rate 2e-4 cosine, 3 epochs, batch 16 effective. Tune only if loss curve says to.
• Eval after every epoch on a held-out domain set. Stop training when eval stops improving.
• Mix general data (1-5%) to preserve general capability.

Most engagement-shaped fine-tunes complete in an evening on a single 24GB GPU. The expensive part is data curation, not training.

Cost shape

Indicative ballpark for a deployment over a year:

Approach	One-time	Recurring
Pure RAG	$5-15k engineering	$200-2000/mo vector DB + retrieval compute
LoRA fine-tune	$5-15k engineering + data	$0 (LoRA serves on existing hardware)
RAG + fine-tune	$10-30k engineering	$200-2000/mo + LoRA serving overhead

If recurring matters more than one-time, fine-tuning's no-marginal-cost is attractive. If knowledge changes frequently, RAG's update cost is unavoidable.

The 80% rule

The first answer is almost always RAG. It's faster to ship, easier to update, and easier to debug. Fine-tuning enters the picture when RAG hits its limits — behavior enforcement, latency, narrow domain reasoning.

We have a heuristic on engagements: spend the first two weeks on RAG, measure honestly, and only escalate to fine-tuning if the measurement says we have to. About 60% of engagements end up RAG-only. About 25% add a LoRA on top of the RAG system. About 15% need continued pre-training + LoRA + RAG; those are the expensive ones where the domain is genuinely far from the base.