Fine-tuning vs RAG: which one does your problem actually need?

The one question that decides it

Ask this before anything else: is the model failing because it does not know something, or because it does not behave the way you want?

A knowledge failure looks like the model confidently stating your refund window is 14 days when your policy says 30. It has no way of knowing your policy. No amount of training on tone or format fixes that, because the fact is simply absent.

A behaviour failure looks like the model knowing the answer perfectly well but returning it as three paragraphs of prose when you needed a JSON object, or using the wrong register, or ignoring the structure you asked for on the second turn of a conversation.

Knowledge failures are retrieval problems. Behaviour failures are fine-tuning problems. Getting this backwards is the single most common and most expensive error in applied AI work, because both approaches produce something that looks like progress for several weeks before the underlying problem resurfaces.

Why teams get it backwards

Fine-tuning feels more substantial. It involves GPUs, training runs, loss curves and a produced artefact. Retrieval feels like plumbing. When a team wants to demonstrate that they are doing serious AI work, the training run is the more satisfying thing to point at.

There is also a genuine intuition behind it: if the model does not know something, teach it. That intuition is reasonable and mostly wrong for production systems. A fine-tuned fact is baked into weights that took hours to produce. When the fact changes — and policies, prices, product names and staff lists change constantly — you retrain. A retrieved fact changes when you update a document.

The reverse error is subtler. A team that has built good retrieval starts using it to fix behaviour, stuffing style guides and formatting instructions into the context window. This works, sort of, at the cost of tokens on every single request forever, and it degrades as the instruction block grows.

What retrieval is genuinely good at

Retrieval-augmented generation puts relevant source material in front of the model at request time. Its strengths follow directly from that.

• Facts that change. Update a document, and the next query reflects it. No retraining, no redeployment.
• Facts that must be attributable. A retrieved answer can cite the document and page it came from, which is the only practical way a reader can verify it.
• Corpora too large to memorise. Ten thousand documents will not fit in a context window and should not be compressed into weights.
• Permission-sensitive content. Retrieval can be filtered by what the user is entitled to see. Weights cannot.

That last point deserves emphasis. If you fine-tune a model on documents that only some staff should see, every user of that model has access to that knowledge, and there is no reliable way to take it back out. This is not a theoretical risk — it is the most common serious data-governance failure in enterprise AI deployments.

What fine-tuning is genuinely good at

Fine-tuning changes how the model behaves, in ways prompting can only approximate.

• Output format reliability. Getting valid, correctly shaped JSON on the first attempt rather than the third. This alone often pays for the training run, because retries are where token budgets quietly disappear.
• Domain register and vocabulary. A model that uses your industry's word for a thing rather than the general-purpose synonym.
• Task structure. Multi-step reasoning patterns specific to your workflow that the base model was never shaped around.
• Latency and cost reduction. A tuned smaller model frequently matches a larger model on a narrow task at a fraction of the serving cost, which is the strongest commercial argument for fine-tuning and the one least often made.
• Language performance. Where a base model handles a language or dialect poorly, tuning on good data moves the metric in ways prompting cannot.

Notice that none of these are facts. They are all shapes of behaviour.

The cost comparison nobody runs first

Teams compare the training cost and stop there. The full picture looks different.

Retrieval carries an ongoing cost per request: the retrieved passages consume input tokens on every call, and there is an ingestion and storage cost for the corpus. It carries almost no maintenance cost when content changes — you update the document.

Fine-tuning carries a one-off training cost and then reduces per-request cost, because the behaviour is in the weights rather than in the prompt. Its maintenance cost is where the surprise lives: every time the underlying requirement shifts, you need new data, a new run, and a new evaluation.

The break-even depends almost entirely on how often your requirements change. For a stable extraction task running at high volume, fine-tuning wins decisively within weeks. For a policy assistant in an organisation that revises policy quarterly, retrieval wins permanently.

The answer is usually both

In production, the two are not alternatives. The strongest systems fine-tune for behaviour and retrieve for knowledge, and the combination is better than either alone by a wide margin.

A well-built support assistant might be tuned to produce a specific response structure, adopt the right register, and know when to escalate — while retrieving current policy, current pricing and the customer's own history at request time. The tuning handles how it responds. The retrieval handles what it knows.

Build the retrieval layer first. It is faster to stand up, easier to evaluate, and it will tell you honestly whether your remaining failures are knowledge failures or behaviour failures. Then tune for whatever is left.

A decision checklist

Work down this list. The first line that matches is your answer.

• The information changes more than once a quarter → retrieval
• A reader needs to verify the answer against a source → retrieval
• Different users must see different subsets of the information → retrieval
• The corpus is larger than a context window → retrieval
• The output format or schema is wrong more than occasionally → fine-tuning
• You are paying for a large model to do a narrow task → fine-tuning
• The model handles your language or dialect poorly → fine-tuning
• Prompts have grown past roughly 500 tokens of pure instruction → fine-tuning

If several lines match across both columns, that is not ambiguity. That is a system that needs both, and you should build it in that order.