Insights

RAG vs Fine-Tuning: Which One Fits Your Data

Retrieval hands a model the facts it needs at question time, fine-tuning changes how it writes and decides. Most document problems are knowledge problems, so they start with retrieval.

Share:
RAG vs Fine-Tuning: Which One Fits Your Data

A company updates its price list every month and its HR handbook twice a year. If a model is fine-tuned on those documents in January, by March it answers confidently from the January price list. If the same documents sit in a retrieval index, replacing one file fixes the answer the same afternoon. That is the core of the RAG vs fine-tuning choice: retrieval supplies knowledge at the moment of the question, fine-tuning changes the model's behavior. They solve different problems, and most document problems are knowledge problems.

What each approach changes

Retrieval-augmented generation (RAG) leaves the model untouched. When a question arrives, a search step finds the most relevant passages in your documents and pastes them into the prompt. The model then answers from text it can see. The original RAG paper from 2020 framed the motivation plainly: for models that store facts only in their weights, updating their knowledge and showing where an answer came from "remain open research problems", and an external, searchable memory addresses both.

Fine-tuning changes the model itself. You train it further on examples of the input and output you want, and the weights shift so that the model produces that kind of output by default. Modern fine-tuning on local hardware usually means a parameter-efficient method such as LoRA, which trains a small set of added weights instead of the whole model. The LoRA authors reported that, compared with fully fine-tuning GPT-3 175B, their method cut the number of trainable parameters by 10,000 times and GPU memory by 3 times. That is why fine-tuning is now within reach of a single workstation, but cheaper training does not change what training is good at.

A simple way to hold the difference:

  • RAG answers "what does the model know right now?"
  • Fine-tuning answers "how does the model behave every time?"

Why fine-tuning is a weak way to add facts

It seems natural to "teach" a model your documents by training on them. The evidence says this works poorly. A 2023 study that compared the two methods for knowledge injection found that RAG consistently outperformed unsupervised fine-tuning, both for knowledge the model had partly seen before and for entirely new facts, and that models "struggle to learn new factual information" this way. The authors saw some improvement when the same fact appeared in many rephrasings, which hints at the real cost: to make a fact stick, you have to manufacture training data around it.

Three practical problems follow from that:

  1. No citations. A fine-tuned model cannot point to the page it learned something from, because the fact is spread across its weights. For contracts, policies or medical notes, an answer without a source is hard to trust and hard to audit.
  2. No clean way to forget. When a policy is withdrawn, there is no "delete this document" for weights. You retrain from a corrected dataset.
  3. Blended facts. A model trained on three versions of a handbook can mix clauses from all three in one confident answer.

Retrieval avoids all three because the source text is visible in the prompt and in the answer's citations. It brings its own failure modes, mostly the search step returning the wrong passages, which is why evaluating a RAG system with real staff questions matters before anyone relies on it.

Decision table by problem type

Problem Start with Why
Answer questions from policies, manuals, contracts RAG Facts change and answers need a source
Search a large archive in plain language RAG The archive is the knowledge; the model only reads and summarizes
Data that changes weekly (prices, stock, tickets) RAG, or a direct database query Retraining cannot keep pace
Always output a fixed JSON schema or form layout Fine-tuning, after prompting fails Format is behavior
Write in the firm's house tone Prompting first, then fine-tuning Tone is behavior, examples in the prompt often suffice
Classify tickets or documents into your own categories Fine-tuning A narrow, repeated decision with clear labels
Use internal jargon and abbreviations correctly Both, often RAG supplies the definitions, light fine-tuning fixes habits
Make a small model do one task as well as a larger one Fine-tuning Narrowing the task lets a smaller model compete on it
Answer only from approved text and say "not found" otherwise RAG plus instructions Grounding is a retrieval property, not a training one

Two rows deserve a note. For tone and format, the cheapest step is a better prompt with two or three examples, and many teams never need to go further. For classification, fine-tuning shines because the output space is small and the training labels are easy to check, which is the situation where a held-out test set gives an honest score.

Cost and maintenance of each

The cost profiles differ in shape as well as size. RAG front-loads engineering and then costs little per change. Fine-tuning front-loads data preparation and repeats that cost whenever the target moves.

What RAG costs

  • Build: a document pipeline (extract text, split it into chunks, embed, store), a search index, and a prompt that asks for cited answers. On one server this is the stack described in private RAG over internal documents.
  • Hardware: an embedding model and the chat model you were going to run anyway. No training run.
  • Per change: re-index the changed file. Minutes, not days.
  • Ongoing work: checking retrieval quality, handling new file types, and enforcing document permissions so that search never returns a file the asker could not open.

What fine-tuning costs

  • Build: a labeled dataset. This is the expensive part, and it is mostly human time.
  • Hardware: a GPU with enough memory for the training method. The QLoRA paper showed that a 65-billion-parameter model could be fine-tuned on a single 48 GB GPU using a 4-bit base model plus LoRA adapters, and smaller models need far less.
  • Per change: rebuild or extend the dataset, retrain, re-test against the held-out set, and redeploy.
  • Ongoing work: every new base model release means deciding whether to retrain your adapter on it, because an adapter trained for one base model does not carry over to another.

A worked comparison

Take an illustrative 40-person firm that wants staff to ask questions about 600 internal documents, about 15 of which change each month.

With fine-tuning, each document would need question-and-answer pairs written around it so the facts stick. At 10 pairs per document and 3 minutes per pair to write and check, the first dataset is 600 x 10 x 3 = 18,000 minutes, or 300 hours. Each month, the 15 changed documents mean 15 x 10 x 3 = 450 minutes, 7.5 hours of new pairs, plus a retraining run and a test cycle, and the old facts still have to be scrubbed from the dataset.

With retrieval, the 15 changed files are dropped back into the ingest folder and re-indexed. The human work each month is checking a sample of answers, a few hours at most, and the model can cite the exact page.

Now flip the problem. The same firm wants incoming emails sorted into 12 internal categories. RAG has nothing to retrieve here; the task is a decision. Labeling 1,000 past emails at 20 seconds each is 1,000 x 20 = 20,000 seconds, about 5.6 hours, and a small fine-tuned model can then sort mail quickly on modest hardware. The labels change rarely, so the training cost is paid once. The details of that kind of run are in fine-tuning a local LLM with LoRA.

Why most document projects start with retrieval

A business usually asks for AI over its documents because staff cannot find or remember what those documents say. That is a knowledge gap, and retrieval addresses it directly. It also has three properties that matter on-premise:

  • Permissions stay enforceable. A retrieval layer can filter results by who is asking. Weights cannot. Once confidential text is trained into a model, anyone who can query the model can potentially draw it out.
  • Mistakes are traceable. A wrong answer from RAG can be traced to the passage that was retrieved, which tells you whether to fix the search, the chunking or the source document.
  • The model stays replaceable. When a better base model appears, swap it in and keep the same index. A fine-tuned adapter would need retraining first.

Fine-tuning earns its place later, once retrieval works and a specific, repeated behavior still falls short: the answers are right but the format is wrong, a classifier is needed, or a smaller model must handle a narrow job to save hardware. At that point the two combine well, with a fine-tuned model reading retrieved passages.

A practical decision rule: write down ten questions staff actually ask. If the right answer to most of them lives in a document, build retrieval first. If the right answer is a label, a format or a style that no document contains, consider fine-tuning, and only after a prompt with examples has been tried and measured.

Frequently asked questions

Can RAG and fine-tuning be used together?

Yes, and that is common in mature systems. Retrieval provides the current facts and citations, while a lightly fine-tuned model follows the house format or handles domain wording better. Build and measure retrieval first, so you can tell which component caused any later improvement.

Does a longer context window remove the need for RAG?

Only for small collections. A long context lets you paste in a few long documents, but it costs memory and time on every request, and a few hundred files will never fit at once. Retrieval still has to pick which text goes in.

Can fine-tuning improve a RAG system without touching the chat model?

Yes. The retrieval step uses its own embedding model, and that model can be fine-tuned on pairs of real staff questions and the passages that answer them. This targets the most common RAG failure, the search returning the wrong passages, while the chat model and its citations stay unchanged. Measure retrieval on a fixed question set before and after, so the gain is visible.

How much training data does a fine-tune need?

It depends on the task. A narrow classification or formatting job can improve with a few hundred to a few thousand clean examples, while broad behavior changes need more. The reliable way to know is to train on increasing slices of your data and watch the score on a held-out test set level off.

Ready to Modernize Your Business?

Get your AI automation roadmap in minutes, not months.

Analyze Your Workflows →