A familiar scene: a team connects a large language model to its internal wiki, asks it about the refund policy, and gets a confident, nicely formatted answer that is completely wrong. The model never saw the policy. It produced something plausible instead. The next question in the meeting is always the same – should we fine-tune it on our documents? Usually the answer is no, or at least not yet. The RAG vs fine-tuning decision is one of the most common in applied AI, and the two approaches solve different problems.
The problem: a model knows the internet, not your company
A general language model has learned language and a vast amount of public knowledge during pre-training. It knows nothing about your product catalogue, contracts, tickets or the policy someone updated on Tuesday. To make it useful at work you need to get that knowledge in front of it. There are two broad routes:
- Retrieval-augmented generation (RAG): at question time, find the relevant passages from your documents and put them into the prompt, so the model answers from them.
- Fine-tuning: continue training the model on your own examples so that its weights change.
They sound interchangeable. They are not.
How retrieval-augmented generation works
A RAG system has two halves.
Indexing (done in advance):
- Split documents into chunks – a few paragraphs each, ideally along headings.
- Turn each chunk into an embedding, a vector of numbers that captures its meaning.
- Store the vectors, the text and metadata (source, date, access rights) in a vector database or a search engine that supports vectors.
Answering (done per question):
- Embed the user's question the same way.
- Retrieve the chunks whose vectors are closest – often combined with classic keyword search, because exact product codes and names are where pure vector search struggles.
- Optionally re-rank the results with a more precise model.
- Send the top chunks plus the question to the language model with an instruction like "answer only from these sources and cite them".
The result: answers grounded in your current documents, with links a user can check. Update a document, re-index it, and the next answer reflects the change. No training run required.
How fine-tuning works
Fine-tuning takes a pre-trained model and trains it further on a dataset of examples – typically pairs of input and ideal output. The weights shift so the model's default behaviour moves toward your examples. Parameter-efficient methods such as LoRA adjust only a small set of added weights, which makes this far cheaper than training from scratch.
What fine-tuning is genuinely good at:
- Format and style. Always reply in this JSON schema, this tone, this structure.
- Specialised tasks. Classifying support tickets into your 40 categories, extracting fields from your invoice layout.
- Shorter prompts. Behaviour you would otherwise spell out in long instructions gets baked in.
What it is poor at: reliably storing facts. A fine-tuned model absorbs facts loosely and can still mix them up, it can't tell you which document an answer came from, and every policy change means preparing data and training again.
RAG vs fine-tuning, side by side
| RAG | Fine-tuning | |
|---|---|---|
| Best for | Facts, documents, changing knowledge | Behaviour, format, narrow tasks |
| Keeping it current | Re-index changed documents | Retrain the model |
| Citations | Natural – you know which chunk was used | Not available |
| Access control | Filter retrieval by user permissions | Very hard – knowledge is in the weights |
| Upfront effort | Data pipeline and search tuning | Curated training examples |
| Main failure mode | Retrieves the wrong passage | Confidently wrong "remembered" facts |
The access-control row deserves attention. If HR documents should only be visible to HR, RAG can enforce that by filtering what gets retrieved for each user. Once information is trained into a model's weights, there is no reliable way to stop it surfacing for someone who shouldn't see it.
Where RAG systems fail (and how to fix them)
RAG is the right default for company knowledge, but a naive build disappoints. The usual culprits:
- Bad chunking. Splitting mid-table or mid-sentence destroys meaning. Chunk along document structure and keep headings with their text.
- Pure vector search. Add keyword search (hybrid retrieval) for names, codes and acronyms.
- No evaluation set. Write 50–100 real questions with known correct sources and measure retrieval quality before tuning prompts.
- Stale or duplicate documents. Three versions of the same policy will produce three different answers. Clean up the source, or rank by recency.
- No "I don't know". Instruct the model to say when the sources don't contain the answer, and test that it does.
Most "the AI is bad" complaints turn out to be retrieval problems. Fix what gets fed to the model before you blame the model.
When to combine them
The two approaches are not rivals. A mature system often uses RAG for knowledge and a small fine-tune for behaviour – for example, a model tuned to always produce a structured answer with citations in your house style, fed by retrieval from your document store. Fine-tuning the embedding model on your domain vocabulary is another targeted use that can noticeably improve retrieval.
A decision checklist
- Does the answer depend on facts that change? → RAG
- Do users need to see where the answer came from? → RAG
- Do different users have different permissions? → RAG
- Is the problem consistent format, tone or a narrow classification task? → Fine-tuning
- Is the prompt growing into pages of instructions and examples? → Consider fine-tuning
- Unsure? → Build RAG first, measure, and fine-tune only for the gaps you can name.
This is the kind of system Mraviko builds for clients, and the order of work above is the one we follow.
The short answer
In the RAG vs fine-tuning debate, think of RAG as giving the model an open book and fine-tuning as changing how it writes. For your company's knowledge – policies, products, contracts, tickets – you almost always want the open book. Save fine-tuning for the behaviours that prompting and retrieval can't get right.
