RAG or Fine-Tuning? How to Teach an AI System Your Business
Retrieval-augmented generation or fine-tuning: what each one does, which business needs it fits, why data quality decides the result, and the order in which to try them.
The short answer
Use retrieval (RAG) when the system must answer from your documents, prices, policies and records, and those facts change. Use fine-tuning when you need a consistent format, tone or classification that examples teach better than instructions. Most business systems need retrieval first, careful prompts second, and fine-tuning only when measured results show a gap.
Weighing this for your business? Request a project review

Our take
Axiom Forge starts with the questions the system must answer and the documents that hold the answers. We build the test set before the pipeline, so every later choice — retrieval, prompts or fine-tuning — is judged against the same real questions.
In this article6 sections · 4 min
Retrieval-augmented generation, described by Lewis and colleagues in 2020, gives a model the right pages at the moment of the question. The system searches your content, passes the relevant passages to the model and asks it to answer from them. The knowledge stays in your documents, where it can be updated and where access can be controlled.
Fine-tuning changes the model itself by training it on examples. It teaches behaviour: a house style, a strict output format, a way of classifying cases. It is a poor way to store facts that change, because every change means another round of training.
Start from what the system must do, not from the technique.
| Need | Better fit | Why |
|---|---|---|
| Answers from documents that change | Retrieval | You update the index, not the model |
| Answers with sources a person can check | Retrieval | The passages used can be shown |
| Private data with access rules | Retrieval with permission filters | Access is enforced when searching |
| A strict output format or tone | Fine-tuning or structured outputs | Examples teach form |
| Classification at high volume | A fine-tuned small model | Lower cost and latency per case |
A retrieval system fails quietly when the content behind it is poor: three versions of the same policy, scanned pages with no text, price lists nobody has updated. The model will answer confidently from whatever it is given.
Most of the work is therefore content work: deciding which sources are authoritative, splitting them into meaningful passages, attaching metadata such as date, product and audience, and removing what is out of date. Permissions belong here too. If an employee may not open a document, the assistant must not quote from it.
Collect fifty to a hundred real questions with the answers your experts consider correct. Run every version of the system against this set and check two things: did it find the right passages, and did the answer stay faithful to them.
Without such a set, teams argue from single examples and change things at random. With it, each change is a measured step. The NIST AI Risk Management Framework gives measurement similar weight: Measure is one of its four core functions, with Govern, Map and Manage.
Retrieval adds an indexing pipeline and a search step to every question. Its running cost follows the volume of content and of questions. Its upkeep is mostly editorial: keeping sources current.
Fine-tuning has a training cost and then a maintenance cost: when the base model is replaced by a newer one, the training usually has to be repeated and tested again.
Begin with a strong general model, clear instructions and a few examples in the prompt. OpenAI’s own optimisation guide notes that prompt engineering may be all a use case needs. Add retrieval when answers need your facts. Measure. Consider fine-tuning only when the test set shows a gap that prompts and retrieval do not close, or when volume makes a smaller specialised model worth its upkeep.
The two methods also combine: a fine-tuned model for form and tone, with retrieval for the facts.
Discuss your project with the founder
Is RAG cheaper than fine-tuning?
Usually to start with, yes: there is no training step and the knowledge can be updated by editing documents. At very high volumes, a small fine-tuned model can be cheaper per request for narrow tasks such as classification.
Does fine-tuning stop a model from making things up?
No. Fine-tuning shapes style and behaviour; it does not guarantee facts. Grounding answers in retrieved passages and showing the sources is the more reliable way to reduce invented answers.
Can we use both?
Yes. A common design uses retrieval for the facts and a fine-tuned or carefully prompted model for the format and tone of the answer.


