Материалы журнала опубликованы на английском языке. Русская версия сайта

AIAI · KNOWLEDGE · ARCHITECTURE

RAG or Fine-Tuning? How to Teach an AI System Your Business

Retrieval-augmented generation or fine-tuning: what each one does, which business needs it fits, why data quality decides the result, and the order in which to try them.

The short answer

Use retrieval (RAG) when the system must answer from your documents, prices, policies and records, and those facts change. Use fine-tuning when you need a consistent format, tone or classification that examples teach better than instructions. Most business systems need retrieval first, careful prompts second, and fine-tuning only when measured results show a gap.

Weighing this for your business? Request a project review

Illustration: RAG or Fine-Tuning?

Our take

Axiom Forge starts with the questions the system must answer and the documents that hold the answers. We build the test set before the pipeline, so every later choice — retrieval, prompts or fine-tuning — is judged against the same real questions.

Gevorg AntonianFounder & CEO, Axiom Forge
In this article6 sections · 4 min

Two different jobs

Retrieval-augmented generation, described by Lewis and colleagues in 2020, gives a model the right pages at the moment of the question. The system searches your content, passes the relevant passages to the model and asks it to answer from them. The knowledge stays in your documents, where it can be updated and where access can be controlled.

Fine-tuning changes the model itself by training it on examples. It teaches behaviour: a house style, a strict output format, a way of classifying cases. It is a poor way to store facts that change, because every change means another round of training.

Which need points to which method

Start from what the system must do, not from the technique.

Choosing between retrieval and fine-tuning
NeedBetter fitWhy
Answers from documents that changeRetrievalYou update the index, not the model
Answers with sources a person can checkRetrievalThe passages used can be shown
Private data with access rulesRetrieval with permission filtersAccess is enforced when searching
A strict output format or toneFine-tuning or structured outputsExamples teach form
Classification at high volumeA fine-tuned small modelLower cost and latency per case

Retrieval is only as good as the content

A retrieval system fails quietly when the content behind it is poor: three versions of the same policy, scanned pages with no text, price lists nobody has updated. The model will answer confidently from whatever it is given.

Most of the work is therefore content work: deciding which sources are authoritative, splitting them into meaningful passages, attaching metadata such as date, product and audience, and removing what is out of date. Permissions belong here too. If an employee may not open a document, the assistant must not quote from it.

Evaluate with real questions

Collect fifty to a hundred real questions with the answers your experts consider correct. Run every version of the system against this set and check two things: did it find the right passages, and did the answer stay faithful to them.

Without such a set, teams argue from single examples and change things at random. With it, each change is a measured step. The NIST AI Risk Management Framework gives measurement similar weight: Measure is one of its four core functions, with Govern, Map and Manage.

Cost and upkeep

Retrieval adds an indexing pipeline and a search step to every question. Its running cost follows the volume of content and of questions. Its upkeep is mostly editorial: keeping sources current.

Fine-tuning has a training cost and then a maintenance cost: when the base model is replaced by a newer one, the training usually has to be repeated and tested again.

A sensible order of work

Begin with a strong general model, clear instructions and a few examples in the prompt. OpenAI’s own optimisation guide notes that prompt engineering may be all a use case needs. Add retrieval when answers need your facts. Measure. Consider fine-tuning only when the test set shows a gap that prompts and retrieval do not close, or when volume makes a smaller specialised model worth its upkeep.

The two methods also combine: a fine-tuned model for form and tone, with retrieval for the facts.

Discuss your project with the founder

Questions leaders ask

Is RAG cheaper than fine-tuning?

Usually to start with, yes: there is no training step and the knowledge can be updated by editing documents. At very high volumes, a small fine-tuned model can be cheaper per request for narrow tasks such as classification.

Does fine-tuning stop a model from making things up?

No. Fine-tuning shapes style and behaviour; it does not guarantee facts. Grounding answers in retrieved passages and showing the sources is the more reliable way to reduce invented answers.

Can we use both?

Yes. A common design uses retrieval for the facts and a fine-tuned or carefully prompted model for the format and tone of the answer.

Sources

  1. Lewis et al. — Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasksarxiv.org (opens in a new tab)
  2. OpenAI — Model optimization guidedevelopers.openai.com (opens in a new tab)
  3. NIST — AI RMF Coreairc.nist.gov (opens in a new tab)

How Axiom Forge can help

Turn the guidance into an accountable product plan.

Axiom Forge connects product direction, UX, design and engineering for aI product development and business process automation. Start with the business outcome, the people who must use the product and the operating constraints behind it.

Founder-led review · NDA on request · No obligation

See services and starting prices

Write to us

Email us — we reply in English, Russian or Armenian.

Our time now:

Analytics preferences

We use Google Analytics and Google Ads to understand which pages help visitors and to measure enquiries. Form contents and personal details are never sent to them.