RAG vs Fine-Tuning vs Prompt Engineering: When to Use Each (2026)

Pick prompt engineering first. Add retrieval when the model needs knowledge it was never trained on. Fine-tune only when you need different behaviour, a rigid output format, or a cheaper model doing one narrow job at volume. Almost nobody should fine-tune to teach a model facts. That single rule resolves the majority of real architecture decisions, and the rest of this guide explains the cases where it does not.

The short answer, and why most teams get it wrong

Form, facts, or phrasing

Three techniques get compared endlessly, but they solve different failure modes. Prompt engineering changes what you asked for. Retrieval changes what the model can see. Fine-tuning changes how the model behaves. If you cannot say which of those three your problem is, you are not ready to choose an architecture.

The sharpest version of the rule is this: fine-tuning is for form, retrieval is for facts. When a model produces the right information in the wrong shape, weights are the lever. When it produces confident nonsense about your internal pricing policy, retrieval is the lever. Teams that mix these up spend months training a model to memorise a document set that changes every Tuesday.

The escalation ladder

Order matters more than selection. The sequence that works for most teams looks like this:

•     Prompt. Optimise the system prompt, examples, and output schema first.

•     Cache or long context. If the corpus is small and stable, preload it. Skip the pipeline entirely.

•     Retrieve. Build retrieval when the knowledge base is large, changing, or needs citations.

•     Tune. Only after the first three are exhausted, and only for behaviour or cost.

•     Distil. Compress a working large-model pipeline into a small specialised model.

Most expensive mistakes are sequencing errors. A team skips to step four because fine-tuning sounds more like real AI, then discovers eight weeks later that the problem was a vague system prompt.

What each approach actually does

Prompt engineering: changing the request

Model weights stay untouched. You are shaping the input through system instructions, few-shot examples, chain-of-thought structure, and enforced output schemas. This category has quietly absorbed far more than it used to. Context assembly, prompt caching, and tool selection now sit inside what practitioners call context engineering, and the discipline has grown large enough that treating it as the cheap option undersells it.

RAG: changing what the model can see

Retrieval-augmented generation embeds your documents, indexes them, then finds and injects relevant passages at query time. The model answers an open-book exam instead of a closed-book one. You gain freshness and source attribution, because you know exactly which chunks were used. You pay for it on every single request in both latency and tokens.

Naive RAG does one retrieval pass. Agentic RAG retrieves, evaluates whether the result is sufficient, identifies gaps, then retrieves again. The second pattern is stronger on complex questions and materially slower.

Fine-tuning: changing the model

Supervised fine-tuning, LoRA and QLoRA adapters, preference methods such as DPO, and reinforcement fine-tuning all modify behaviour by updating parameters. Full fine-tuning is effectively obsolete for most teams. A thin adapter on a strong base model is the realistic path.

Fine-tuning for knowledge injection fails for a specific reason: catastrophic forgetting. Pushing new facts into weights degrades capabilities the model already had, and the resulting answers are stale, unverifiable, and impossible to attribute to a source.

The two options missing from most comparisons

Long context with prompt caching, and cache-augmented generation, both deserve a seat at this table. CAG precomputes key-value caches for the entire relevant document set and uses them directly at generation time, removing retrieval from the request path. Where a knowledge base is well-defined and small enough to preload, a full vector pipeline is infrastructure you are maintaining for no benefit.

What the research actually shows

Most comparison articles cite one paper and declare the matter settled. The literature is messier than that, and the mess is where the useful signal lives.

The study everyone cites

Ovadia and colleagues, publishing at EMNLP 2024, compared unsupervised fine-tuning against retrieval across knowledge-intensive tasks. Their finding was consistent: while unsupervised fine-tuning produced some improvement, retrieval outperformed it both for knowledge the model had already encountered during training and for entirely new information. That result is the foundation of the standard advice to use retrieval for facts.

The studies that disagree

A controlled comparison on MedQA-USMLE at the four-billion-parameter scale found close to the opposite. Domain fine-tuning delivered a 6.8 percentage-point gain in majority-vote accuracy over the general baseline, moving from 46.4 percent to 53.3 percent. Retrieval over a medical explanations corpus produced no statistically significant gain in either model, and on the domain-tuned model the point estimate was slightly negative. At that scale, on that benchmark, knowledge encoded in weights beat knowledge supplied in context.

Microsoft's agricultural case study found something different again. Fine-tuning raised accuracy by more than six percentage points, and that gain stacked with retrieval, which added roughly five points on top. Neither technique cancelled the other out.

Reconciling the contradiction

These results are not in conflict once you notice what varies between them. Three factors decide the outcome:

•     Model scale. Small models benefit far more from having domain knowledge baked in. Larger models already carry enough general competence that retrieval is the cheaper way to close the gap.

•     Knowledge popularity. Obscure, low-frequency facts respond better to fine-tuning than well-represented ones, and even that advantage shrinks as models grow.

•     Task shape. Reasoning-heavy multiple-choice work behaves differently from open-ended document question answering.

The practical takeaway: if you are running a small open-weights model on a narrow specialist domain, the standard advice may not apply to you. If you are calling a frontier model on general business documents, it almost certainly does.

A decision framework you can run in ten minutes

As discussed above, the answer depends on your specifics. These five questions get you there faster than any comparison table.

•     Is the output wrong because information is missing, or because the model is behaving incorrectly? This is the primary fork and everything else follows from it.

•     How often does the ground truth change? Anything that updates weekly disqualifies fine-tuning immediately.

•     How large is the knowledge base in tokens? Below roughly 200,000 tokens and stable, preloading beats retrieval infrastructure.

•     What is your query volume? Fine-tuning has to amortise its upfront cost across requests, and low volume means it never does.

•     Do you need citations, auditability, or the ability to delete a record? Data in weights cannot be attributed, audited, or selectively removed, which ends the discussion in regulated sectors.

What each approach costs

Accuracy rarely decides these arguments. Cost does, and the two approaches have opposite cost shapes.

Fine-tuning front-loads spend into training, then gives you a flat and predictable per-request cost. It can also cut per-request tokens sharply, because a tuned model no longer needs a long system prompt attached to every call. Against that, several providers apply an inference premium on custom models, so the saving is not automatic and needs checking against your specific vendor's pricing before you commit.

Retrieval inverts this. Setup is comparatively cheap. Then you pay embedding, vector search, and retrieved-context tokens on every query, forever. At high volume that recurring tax is the dominant line item.

THE NUMBER THAT DECIDES IT

Work out your monthly request volume for the single task you are considering tuning. Below a few tens of thousands of requests a month on one narrow task, training cost has no realistic path to paying itself back, and prompt or retrieval work wins on economics alone.

The cost nobody budgets

Engineering time dwarfs compute for most teams. Prompt iteration is hours. A production retrieval pipeline with chunking, reranking, and evaluation is weeks. A fine-tuning workflow needs dataset construction, training runs, and a regression suite before anyone can trust it. These are also different skill sets with different salaries attached, which makes the choice a hiring decision as much as a technical one.

When to use each, in practice

Translating the framework into concrete situations:

Combining approaches

The interesting production systems do not choose. The agricultural results above showed fine-tuning and retrieval gains stacking rather than competing, and that pattern repeats across domains.

Retrieval-augmented fine-tuning, known as RAFT, is the most useful formalisation. The model is trained on questions paired with retrieved documents, including deliberately irrelevant distractor documents, so it learns to reason over retrieved context and ignore bad retrievals. That targets the single most common retrieval failure mode directly.

A mature production stack usually layers all of it: an engineered system prompt with a tool list, retrieval over proprietary documents, and a small tuned adapter handling high-volume routing or classification. Each technique applied where it is strongest.

The counterweight is real. Combining approaches multiplies the surfaces that can fail, and hybrid systems do not reliably beat the best single approach. Add the second technique when you have measured that the first one is insufficient, not before.

How teams get this wrong

The retrieval quality trap deserves particular attention. Most conclusions that retrieval does not work are chunking problems, embedding problems, or reranking problems wearing an architecture costume. One survey of the field attributed the large majority of errors in production language model applications to incomplete or poorly structured context rather than insufficient model capability. Fine-tuning is the wrong response to bad retrieval, and an expensive one.

Decide with evaluations, not opinions

The next step is the one almost every team skips. You cannot choose between these approaches without a way to measure whether the choice helped.

Assemble fifty representative examples with known-good answers. One afternoon of work. Then run a structured bake-off: score the naive prompt, score an optimised prompt, score retrieval, and only then consider tuning. Stop at whichever step clears your quality bar, because every subsequent step costs more and buys less.

Measure more than accuracy. Track latency at the ninety-fifth percentile, cost per thousand queries, hallucination rate under adversarial input, and how much maintenance each option demands per month. An approach that wins on accuracy and loses on all four of those is not the winner.

BEFORE YOU FINE-TUNE, CONFIRM ALL OF THESE

You have an evaluation set. You have exhausted prompt variations. You have tried retrieval. Your task is narrow and stable. Your volume justifies the training cost. Your knowledge does not change weekly. You have a rollback plan. You can explain what fine-tuning will fix that the previous steps could not. Most teams fail on the first item.

The 2026 shift nobody has priced in

In May 2026, OpenAI notified developers it was winding down its self-serve fine-tuning platform. Organisations that had not previously run fine-tuning lost the ability to create training jobs straight away, access tightened further in July, and from 6 January 2027 no existing customer will be able to create new fine-tuning jobs at all. Inference on already-deployed custom models continues, tied to the lifecycle of the underlying base model.

The stated reasoning matters more than the deadline. Newer base models follow instructions and formats well enough that prompt-based approaches are now cheaper and faster, so fewer use cases genuinely require fine-tuning. The full deprecation schedule is published on OpenAI's official deprecations page.

This does not end fine-tuning as a technique. Google's Vertex AI and Mistral still offer it, and open-weights LoRA adapters are unaffected. What it does end is the assumption that fine-tuning is a safe default you can reach for on any platform. If you built a product on a custom OpenAI model, you now have a migration with a date attached.

So is RAG dead?

The strongest argument against retrieval is that context windows grew large enough to make it unnecessary. That argument was built on a growth curve that has since flattened.

Two things undercut the replacement thesis. Enterprise corpora run to millions of documents, so a one-million-token window covers small projects rather than production knowledge bases. And long inputs degrade: models reliably lose accuracy on facts buried in the middle of a large context, an effect well documented enough to have its own name in the literature.

Retrieval also has not stood still. It absorbed reranking, hybrid search, graph traversal, and agentic multi-step loops, and it now sits inside the wider practice of context engineering alongside caching and tool use. Treating retrieval and long context as rivals is the category error. They are two settings on the same dial, and the question is how much context to assemble and how precisely, not which camp to join.

Which brings the decision back to where it started, with one addition worth carrying into your next architecture review: the technique you choose matters considerably less than whether you can measure the difference it made. Teams with a fifty-example evaluation set and a mediocre architecture consistently outperform teams with an elegant architecture and no way to tell whether it works.