Prompting vs RAG vs fine-tuning vs distillation: a decision guide for 2026

Every team building on LLMs reaches the same fork: the answers are not good enough, and four levers are on the table. Write a better prompt, add retrieval, fine-tune the model, or distil a smaller one. People pick by fashion. Fine-tuning feels like the serious option, so it gets chosen to fix problems it cannot fix.

The map has also moved. In May 2026 OpenAI announced it is winding down its self-serve fine-tuning platform, saying newer base models follow instructions and formats so well that prompting is now cheaper and faster. Google and AWS still offer managed tuning, and open models can be tuned anywhere. If you last compared these options a year ago, parts of your mental model are out of date.

This is the framework I use with clients. It separates the levers by what they change, compares cost, data and upkeep, lists who offers what as of 2 October 2026, names the mistakes I see most often, and ends with a decision tree and a checklist.

What each lever actually changes

The cleanest way to think about it is to ask what is wrong with the output. There are three kinds of wrong. The model does not know something (knowledge). It does the wrong thing with what it knows (behaviour). Or it produces the right content in the wrong shape (format). A fourth concern, cost and latency, is separate: the output is fine but too expensive.

Who offers what in 2026

Before choosing a method, check that your provider still sells it. This is the state I could verify from provider documentation and announcements on 2 October 2026.

Three practical consequences. First, "which model can I tune?" now decides more than "which method?": the frontier models most teams actually run in production are mostly not tunable, because I could find no managed fine-tuning for current Claude or GPT-5-class models. Second, RFT is real but young: Google lists it as pre-GA, and Microsoft's own RFT posts recommend starting with deterministic graders. Third, the open-weight route (LoRA on Gemma, Qwen or Llama) is the only one that is portable, and it comes with the operations burden of hosting.

Start with prompting, but measure it

OpenAI's own optimisation guide frames the work as a loop of evals, prompt engineering and fine-tuning, and puts a baseline eval first. I agree with the order, and I would add one rule: you may not conclude that prompting failed until you have tried the strong version of it. That means a clear task statement, a structured output schema, three to ten real examples including hard cases, and a model that is strong enough for the task.

• Put the format contract in a schema or structured-output mode, not in prose.
• Use few-shot examples taken from real failures, not invented ones.
• Cache the stable prefix. A long, repeated prompt is mostly a cost problem, and caching fixes much of it (see cost, latency and routing).
• Try a stronger model before tuning a weaker one. Many "fine-tuning problems" are model-size problems.
• Write the eval set first. Without it you cannot tell if any later change helped (see evals for product features).

Google's tuning documentation gives the same split: prompting suits limited labelled data and rapid prototyping, while tuning is most effective when you have a sizeable labelled dataset (it suggests 100 examples or more) and a task that advanced prompting cannot solve. Tuning's upside it names itself: higher quality on your task and lower latency and cost thanks to shorter prompts.

RAG for knowledge, not fine-tuning

If the model is missing facts, give it the facts at query time. This is where the most expensive mistake happens, so the evidence is worth stating. In Fine-Tuning or Retrieval?, Ovadia and colleagues compared unsupervised fine-tuning with RAG and found that RAG consistently outperformed it, for knowledge the model had seen in training and for entirely new knowledge; models struggled to learn new facts through fine-tuning at all. In Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?, Gekhman and colleagues found that examples introducing new knowledge are learned more slowly, and once learned they linearly increase the tendency to hallucinate. Their conclusion: models acquire factual knowledge mostly in pre-training, and fine-tuning teaches them to use it more efficiently.

The practical reasons are as strong as the research ones. Facts change, and a fine-tuned fact cannot be updated, cited or deleted for a GDPR request without retraining. RAG gives you sources, permissions per document and updates in minutes. The cost is a pipeline you have to build well: chunking, hybrid search and reranking, which I cover in the RAG pipeline guide and in RAG in 2026: hybrid, agentic or long context.

A useful test: if the right answer exists in a document someone could hand you, it is a retrieval problem. If no document could contain it, because the problem is how to respond, it is not.

Supervised fine-tuning for behaviour and format

SFT is the right tool when the problem is consistent behaviour. OpenAI's guidance lists classification, nuanced translation, generating content in a specific format and correcting instruction-following failures, and warns against using SFT to add entirely new knowledge. My additions from projects: strict house style, extraction into a fixed schema where prompts are too long, and compressing a 3,000-token prompt into the weights so every call gets cheaper.

Data. The numbers providers quote are small. OpenAI accepted as few as 10 examples and saw improvements from 50 to 100; Google's guidance says hundreds of labelled examples; AWS's Claude 3 Haiku fine-tuning took between 32 and 10,000 lines. Quantity is not the hard part. OpenAI's best-practice guide says it well: if your model has grammar, logic or style problems, check whether your data has the same problems, and a smaller amount of high-quality data beats a larger amount of low-quality data. Watch the balance too: training data with 60 percent refusals will produce a model that over-refuses.

Method. Parameter-efficient tuning keeps this affordable. The LoRA paper reports roughly 10,000 times fewer trainable parameters and about three times less GPU memory than full fine-tuning of GPT-3 175B, with no added inference latency. Managed services hide this; on open models you choose between LoRA and full tuning yourself.

Maintenance. A fine-tune is a fork of one specific base model. When that model is deprecated, you retrain, and your training data and eval set are the assets that survive. Some platforms also change how you pay: AWS requires Provisioned Throughput to run a customised Claude 3 Haiku, which turns a per-token cost into a standing one.

Preference tuning and RFT: only with a signal

DPO trains on pairs: a prompt, a preferred output and a non-preferred one. OpenAI recommends it for summarising with the right emphasis and for chat with the right tone and style, and its cookbook puts the data need at thousands to tens of thousands of examples. Google recommends running SFT first and preference tuning second, and OpenAI's pipeline likewise lets SFT teach the exact wording before preference training. DPO is a polish step for taste, not a way to teach a task.

RFT is different in kind: the model generates answers during training and a grader scores them. OpenAI's documentation says to start small, from several dozen to a few hundred examples, to see if RFT is useful at all. It fits tasks with a checkable result such as code that passes tests, structured extraction with a known answer or constrained reasoning. Google offers string-match, Gemini-autorater and code-execution rewards, up to 16 combined; AWS reports an average 66 percent accuracy gain over base models for its RFT, which is a vendor claim from selected cases and not a guarantee. RFT does not work where the model has no initial skill or the signal is vague, and a weak grader gets exploited. Writing the grader is the real project; it is an eval by another name, so my guide to evals applies directly.

Distillation: buying back cost and latency

Distillation is the one lever aimed at the bill. A strong teacher model generates outputs on your real inputs, and a smaller student is tuned on them. AWS describes Bedrock Model Distillation as up to five times faster and up to 75 percent cheaper than the original large model, with less than two percent accuracy loss, particularly for RAG use cases. Those are AWS's figures for its own product, so measure on your data. The constraints are instructive: teacher and student must be from the same family, and AWS itself advises using the student as is if it already performs well.

Google's documentation for open models makes the same point from the other side: distillation works best when the teacher is substantially more capable than the student, for example on multi-step reasoning, and gives smaller gains when the student is already close or the task is short-form retrieval. My rule: distil only a stable task with real volume, after you have an eval that proves the student matches the teacher. If your goal is routing between a small and a large model instead, see typed decisions for LLM routing.

A decision tree

The order of the questions is the point. Each one is cheaper to try than the next, and each answer rules out the more expensive levers.

Two notes on reading it. The levers combine: a production system is often a tuned or well-prompted model, with retrieval for facts, behind an eval gate. And the tree restarts whenever the base model changes, because a better model can make yesterday's fine-tune unnecessary, which is exactly what OpenAI says it has seen.

Mistakes I see most often

• Fine-tuning for knowledge. The model recites your product catalogue in training and invents prices in production. Use retrieval.
• Tuning before measuring. No baseline, no eval, so nobody can say whether the tuned model is better. It is often worse on cases nobody tested.
• Dirty training data. Examples copied from past model output or from inconsistent human answers teach the inconsistency.
• Tuning a weak model to rescue a bad task definition. If two humans disagree on the right answer, no method will fix it.
• Ignoring the base-model lifecycle. The fine-tune dies with its base model, and in 2026 the provider may stop offering tuning at all.
• Skipping a regression set. Tuning for one behaviour can quietly damage others, including safety behaviour.
• Leaving thinking on in a tuned task. Google advises setting the thinking budget to 0 (or the level to minimal on Gemini 3 and above) for tuned tasks, which can improve performance and cut cost.

Evaluation and maintenance checklist

Whatever lever you pull, the same discipline applies. This is the checklist I would put in a pull request template for any change to prompts, retrieval or models.

1. Freeze an eval set from real traffic, including failures and edge cases, before changing anything.
2. Record the baseline for the current prompt and model, with quality, latency and cost per task.
3. Change one lever at a time and re-run the full set, not just the cases you were looking at.
4. Keep a regression set for behaviours you must not lose: refusals, safety, formatting.
5. Version the training data, the grader or judge prompt and the base-model identifier next to the tuned model.
6. Set a retraining trigger: base-model deprecation, a drift in eval score or a changed requirement.
7. Calculate break-even: tuning and hosting cost against the tokens you save per month.

What I would do

With a new LLM feature, I would build the eval set in week one, reach a good prompt in week two, and add retrieval the moment facts matter. I would consider fine-tuning only when a stable behaviour keeps failing after that, and only on a platform I trust to still offer it in two years, which today points to Google, AWS or open models rather than a closed API that is retreating. I would try RFT only with a deterministic grader, and distillation only when the monthly bill is large enough to pay for the project.

For most European B2B products, prompting plus retrieval plus a good eval gets 90 percent of the value. That is a judgement from my projects, not a measured number, and your eval set should overrule it. The skill is knowing which kind of wrong you are looking at.

Sources

1. OpenAI community: OpenAI self-serve fine-tuning availability (wind-down announcement)
2. Tessl: OpenAI is shutting down self-serve fine-tuning
3. OpenAI API docs: Model optimization
4. OpenAI API docs: Supervised fine-tuning
5. OpenAI API docs: Direct preference optimization
6. OpenAI API docs: Reinforcement fine-tuning
7. OpenAI API docs: Fine-tuning best practices
8. OpenAI Cookbook: Choosing between SFT, DPO and RFT
9. Google Cloud: Introduction to tuning (Gemini)
10. Google Cloud: About supervised fine-tuning for Gemini models
11. Google Cloud: About preference tuning for Gemini models
12. Google Cloud: Reward functions for reinforcement learning fine-tuning
13. Google Cloud: Supervised and distillation fine-tuning for open models
14. AWS: Fine-tuning for Claude 3 Haiku in Amazon Bedrock is now generally available
15. AWS: Amazon Bedrock now supports reinforcement fine-tuning
16. AWS: Amazon Bedrock Model Distillation (preview announcement)
17. AWS: Bedrock reinforcement fine-tuning adds open-weight models
18. Microsoft Foundry blog: What is new in Foundry fine-tuning, April 2026
19. Microsoft: Announcing new fine-tuning models and techniques in Azure AI Foundry
20. Ovadia et al.: Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs
21. Gekhman et al.: Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?
22. Hu et al.: LoRA: Low-Rank Adaptation of Large Language Models