RAG vs fine-tuning: which one do you need?

RAG vs fine-tuning is the question behind most company chatbots: do you give the model the right information when someone asks, or do you change the model so it behaves the way you want? RAG (retrieval-augmented generation) adds knowledge to the prompt at question time. Fine-tuning trains a model on examples so its behavior changes. This guide compares them using what OpenAI, AWS, Anthropic and the original research papers say, checked on October 6, 2026, and ends with a decision guide.

What is the difference between RAG and fine-tuning?

Table: RAG and fine-tuning side by side

The first four rows follow from how each method works and from AWS's description of RAG, which lists avoiding retraining, access to current information and source attribution among its benefits. The last row is our summary, not a vendor figure: price both with your provider's pricing page.

What does OpenAI say about fine-tuning?

OpenAI describes fine-tuning in its model optimization guide this way: "Fine-tuning lets you take an OpenAI base model, provide the kinds of inputs and outputs you expect in your application, and get a model that excels in the tasks you'll use it for." It lists these methods: supervised fine-tuning, direct preference optimization, reinforcement fine-tuning and vision fine-tuning.

It recommends fine-tuning when you need to give the model more examples than fit in a single context window, reduce token costs at scale, train on proprietary data or make a smaller model good enough for a task. Just as important, it says to start elsewhere: write evals that measure your model's output to establish a baseline, and note that "the prompt engineering process may be all you need in order to get great results".

How does LoRA make fine-tuning cheaper?

Training every weight of a large model is expensive. LoRA (low-rank adaptation), from a 2021 paper by Edward Hu and colleagues, freezes the pre-trained weights and injects small trainable matrices into each layer. For GPT-3 175B trained with Adam, the authors report reducing the number of trainable parameters by 10,000 times and the GPU memory requirement by 3 times, with model quality on par with or better than full fine-tuning and no extra latency at inference. Whatever method you use, the adapted model still has to fit in memory when you run it, which is the question in how much VRAM an LLM needs.

What does RAG add?

RAG leaves the model alone and changes what it reads. According to AWS, the system converts your documents into embeddings in a vector database, matches the user's question against them, adds the best passages to the prompt and keeps the index updated. Our guide to what RAG is walks through the steps and Anthropic's measured retrieval failure rates (5.7% at baseline, 1.9% with its best combination of techniques on its own test set). Retrieved text also takes up room in the context window, so the cost of RAG grows with how much you retrieve.

How to choose: a short decision guide

Ask these questions in order:

1. Can a better prompt solve it? OpenAI says to try prompting and evals first. If the answer is yes, you need neither.
2. Is the model missing information? Private documents, prices, policies or anything that changes weekly: use RAG. Retraining a model for facts that will be out of date next month is hard to maintain, and AWS lists avoiding retraining as a benefit of RAG.
3. Do answers need to cite where they came from? RAG, because the passages are known and can be shown.
4. Is the model missing behavior? A fixed output format, a house tone or a narrow task done thousands of times from examples: fine-tuning, which OpenAI also recommends when you want to cut tokens at scale or use a smaller model.
5. Do you need both? Many systems fine-tune for behavior and use RAG for knowledge. They are complementary, so the answer to "RAG or fine-tuning" is sometimes "both, in that order of effort: prompt, then RAG, then fine-tuning".

Common mistakes

• Fine-tuning to teach facts. Information that changes is easier to maintain in documents you can update than in weights you must retrain, and a fine-tuned model has no built-in way to cite a source.
• Building RAG without measuring retrieval. If the right passage is not retrieved, the best model cannot use it. Test retrieval on a list of real questions before judging the answers.
• Skipping the evals. Without a baseline you cannot tell whether either method helped.
• Sending too much text. More retrieved passages cost more tokens and can dilute the answer, as the vendors warn about long contexts.

What to check before you invest

Which of your problems is knowledge and which is behavior; how often the information changes; whether answers must cite sources; who can see the documents you index; and what both options cost with your provider. For estimating the prompt side, use our LLM API cost calculator.