What is RAG (retrieval-augmented generation)?

What is RAG? RAG, short for retrieval-augmented generation, is a technique where a system looks up relevant information in an outside knowledge base and adds it to the prompt before a language model answers. Amazon Web Services describes it as optimizing the output of a model "so it references an authoritative knowledge base outside of its training data sources before generating a response". This guide explains what RAG is, how it works step by step, when it is worth building and when you can skip it, using the primary sources listed at the end (checked October 6, 2026).

What does RAG mean, and where does it come from?

The name comes from a 2020 research paper, "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks" (Patrick Lewis and colleagues, arXiv:2005.11401). The authors describe RAG as models "which combine pre-trained parametric and non-parametric memory for language generation": the parametric memory is what the model learned in training, and the non-parametric memory is an outside store it can consult. Today the word usually means the simpler product pattern: search first, then ask the model with the results in the prompt.

How does RAG work?

AWS lists four steps, which most RAG systems follow in some form:

1. Create the external data. Documents are converted into numerical representations (embeddings) and stored in a vector database.
2. Retrieve the relevant information. The user's question is converted into a vector and matched against the stored vectors.
3. Augment the prompt. The retrieved passages are added to the prompt, with instructions on how to use them.
4. Keep the data current. The documents and their embeddings are updated on a schedule or in batches.

The model then answers using the passages in front of it. Because those passages can be shown to the reader, a RAG system can cite where an answer came from, which AWS lists as one of its main benefits for trust.

Table: what each stage of a RAG pipeline does

What problem does RAG solve?

A model only knows what was in its training data and what is in the prompt. RAG addresses three limits, according to AWS: it avoids retraining a model every time your information changes, it can connect the model to current sources, and it lets users verify answers through source attribution. It also keeps private documents out of the model's weights, because they live in a store you control.

It does not make a model infallible. Retrieval can fail: Anthropic measured a baseline failure rate of 5.7% on its own test set, meaning that in those tests the right passage was missing from the top results about one time in eighteen. Whatever is retrieved is only as good as the documents behind it, and even with good passages a model can state something they do not say, which is why grounding answers in quotes and citations is one of the mitigations in our guide to model hallucinations.

Do you need RAG or can you paste everything into the prompt?

Anthropic's guidance is to skip RAG when the knowledge base is small: if it is smaller than about 200,000 tokens (roughly 500 pages), include it directly in the prompt. Today's windows are much larger, with 1M tokens listed for current Claude models and 1,050,000 for GPT-6.1 Sol, but a bigger window does not remove two problems: you pay for every token you send, and vendors warn that accuracy can fall as the window fills. Our guide to the context window covers the numbers and the costs.

A simple rule: if all your material fits comfortably in one prompt and changes rarely, try the prompt first. If it is large, changes often or must be cited, use retrieval.

How can you make RAG better?

Anthropic's "contextual retrieval" work gives measured ideas. Traditional chunking "often destroys context", so it prepends an explanation of where each chunk comes from before embedding it. Combined with a lexical ranking function called BM25, which is good at exact words, codes and identifiers, and with a reranking step, it reports fewer failed retrievals than the baseline:

Table: Anthropic's reported results on its test set

These are Anthropic's figures on its own data. Your documents will behave differently, so measure your own retrieval: write a list of real questions, note which passage should answer each, and check how often it is retrieved before you judge the model's answers.

RAG, fine-tuning and agents

RAG puts knowledge into the prompt at question time. Fine-tuning changes the model itself, which suits a fixed style or task rather than facts that change. They are not alternatives for every case, and many systems use both: our guide to RAG vs fine-tuning gives a decision guide. An agent can use retrieval as one of its tools, and protocols such as MCP standardize how a model connects to a document store or database.

What to check before you build one

• Which documents go in, and who is allowed to see them: a RAG store can expose anything you index.
• How the knowledge is updated, because stale embeddings give stale answers.
• Whether answers show their sources, so a reader can check them.
• A test set of real questions, to measure retrieval separately from generation.
• The cost of the extra tokens, which you can estimate with our API cost calculator.