What is RAG?
Two pipelines rather than one: indexing when documents arrive, querying per request. Each of the four stages introduces its own failure, and retrieval always returns something even when your corpus cannot answer.
Giving the model the page before you ask the question
A model knows what was in its training data. It does not know your contracts, your tickets, or last Tuesday's incident report, and no amount of prompting makes it.
Retrieval-augmented generation is the obvious response: before asking, go and find the relevant passages, put them in the prompt, and ask the model to answer from them. The name is longer than the idea.
Everything interesting is in the word "relevant", which is where the engineering lives.
Two pipelines, not one
RAG is usually drawn as a single flow. It is two, and they run at different times on different schedules.
Indexing happens when documents arrive. Split each document into passages, turn each passage into a vector, and store it with enough metadata to find its way home. This is a batch job. It is also the expensive one, because you pay to embed the whole corpus.
Querying happens per request. Embed the question, find the nearest passages, optionally re-rank them, assemble a prompt, and generate. This is the one with a latency budget.
Most RAG problems are indexing problems discovered at query time. If the right passage was never a chunk, no retrieval strategy will find it.
The four stages, and what each one can ruin
| Stage | Decision | Failure it introduces |
|---|---|---|
| Chunking | where passages split | an answer cut in half |
| Embedding documents | what text becomes a vector | a passage nobody can find |
| Retrieval | which passages come back | the right document, ranked eleventh |
| Generation | what the model does with them | a confident answer the passages do not support |
Reading that table as a debugging order is the most useful thing here. When an answer is wrong, find out which stage lost it before changing the prompt. Retrieve the passages for the failing question and look at them. If the answer is not in what came back, the prompt was never the problem.
What it is not
It is not fine-tuning. Fine-tuning changes how a model behaves. Retrieval changes what it can see. If the complaint is "it doesn't know our product", that is retrieval; if it is "it doesn't write like us", that is closer to fine-tuning or examples. RAG vs fine-tuning works through the distinction.
It is not a memory. Nothing persists between requests except what you put in the store. A conversation that needs to remember what was said three turns ago needs state and historical context, which is a different problem with different answers.
It is not always the answer. A large context window means some corpora fit in the prompt entirely, and a system with ten documents does not need a vector store. RAG use cases covers the conditions where retrieval genuinely wins and the shapes where it does not.
The part everyone underestimates
A retrieval system always returns something. Ask it a question your corpus cannot answer and it will hand back the closest passages it has, with scores that look much like the scores for questions it can answer.
The model then answers from them, because that is what you asked it to do.
This is the single most common way a RAG system embarrasses its owners, and it is not fixed anywhere in the retrieval half. It is fixed in generation, by instructing the model to refuse and by giving it something to refuse with, and it is caught in evaluation by including questions the corpus cannot answer in the test set. Both are covered in generation.
Further reading
- Anthropic, Introducing contextual retrieval: measured improvements at each stage of the pipeline.
- RAG use cases: whether retrieval is the right shape for your problem at all.
- Chunking strategies that survive production: the decision the rest of the pipeline inherits.
- Generation: the last stage, and where refusal has to live.
Knowledge check
Question 1 of 4
Sign in to save your progress and pick up where you left off.