RAG Use Cases

What retrieval is for

Retrieval-augmented generation has a narrower original definition than current usage suggests, and the narrow version is the useful one.

The 2020 paper that named it built it for knowledge-intensive tasks, meaning questions whose answers live in a body of text rather than in the model's weights. It reported that retrieval-augmented models generate "more specific, diverse and factual language" than a comparable model working from its parameters alone.

Knowledge-intensive is the operative phrase. The technique was designed for questions with an answer sitting in a document somewhere, and you needed to find the document. Everything else retrieval gets used for is an extension of that, and the extensions are where it disappoints.

The rest of the rag section on this site is about building a pipeline, chunking, storing in a vector DB, evaluating what comes back. What follows is the question those assume you answered.

Three conditions that make retrieval the right tool

Three things have to be true at once. Two is not enough.

The answer is in text you have. Not derivable from it, but in it, as a passage someone could point at. If answering requires combining nine documents in a way none of them states, that is a different problem.

There is too much text to send. If your entire corpus fits comfortably in a prompt, retrieval is machinery you are building to solve a problem you do not have. This threshold has moved, and the last section is about that.

It changes, or you need to cite it. A corpus that never changes and needs no attribution is a candidate for other approaches. One that updates weekly, or whose answers users will challenge, wants retrieval. Updating an index is cheap, and a retrieved passage comes with a source attached.

The third condition is the one people under-weight. Citation is not a feature you add later. A system that produces an answer with no traceable source is a system nobody can audit, and for support, legal, medical or financial content that is disqualifying on its own.

Four problem shapes that fit retrieval

Answering from documentation. The canonical case. A support assistant over product docs, an internal helpdesk over policies, a developer tool over API references. Large corpus, frequent updates, users who need the link.

Search that answers instead of listing. Replacing a results page with a direct answer plus its sources. The corpus already existed; what changes is the output.

Grounding a task in a specific record. A drafting tool pulling this customer's history, a review tool pulling this contract's precedents. The retrieval is narrow and often a database query rather than a similarity search, which is fine. Retrieval does not have to mean embeddings.

Working with a corpus too large or too private to train on. Internal documents, current data, anything under a retention rule. Retrieval reads at request time and stores nothing in the model, which is a compliance property as much as a technical one.

Four problem shapes that do not, and why

These fail for one reason, and it is worth naming once: retrieval returns the passages most similar to your query. Every failure below is that mechanism being asked for something it does not do.

Questions needing every document. "How many of our contracts include an arbitration clause?" Retrieval returns the top k. Top k of four hundred contracts is not an answer to a counting question, and the model will produce a confident number from the subset it saw. This is the failure that most looks like success.

Aggregation and comparison across a corpus. "Which region files the most complaints?" Same problem. You need a query over structured data, and the right architecture is a tool that runs SQL, function calling, not retrieval.

Anything where the answer is a calculation. "What is this customer's outstanding balance?" There is no passage containing it. Retrieving documents that mention balances gets you a plausible-looking wrong number.

Queries whose key terms are exact strings. Error codes, part numbers, ticket IDs, surnames. Embeddings encode meaning, and ERR_CONN_4021 has none. It lands somewhere arbitrary and you get documents about connection errors generally. The fix is hybrid search, combining keyword and vector matching, which Weaviate covers.

A pattern across all four: retrieval fails quietly. It does not return an error. It returns plausible documents and a fluent answer built on the wrong evidence, which is why evaluating retrieval is not optional.

Three alternatives to retrieval, and telling which you needed

Before building a pipeline, check whether one of these is the real answer.

Put it in the prompt. If the material is small and stable (a style guide, a policy page, a product list), include it. No index, no chunking, no embedding model, and prompt caching makes the repeated cost small. This is the alternative most often skipped, and the one that most often would have worked.

Call a tool. If the answer lives in a database, an API, or a calculation, give the model a function instead of a search index. Anything involving current state, exact values, counting, or aggregation belongs here. This is the correct answer to three of the four failures above.

Fine-tune. If the problem is how the model responds rather than what it knows (tone, format, a domain's conventions) retrieval will not help and fine-tuning might.

The diagnostic is one question: is the problem knowledge, behavior, or computation? Knowledge from documents is retrieval. Computation or current state is a tool. Behavior is fine-tuning or a better prompt. Most retrieval projects that disappoint were answering a computation question with a search index.

How a long context window moved the threshold

The second condition, "too much text to send", is the one that has changed since the paper, and any honest article on this subject has to say so.

Current frontier models document context windows of 1M tokens. A corpus that unambiguously required retrieval a few years ago may now fit in a single request, and for a few hundred pages the simplest architecture is to send them.

But the threshold moved; it did not disappear, and two things stop it from disappearing. Cost scales with every token sent, so a large prompt repeated across every request is a bill that grows with traffic rather than with corpus size. And quality does not hold: context is "a finite resource with diminishing marginal returns", models have an "attention budget", and recall degrades as tokens accumulate. Long context processing covers what that looks like in practice.

So the question is no longer "does it fit" but "is retrieving the relevant part cheaper and more accurate than sending all of it", and for a corpus of a few hundred pages the answer is now often no, where a few years ago it was always yes.

The practical order: try the whole thing in the prompt. Measure cost and quality. Build the pipeline when one of them stops being acceptable. That gets a working system today and a retrieval pipeline only if you turn out to need one, which is the opposite of how most of these projects start.

Further reading

• Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks: the original framing, and the word "knowledge-intensive" doing real work.
• Chunking strategies that survive production: the pipeline, once you have decided you need one.
• Context evaluation: finding out whether retrieval is returning the right thing.
• Long context processing: the alternative that moved the threshold.