Retrieval Process

Retrieve wide, then narrow

The instinct is to ask the store for the five best passages and pass them on. The better shape is two stages: get a generous candidate set cheaply, then spend something to order it properly.

Anthropic's contextual retrieval evaluation uses exactly that split. It retrieves the top 150 chunks, then reranks to select the top-20 for the model. The reranking step alone takes the top-20 failure rate from 2.9% to 1.9%, a 67% reduction against the 5.7% baseline.

The reason the two-stage shape works is that the two stages are good at different things. A vector search over millions of passages has to be fast, so it uses an approximate index and a cheap distance. A reranker looks at the query and the passage together and can afford to, because it is only looking at 150.

The stages, in order

Prepare the query. Embed it with the same model the corpus used, and apply whatever task convention that model expects. Encoding queries as passages is accepted silently and degrades everything.

Filter before you search, not after. A filter applied after the vector scan throws away results you already paid for and can return fewer than you asked for. Performing similarity search covers the ordering; RAG with dynamic filters covers the case where the filter is a tenancy boundary and a bug is a breach.

Search both indexes. Vector for meaning, BM25 for terms, fused. Anthropic's evaluation is blunt about it: embeddings combined with BM25 beat embeddings on their own.

Rerank. A cross-encoder scores each candidate against the query directly. This is the expensive step per candidate, which is why it runs on 150 rather than on the corpus.

Assemble. Take the top n, deduplicate, and order them for the prompt.

Choosing how many to pass on

Two numbers, and they answer different questions.

Candidates. How many the first stage returns. Bounded by what reranking costs. More candidates raise the chance the right passage is in the set at all, which is the only thing the first stage can be judged on.

Final passages. How many reach the model. Bounded by context and by attention. This is where people over-supply: twenty passages of eight hundred tokens is sixteen thousand tokens of context, paid on every request, and long context processing covers what happens to recall when the relevant passage is buried among fifteen irrelevant ones.

Measure the two separately. Recall at the candidate stage tells you whether retrieval can succeed; precision in the final set tells you whether it does.

Deduplication matters more than it sounds

Chunks overlap by design, and documents get re-ingested. Without a deduplication pass, the top five passages are frequently three copies of one paragraph.

Wasted tokens are the smaller cost. A model given the same claim three times treats it as better supported than a claim stated once, so duplication quietly biases the answer toward whatever happens to be chunked twice.

Knowing whether it works, without labelling everything

"Is the answer good" confounds retrieval with generation. The measurement that matters here is narrower: did the passage containing the answer come back?

You can answer that with far less labelling than you expect. Take fifty real questions and note, for each, which document answers it. Then run retrieval and check whether that document appears in the candidates, and where it lands after reranking. Two numbers, both cheap, both actionable:

• Recall at the candidate stage: if this is low, nothing downstream can help.
• Position after reranking: if the right passage is present but ranked fourteenth, the reranker is the problem, not the index.

Include questions the corpus cannot answer, and check what scores come back for them. If those scores are indistinguishable from answerable ones, a similarity threshold will not save you and refusal has to be handled in generation.

Further reading

• Anthropic, Introducing contextual retrieval: the 150-to-20 shape and the measured effect of reranking.
• Performing similarity search: what the store does with the query, and filter ordering.
• Semantic search: the two indexes and how they are fused.
• Generation: what happens to the passages this produces.