Generation

The last stage is where retrieval gets believed

Retrieval hands over some passages. Generation decides what the reader is told.

Every failure earlier in the pipeline becomes visible here and only here, which is why the generation prompt attracts blame it often does not deserve. It is also where the one failure retrieval cannot fix has to be handled: the question your corpus cannot answer.

Passages are input, not instructions

The first structural decision is separating the retrieved text from your own instructions. The passages came from documents, and documents can contain anything, including a sentence that reads like a command.

Mark the boundary explicitly. Put the context in a tagged block, keep your instruction outside it, and say in the instruction that the material inside is reference text. This is the same reasoning as prompt injection attacks: the model reads your instruction and the retrieved document as the same kind of thing, so the separation has to be stated rather than assumed.

<context>
  <document id="doc-41" title="Refund policy">…</document>
  <document id="doc-88" title="Shipping exceptions">…</document>
</context>

Answer the question using only the documents above. Cite the document id for
each claim. If the documents do not contain the answer, say so and do not
answer from general knowledge.

Citations you can check

Asking a model to cite its sources produces citations. It does not produce correct ones: a citation is text like any other output, and a plausible document ID is easy to generate.

Anthropic's citations feature moves that out of the text. Enable it on a document block and the API returns structured citations, with a guarantee worth quoting:

Because the API parses citations into the response formats described in the following sections and extracts cited_text directly, citations are guaranteed to contain valid pointers to the provided documents.

The granularity depends on the document type:

The economics are better than the prompt-based version too. cited_text does not count toward output tokens, and does not count toward input tokens when passed back in later turns. Anthropic also reports that in their evaluations the feature is "significantly more likely to cite the most relevant quotes from documents than purely prompt-based approaches".

If you are on a provider without an equivalent, the fallback is to have the model emit document IDs and then verify each one against the set you actually supplied, dropping any that do not resolve. That catches the invented citation; it does not check that the passage supports the claim.

Refusal is a feature you build

A retrieval system always returns something. Generation is where "nothing relevant came back" has to become "I don't know".

Three things make it work, and all three are needed:

Instruct it. Say explicitly what to do when the documents do not contain the answer. Without that sentence the model will answer anyway, because answering is what it was asked to do.

Give it the signal. If your retrieval scores are meaningful, pass a threshold decision into the prompt rather than making the model infer relevance from passages you already decided to include. Including a passage is itself an implicit claim that it is relevant.

Test it. Put questions the corpus cannot answer into the evaluation set and assert the refusal. This is the case a happy-path test set never contains and production always does.

What to measure here, separately from retrieval

Judging "was the answer good" tells you nothing about which stage to fix. Two narrower questions do:

Faithfulness. Is every claim in the answer supported by a supplied passage? This is answerable by a judge with the passages in hand, which is the shape model-based evals describes. It also has a defensible right answer, unlike "is this helpful".

Refusal rate on unanswerable questions. A single number that should be near one, and that quietly drops whenever someone tunes the prompt for helpfulness.

Both are cheaper than they sound and both survive a prompt change, which a collection of remembered examples does not.

Further reading

• Anthropic, Citations: the document types, the citation formats, and the validity guarantee.
• Retrieval process: what arrives here and how it was ordered.
• Prompt injection attacks: why retrieved text needs a boundary around it.
• Model-based evals: judging faithfulness against the passages supplied.