Context Evaluation

Separating a bad answer from a bad retrieval

A user asks about the refund window and gets the wrong number. Two different things could have happened.

The right passage reached the model and the model ignored it or misread it. Or the passage never arrived, and the model answered from whatever else was in front of it.

These need opposite fixes. The first is a generation problem: prompt wording, a different model, a stricter instruction. The second is a retrieval problem, and no amount of prompt engineering touches it. A model cannot quote a document it was never given.

So measure the context first. Until you know whether the answer was available, every change you make to the prompt is a guess.

Measuring how well retrieval ranked the right passages

Context precision asks whether relevant chunks were ranked above irrelevant ones. Ragas computes it as the mean of precision at each rank position, which rewards putting the right passage early rather than burying it somewhere in the list.

Position is not a technicality. Models read long contexts unevenly, and material in the middle gets retrieved less often than the same material at the edges, which is the finding in Lost in the Middle. A retrieval that ranks the answer eighth out of ten has technically succeeded and practically failed.

Low precision means you are sending noise. That costs tokens, and it pushes the useful passage further down.

Measuring how much of the answer retrieval missed

Context recall asks whether everything needed arrived. Ragas computes it from claims: the number of claims in the reference answer supported by the retrieved context, divided by the total number of claims in the reference.

Take the answer you consider correct, break it into claims, and check each one against what retrieval returned. If your reference answer makes four claims and the retrieved context supports three, recall is 0.75, and one quarter of the answer was never available to the model.

Low recall is the harder failure. The information exists in your corpus and your search did not find it, which points at chunking, the embedding model, the metric, or a filter.

What each measurement costs you in labels

Here is the asymmetry that decides where to start.

Recall needs ground truth. Ragas is explicit that "calculating context recall always requires a reference to compare against," and there is no way around it: you cannot measure what was missed without knowing what should have been found. Someone has to write correct answers for a set of real questions.

Precision does not have to. Alongside the reference-based version, Ragas provides LLMContextPrecisionWithoutReference, which estimates relevance by comparing retrieved contexts against the generated response.

So you can start today. Take a few hundred real questions from your logs, run them through retrieval, and score precision without labeling anything. That tells you whether you are sending noise, which is the more common problem and the cheaper one to fix.

Then build the labeled set for recall, knowing it is the expensive half. Fifty carefully written question-and-answer pairs beat five hundred careless ones, because a wrong reference answer produces a wrong score with no signal that anything is off.

Reading retrieval quality separately from answer quality

Report the numbers apart. A single end-to-end score hides which half broke.

That last row is the one teams miss. The answer was right and retrieval failed, so the model filled the gap from what it learned during training. It looks like success until the question is about your policies rather than public knowledge, and then the same mechanism invents a plausible number.

Watching both numbers catches it. Watching one does not.

Turning a containment check into an assertion

Not all of this needs a model. "Did the retrieved context contain document 47" is a set membership test, and on questions where you know which document holds the answer, that check is a computed one that runs in milliseconds and never disagrees with itself.

Use it for the cases you care most about, and keep the scored metrics for the broad picture. Deterministic evals covers where that line falls.

Further reading

• Ragas, Context precision: the ranking measure, and the reference-free variant.
• Ragas, Context recall: the claim-based formula and the reference requirement.
• What is a context layer: the thing these measurements are measuring.