Evaluation Metrics

Pick the metric that matches the question

A metric is a question with a number attached. Most confusion about evaluating model output comes from reaching for a number before deciding what it should answer.

Three questions cover nearly everything, and they need different machinery:

• Did it produce the right answer? There is a correct output and you can compare against it.
• Did it stay faithful to what it was given? There is no single right answer, but there is source material the answer must not contradict.
• Is it good? No correct answer and no source. The hardest to measure and the one people try first.

When there is a right answer

If your task has one, use it. These are cheap, deterministic, and the tier deterministic evals says to exhaust before paying for anything else.

Exact match. The output equals the expected value. Right for classifications, extracted fields, routing decisions.

Precision, recall and F1. For anything that returns a set rather than one value: extracted entities, retrieved documents, tags. Precision is how much of what you returned was right; recall is how much of what was right you returned; F1 is their harmonic mean. Reporting only F1 hides which of the two is failing, and they usually need different fixes.

Pass rate on a check. Anything expressible as code: the JSON parses, the number falls in range, the forbidden phrase is absent. Binary per case, a percentage across the set.

The trap here is a metric that looks exact and is not. String equality on a free-text answer measures formatting, not correctness, and will report a regression when the model adds a full stop.

When there is source material

This is the RAG case, and it has a family of metrics that need no ground-truth answer, which is what makes them usable on real traffic rather than only on a labelled set.

Faithfulness is the clearest. Ragas defines it as measuring "how factually consistent a response is with the retrieved context", computed as:

Faithfulness Score = (Number of claims supported by context) / (Total claims in response)

by identifying the claims in the response and checking each against the context. It ranges from 0 to 1, and it needs the question, the response and the retrieved contexts, and not a reference answer.

That property is worth dwelling on. You can compute faithfulness on production traffic you have no labels for, which makes it one of the few meaningful numbers available continuously rather than at release time.

The same family covers retrieval separately from generation, which is the split retrieval process argues for: a low faithfulness score with good retrieval is a generation problem, and the same score with bad retrieval is not.

When there is neither

Helpfulness, tone, whether an explanation lands. No reference, no source, and no formula.

This is where a judge model comes in, and where the number is only as good as the rubric behind it. Model-based evals covers writing one that means the same thing twice, validating it against human ranking rather than against absolute scores, and the ways a judge fails.

The honest position: these numbers are directional. They are useful for comparing this week's prompt against last week's and misleading as absolutes.

Reading a number without fooling yourself

An average hides the distribution. A faithfulness mean of 0.9 is consistent with every answer scoring 0.9 and with nine perfect answers and one fabrication. The second is the one that matters. Look at the count below a threshold, not the mean.

Rates need denominators. "Twelve failures" means nothing without how many cases ran. This sounds obvious and is the most common way an eval report misleads.

Variance is the baseline. The same input run twice gives different output, so a metric that moves by two points may not have moved at all. Run the set more than once before believing a change, which is the discipline regression testing is built around.

Record what produced the number. Model, prompt version, date. A metric without those is not comparable to next month's, and a judge upgrade will shift every score at once.

Further reading

• Ragas, Faithfulness: the definition, the formula, and the inputs it needs.
• Deterministic evals: everything computable, which is where to start.
• Model-based evals: the tier for questions with no right answer.
• Regression testing: comparing rates over a fixed set rather than single verdicts.