LLM Evaluations

Why the unit-test instinct fails against a model

A unit test asserts that a function returns a specific value. Run it twice, get the same answer twice, and a failure means something changed.

Point that at a model and it breaks immediately. The same prompt can produce two different sentences that are both correct, so a test comparing output to a stored string fails for reasons unrelated to quality. Loosen it until it stops failing and it now passes for answers that are wrong.

The shift is from asserting one output to measuring quality across many examples. You stop asking "did this call return the right string" and start asking "across two hundred real cases, did this change make the system better or worse."

What an evaluation set is, and where the cases come from

An evaluation set is a fixed collection of inputs with some notion of what a good outcome looks like, that you re-run after every change.

Three sources, in order of value.

Real inputs from your logs. The questions users ask, including the badly phrased ones. Invented test cases are cleaner than reality and miss what reality does.

Every failure you have already seen. A bug report is an evaluation case. The answer that was wrong last month should be in the set forever, so the next model upgrade cannot quietly reintroduce it.

The cases you are most afraid of. The refund that must never be approved automatically, the medical question that must be declined, the competitor's name that must not appear.

Keep the set small enough to maintain. Fifty cases somebody curates beat five hundred that nobody has looked at since they were generated, because a wrong expected answer produces a wrong score with no signal that anything is off.

Three layers, and which questions belong to each

Not every question needs the same instrument.

Computed checks run in your code and return the same verdict every time. Does it parse, does it validate, is the label one of four values, does the refund amount stay under the charge. These are instant and free. Deterministic evals covers the tier, and the useful surprise there is that computed does not mean exact: a similarity score is still a computation.

Model judges answer questions a computation cannot. Is this reply helpful, is the tone right, does it follow the policy in spirit. You pay tokens, and you accept that the judge's verdict varies between runs.

Human review is what you use when the judgment matters enough to be worth a person, and to check that your judge agrees with people. A judge nobody has validated against human opinion is a number whose meaning you have not established.

The mistake to avoid is skipping the first tier. People hit a case exact matching cannot handle and jump straight to a judge model, which turns a free instant suite into one that costs money and returns different numbers each run.

Where evaluations run, and what they gate

Run them at the moments where something could regress: a prompt change, a model version change, a retrieval change, a chunking change.

Split them by cost. Computed checks are cheap enough to block a merge, so put them in CI as a gate. Judged metrics belong as a reported signal, or behind a threshold loose enough to catch real regressions rather than noise. A build that fails because a judge scored 0.71 instead of 0.74 teaches the team to ignore the gate. DeepEval covers what that looks like in a test runner, including why most of its metrics are judged.

Store the trace identifier with each result. A score tells you something regressed; the trace tells you which step did it.

Build the set before the feature

OpenAI's own optimization workflow puts it first: build evals, then prompt the model with context and instructions, then consider fine-tuning.

Evaluations come before prompting, not after. Without a measurement built beforehand, "this prompt feels better" is your only evidence, and every later decision, whether to switch models, whether retrieval helped, whether fine-tuning was worth it, rests on nothing.

The set is also the thing that makes the rest of this section usable. It is what context evaluation measures against, what adversarial findings become, and what tells you whether a change to any part of the system helped.

Further reading

• OpenAI, Evals and Graders: running a grader over a dataset, and the grader types.
• Deterministic evals: the computed tier, in depth.
• Model-based evals: the judged tier.