Deterministic Evals

What makes an evaluation deterministic

A deterministic evaluation is one your code computes. Same output in, same verdict out, every run, with no model involved in the judging.

The alternative is a judge model: you ask another model whether the answer was good. That has its place, and it costs money, adds latency, and returns a different verdict on Tuesday than it did on Monday.

The line between them is not exact matching against fuzzy matching. It is computed against judged. A similarity score is a computation, so it belongs on this side even though it returns 0.83 rather than pass or fail.

The tiers of computed checks

OpenAI's grader types map the territory. Three of them compute and one judges, which is the split that decides whether a check costs a model call.

Exact comparison. string_check compares your output to a reference with eq, neq, like, or ilike, and returns 0 or 1. Use it for classification labels, extracted identifiers, routing decisions: anything with one right answer you wrote down in advance.

Computed similarity. text_similarity scores closeness with metrics including fuzzy_match, bleu, gleu, meteor, cosine, and rouge variants. The result is a number in a range rather than a verdict, and it is still deterministic. Run it a thousand times on the same pair and you get the same score a thousand times.

Your own code. A python grader runs a grade() function you write and returns a float, with libraries like rapidfuzz, scikit-learn, and pandas available. Anything you can express as code belongs here, which is a much wider category than the built-in list suggests.

A judge model. score_model prompts a separate model to grade the output. This is the other side of the line, and a multi grader can combine it with the computed ones under a weighted expression.

Why fuzzy does not mean non-deterministic

Getting this distinction wrong costs money.

The common path: someone writes exact-match assertions, watches them fail on answers that are correct but worded differently, concludes that assertions cannot handle natural language, and reaches for a judge model. Every evaluation run now costs tokens and returns a different number each time.

The middle tier was skipped. A rouge or cosine score tolerates rephrasing while staying a computation. You set a threshold, and the threshold holds. You have not given up rigor for tolerance; you have chosen a measure that allows variation in wording without allowing variation in verdict.

Reach for a judge when the question requires judgment. "Is this reply polite and on-brand" is not computable. "Does this reply mention the refund window" is, in one line of Python.

Computed checks for a support-reply feature

On a support-reply feature, the computed tier covers more than you would guess:

• The response parses as JSON.
• It validates against the schema, with every required field present.
• status is one of three known values.
• confidence is a number between 0 and 1.
• article_ids are IDs that exist in your help center.
• The reply does not contain a phrase your policy forbids.
• The reply mentions the refund window when the ticket asked about refunds.
• The refund amount never exceeds the original charge.

None of these tells you the reply was good. Every one of them tells you the reply was not broken, which is a different question and the one that catches regressions.

Ordering computed checks before judged ones

Order your suite so computed checks run before any judged ones.

They finish in milliseconds and cost nothing, so you can run the whole set on every change to a prompt, a model version, or a retrieval step. A judge model over the same set costs tokens each time, which is what turns an evaluation suite into something people run before releases instead of on every commit.

There is a second reason. When a schema check fails, the output was malformed, and asking a model to rate the quality of malformed output wastes a call to learn something you already know. Fail fast and skip the judge.

This is also where the findings from adversarial testing belong. An attempt that got the model to promise an out-of-policy refund becomes a deterministic check that the forbidden phrase never appears, and it runs forever after.

What determinism cannot buy you

Every check above tests a property you thought to write down.

They will not tell you the tone was wrong, the answer was unhelpful but correct, or the model missed the point of the question. That is the ceiling, and no amount of cleverness in the computed tier gets past it.

Determinism buys reliability and cost, not coverage. Use it for everything it can reach, so that the expensive judgment is spent on the questions that need judgment. See LLM evaluations for how the layers fit together.

Further reading

• OpenAI, Graders: the grader types, their operations, and the metrics above.
• OpenAI, Evals: running a grader over a dataset.
• Model-based evals: the other side of the line.