Regression Testing

The question a regression suite answers

There are two questions about a model-backed feature, and they need different machinery.

"Is this good?" is what evaluation asks. You run a set of cases, you score them, you get a number, and you argue about whether the number is acceptable. LLM evaluations is about that question.

"Is this worse than last week?" is the other one, and it is the question you ask every time you touch a prompt. It needs no absolute standard. It needs the same cases run before and after, and a comparison.

The second is easier and more useful, which is a good thing, because it is also the one you will ask far more often.

Building the set out of failures you already had

The instinct is to sit down and write test cases. It produces a bad set, because you write the cases you can imagine, and the cases you can imagine are the ones your prompt already handles.

The set worth having is built the other way: every case in it is something that went wrong.

A user reports a bad answer. Before fixing it, save it: the exact input, the context it ran with, and what an acceptable answer would have been. Then fix it. The case stays.

Do that for six months and you have something nobody could have written up front: a set weighted toward your application's real failure modes, in your users' real phrasing, including the ambiguous requests and the edge cases and the one where the right answer is "I don't know." The cost is a few minutes per incident, during an investigation you were doing anyway. If you have a tracing platform, this is the trace-to-dataset path LangSmith is built around; if you do not, a directory of JSON files works.

Seed it with a dozen cases covering the ordinary path so a catastrophic regression is caught, then let failures grow it. A set of forty real failures is worth more than four hundred invented ones.

Reading a run: the diff, not the pass rate

Run the set against your change. You get a pass rate. Ignore it.

The pass rate is the number people report and the least informative thing produced. Eighty-four percent tells you nothing on its own, not whether that is good, not whether it moved, not what it consists of.

What to read is the diff against the previous run, in three groups:

• Cases that passed and now fail. The reason the suite exists. Any one of these blocks the change until explained.
• Cases that failed and now pass. Confirmation your fix worked, and occasionally a surprise that tells you the change did more than intended.
• Cases that changed output without changing verdict. The early warning. Nothing is broken, but a prompt edit that reshapes answers across forty cases did something larger than you thought.

Two runs with identical pass rates can be completely different events: one where nothing moved, and one where six cases broke while six others were fixed. The aggregate hides the thing you needed to see.

Nondeterminism, and why a gate compares rates

The obvious objection: the same input does not produce the same output twice, so what does "the case passed" mean?

The constraint is real and documented. Even at temperature zero, "the results will not be fully deterministic", and sampling parameters explains why pinning the temperature is not a reproducibility guarantee.

Three things make a suite work anyway.

Check properties, not strings. A case should assert what must be true: the right document was cited, the JSON parses, the refund amount is 40, the answer declines to give medical advice. Those survive rephrasing. Deterministic evals covers writing checks like this, and they are the backbone of a suite because they are stable and free.

Run the flaky ones more than once. A case passing three times in five is information, and its useful form is a rate rather than a verdict. Where a case matters enough, run it several times and record how often it passed.

Gate on the aggregate, not the case. This is where the pass rate finally earns its keep, not as a quality score but as a noise filter. One case flipping is probably sampling. Six flipping together is a regression. Set the threshold by running the suite twice against no change at all and seeing how much it moves on its own; that is your noise floor, and your gate goes outside it.

Four changes that force a rerun

Anything altering what the model sees or which model sees it:

Prompt edits. The obvious one, including edits that look cosmetic. Reordering sections or rewording an instruction can shift behavior across cases that had nothing to do with the wording.

Model version changes. Less optional than it sounds. Every model ID is "a pinned snapshot" with a published retirement date (one current model's commitment is "Not sooner than October 15, 2026") so migration is scheduled whether or not you scheduled it. This is the moment a saved set pays for its entire existence: the difference between measuring a migration and gambling on one.

Retrieval changes. New chunking, a different embedding model, a reranker, a re-indexed corpus. The prompt is unchanged and its contents are not, which is the same thing from the model's point of view.

Tool changes. A renamed parameter, an altered description, a tool's output format changing. Tool descriptions are prompt text.

The uncomfortable one is the third and fourth: they are usually made by someone who does not think of themselves as touching the model. Re-indexing a corpus is a data change with a model-behavior consequence, and it should trip the same gate as a prompt edit.

When a failing case is the suite being wrong

Not every failure is a regression. Sometimes the expected answer has expired.

The policy changed and the old correct answer is now wrong. The product renamed a feature. A case was recorded with an expectation that was too strict, and three acceptable answers fail it.

Update the case. This is normal, and refusing to do it produces a suite everyone learns to ignore, which is worse than having none.

What matters is that updating an expectation is a deliberate act with a record. Change it in the same commit as the code, say why in the message, and have someone else look. The failure mode to guard against is not stale cases; it is updating an expectation to match what the model now produces, because that turns a test into a snapshot of current behavior and quietly deletes the regression it was about to catch.

A rule that holds up: you may change an expectation when you can state what changed in the world to make the old one wrong. If the only thing that changed is the output, it is a regression.

Further reading

• Anthropic, Messages API reference: the temperature-zero caveat that shapes how a gate has to work.
• Anthropic, Models overview: pinned snapshots and retirement dates, the rerun trigger nobody schedules.
• Deterministic evals: writing checks that survive rephrasing.
• LLM evaluations: the other question, and the tiers this sits inside.