Human Evals

The tier you reach for last and cannot replace

LLM evaluations sets out three layers, and this is the expensive one. Deterministic evals covers what code can check, and model-based evals covers a judge model, including using people to validate that judge by ranking.

This page is people as the measurement itself: when there is no substitute, and how to run it so the numbers mean something.

Two situations leave you no alternative. The first is a judgement no rubric survives, where taste, tone or domain expertise decides. The second is the question of whether your judge and your metrics are measuring the thing you care about, which cannot be answered by the same machinery that produced them.

Agreement before scores

The first number to produce is not quality. It is whether two people looking at the same output agree.

Give three reviewers the same twenty cases before you run anything larger. If they disagree often, the rubric is the problem, not the model, and every score you collect afterwards is noise wearing a decimal point.

The usual cause is a rubric asking for a judgement the reviewer cannot make from what is in front of them. "Is this helpful" needs to know what the user wanted. "Does this answer the question asked, using only the provided context" is answerable from the page.

Fix the rubric and re-measure agreement. Only then collect scores.

Comparison beats rating

People are unreliable at absolute scales and good at picking a winner.

A reviewer asked to score an answer 1 to 5 will drift over a session, anchor on whatever they saw first, and use a different scale from the reviewer next to them. The same reviewer shown two answers and asked which is better will be consistent, and so will their colleague.

So prefer pairwise comparison. It gives you what you usually want anyway, which is whether this week's change beat last week's, and model-based evals makes the same argument for validating a judge.

Where an absolute number is unavoidable, anchor the scale with worked examples of each point rather than adjectives.

Sizing the set

Human review is bounded by attention, not by budget. Quality falls off well before a reviewer runs out of time.

Plan around fifty to a hundred cases per reviewer per session, in blocks with breaks, and stratify the sample rather than taking the newest hundred. You want the hard cases, the long inputs, the ones where retrieval returned little, and the categories your traffic is thin on. A random sample of production traffic will be dominated by the easy majority and will tell you what you already know.

Keep a fixed subset that never changes, so the number is comparable over time, and rotate the rest so you keep finding new failures. That is the same split regression testing argues for.

Who reviews changes what you learn

Three options, and they answer different questions.

Your own team is fast, free and biased. They know what the system is supposed to do, so they read past errors a user would trip on. Fine for triage, unreliable as a quality signal.

Domain experts are the only option when correctness needs knowledge. A clinician, a lawyer, an accountant. Expensive, slow to schedule, and worth it for the cases where being wrong matters.

Actual users, through thumbs, corrections, or abandonment. The most honest signal and the least controlled, since you learn what they disliked and rarely why.

Most teams need the first for daily work and the second for anything they intend to claim.

Feeding it back

A human review that ends in a spreadsheet has paid for itself once.

Every case a reviewer marked wrong is a case for your evaluation set, with the reviewer's reasoning attached. Over a few rounds this converts expensive judgement into a cheap suite you can run on every change, which is how the expensive tier stops being the one you run.

Track the disagreement rate between your judge model and your reviewers on the same cases. When it drifts, the judge needs revalidating, and that is the number that tells you when.

Further reading

• LLM evaluations: the three layers, and which questions belong to each.
• Model-based evals: the judge, and validating it against people by ranking.
• Evaluation metrics: what to measure once you know who is measuring.
• Regression testing: the fixed set that makes a number comparable.