Intermediate
Production AI

Knowing whether it works

The three tiers of evaluation, and which questions each one can actually answer.

What you will be able to do

Gate a change on evidence rather than on how the last five answers looked.

Prerequisites
A feature in front of people, or close to it.
Length
5 lessons · 22 min
Start course

Syllabus

  1. 1LLM EvaluationsStop asserting one output and start measuring across many. Three layers, and the cheap one people skip on the way to a judge model.4 min
  2. 2Evaluation MetricsThree questions need different machinery: is there a right answer, is there source material, or neither. Faithfulness needs no reference answer, which is what makes it computable on production traffic.3 min
  3. 3Deterministic EvalsAn evaluation is deterministic when it is computed rather than judged by a model, which is not the same as exact rather than fuzzy. A similarity score is computed, so it sits between exact matching and a judge model, and it costs nothing to run.4 min
  4. 4Model-Based EvalsA judge answers what code cannot decide, and costs a model call per case while being able to be wrong. Validate it on ranking rather than absolute scores, and test it for grader hacking, position bias and verbosity bias.4 min
  5. 5Regression TestingNot "is this good" but "is this worse than last week". Build the set from failures you already had, and read the diff rather than the pass rate.7 min