Knowing whether a change helped — the only thing separating engineering from vibes.
Sign in to track your progress across sections.
10 articles
Listed inDeterministic EvalsEvaluationon
An evaluation is deterministic when it is computed rather than judged by a model, which is not the same as exact rather than fuzzy. A similarity score is computed, so it sits between exact matching and a judge model, and it costs nothing to run.
Listed inRagasEvaluationon
Faithfulness, answer relevance, and context precision — how the metrics are computed and what they miss.
Listed inLLM EvaluationsEvaluationon
The open-source eval framework and registry — a good template for structuring your own graded test set.
Listed inLLM EvaluationsEvaluationon
Stop asserting one output and start measuring across many. Three layers, and the cheap one people skip on the way to a judge model.
Listed inDeepEvalEvaluationon
Evaluations in pytest. Almost every built-in metric calls a judge model, which decides what belongs in a blocking gate and what does not.
Listed inRegression TestingEvaluationon
Not "is this good" but "is this worse than last week". Build the set from failures you already had, and read the diff rather than the pass rate.
Listed inModel-Based EvalsEvaluationon
A judge answers what code cannot decide, and costs a model call per case while being able to be wrong. Validate it on ranking rather than absolute scores, and test it for grader hacking, position bias and verbosity bias.
Listed inEvaluation MetricsEvaluationon
Faithfulness, relevance, latency, cost — picking metrics that map to the product.
Listed inRagasEvaluationon
A RAG-specific eval framework for faithfulness and context precision.
Listed inHuman EvalsEvaluationon
Rubrics, annotator agreement, and when humans are the only ground truth.