DeepEval

Evaluations as tests, in the runner you already have

Most evaluation work starts in a notebook and stays there. The results are real, and nobody sees them again after the afternoon they were produced.

DeepEval's premise is that evaluations belong where your tests are. The documentation puts it plainly: "deepeval plugs into Pytest, so deepeval test run collects and runs your eval files the same way pytest would."

Test discovery works as it does now. Files, functions, fixtures and assertions are the ones your team already knows. What changes is the command, and what a failure means.

What a test case carries

The unit is an LLMTestCase, and its fields tell you what the framework expects you to evaluate:

• input, the user query.
• actual_output, what your system produced.
• expected_output, the ideal answer, optional and used by comparison metrics.
• retrieval_context, for retrieval-augmented systems.

Multi-turn work uses ConversationalTestCase with Turn objects carrying role and content.

retrieval_context is the field worth noticing. Its presence means the framework expects you to evaluate what retrieval returned alongside what the model said, which is exactly the split that keeps you from blaming generation for a retrieval failure. Context evaluation covers why those are separate measurements.

Which metrics call a model

Here is the fact that should shape how you use this tool. From the documentation: "almost all deepeval metrics including GEval are LLM-as-a-Judge metrics", defaulting to OpenAI with other providers configurable.

GEval is the general-purpose one, a research-backed metric where you describe the criterion in natural language and a model scores against it. That is powerful and it is a model call.

So a suite of built-in metrics is a suite of model calls. Each run costs tokens proportional to your case count, and each run can return a different number for the same input, because a judge is not deterministic.

None of that makes the metrics wrong. They answer questions a computation cannot: whether an answer is helpful, whether the tone fits, whether a reply followed a policy in spirit. It does mean you should know which of your tests are measurements and which are opinions.

Which metrics belong in a blocking CI gate

A gate that fails a build should fail for a reason the author can act on.

Computed checks belong in the gate. Schema validation, required fields, enum membership, forbidden phrases, numeric bounds. They are instant, free, and give the same verdict every run, so a failure means something changed in the commit. Deterministic evals covers the tier, and the useful point there is that computed does not mean exact matching: a similarity score above a threshold is still a computation.

Judged metrics belong as a reported signal, or behind thresholds loose enough to catch real regressions rather than noise. A build that fails because a judge scored 0.71 instead of 0.74 on a sample of forty teaches the team to rerun the job until it passes, and a gate people rerun until it passes is not a gate.

Two practices make judged metrics usable in CI. Run them on a fixed set rather than a sample, so variation comes from the judge rather than from which cases you drew. And read the trend across builds, not one number: a single run of a nondeterministic metric is one sample.

Then budget for it.

Two hundred cases against several judged metrics is several hundred model calls every run, which makes a per-commit gate a standing bill. Most teams settle on computed checks per commit, with the full judged suite running nightly or before a release.

Further reading

• DeepEval, Getting started: the pytest integration, test case shape, and the note that almost all metrics are LLM-judged.
• LLM evaluations: the layers this tool spans.
• Deterministic evals: what to put in the blocking gate.