Ragas

A metric library for retrieval systems

Ragas scores RAG output. It is a Python library rather than a platform, and the thing it gives you that a general framework does not is a set of metrics designed around retrieved context.

DeepEval is the neighbouring tool and takes a different shape: evaluations as tests in the runner you already have. Evaluation metrics uses Ragas for the faithfulness definition. This page is running it.

pip install ragas

There is also uvx ragas quickstart rag_eval, which writes a working project rather than a snippet.

Three objects

SingleTurnSample is one test case. Its fields name what a RAG evaluation needs: user_input, response, retrieved_contexts, and reference where you have a ground-truth answer.

EvaluationDataset holds the collection, built as EvaluationDataset(samples=data_samples).

evaluate runs your chosen metrics across the dataset.

The shape is worth noticing. retrieved_contexts is a first-class field rather than something you stuff into the prompt string, which is what makes the retrieval-aware metrics possible at all.

The field that decides which metrics you can run

reference is optional, and whether you have it splits the metric library in two.

Metrics needing a reference compare your answer to a known correct one. They are stronger, and they are limited to a labelled set someone had to build.

Metrics needing only retrieved_contexts ask whether the answer is supported by what was retrieved. Faithfulness is the clearest: claims in the response divided by claims the context supports.

That second group is the reason to reach for Ragas. You can compute it on production traffic you have no labels for, which makes it one of the few meaningful quality numbers available on a schedule rather than only at release. Build the unlabelled path first, and add reference-based metrics as your labelled set grows.

Configuring the judge

Most Ragas metrics call a model, so you supply one. llm_factory is the entry point:

llm_factory("gpt-4o")
llm_factory("claude-3-5-sonnet-20241022", provider="anthropic")
llm_factory("mistral", provider="ollama", base_url="http://localhost:11434")

The Ollama line matters more than it looks. It means you can score data that cannot leave your network, which is usually the objection that stops an evaluation programme.

Two consequences of a judged metric, both covered in model-based evals. The scores move when the judge model changes, so pin it and record which one produced a number. And the run costs model calls in proportion to your dataset, so a nightly sweep over ten thousand cases is a real line on the bill.

Reading a result

Each score carries .value and .reason, and results are written to CSV under evals/experiments/.

.reason is the field to use. A faithfulness score of 0.6 tells you something is unsupported; the reason tells you which claim, which is the difference between a number and a fix. Read the reasons on the worst dozen cases before you read the mean, because evaluation metrics is right that an average hides the distribution that matters.

Where it fits in a pipeline

Ragas is a measurement library, not a gate, and treating it as one causes trouble. Judged metrics vary between runs, so a threshold on faithfulness will fail a build for reasons unrelated to your change.

The arrangement that works: computed checks block CI, as deterministic evals argues; Ragas runs on a schedule and on release candidates, compared against the previous run over a fixed set rather than against an absolute bar.

Use it to answer whether retrieval or generation is at fault, which is the split retrieval process cares about. A low faithfulness score with good retrieval is a generation problem. The same score with poor retrieval is not, and Ragas scores those separately so you can tell.

Further reading

• Ragas, Evaluate a simple RAG: the objects, the metric split, llm_factory and the result fields.
• Evaluation metrics: the faithfulness formula, and reading a number honestly.
• DeepEval: the other tool, shaped as tests.
• Model-based evals: what a judged metric costs you in stability.