Arize AI

Tracing and evaluation in the same tool

Most teams end up with two systems: one that records what happened, and another that scores whether it was any good.

Keeping them apart costs you the connection. A score drops and you go looking for the requests behind it; a trace looks slow and you cannot tell whether the answer was worth the wait.

Arize builds both into one product. Phoenix is the open-source half, and the pairing is the reason to look at it rather than any single feature.

Phoenix: open source, local, OpenTelemetry-native

Phoenix captures "model calls, retrievals, tool usage, and custom logic" as traces, so a single run of your application is visible step by step. It supports evaluation through model-based evaluators, code-based checks and human annotations. And it carries prompt engineering tooling: versioning, and experiments that compare changes against the same inputs.

Two properties matter more than that list.

It is built on OpenTelemetry, instrumented through OpenInference. That is the same standard tracing and logging recommends instrumenting against, which means the work is not tool-specific. If you emit GenAI spans following the semantic conventions, you have not married a vendor.

It runs locally. A single command serves it on localhost:6006, and self-hosting works under Docker or Kubernetes. For anyone who cannot send prompts and completions to a third party, that is the difference between using observability tooling and building it.

Auto-instrumentation covers LangChain, LlamaIndex, DSPy, OpenAI, Bedrock and Anthropic, so an existing application gets traces without a rewrite. That shortens the distance between deciding to instrument and having something to look at, which is usually where the effort dies.

Running experiments against production examples

The prompt-experiment feature is the part that uses both halves at once.

You take real examples from your traces, change something, and run the comparison against identical inputs. That answers the question every prompt change raises, which is whether it helped on the cases you see rather than on the three you tested by hand.

This is what LLM evaluations describes, wired to your trace data so the cases come from production rather than from imagination.

Where the hosted platform adds something

AX is the commercial product. It adds Signal, which groups recurring failures into ranked issues, Alyx for querying your data in natural language, controlled experiments, and human annotation workflows.

Signal is the one that addresses a real operational problem. At volume, individual traces stop being readable, and what you want is "these forty failures are the same failure." Grouping is what turns observability from a debugging tool into a monitoring one.

What the documentation does not claim

The seeded description of this tool on this site promised drift detection and embedding analysis. Neither is documented, on the open-source product or the commercial platform, so neither is claimed here.

If you are specifically looking for distribution drift over embeddings, verify against current documentation before choosing a tool on that basis. What is documented, and what makes this worth considering, is the OpenTelemetry-native tracing, the evaluation in the same place, and the option to run the whole thing on your own machine.

Further reading

• Arize, Phoenix: tracing, evaluation, prompt experiments, and the OpenTelemetry foundation.
• Arize, AX: what the hosted platform adds.
• Tracing and logging: the instrumentation model this consumes.