LLM Observability

Why the usual telemetry is not enough

A model call is an HTTP request, so your existing tooling already records that it happened, how long it took, and whether it returned 200. All true, and none of it explains why the answer was wrong.

What is missing is everything specific to the call: which model answered, what it was asked, what it cost, why it stopped. A 200 that took 1.2 seconds tells you nothing about a response truncated mid-sentence.

There is a standard, and it is worth following

OpenTelemetry publishes GenAI semantic conventions, which means the attribute names are settled rather than invented per team.

The span name convention is:

Span name SHOULD be {gen_ai.operation.name} {gen_ai.request.model}

Required on an inference span:

• gen_ai.operation.name: chat, embeddings, generate_content
• gen_ai.provider.name: openai, anthropic, gcp.vertex_ai
• gen_ai.request.model: the model you asked for

Recommended, and the ones that earn their place:

• gen_ai.usage.input_tokens, gen_ai.usage.output_tokens
• gen_ai.response.finish_reasons
• gen_ai.response.model: the model that actually answered
• gen_ai.response.id
• gen_ai.request.temperature, top_p, top_k

Two of those are quietly important. gen_ai.response.model against gen_ai.request.model catches an alias resolving somewhere you did not expect, which is a cost change nobody announced. And finish_reasons is how you find truncation, which otherwise arrives as a successful response containing half an answer.

One caveat before you build on it: the specification is marked Development status, so the names can still change. Following it is still better than inventing your own, but pin a version and expect to revisit.

What to record beyond the standard

The conventions describe a model call. An application is more than one.

A feature label. Nothing in the API says which of your features made the call, and cost per feature is the number anyone asks for.

The cache fields, separately. Anthropic returns cache_read_input_tokens and cache_creation_input_tokens alongside input_tokens. Collapsing them into one figure hides whether caching is working at all. Cost and latency monitoring covers why that matters to the bill.

Retrieval, as its own span. For a RAG feature, what was retrieved and with what scores. Without it, a bad answer is indistinguishable from bad retrieval.

The whole trace, for an agent. A loop that ran nine times is nine spans under one parent, and the shape of the trace is the debugging information. Tracing and logging covers what a span should carry.

Prompts and completions are the hard question

The most useful thing to record is the thing you should be most careful about.

The conventions say plainly that these attributes "are likely to contain sensitive information including user/PII data", and point to separate guidance on recording content on attributes and uploading content to external storage.

The tension is real and does not resolve neatly. Without the prompt and the completion, a report of a bad answer is unreproducible. With them, your observability backend is now a store of customer data, subject to whatever rules that data came with.

The usual settlements, none free:

• Record a hash or an ID of the prompt, and keep the text in a system you already treat as sensitive.
• Sample: full content on a small percentage, metadata on everything.
• Redact before export, accepting that redaction is imperfect.
• Record content only in non-production environments, accepting that you cannot debug production reports.

Decide this before you instrument, because the default in most libraries is to capture content and the default is not a decision.

Where to start

Instrument one feature end to end rather than every call shallowly. One trace with the model span, the retrieval span, the token counts and the finish reason will answer more questions than a dashboard of averages across everything.

Then add the two alerts that catch real regressions: finish reasons that are not what you expect, and cost per request. Both are ratios rather than totals, for the reason cost and latency monitoring gives. A total grows when the product succeeds, so an alert on it fires on good news and gets muted.

Further reading

• OpenTelemetry, GenAI spans: the span naming convention, the attributes, and the content-capture warning.
• Tracing and logging: what a span is and what belongs on one.
• Cost and latency monitoring: the fields that make a bill reconstructable.
• Langfuse: one implementation of this, if you would rather not assemble it.