Tracing & Logging

Why one log line per model call is not enough

A user tells you the answer was wrong. You open your logs and find the prompt and the completion.

They look fine. The prompt asked a reasonable question, and the model gave a reasonable answer from what it was given. The failure was earlier: retrieval returned the wrong section of the handbook, or a tool call timed out and your code carried on with a default, or a summarization step dropped the sentence that mattered.

One answer is many steps, and the step that broke is usually not the one you logged.

Modeling a request as a trace

A trace is one request, and spans are the operations inside it, nested to show what happened within what.

POST /support/draft-reply                        1,840ms
├─ embed query                                      41ms
├─ vector search (k=8)                              88ms
├─ rerank candidates                               210ms
├─ tool: lookup_charge                             402ms
└─ chat completion                               1,090ms

Now the same complaint is answerable. You can see which chunks came back, what the charge lookup returned, and what went into the prompt. Without the tree you have a total latency and a guess.

Instrument the boundaries where something can go wrong: every model call, every retrieval, every tool invocation, and any step that transforms context.

What to record on a span

OpenTelemetry publishes semantic conventions for this, so you do not have to invent attribute names and neither does the next tool you adopt.

Two of those pay for themselves.

Request model and response model are separate attributes. When they differ, the provider served something other than what you asked for, and you want that visible rather than inferred from a quality dip.

finish_reasons is the field that tells you generation hit the output limit instead of finishing. A truncated response comes back as a success, so without this attribute on the span, truncation looks like a model that gave a short answer. Inference covers why that matters.

Record your own prompt version too. When quality moves, the first question is what changed, and a version on the span answers it in one query.

Naming spans without exploding cardinality

The convention names a span {gen_ai.operation.name} {gen_ai.request.model}, so chat gpt-5.5 rather than one name per request.

Retrieval spans use {gen_ai.operation.name} {gen_ai.data_source.id}. And the exception is instructive: fetching a response by identifier uses only the operation name, because response identifiers are high cardinality.

That rule generalizes. Anything unbounded belongs in an attribute, not in the span name. Put a user ID or a request ID in a name and your backend creates a new series per request, which gets expensive and makes aggregation useless. Names group; attributes distinguish.

Deciding what not to record

Prompts and completions carry whatever your users typed, which on a support system means names, addresses, and account details.

OpenTelemetry marks message content as opt-in, and states why: these attributes are "likely to contain sensitive information including user/PII data." So capturing prompt text is a deliberate decision with a privacy consequence, not the default.

Decide three things before you turn it on:

Who can read it. Trace access is production data access. If your traces carry message content, the people who can open a trace can read customer conversations.

How long you keep it. Token counts and latencies are cheap and useful for months. Message bodies are neither. Different retention for different attributes is normal.

What you redact. Strip card numbers and credentials before the span is written, not in the backend. Once it has left your process you have lost the chance.

A workable default: capture metadata everywhere, capture content behind a flag you can turn on for an investigation and off afterward.

Connecting a trace to a complaint

A trace only helps if you can find the one you want.

Put your own request identifier on the root span and return it to the client. When a user reports a bad answer, you have the trace instead of a timestamp and a search.

Do the same for evaluation. When a scored run fails, store the trace ID with the score. The score tells you something regressed; the trace tells you which step. That link is what turns LLM evaluations from a number on a dashboard into a debugging path.

Further reading

• OpenTelemetry, GenAI spans: span naming, the attribute table, and the opt-in rule for content.
• Chip Huyen, Building a generative AI platform: where observability sits among the other components.
• PostHog: joining traces to what the user did next.