Production Monitoring

From having the data to being told

Three articles here cover what to collect. LLM observability is which fields, tracing and logging is what a span carries, and cost and latency monitoring is reconstructing the bill.

Collecting it is not monitoring. Monitoring is the part where something reaches a person who can act, and where the rest stays in a dashboard nobody has to watch.

The question this page answers: of everything you record, what deserves to wake someone up?

Decide what you promise before you decide what to alert on

An alert without a stated expectation is a guess about what counts as bad.

Write down two or three numbers you intend to hold, with a target and a window. Ninety-five percent of requests answered in under four seconds over a rolling hour. Fewer than one percent of requests ending in an error over a day. Under two percent of answers refused for a feature where refusals are wrong.

Those are worth arguing about once, in advance, with whoever owns the product. After that the alert threshold is arithmetic rather than opinion, and an alert firing means a promise is being broken rather than a number looking high to whoever is on call.

The four signals specific to a model call

Ordinary application monitoring already covers error rate and latency. Four more carry information only an LLM system has.

Finish-reason mix. A rise in truncation is a quality regression that returns HTTP 200 every time. Nothing else will show it.

Refusal rate. The model declining is a successful request and often a failed feature. Track it per feature, since the acceptable rate for a moderation tool and a coding assistant are different numbers.

Fallback and retry rate. If you fail over to a second provider or a smaller model, the rate you do it at is a health signal for the first one. A silent fallback that becomes permanent is a cost and quality change nobody decided.

Tokens per request. Cost and latency monitoring argues for ratios over totals, and this is the ratio that catches a prompt change, a context leak, or a retrieval step that started returning far more than it used to.

Page, ticket, or dashboard

Sort every signal into one of three before you configure it.

Page a human for things that are broken now and that a person can fix now: the provider is erroring, the error rate crossed your stated bar, latency is past the promise. Keep this list short enough that being paged still means something.

Open a ticket for degradation with no emergency: refusal rate drifting up over a week, token cost per request climbing, one tenant's latency worse than the rest. Real problems, and nobody should be woken for them.

Leave it on a dashboard for everything you want when you are already investigating. Distributions, per-model breakdowns, retrieval scores. These answer questions; they do not raise them.

The failure here is one-directional. Teams put too much in the first bucket, the pages get muted, and then the real one arrives in a muted channel.

Alert on the thing users feel

Two habits keep a monitoring setup honest.

Use percentiles, not means. A mean latency inside your target is consistent with a slow tail that is the only part anyone complains about. Alert on p95 or p99, and pick which one deliberately.

Use ratios, not counts. A count of errors grows when the product succeeds, so an alert on it fires on good news and gets muted. This is the same argument cost and latency monitoring makes about totals.

Make the alert lead somewhere

An alert that says error_rate high costs the responder twenty minutes of orientation. The same alert carrying the feature, the model, a link to a filtered trace view for the window, and the last deploy costs them two.

Write that down while you configure the alert, not the first time it fires. The minimum useful runbook is three lines: what this measures, the two most common causes, and where to look first.

And close the loop the other way. When a user complaint arrives that your monitoring did not catch, that is the signal telling you which of these four is missing. Tracing and logging covers connecting a complaint to a trace; this is the same connection, used to find the gap in your alerts.

Further reading

• LLM observability: the fields worth recording, and the standard for naming them.
• Cost and latency monitoring: ratios over totals, and reading a distribution.
• Tracing and logging: what a span carries, and finding the one behind a complaint.
• Regression testing: catching it before production does.