Cost & Latency Monitoring

Why one token count cannot tell you what a request cost

A single number for tokens used is not enough to compute a bill, because the tokens are not priced the same.

On Anthropic's published rates, checked 2026-09-16, Claude Opus 5 is $5 per million input tokens and $25 per million output tokens, so an output token costs five times an input token. A cache read is billed at 0.1x the base input price, and a cache write at 1.25x or 2x depending on duration. Batch requests are discounted 50%.

Run those through a cached, batch workload and a token total computed at the base input rate can be wrong by close to an order of magnitude. Not slightly off. Wrong enough that a cost-per-feature dashboard built on it will rank your features in the wrong order.

The fix is to record the categories separately, and they are already in the response. A Messages API response reports input_tokens, output_tokens, cache_read_input_tokens and cache_creation_input_tokens. Read them rather than counting tokens yourself, which also avoids the estimation error stop estimating tokens covers.

The fields to record, and why each one is separate

Per request, at minimum:

• input_tokens and output_tokens, separately, because they are priced differently.
• cache_read_input_tokens and cache_creation_input_tokens, separately, because a read is a tenth of base and a write is a premium over it. Collapsing these into "input" hides whether your caching strategy is working at all.
• The model ID. Prices differ per model, and a feature that silently moved to a different model is a cost change you want attributed rather than mysterious.
• A feature label. The unit you care about is cost per feature, and nothing in the API tells you which of your features made a call.
• Time to first token and total duration, as two numbers.
• Whether the request used a batch endpoint or a server tool, since batch is discounted and server tools such as web search add per-use charges outside tokens entirely.

That last point is the one that catches people building a token-only cost model: some charges are not tokens, so a system that reconciles perfectly against token counts can still miss a line on the invoice.

Tracing and logging covers where these fields belong on a span and what should stay off one.

Two latencies that move for different reasons

Record both, because they respond to different changes.

Time to first token is dominated by queueing and prompt processing. It moves when your prompt grows, when caching starts or stops working, or when a provider is under load.

Total duration is dominated by how many tokens get generated. Inference covers why: reading happens in one pass, writing happens token by token.

Keeping them apart tells you what changed. Total time up and time to first token flat means answers got longer, which is usually a prompt change. Both up together points at the provider or at a prompt that grew. Time to first token up alone, with everything else steady, is often a cache that stopped hitting, which is why the cache fields above are worth recording.

Which one you optimise depends on the interface. Streaming responses improves the first and leaves the second alone, which is the right trade for something a person is reading and no help at all for a batch job.

Reading a distribution instead of an average

A mean latency is close to useless for a model call, because the distribution is wide and skewed. A feature whose average is 2 seconds may be serving half its users in under 1 and a tenth of them in over 10, and the tenth is the group writing to support.

Record percentiles rather than an average, and look at the high end. The same applies to cost per request: a mean hides the long conversations and large retrievals that make up most of the bill.

Which percentile to watch depends on the product and the volume, so no single one is recommended here. Look at the shape once before choosing, because whether your tail is twice the median or twenty times decides how much attention it deserves.

Alerting on ratios rather than totals

Total spend grows when the product succeeds. An alert on it fires on good news and gets muted, which is how a real cost problem goes unnoticed for a month.

Ratios stay flat when nothing is wrong, which makes them the things worth alerting on:

• Cost per request, or per conversation. Rises when prompts grow, when caching breaks, or when something silently switched model.
• Cache hit rate. Falls the moment something makes your prefix vary per request, which is easy to do by accident and quantifiable in money.
• Output tokens per request. Rises when a prompt change makes answers longer, which is the most common cause of a latency and cost regression arriving together.
• Cost per successful outcome, if you can define success. Rising cost per request is ambiguous; rising cost per resolved ticket is not.

None of these need a threshold from an article. Watch them for a fortnight, see what normal looks like for your workload, and set the alert outside that band.

Further reading

• Anthropic, Pricing: the per-model rates, cache multipliers and batch discount quoted above.
• Tracing and logging: what a span is, and what belongs on one.
• Inference: why output length drives latency and where token costs accumulate.
• Prompt caching: the discount the cache fields are measuring.