Inference

What is inference?

Inference is the model answering. Training is how a model came to exist; inference is every call you make to it afterward. When you send a prompt and get text back, you paid for inference.

The word matters because it names the thing you are billed for, wait on, and have to keep running. Training happened once, somewhere else. Inference happens on every request your users make.

What happens during one inference request

1. Your code assembles a prompt. System instructions, the conversation so far, whatever documents you retrieved, and the user's message.
2. The provider turns that text into tokens and checks it against the model's context limit and your account's rate limits.
3. The model reads the prompt in one pass. This step is fast relative to what follows.
4. The model writes the answer one token at a time. Each token depends on the ones before it, so this part cannot be parallelized away.
5. Tokens return to you, either all at once when the answer is finished or piece by piece if you asked for streaming.

Step 4 is where the time goes, and knowing that changes how you make things faster.

Time to first token against total response time

Time to first token is how long the user stares at nothing. Total time is how long until the answer is complete. They move for different reasons, and confusing them leads to work that does not help.

Streaming improves the first number and leaves the second alone. The answer takes just as long to finish, but the user starts reading after the first few words instead of after the last. For a chat interface that is most of the perceived speed problem solved. For a job that parses the whole response before doing anything, streaming buys you nothing.

Why output length dominates latency

Output length dominates. The model generates one token at a time, so asking for five paragraphs takes about five paragraphs' worth of generation steps. Asking for a single label is quick even when the prompt is long.

Input length matters less than people expect, because reading happens in one pass. A long prompt costs you money and context budget more than it costs you time.

Two levers follow from this. Cap the output when you know the shape of the answer you want, and reuse a stable prefix across calls through prompt caching when your system instructions and retrieved documents repeat. For current numbers on either, read your provider's guidance: OpenAI publishes a latency guide and Anthropic documents caching behavior.

Where token costs accumulate

Providers bill per token, and input and output carry different prices, with output the more expensive of the two at the major providers.

Three things follow that teams tend to miss. A retry costs the same as the original call, so a request that fails after generating most of an answer bills you for what it produced and bills you again for the second attempt. Conversation history compounds, because turn ten resends turns one through nine and the cost of a long thread grows faster than the number of messages in it. And the expensive part of a prompt is more often what you attached to it than what you wrote, which is why retrieved documents deserve a budget of their own.

Measure this per request rather than guessing. See cost and latency monitoring.

Four inference failures to handle in code

Rate limits. Providers cap both how many requests and how many tokens you may use in a period, so a burst of small calls and a single enormous call can both trip a limit. OpenAI documents both dimensions. Handle a limit response by backing off and retrying rather than by failing the user's request on the first refusal.

Truncation. An answer that hits your output cap stops mid-sentence and returns as a success. Check the field your provider uses to report why generation stopped, and treat a length-based stop as a failure your code has to handle.

Timeouts. A long generation can outlast an HTTP timeout somewhere between you and the model, and the work is billed even though you never see the result. Streaming avoids this by keeping the connection producing output.

Overload. Providers return an overload condition when capacity is short, and Anthropic lists it alongside the other error types. Treat it as temporary, retry with backoff, and consider a fallback model if the request matters more than which model serves it.

The pattern across all four: model calls fail in ways that ordinary HTTP clients do not expect, and each one needs a decision made in advance about what the user sees.

Further reading

• OpenAI, Latency optimization: which levers move which number.
• OpenAI, Rate limits: how the caps are expressed and what to do when you hit one.
• Anthropic, Prompt caching: what counts as a reusable prefix and what invalidates it.
• Anthropic, Errors: the error types to branch on.