Streaming Responses

What streaming changes about a model call

Streaming delivers the answer as the model produces it, instead of waiting for the last token. The first words reach your user in a fraction of the time.

It does not make generation faster. Total time from request to final token is the same, so a background job that parses a complete response gains nothing from streaming and takes on the handling below for free. Turn it on where somebody is watching.

Reading the event sequence

A streamed response arrives as server-sent events. Anthropic's sequence runs:

1. message_start, carrying a message object with empty content.
2. For each content block, a content_block_start, a run of content_block_delta events, and a content_block_stop.
3. One or more message_delta events.
4. A final message_stop.

You accumulate the deltas. Nothing else in the stream gives you the text.

ping events appear anywhere and carry nothing. Ignore them. A parser that treats every event as a delta will insert empty strings into your output or throw on a field that is not there.

Where the finish reason arrives

In message_start, stop_reason is null. It shows up later, in a message_delta near the end of the stream.

That ordering has a consequence worth designing around. You cannot know why generation ended until it has ended. The answer that hit the output limit and the answer that finished properly look identical while they are arriving, so any decision that depends on completion belongs after the stream closes, not during it.

Token counts in message_delta.usage are cumulative. Read the last one; summing them inflates your numbers.

Errors that arrive after a successful response

This is the part that breaks applications written against the non-streaming mental model.

The HTTP response was 200. Your user is reading a paragraph. And then an error event arrives in the stream:

event: error
data: {"type": "error", "error": {"type": "overloaded_error", "message": "Overloaded"}}

Anthropic notes that this error "would normally correspond to an HTTP 529 in a non-streaming context." Without streaming you would have caught a 529 and retried before anyone saw anything. With streaming, the failure lands in the middle of visible output.

Decide what your interface does about it before it happens. You have half an answer on screen and no way to complete it. Retrying from the top means the user watches the text restart, which is honest but jarring. Leaving the fragment and marking it incomplete is usually better, because a truncated answer that says it is truncated beats a confident half-answer.

Whatever you choose, do not silently keep the fragment. That is how a partial sentence gets stored as if it were a finished reply.

Streaming and structured output together

You cannot validate a schema against an object you have not finished receiving.

The guidance is to accumulate the deltas and parse once content_block_stop arrives, or to use a parser built for partial JSON. Either way, anything you render before the block closes is provisional.

That rules out streaming a half-built object straight into your interface as final state. It does not rule out streaming at all. Show progress while the block fills, then validate and commit when it closes.

Further reading

• Anthropic, Streaming messages: event types, cumulative usage, and the mid-stream error example.
• Inference: time to first token against total time, and why output length dominates.
• Structured output: schema guarantees, and why streaming complicates them.