Why Responses Stream

Token-by-token delivery is not a UI trick. It is the raw shape of generation, and it changes perceived latency more than any optimization.

Text appearing word by word looks like a deliberate design choice — a typewriter effect to seem thoughtful.

It is the opposite. Streaming is what generation actually looks like. Waiting for the complete response is the added step.

It falls out of the mechanism

Generation is a loop: predict a token, append it, predict again. Each token exists the moment it is sampled, hundreds of milliseconds before the response finishes.

So there are two options. Hold each token until all of them exist, then send everything. Or send each one as it appears.

Streaming is the second, and it requires no extra machinery — it is the absence of buffering.

Two latency numbers

Streaming splits latency into measurements that behave differently.

Time to first token measures prefill — processing your entire prompt before generation starts. It scales with prompt length, and steeply at long lengths since attention cost is quadratic.

Tokens per second measures the decode loop. Roughly constant per token, dependent on model size and load.

Total time is the first plus (output length ÷ the second). Which number to optimize depends on your situation: a long prompt with a short answer is prefill-bound, and shortening the prompt is the fix. A short prompt with a long answer is decode-bound, and only a smaller model or less output helps.

Without streaming you cannot distinguish these, which makes performance work guesswork.

Why it matters for interfaces

A response taking eight seconds feels broken in silence and acceptable when text is arriving.

Reading speed is the reason. Most people read slower than models generate, so once output starts flowing, the remaining generation happens behind your reading. The perceived wait is time-to-first-token, not total time — which is why streaming improves felt performance more than most actual optimizations.

It also enables early exit. A user who sees the answer going wrong can stop it, saving tokens and their time.

Mechanics

Providers stream over server-sent events or a similar protocol. You get a sequence of small events, each carrying a fragment of text, terminated by a final event with stop reason and token counts.

Three things to handle:

Fragments are not words. Events carry tokens, which may be partial words or bare punctuation. Concatenate; never assume boundaries.

Errors can arrive mid-stream. A request that started successfully can fail partway. Handle failure after partial output, which is genuinely awkward if you have already shown text to a user.

Token counts come at the end. Cost accounting happens in the final event.

For browser applications, the stream has to be relayed through your backend, since the API key cannot be in client code. Your server consumes the provider stream and re-emits to the client.

When not to stream

Structured output. Partial JSON is invalid JSON. If you are parsing the result, wait for it — streaming buys nothing when nothing can be displayed incrementally.

Pipeline stages. If the output feeds directly into another call, there is no observer to benefit.

Batch processing. Nobody is waiting.

Post-processing before display. If you validate or reformat before showing anything, you need the whole response anyway.

The rule: stream when a human is watching text appear. Otherwise it is complexity for nothing.

In agent loops

Agents present a different case. Individual model calls may not be worth streaming — the interesting output is tool calls, not prose.

But the loop should stream its progress. Twelve steps of silence is unacceptable, and showing which tool is running keeps the run legible. Stream at the loop level rather than the token level.

What to remember

• Streaming is the natural shape of generation; buffering is the added step.
• It separates time to first token (prefill, scales with prompt length) from tokens per second (decode, roughly constant).
• Perceived latency is dominated by time to first token, because reading is slower than generation.
• Events carry token fragments, errors can arrive mid-stream, and token counts come last.
• Skip it for structured output, pipeline stages, and batch work — stream agent progress instead of tokens.

Next: Cutting Your API Bill