When AI answers feel slow, add streaming responses first

Once you add an AI feature to a service, there is one comment you are sure to hear: "The answers are good, but they take too long." The longer the answer, the longer the model has to finish every last word before anything appears, so users stare at a blank screen in the meantime.

This post covers streaming responses — showing the answer on screen bit by bit as it is generated — based on Anthropic's and OpenAI's official documentation. It is part 4 of our AI feature development series, following demo-to-production, API costs and structured outputs.

What changes with streaming

A regular request returns the answer all at once after it is fully generated. Turn on the stream option when you make the request, and the server keeps sending the answer in small pieces. The mechanism used here is SSE (Server-Sent Events, a web standard where the server keeps a connection open and continues sending data over it).

The answer may finish at about the same time, but the first words reach the screen much sooner. Users can see right away that the AI is working, and they can read the beginning while they wait.

Anthropic's documentation notes that for requests with a large output length (max_tokens), the SDK requires streaming to avoid HTTP timeouts. If your feature generates long reports or documents, it is best to design it for streaming from the start.

The receiving side processes "events" in order

A streamed answer does not arrive as plain text; it arrives as events, each with a type name.

• Anthropic: message_start → content_block_start → content_block_delta (text pieces, many times) → content_block_stop → message_delta → message_stop
• OpenAI (Responses API): response.created → response.output_text.delta (text pieces, many times) → response.completed

Your screen code appends the text pieces (deltas) and displays them, then saves the result or moves to the next step when the final event arrives. Anthropic's documentation notes that ping events may be mixed in and new event types may be added in the future, so unknown events should be skipped without raising an error.

Decide in advance what happens when the stream drops

Because a streaming connection stays open for a long time, things can go wrong partway through. Anthropic's documentation notes that an overloaded_error event can arrive mid-stream when usage is high, and OpenAI also defines a separate error event during streams.

• Don't erase what was already received: keep the text up to the break on screen and show a "continue" button
• Decide how to resume: Anthropic's documentation describes, per model version, a recovery method that passes the received part into the next request so the model continues from there
• Save only after the final event: saving intermediate pieces leaves half-finished answers behind
• Add a stop button: let users cut the answer off right away if it is not what they wanted

Using it with structured outputs and content review

There is one thing to watch when combining streaming with the structured outputs (JSON) covered in the previous post. JSON that arrives in pieces is not complete data until it finishes. Anthropic's documentation also notes that tool inputs arrive as partial JSON strings, so you should collect them and parse them only after the block ends.

OpenAI's documentation also points out that partial results during streaming are harder to moderate. If the screen is shown directly to customers, add at least a minimal check per piece, or design it separately from screens that are published only after review.

Summary: the order to add it

1. Turn on the stream option first for features with long answers
2. Append only the text-piece events and show them on screen
3. Save on the final event, and show a "continue" option on error events
4. Process results that need JSON parsing or review only after everything has arrived

With less waiting, the same AI feature feels much faster. Try it first on the one screen with the longest answers, then expand from there.

References

• Anthropic, Streaming Messages: https://platform.claude.com/docs/en/build-with-claude/streaming
• OpenAI, Streaming API responses: https://developers.openai.com/api/docs/guides/streaming-responses

SOL LABS AI

We build web apps, mobile apps and business systems, then add and run the AI they need. We build and operate our own meeting notes, contract and company analysis AI.

See our projects