Streaming Responses

Waiting 20 seconds for a full answer feels broken; seeing words appear immediately feels fast. Streaming sends text as it is generated.

Streaming with the SDK

with client.messages.stream(
    model="claude-opus-5-5",
    max_tokens=64000,
    messages=[{"role": "user", "content": "Write a beginner’s guide to Git branches."}],
) as stream:
    for text in stream.text_stream:
        print(text, end="", flush=True)
    final = stream.get_final_message()     # full message, usage and stop_reason

print("\n", final.stop_reason, final.usage.output_tokens)

Use streaming for long outputs and large max_tokens: it also avoids HTTP timeouts on long generations.

Streaming through Flask (Server-Sent Events)

from flask import Response, stream_with_context
import json

@ai_bp.post("/stream")
@login_required
@limiter.limit("20 per hour")
def stream_answer():
    question = validated_question()            # same checks as before

    def generate():
        try:
            with client.messages.stream(
                model=current_app.config["AI_MODEL"],
                max_tokens=64000,
                system=SYSTEM,
                messages=[{"role": "user", "content": question}],
            ) as stream:
                for text in stream.text_stream:
                    yield f"data: {json.dumps({'text': text})}\n\n"
                final = stream.get_final_message()
            yield f"data: {json.dumps({'done': True, 'stop_reason': final.stop_reason})}\n\n"
        except anthropic.APIError:
            yield f"data: {json.dumps({'error': 'The assistant failed.'})}\n\n"

    return Response(stream_with_context(generate()), mimetype="text/event-stream",
                    headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"})

Reading the stream in the browser

EventSource only supports GET, so for POST use fetch and read the body:

const res = await fetch("/api/ai/stream", { method: "POST", headers, body });
const reader = res.body.getReader();
const decoder = new TextDecoder();
let buffer = "";
while (true) {
  const { value, done } = await reader.read();
  if (done) break;
  buffer += decoder.decode(value, { stream: true });
  const events = buffer.split("\n\n");
  buffer = events.pop();
  for (const evt of events) {
    const data = JSON.parse(evt.replace(/^data: /, ""));
    if (data.text) output.textContent += data.text;
    if (data.error) output.textContent = data.error;
  }
}

Deployment notes

• Each open stream holds a server worker or thread. Use Gunicorn with threads (or an async worker) and set the worker timeout above your longest generation.
• Proxies may buffer responses; the X-Accel-Buffering: no header disables buffering in Nginx.
• Save the final answer and its usage after the stream finishes, not before.

Exercise

Convert your ask endpoint to streaming, show a typing cursor while text arrives, and disable the send button until the stream ends.

Check yourself

• Why does streaming improve perceived speed?
• Why use fetch instead of EventSource for POST streams?
• What deployment setting must exceed your longest generation time?