Streaming LLM Responses to a Browser With SSE

If your app calls a model and the user stares at a spinner for thirty seconds, streaming is the fix — and Server-Sent Events is the right tool for it, not WebSockets. SSE is one-way, runs over ordinary HTTP, and the browser's EventSource reconnects on its own. On a small container it needs four things to actually work: flush after every chunk (the most common reason streaming silently doesn't), send periodic heartbeats so intermediaries don't close a quiet connection, cancel the upstream model call when the client disconnects so you stop paying for output nobody will read, and expect every deploy to cut every stream at once, because the container is replaced with no overlap.

Why SSE rather than WebSockets#

Token streaming is a one-way problem: the server has something to say, the client just listens. SSE matches that exactly, and it comes with things you'd otherwise build.

• It's plain HTTP. One route, one HTTPS port binding, no protocol upgrade.
• Reconnection is built in. EventSource retries automatically, and sends Last-Event-ID so you can resume rather than restart.
• It's simpler to reason about when things go wrong, because it's a request and a response body that hasn't finished.

Reach for WebSockets when the client also needs to send messages mid-stream — collaborative editing, live interaction. For "show me the answer as it arrives", SSE is less machinery for the same result.

Flush, or nothing streams#

The failure that wastes the most time: everything looks correct, and the response arrives all at once at the end.

Almost always, something is buffering. Your framework may collect the response body, and any proxy in the path may hold it until it's complete. The fixes are boring and specific:

• Flush after each chunk in your handler. In most frameworks this is an explicit call or a generator that yields — check that yours actually writes through rather than accumulating.
• Set the right headers. Content-Type: text/event-stream and Cache-Control: no-cache are required. Adding X-Accel-Buffering: no costs nothing and disables buffering in nginx-family proxies if one is ever in the path.
• Disable compression for the stream. A compressor that waits for enough bytes to be worth compressing is a buffer by another name.

On this platform your route carries TLS termination and hostname routing and nothing else — no buffering or timeout middleware is configured for it — so if your stream arrives in one lump, look at your own framework first.

Send heartbeats#

Even with nothing configured to time you out, connections pass through networks you don't control: mobile carriers, corporate proxies, home routers. A connection that goes quiet for a while can be dropped by any of them.

SSE has a cheap answer — a comment line, which clients ignore:

: keepalive

Send one every 15–30 seconds during any gap in output. This matters more than you'd think for LLM streaming specifically, because there's often a long pause before the first token while the model thinks.

Cancel when the client leaves#

The one that costs money. A user closes the tab, and unless you notice, your connection to the model provider keeps going and you keep paying for tokens nobody will ever see.

Most frameworks expose client disconnection — a cancellation token, a closed-request signal, an exception on write. Wire it to cancel the upstream request, don't just break out of your own loop.

This is worth more here than on a platform with connection draining, because every deploy drops all connections at once. Without cancellation, deploying during busy traffic leaves a batch of in-flight model calls completing into nothing. It's the same discipline as the rest of LLM cost control.

Deploys cut streams — design the client for it#

There's no overlap between the old container and the new one, and roughly a second between the stop signal and the kill. A half-finished response is simply cut.

Two things make that survivable:

• Send an event ID with each chunk, so a reconnecting client can send Last-Event-ID and you can resume rather than regenerate. Regenerating means paying twice for the same answer.
• Expect the stampede. Every client reconnects in the same instant because they all disconnected in the same instant, so EventSource's default retry needs jitter — set the retry: field to varied values, or handle reconnection yourself.

If resuming is too complex for your app, the honest fallback is to tell the user the connection dropped and offer a retry button. That's better than a truncated answer presented as complete.

Don't accumulate the whole response#

A streaming handler that also builds the full text in memory to save it afterwards is fine for chat-sized replies and less fine when you're running many concurrent streams on a small allocation. If you need the complete answer stored, append to your database as you go, or write chunks and assemble later.

Concurrency is also worth a thought: each open stream is a held connection and, in some frameworks, a held worker. On a fractional CPU an async framework handles many idle streams comfortably, while a thread-per-request one may not — with LLM streaming, "idle" is most of the time, so async is the better fit.

FAQ#

Should I use SSE or WebSockets for LLM streaming?#

SSE. It's one-way like the problem, runs over plain HTTP, and reconnects automatically. Use WebSockets only if the client must send messages mid-stream.

Why does my stream arrive all at once?#

Something is buffering. Flush after each chunk, set Content-Type: text/event-stream and Cache-Control: no-cache, and turn off compression for that response.

Do I need to keep the connection alive?#

Yes, send an SSE comment every 15–30 seconds. Networks between you and the user drop quiet connections, and there's often a long pause before the first token.

Do I still pay if the user closes the tab?#

Yes, unless you cancel the upstream call on disconnect. The provider finishes generating regardless.

What happens to streams when I deploy?#

They're all cut at once — the container is replaced with no overlap. Send event IDs so clients can resume, and add jitter to reconnects.

Can I resume a stream after a reconnect?#

Yes, if you emit event IDs. The client sends Last-Event-ID and you continue from there instead of regenerating.

Wrapping up#

Streaming is one of the few changes that makes an AI app feel dramatically better without touching the model, and most of the work is unglamorous plumbing: write through instead of buffering, keep the connection warm through the thinking pause, hang up on the provider when the user hangs up on you. Add event IDs so a redeploy costs a reconnect rather than a regenerated answer, and the fact that your container gets replaced regularly stops being something your users experience.