Streaming LLM responses with Node.js and Server-Sent Events

A model that takes eight seconds to answer feels broken if the screen sits still for eight seconds. The same answer, appearing word by word from the first second, feels fast. Streaming LLM responses to the browser is one of the cheapest UX wins in an AI feature, and with Node.js and Server-Sent Events (SSE) it takes very little code.

I built streaming into the in-app AI assistant for the Recond Flutter app: replies stream in, and the UI shows what the agent is doing while it works. The lessons from that carry straight over to a web client, so this post covers the server, the browser and the proxy in between.

Why Server-Sent Events

For streaming a model's reply, the data flows one way: server to client. SSE is built for exactly that. It runs over plain HTTP, works through most proxies and load balancers, and has a tiny text format: lines starting with data:, separated by a blank line. Browsers ship a built-in client for it, EventSource.

WebSockets also work, and they make sense when the client needs to send a steady stream of messages back, like live collaboration. For "send a prompt, stream back an answer", SSE is less to build and less to operate.

The Express endpoint

The endpoint sets the SSE headers, opens a streaming request to the model API and forwards chunks as events. Here it is with a generic streaming endpoint as the upstream:

import express from "express";

const app = express();
app.use(express.json());

app.post("/api/chat/stream", async (req, res) => {
  res.writeHead(200, {
    "Content-Type": "text/event-stream",
    "Cache-Control": "no-cache",
    Connection: "keep-alive",
    "X-Accel-Buffering": "no",
  });

  const send = (event: string, data: unknown) =>
    res.write(`event: ${event}\ndata: ${JSON.stringify(data)}\n\n`);

  // If the user closes the tab, stop paying for tokens nobody will read.
  const upstream = new AbortController();
  res.on("close", () => upstream.abort());

  const heartbeat = setInterval(() => res.write(": ping\n\n"), 15_000);

  try {
    send("status", { text: "Thinking" });
    const modelRes = await fetch(process.env.LLM_STREAM_URL!, {
      method: "POST",
      headers: {
        "content-type": "application/json",
        authorization: `Bearer ${process.env.LLM_API_KEY}`,
      },
      body: JSON.stringify({ input: req.body.message, stream: true }),
      signal: upstream.signal,
    });
    if (!modelRes.ok || !modelRes.body) throw new Error(`Upstream returned ${modelRes.status}`);

    // Assumes the placeholder upstream streams plain text. Real model APIs stream
    // their own event format, which you'd parse here before forwarding.
    const decoder = new TextDecoder();
    for await (const chunk of modelRes.body) {
      send("token", { text: decoder.decode(chunk, { stream: true }) });
    }
    send("done", {});
  } catch (err) {
    if (!upstream.signal.aborted) send("error", { message: "Generation failed" });
  } finally {
    clearInterval(heartbeat);
    res.end();
  }
});

Details that matter:

• Named events. status, token, done and error let the client handle each one differently.
• Abort on disconnect. Listening for close on the response and aborting the upstream request stops generation when the user leaves. Otherwise you keep paying for the rest of the answer.
• Heartbeats. A comment line (starting with :) every 15 seconds keeps idle connections from being cut while the model is thinking or a tool call is running.
• Generic errors to the client. Log the real error server side and send the browser a safe message.

Show what the agent is doing

For a plain chat reply, tokens are enough. For an agent that searches records, calls tools or builds a result before it writes anything, the first visible token can take a while. That gap is where users decide the feature is stuck.

In the Recond assistant, the UI shows what the agent is doing while it works, and a loader holds the place of a card while it renders. In SSE terms, that is the status event: send one each time the agent starts a step ("Searching records", "Reading results") and render it as a small line above the reply. It costs one send() per step and changes how the wait feels. I cover more of these patterns in UX patterns for AI features.

Reading the stream in the browser

EventSource is the simplest client: create it with a URL, add listeners for each named event and close it on done. Its limitation is that it only makes GET requests, so you can't send a prompt in the body. Two ways around it:

1. Create, then stream. POST the message to create a reply, get back an ID, then open new EventSource("/api/replies/<id>/stream"). This also makes it easy to reconnect.
2. Use fetch and read the body. POST with fetch, then read res.body.getReader() in a loop, decode with TextDecoder and split on blank lines to get events. A little more parsing, and full control over the request.

My default for chat UIs is the second one, because the prompt and auth headers go in a normal POST.

Pace the stream so it reads as typing

Raw model output arrives in uneven bursts: nothing for a moment, then a chunk of several words. Rendering each burst as it lands makes text jump. In the Recond assistant, the streamed reply is paced so it reads as typing.

Here is how I'd approach it in a browser. Push incoming text into a buffer, and render from the buffer on a steady timer (a few characters per animation frame), speeding up when the buffer grows so the display never falls far behind the model. When done arrives, flush whatever is left. The total time barely changes and the reading experience gets much calmer.

Turn off proxy buffering

The classic failure: streaming works on localhost and arrives all at once in production. The cause is almost always a reverse proxy buffering the response. Nginx buffers proxied responses by default. The X-Accel-Buffering: no header in the endpoint above disables it per response, and I also set it in the location block:

location /api/chat/stream {
    proxy_pass http://app_upstream;
    proxy_http_version 1.1;
    proxy_set_header Connection "";
    proxy_buffering off;
    proxy_cache off;
    proxy_read_timeout 300s;
}

Check every layer between the browser and Node: a load balancer's idle timeout, a CDN that buffers, compression middleware that holds output until it has enough bytes. Heartbeats handle idle timeouts, and the rest need configuration. If you deploy on AWS like I do, I cover the infrastructure side in deploying AI-powered Node.js apps on AWS.

The short version

Send named SSE events from an Express endpoint, abort the upstream call when the client disconnects and add heartbeats. Emit status events so users can see the agent working, pace the rendered text so it reads as typing, and turn off buffering in Nginx and every other proxy in the path.