Article

LLM Streaming in Production: Stabilizing Real-Time Responses with SSE Event Boundaries, Backpressure, and Reconnection

This article tackles production failures in LLM streaming, diving deep into SSE event boundaries, backpressure control, reconnection strategies, proxy timeouts, and frontend rendering optimizations to help engineering teams build reliable, observable streaming response pipelines for stable real-time interactions.

LLM Streaming in Production: Stabilizing Real-Time Responses with SSE Event Boundaries, Backpressure, and Reconnection

Streaming output is not as simple as “printing tokens as they are generated.” It is a real-time stateful link lasting from seconds to minutes, involving model generation speed, upstream event formats, server-side buffering, reverse proxies, browser parsing, mobile networks, frontend rendering, and users closing pages. If any link is handled roughly, seemingly random failures will occur.

1. The Core Problem: Why Streaming is Much Harder Than Regular HTTP

Once LLM applications go into production, users are often unwilling to wait for the complete response to finish. Long answers, code generation, analysis reports, and Agent task progress all require early display of intermediate results. Consequently, many systems enable the streaming mode of the model API, forwarding upstream tokens or events to the frontend.

However, streaming output in production is prone to these typical failures:

PhenomenonRoot Cause
First token appears quickly, subsequent tokens stallProxy buffering not disabled, or server not handling backpressure
Frontend occasionally receives half a JSONParsing by network chunk instead of SSE event boundary
User closes page, upstream continues generating, burning moneyClient disconnect event not listened to, upstream request not cancelled
Proxy buffers events until a certain size before releasingNginx/Ingress/CDN has response buffering enabled
Retry after disconnection generates a different answer, causing UI duplication or semantic driftNo event sequence number and replacement strategy, old and new content silently concatenated
Slow client exhausts server connection pool and memoryNo per-connection buffer limit, no backpressure control

This article focuses not on model inference itself, but on the engineering governance of the LLM streaming output pipeline: how to define event boundaries, implement backpressure, handle disconnections, and make the SSE gateway observable, fallback-capable, and load-testable in production.

2. Core Principles: Upgrading Token Streams to Event Streams

SSE is an Event Protocol, Not Just Text with Newlines

OpenAI’s streaming responses use Server-Sent Events with stream=true, and Chat Completions streaming results are returned incrementally in a data-only SSE format. Anthropic more explicitly breaks the stream into named events:

  • message_start
  • content_block_start
  • content_block_delta
  • message_delta
  • message_stop
  • ping
  • error

This means production systems should not treat upstream responses as simple string concatenation but should convert them into an internal unified event model. For example:

type StreamEvent = {
  requestId: string;
  seq: number;
  provider: "openai" | "anthropic" | "other";
  type:
    | "message_start"
    | "text_delta"
    | "tool_input_delta"
    | "heartbeat"
    | "usage"
    | "error"
    | "message_stop";
  textDelta?: string;
  partialJson?: string;
  finishReason?: string;
  usage?: { inputTokens?: number; outputTokens?: number };
  createdAt: string;
};

This intermediate layer has three core values:

  1. Frontend is not directly bound to vendor event formats: Changing model vendors only requires modifying the adapter, not rewriting the frontend renderer.
  2. Differentiates event semantics: Text deltas, tool parameter deltas, heartbeat events, error events, and final usage each go through their own channels. Treating everything as text can easily lead to prematurely rendering partial JSON from tool calls or mistaking heartbeats for model output.
  3. Provides stable sequence numbers for reconnection: Even if generation cannot continue from the upstream model, the gateway layer knows which events have been sent to the client, which are only in the server buffer, and which have been persisted in the ledger.

Event Boundaries Must Be Determined by Protocol, Not Guessed by Characters

MDN states that SSE uses the text/event-stream MIME type, where each notification is a text block terminated by a pair of newlines. The WHATWG HTML standard also defines the parsing of fields like data, event, id, and retry.

In the LLM context, the most common mistake is parsing by network chunk for business logic. Network chunks are just transport layer segments, not SSE events, not JSON objects, and certainly not complete tokens.

Correct approach:

  • First, parse event blocks according to SSE rules
  • Then, dispatch based on the event’s event field or payload type
  • text_delta can be rendered incrementally
  • input_json_delta or partial JSON can only be accumulated until tool input ends before parsing
  • Only close the UI state upon message_stop or a final event

If you see Unexpected end of JSON input on the frontend, it’s usually not the model’s fault, but code parsing partial JSON as complete JSON.

3. Engineering Implementation: Building a Reliable Streaming Gateway

1. Gateway for Protocol Normalization

It is recommended to add a lightweight Streaming Gateway between the application service and the model vendor, responsible for at least four tasks:

  • Convert different vendors’ streaming formats into internal StreamEvent
  • Generate requestId + seq for each event
  • Unify the SSE format output to the frontend
  • Manage cancellation, timeouts, heartbeats, errors, and metrics

Simplified output format example:

event: text_delta
id: req_123:42
data: {"seq":42,"textDelta":"next segment of text"}

For standard web text generation scenarios, SSE is usually simple enough: browsers natively support event streams, and the server can easily output via HTTP.

2. Define Connection Lifecycle

Each streaming request should have at least the following states:

accepted → upstream_started → first_event_sent → streaming → completed
                |                    |                 |
                |                    |                 → client_aborted
                |                    → upstream_error

It is recommended to record at least these metrics:

MetricMeaning
time_to_first_event_msTime from request entering gateway to first visible event sent
stream_duration_msDuration of the streaming connection
events_sent_totalNumber of events successfully written to the client
client_abort_totalNumber of client-initiated disconnections
upstream_cancel_totalNumber of successful upstream cancellations after client disconnect
stream_buffer_bytesPending write buffer per connection
final_usage_missing_totalNumber of requests that did not receive final usage

Recording only “request success/failure” is insufficient, because streaming requests are often in a semi-successful state: upstream has generated, but client disconnected; frontend received partial tokens, but final usage never returned; model returned an error event, but the HTTP status code is still 200.

3. Backpressure: Don’t Write Indefinitely to Slow Clients

The Node.js documentation explains a typical problem: when the receiver is slower than the sender, data accumulates in the buffer; without a backpressure mechanism, memory and GC pressure are amplified.

LLM streaming is the same. The model might emit a delta every few tens of milliseconds, but the client might be on a weak network, a background tab, a mobile network, or a low-performance WebView. If the server doesn’t care whether the write succeeded, it will eventually pile all events into memory.

The core principle is to respect the return value of res.write():

import { once } from "node:events";
import type { Response } from "express";

async function writeSse(res: Response, event: string, id: string, data: unknown) {
  const payload = `event: ${event}\nid: ${id}\ndata: ${JSON.stringify(data)}\n\n`;
  if (!res.write(payload)) {
    await once(res, "drain");
  }
}

This code expresses a baseline: when the downstream cannot write, the gateway should pause further writes instead of continuing to stuff events into memory.

A more complete implementation also needs:

  • A maximum pending bytes limit per connection
  • A maximum concurrent streaming connections limit per user
  • Immediate cancellation of the upstream model request upon client disconnect
  • Small-batch rendering on the UI side to avoid expensive DOM updates per token
  • Reduced refresh frequency for background tabs or weak mobile networks

4. Heartbeats: Distinguish “Model is Still Thinking” from “Connection is Dead”

SSE supports comment lines and ping events. Production systems can set up two types of heartbeats:

  • Upstream heartbeat: Determine if the model vendor connection is still alive
  • Downstream heartbeat: Prevent browsers, proxies, or load balancers from closing idle connections

A recommended practice is to send heartbeats that don’t affect business semantics:

: keep-alive

Or:

event: heartbeat
data: {"ts":"2026-07-09T15:00:00-04:00"}

The frontend should only update the connection status upon receiving a heartbeat, not treat it as model output.

5. Reconnection: Don’t Promise “Resume in Place”

Reconnection is the most underestimated part of LLM streaming. Regular file downloads can resume via Range requests, but LLM generation is not file reading. Most model APIs cannot continue the same sampling process from “the 154th token.”

Production systems should categorize recovery into three types:

Recovery TypeStrategyUse Case
Frontend reconnection display recoveryFrontend sends Last-Event-ID or lastSeq, gateway replays saved eventsOnly restores display, not upstream generation
Regeneration after upstream failureRe-initiate request with the same prompt, UI clearly defines replacement strategyUpstream failed but business allows retry
Unrecoverable failureEnter manual confirmation or task state machine, don’t pretend to be “recovering”Involves tool calls, side effects, payments, orders, database writes

Practical rule: Only events that have been persisted and validated for order can be replayed; if you are not confident in semantic consistency for recovery, clearly fail and allow the user to regenerate.

4. The Proxy Layer: Many “Streaming Failures” Are Actually Buffering Configuration Issues

When streaming output passes through Nginx, API Gateway, Ingress, CDN, or enterprise proxies, it may be buffered. The result is that the backend writes as it generates, but the frontend doesn’t receive anything until the buffer is full or the response ends.

Before going live, at least check:

  • Whether the response header includes Content-Type: text/event-stream
  • Whether unnecessary response compression and proxy buffering are disabled
  • Whether Cache-Control: no-cache is set
  • Whether upstream / read timeouts are long enough
  • Whether long-lived connections can pass through the gateway and load balancer
  • Whether testing has been done on mobile networks and browser background tabs

Do not test streaming only by connecting directly to the backend locally. Any layer of proxy in the production path can change behavior.

5. Frontend Rendering: Don’t Let the UI Become the Bottleneck

A common frontend mistake is to update the entire Markdown immediately upon receiving each delta. In long answers, this triggers a lot of layout, syntax highlighting, and scroll calculations.

A safer strategy:

  • Token deltas enter an in-memory buffer
  • Batch refresh the UI at a 30–100ms window
  • Perform Markdown parsing and code highlighting in stages
  • Display code blocks being generated as plain text first, then fully highlight after completion
  • When users scroll to view history, don’t force auto-scroll to the bottom

For Agent scenarios, also separate “model text,” “tool calls,” “tool results,” and “system progress” into different event types. Do not mix tool call JSON directly into natural language output.

6. Applicable and Non-Applicable Scenarios

Scenarios Suitable for Streaming

  • Online chat, intelligent customer service, knowledge Q&A
  • Code generation, SQL generation, report generation
  • Agent task progress display
  • Long text analysis, document summarization, review comment generation
  • Multi-vendor model gateways requiring unified frontend event format
  • Businesses sensitive to TTFT, user wait experience, and mid-cancellation costs

Scenarios Where Overusing Streaming is Inappropriate

  • Offline batch processing tasks
  • Requests requiring complete structured JSON before proceeding to the next step
  • Compliance scenarios requiring one-time signing, auditing, and archiving of results
  • Backend tasks where downstream systems only accept complete responses

In these scenarios, regular non-streaming calls or asynchronous task queues are often more reliable.

7. Common Misconceptions

Misconception 1: Enabling stream=true Automatically Means Better User Experience

Not necessarily. Streaming only improves the “time to start seeing content.” If frontend rendering is slow, proxy buffering is not disabled, heartbeats are missing, and disconnections are unrecoverable, the user experience can be worse.

Misconception 2: A Network Chunk is a Token

No. Network chunks, SSE events, JSON deltas, and model tokens are different layers. Production code must parse by protocol, not guess business boundaries by chunks.

Misconception 3: Simply Retry and Concatenate After Disconnection

This is very risky. Regeneration may produce different content, especially with high temperature, tool calls, or context changes. Unless there is a strict event ledger and replacement strategy, do not concatenate two different generation processes.

Misconception 4: Only Monitoring HTTP 200 is Sufficient

A streaming request might return HTTP 200 but have an error event midway; it might also have sent half the content to the frontend but lost the final usage. Event-level status must be monitored.

8. Go-Live Checklist

Protocol Layer

  • Unified vendor streaming events into internal StreamEvent
  • Differentiated text delta, tool input delta, heartbeat, error, stop
  • Parsed by SSE event boundaries, not network chunks
  • Handled partial JSON, not prematurely parsed tool parameters
  • Generated requestId + seq for each event

Connection Layer

  • Client disconnect cancels upstream request
  • Connection-level maximum duration exists
  • Idle timeout and heartbeat exist
  • Proxy layer has disabled unnecessary buffering
  • Tested on weak networks, mobile, background tabs, and page refresh

Backpressure Layer

  • Server respects write return value or equivalent backpressure signal
  • Per-connection buffer has a limit
  • Per-user streaming concurrency has a limit
  • Slow clients do not occupy memory indefinitely
  • Frontend rendering uses batching

Recovery Layer

  • Defined replayable events and unrecoverable events
  • Recorded lastSeq or Last-Event-ID
  • Regeneration does not silently concatenate with old output
  • Tool calls and side-effect requests have idempotency and manual confirmation strategies
  • Compensation ledger or flag exists for missing final usage

Observability Layer

  • Recorded TTFE, stream duration, event count, client abort, upstream error
  • Differentiated upstream error, gateway error, client abort
  • Metrics by vendor, model, tenant, and endpoint dimensions
  • Sampled event sequences saved for troubleshooting
  • Load test scripts for the streaming pipeline

9. Conclusion

The value of LLM streaming lies in reducing perceived user wait time, not in allowing the system to skip reliability design. Production-grade streaming output must address at least five issues:

Clear event boundaries, controllable backpressure, strategic reconnection, non-buffering proxies, and explainable metrics.

If your team is just starting, it’s recommended to first build a unified Streaming Gateway: adapt vendor events to internal events, output standard SSE, record event sequence numbers, and handle cancellation and heartbeats. Once the pipeline is stable, gradually add reconnection replay, frontend batch rendering, event ledgers, and multi-vendor compatibility.

Don’t treat streaming as a simple toggle. It is a real-time pipeline spanning models, gateways, proxies, browsers, and user behavior. Only by incorporating all these links into the design can you truly achieve a stable real-time response experience.

References

FAQ

Is SSE mandatory for LLM streaming?
No. SSE is simple enough for unidirectional token reception in browsers. For bidirectional real-time audio, continuous input, or low-latency interaction, WebSocket or WebRTC are more suitable.
Can streaming resume from the last token after disconnection?
Most model APIs cannot resume generation from an arbitrary token offset. Production systems should save stable events, request ledgers, and recovery strategies, rather than assuming seamless upstream continuation.
Why is backpressure control needed for streaming?
Clients, browsers, proxies, or mobile networks may be slower than the model output. Without backpressure, the server accumulates buffers, increases memory usage, and amplifies resource waste after disconnection.
Should timeouts for streaming and regular endpoints be configured separately?
Yes. Regular endpoints focus on total response time, while streaming focuses on time-to-first-event, event interval, total connection duration, and idle time. Using regular endpoint timeouts for streaming may incorrectly disconnect the model during normal generation.
特效配置