I streamed a model's reply into my chat UI. Some characters came out as garbled diamonds: � (U+FFFD The replacement character, shown as �. A decoder prints it when bytes do not form a valid character.). The reply arrived as SSE Server-Sent Events. A server keeps one reply open and sends text events, each ended by a blank line. events over a TCP The protocol that carries bytes between two machines in order. It may split or join them anywhere. connection. The network cut it into Chunk One piece of a reply as it arrives from the network. Its edges have nothing to do with characters., and sometimes a cut landed in the middle of a character. My code decoded each chunk by itself, so both halves of that character broke.
Text on the wire is Byte The smallest unit a computer stores or sends. This essay writes each one as two characters, such as F0.. In UTF-8 The usual way to store text as bytes. Each character takes one to four bytes., a plain letter takes one byte and an emoji like 🚀 takes four. Chinese, Japanese and Korean characters take more than one byte, too. A chunk can end between any two bytes. My code ran TextDecoder The standard API that turns bytes into text. Unless told a stream continues, it decodes each call alone. on each chunk alone. A decoder that sees half a character renders it as a replacement diamond in my UI.
I built the simulator below to show the failure in bytes. It opens on the same 🚀, cut after its second byte. Type your own text: only characters of two or more bytes, such as emoji and accented letters, can tear. Then shrink the chunk size, watch characters tear at the cuts, and see them heal through my streaming decoder:
Type your own text, shrink the chunk size, and watch multibyte characters tear at a chunk boundary, then heal with streaming decode.
Beginner tutorials assume a network call returns at once, with tidy text. In production I found that network packets ignore character and JSON A common text format for structured data. A parser needs the whole object; half of one fails. boundaries. So a reliable stream needs a reassembler in Node.js. It keeps partial bytes until a character is whole, and partial text until an event is whole. Only then does it decode and parse.
Why the stream breaks, and when I need the fix
The failure anatomy: a character or a JSON object cut in two
I stream model output with Server-Sent Events (SSE) or HTTP chunked transfer encoding. Either way, my runtime receives raw bytes, as Buffer or Uint8Array chunks. A critical mistake is to treat each chunk as a complete token. A chunk is only whatever the network delivered next.
A. A character cut in two. UTF-8 characters (IETF RFC 3629) take 1 to 4 bytes: ğ is 2 bytes and 🚀 is 4 bytes. TCP packet boundaries (IETF RFC 9293) can fall inside those bytes. Then chunk.toString('utf-8') decodes a partial character and produces \uFFFD, the Unicode replacement character. That character is lost for good in my output stream.
B. A JSON object cut in two. In structured output and tool-calling modes, models send JSON frames inside SSE events. One JSON object often spans several network chunks. If I call JSON.parse(chunk) on one of them, it throws at once: SyntaxError: Unexpected end of JSON input.
Before I write a backend stream parser, I check whether my architecture needs one at all. I hit chunk boundary crashes when I deployed backends, for two different reasons:
Hiding the API key (the anti-pattern)
I often see engineers proxy model calls through a Cloud Function only to hide the API key from the client. When hiding the key is the only goal, a backend proxy is an anti-pattern. It adds latency, compute cost and stream parsing work.
Trusted backend execution (the mandatory pattern)
Some of my agent logic cannot live on the client. One example is a workflow that runs RAG Retrieval-Augmented Generation. My code looks up documents first and gives them to the model with the question. against a private vector database. Others are tool calls against third-party endpoints, and proprietary reasoning loops I must protect.
In this second case, my Node.js code parses the raw HTTP chunks itself. Then it streams them back to the client. Here I must manage the bytes explicitly, and this is where my parser lives.
Get the next essay by email
One email per new essay: a production failure, taken apart, and the code that fixed it. Reply to any of them to unsubscribe.
The old way vs. the new way
This table sets my naive parser beside my production reassembler. The reassembler is stateful: it remembers leftover bytes and text from one chunk to the next.
| Failure mode | The old way (naive parser) | The new way (stateful reassembler) |
|---|---|---|
| A character cut in two UTF-8 byte split | Decodes each chunk alone. chunk.toString('utf-8') corrupts multi-byte sequences into \uFFFD. | Holds the leftover bytes. StringDecoder A Node.js decoder that keeps the bytes of an unfinished character and joins them to the next chunk. holds incomplete bytes in my internal buffer until complete. |
| A JSON object cut in two Split JSON payloads | Parses half an object. JSON.parse(rawString) crashes my runtime on partial frames. | Waits for the blank line. Delimiter-based line buffering isolates complete data: blocks before parsing. |
| Two events in one chunk Fused SSE packets | Keeps the first event, drops the rest. Processes first payload, dropping trailing events in the same chunk. | Handles every event in the chunk. Loops through every blank line (two line endings in a row, each \n, \r\n or \r) within my combined buffer. |
| One malformed event Error recovery | One bad event ends the stream. Stream crashes, terminating my client connection abruptly. | Flags the bad event and keeps going. Marks a malformed JSON frame with a parseError field and keeps the underlying transport alive. |
Bytes become text
Text becomes events
Only whole events reach JSON.parse
- Byte decoder. It takes raw
BufferorUint8Arraynetwork chunks. When a TCP packet split cuts a UTF-8 character, it keeps the incomplete bytes in memory. Then it joins them to the next chunk. - Event buffer. It collects decoded text until a blank line closes the event. Then it cuts off the complete events and keeps the partial tail. A blank line is two line endings in a row, each
\n,\r\nor\r, so mixed endings split too. A tail still open when the stream ends is reported as atornevent after the complete ones, never parsed. A trailing comment line does not count as a tail. - Safe JSON parsing. It runs JSON parsing on each whole event by itself and keeps the exact text in
raw. A plain-text token like1.50comes out ofdataas the number 1.5, so a text stream readsraw. A broken object or array carries aparseErrorinstead of passing as text, and my stream keeps running.
Show the shell command (reproduce locally)
node utf8_tear.jsProduction-ready TypeScript implementation
Below is my production-tested SafeSseReassemblyStream class. I deploy it as a Node.js Transform stream on Cloud Functions or Firebase App Hosting endpoints. It keeps leftover bytes in a StringDecoder and leftover text in a string buffer:
// Resilient SSE & UTF-8 Stream Reassembler for Node.js / Cloud Run
import { StringDecoder } from 'node:string_decoder';
import { Transform, type TransformCallback } from 'node:stream';
export interface SseEvent<T = unknown> {
event?: string;
data: T;
id?: string;
retry?: number;
parseError?: string; // set when an object or array payload fails JSON.parse
raw: string; // the data lines exactly as sent, for plain-text token streams
}
// SSE ends each event with a blank line: two line endings in a row, each LF, CRLF or CR.
// A CR counts as a line ending on its own only when no LF follows it.
const EVENT_BOUNDARY = /(?:\r\n|\r(?!\n)|\n){2}/;
const MAX_BUFFER_CHARS = 1_000_000;
export class SafeSseReassemblyStream extends Transform {
private readonly decoder: StringDecoder;
private buffer: string;
constructor() {
super({ readableObjectMode: true });
this.decoder = new StringDecoder('utf8');
this.buffer = '';
}
public _transform(
chunk: Buffer | Uint8Array | string,
_encoding: string,
callback: TransformCallback
): void {
try {
const text = typeof chunk === 'string'
? chunk
: this.decoder.write(Buffer.isBuffer(chunk) ? chunk : Buffer.from(chunk));
this.buffer += text;
let match: RegExpExecArray | null;
while ((match = EVENT_BOUNDARY.exec(this.buffer)) !== null) {
const rawEvent = this.buffer.slice(0, match.index);
this.buffer = this.buffer.slice(match.index + match[0].length);
const parsed = this.parseSseBlock(rawEvent);
if (parsed) {
this.push(parsed);
}
}
if (this.buffer.length > MAX_BUFFER_CHARS) {
throw new Error('SSE event passed MAX_BUFFER_CHARS without a closing blank line');
}
callback();
} catch (error) {
callback(error instanceof Error ? error : new Error(String(error)));
}
}
public _flush(callback: TransformCallback): void {
const leftover = this.buffer + this.decoder.end();
this.buffer = '';
// An event with no closing blank line is torn: never parse it, report it in-band.
// callback(err) here would destroy the stream and drop events the reader has not taken yet.
if (leftover.split(/\r\n|\r|\n/).some((line) => line.trim() !== '' && !line.startsWith(':'))) {
this.push({ event: 'torn', data: null, raw: '', parseError: 'Stream ended mid-event; partial frame discarded' });
}
callback();
}
private parseSseBlock(rawBlock: string): SseEvent | null {
let eventType: string | undefined;
let id: string | undefined;
let retry: number | undefined;
const dataLines: string[] = [];
for (const line of rawBlock.split(/\r\n|\r|\n/)) {
if (line.startsWith(':')) continue; // comment or heartbeat
// A line with no colon is a field with an empty value, e.g. a bare "data".
const colonIdx = line.indexOf(':');
const field = colonIdx === -1 ? line : line.slice(0, colonIdx);
const value = colonIdx === -1 ? '' : line.slice(colonIdx + 1).replace(/^ /, '');
switch (field) {
case 'data': dataLines.push(value); break;
case 'event': eventType = value; break;
case 'id': id = value; break;
case 'retry': if (/^\d+$/.test(value)) retry = Number(value); break;
}
}
if (dataLines.length === 0) return null;
const rawData = dataLines.join('\n');
let data: unknown = rawData;
let parseError: string | undefined;
try {
data = JSON.parse(rawData);
} catch (error) {
// Plain-text tokens stay strings; a broken object or array is flagged, not hidden.
if (/^\s*[{[]/.test(rawData)) parseError = String(error);
}
return { event: eventType, data, raw: rawData, id, retry, parseError };
}
}StringDecoder. I also keep transport delimiters apart from application payloads. A partial event never reaches my parser, so it never crashes my stream in production.Industry validation and benchmarks
- When Tokenizers Fail: Byte-Level Chunking for Zero-Shot Transfer to Low-Resource Languages (Aug 2026): Confirms that byte-level boundary misalignment across subword tokenizers and transport streams corrupts multi-byte UTF-8 sequences unless explicit stateful byte-boundary buffering is enforced.
- LeanStream: A Speculate-and-Refine Streaming Framework for Efficient on-Device LLM Inference (Sep 2026): Confirms that streaming token boundaries across network and storage buffers require stateful buffer alignment to prevent boundary corruption.
- IETF RFC 9293: Transmission Control Protocol (TCP) Specification
- IETF RFC 3629: UTF-8, a transformation format of ISO 10646
- WHATWG HTML Standard: Server-Sent Events (SSE) Protocol
- Node.js StringDecoder Core API Specification
- Firebase AI Logic Documentation
- Firebase App Check Attestation
- Cloud Run Streaming and HTTP/2 Ingress