TCP Chunk Tearing and Multi-Byte UTF-8 Reassembly in LLM Streams
LLM streaming endpoints emit UTF-8 text chunks over HTTP/2 or Server-Sent Events (SSE). Because TCP packet boundaries operate independently of UTF-8 character encoding, multi-byte sequences (such as emojis or complex punctuation) can tear cleanly across chunk boundaries.
Passing raw chunk slices directly into JSON.parse() or text decoders causes intermittent SyntaxError crashes. Use Node.js string_decoder.StringDecoder('utf8') or a stateful byte buffer to hold incomplete multi-byte sequences until the trailing bytes arrive.
import { StringDecoder } from "node:string_decoder";
export async function* decodeSafeStream(rawByteStream) {
const decoder = new StringDecoder("utf8");
for await (const chunk of rawByteStream) {
// StringDecoder preserves trailing partial UTF-8 bytes across iterations
const safeText = decoder.write(chunk);
if (safeText) yield safeText;
}
const finalChunk = decoder.end();
if (finalChunk) yield finalChunk;
}