Why can the UI still show half an emoji even when every streaming delta is a valid string?

A valid Unicode string can still end in the middle of an extended grapheme cluster. For example, a base emoji, skin tone modifier, and ZWJ sequence can be spread across multiple deltas. The rendering layer must hold the last unconfirmed grapheme cluster and only commit it after receiving subsequent deltas or when the stream ends.

Should streaming output use NFC or NFKC?

NFC is generally preferred for display and persistence. NFKC collapses compatibility characters like fullwidth, circled, and superscript forms, which can alter code, mathematical expressions, or business identifiers. Use NFKC only when there is an explicit need for search or identifier normalization.

Why can't I normalize each delta individually and then concatenate them?

Unicode normalization forms are not closed under string concatenation. Two individually normalized fragments may not produce a normalized result when concatenated. Therefore, you must maintain a boundary buffer and re-normalize the entire string before persistence, comparison, or hashing.

LLM Unicode Output Governance in Production: Incremental UTF-8 Decoding, Grapheme Clusters, and NFC Boundary Buffering to Prevent Garbled Text and Truncation

Background: Garbled Text Is Usually Not a Model Error

Streaming LLM applications often assume that “receiving a delta” means “receiving a piece of text that can be immediately and stably displayed.” This assumption holds only for pure ASCII. Once you enter the world of multilingual text, emoji, combining diacritical marks, and complex writing systems, the output pipeline faces at least three types of boundaries:

Network byte boundaries: TCP, HTTP, SSE, or WebSocket fragments can split a UTF-8 multi-byte sequence in the middle.
API delta boundaries: The string delivered by the server may end in the middle of a combining mark, variation selector, or ZWJ emoji sequence.
User-visible character boundaries: What a user perceives as a single “character” is often an Extended Grapheme Cluster composed of multiple Unicode code points.

Consequently, bytes.length, the number of code points, JavaScript’s string.length, the model’s token count, and the number of characters a user sees are not the same metric. Mixing these concepts leads to common issues:

Occasional replacement characters �;
Emoji initially appearing as two icons, then merging or jumping;
Accented characters, South Asian scripts, and Hangul combinations briefly misaligned during incremental rendering;
Inconsistent cursor, truncation, sensitive word highlighting, and offset positioning;
Cache keys, database unique keys, or content hashes producing different results for visually identical text.

The focus of governance is not to add a try/catch on the frontend, but to build a layered state machine from bytes to final text.

Core Principles: First Distinguish Four Units of Measurement

UTF-8 Bytes

UTF-8 uses one to four bytes to encode a single Unicode scalar value. Network fragmentation can occur at any byte position, so you cannot perform stateless decoding on each binary chunk independently. The WHATWG Encoding Standard specifies a streaming decoding approach for fragmented input: maintain the decoder state across successive calls, and perform a final flush at the end.

Unicode Code Point

A code point is a Unicode number, e.g., U+0065. But the é a user sees could be either a single U+00E9 or U+0065 followed by U+0301. Both are visually equivalent but have different binary representations.

UTF-16 Code Unit

Browser and JavaScript string indices typically count UTF-16 code units. A supplementary plane character occupies two code units, so text.length does not directly represent the number of code points, let alone the number of user-visible characters.

Extended Grapheme Cluster

Unicode UAX #29 defines the extended grapheme cluster as the default boundary that more closely matches a user’s perception of a character. Combining diacritical marks, emoji skin tone modifiers, flag regional indicators, and ZWJ emoji sequences all require grapheme cluster boundaries as the basic unit for display, deletion, truncation, and cursor movement.

The following table summarizes the differences between these four units:

Unit of Measurement	Definition	Typical Use Case	Common Misconception
UTF-8 Byte	One to four bytes encode one code point	Network transmission, storage capacity, hashing	Using byte offsets for UI truncation
Code Point	Unicode scalar value (U+0000–U+10FFFF)	Character property lookup, case conversion	Treating it as “the character the user sees”
UTF-16 Code Unit	The value of JavaScript `string.length`	Browser DOM indexing	Representing the number of characters
Extended Grapheme Cluster	A user-perceived “single character”	Display, cursor, truncation, deletion	Using a custom regex instead of the full segmentation algorithm

Production Pipeline: A Five-Layer State Machine

Layer 1: Byte Streams Only Enter an Incremental UTF-8 Decoder

On the browser side, use a stateful TextDecoder:

const decoder = new TextDecoder("utf-8", { fatal: true });

async function consume(source: AsyncIterable<Uint8Array>) {
  for await (const bytes of source) {
    const text = decoder.decode(bytes, { stream: true });
    if (text) acceptDecodedDelta(text);
  }
  const tail = decoder.decode(); // flush end-of-stream state
  if (tail) acceptDecodedDelta(tail);
}

In production, use strict decoding within the protocol. When encountering illegal bytes, log the error and terminate the current response, rather than silently replacing them with � and continuing to persist. Replacement mode is suitable for best-effort display, but not as an authoritative source for auditing, cache keys, or business data.

If the SDK already delivers complete strings instead of raw bytes, incremental UTF-8 decoding should be handled by the SDK or transport layer; the business layer should not re-encode and decode.

Layer 2: Deltas Represent Transmission Increments, Not Character Commit Points

Even if every delta is a valid Unicode string, boundaries can still fall within the following sequences:

e followed by a combining accent;
Emoji followed by a skin tone modifier;
Emoji, ZWJ, and a subsequent emoji;
Two Regional Indicators forming a flag;
Devanagari, Tamil, and other scripts’ consonants, viramas, and vowel signs;
Hangul Jamo combinations.

Therefore, after receiving a delta, it should first be appended to a text assembler, and then the grapheme boundary layer determines which content can be stably committed.

Layer 3: Retain the Last Unconfirmed Grapheme Cluster

A practical strategy: each time, concatenate the new delta with pendingTail, then re-segment by extended grapheme clusters. All clusters except the last can be committed; the last one remains in the buffer until more content is received or the stream ends.

class GraphemeCommitter {
  private pendingTail = "";
  private readonly segmenter = new Intl.Segmenter("und", {
    granularity: "grapheme",
  });

  push(delta: string, final = false): string {
    this.pendingTail += delta;
    const parts = [...this.segmenter.segment(this.pendingTail)].map(
      (item) => item.segment
    );

    if (final) {
      const output = parts.join("");
      this.pendingTail = "";
      return output;
    }

    if (parts.length <= 1) return "";

    const stable = parts.slice(0, -1).join("");
    this.pendingTail = parts.at(-1) ?? "";
    return stable;
  }
}

This code is suitable for explaining the boundary strategy, but before going live, you must also pin the runtime and Unicode/ICU version, and use the official Unicode Grapheme Break test suite for regression. Do not replace the full segmentation algorithm with a custom “is it a combining character” check or a few emoji regexes.

Layer 4: Normalization Must Account for Concatenation Boundaries

Unicode UAX #15 defines NFC, NFD, NFKC, and NFKD. For chat output that preserves the original text semantics, NFC is the default choice: it unifies canonically equivalent combining forms without aggressively collapsing compatibility characters like NFKC does.

The key pitfall is that normalization forms are not closed under string concatenation. Performing normalize("NFC") on each delta individually and then concatenating the results does not guarantee that the final string is still NFC. In engineering, adopt a two-tier strategy:

For real-time display, only commit complete grapheme clusters; avoid making final judgments on unstable tails.
After the stream ends, perform NFC once on the complete message, and use this result as the authoritative text for persistence, comparison, search indexing, and hashing.
If you must persist while generating, keep an overwritable tail window; only commit an immutable prefix after it crosses a stable boundary.

Do not default to NFKC on code snippets, mathematical expressions, usernames, or external business IDs. Compatibility normalization may collapse fullwidth, circled, superscript, or specific symbols into another representation.

Layer 5: The Final Event Is for Reconciliation, Not Just Closing the Connection

When the stream ends, perform a complete Finalize:

Flush the UTF-8 decoder.
Flush pendingTail.
Perform NFC on the full text.
Recalculate byte count, code point count, grapheme count, and token count.
Reconcile the final text with the incrementally assembled result.
Generate a content hash and persist it.
Run final parsing for Markdown, code blocks, highlighting, or sensitive information processing.

If the vendor also provides the final complete text, prioritize it as the authoritative result, compare it with the locally assembled delta value, and log any discrepancies—do not unconditionally overwrite and lose diagnostic evidence.

Data Model: Don’t Just Store a Single `length`

Recommend recording different semantic counts for streaming messages:

{
  "utf8_bytes": 1842,
  "unicode_code_points": 917,
  "utf16_code_units": 944,
  "grapheme_clusters": 861,
  "model_tokens": 532,
  "normalization_form": "NFC",
  "unicode_runtime_version": "pinned-by-runtime",
  "assembly_hash": "sha256:..."
}

These fields don’t all need to be retained long-term, but during troubleshooting, you must be able to distinguish between “model token limit exceeded,” “UI character truncation,” “UTF-16 offset error,” and “hash changed after normalization.”

For annotations, highlighting, and audit hit positions, it is recommended to define a clear Offset Contract:

Offset Type	Use Case	Example
UTF-8 Byte Offset	Network layer positioning, storage sharding	`byte_offset`
UTF-16 Code Unit	JavaScript DOM indexing	`js_offset`
Code Point Index	Unicode property processing	`cp_index`
Grapheme Index	User interface highlighting, truncation	`grapheme_index`

In cross-service protocols, do not just write start and end.

Engineering Implementation: Build Tests Around Boundaries, Not Language-Specific Examples

Byte Fragment Replay

For the same UTF-8 data, split it at every possible byte position and feed each chunk into the incremental decoder. All split methods should produce the same final text. Also cover:

Incomplete bytes remaining at the end of the stream.
Illegal continuation bytes.
BOM.
Both strict and replacement error strategies.

Delta Fragment Replay

Split the same Unicode string at every code point boundary, and verify that any combination of deltas produces the same final NFC text. Key samples should include:

Precomposed characters vs. base + combining mark.
Emoji skin tones, variation selectors, and ZWJ family sequences.
Flag sequences.
Hangul Jamo.
Indic conjuncts.
Right-to-left text and bidirectional control characters.

Official Conformance Tests

Include Unicode’s NormalizationTest.txt and GraphemeBreakTest.txt in your CI. Whenever you upgrade the JDK, Node.js, browser engine, Python, ICU, or OS image, re-run these tests, as the Unicode data version carried by the runtime may change.

UI Behavior Tests

Automated tests should not only assert the final string, but also check for the absence of the following during the process:

�;
Half an emoji or temporary splitting;
Cursor landing inside a grapheme cluster;
Truncation leaving an isolated combining character;
Markdown parser frequently re-rendering due to intermediate code fences.

In the real-time UI, the last pending grapheme can be rendered separately as an “uncommitted tail” to avoid full DOM reconstruction.

Observability: Log Error Types, Not Full Original Text

It is recommended to provide at least the following metrics:

Metric Name	Meaning
`llm_utf8_decode_error_total`	Illegal or incomplete UTF-8 sequences
`llm_grapheme_pending_size`	Distribution of uncommitted tail lengths
`llm_normalization_changed_total`	Number of responses that changed after final NFC
`llm_stream_final_mismatch_total`	Discrepancies between local delta assembly and server final value
`llm_offset_contract_violation_total`	Offsets out of bounds or landing inside a grapheme cluster
`llm_unicode_finalize_latency_ms`	Latency of final normalization and reconciliation

By default, logs should save the event type, a summary of Unicode code points, length, and content hash. Do not retain complete sensitive output long-term for troubleshooting garbled text.

Applicable Scenarios

This approach is especially suitable for:

Multilingual chat, translation, and customer service systems.
Real-time subtitles, LLM polishing after speech-to-text.
Products that generate emoji, mathematical symbols, or code.
Applications requiring character-level annotation, redaction, highlighting, and audit positioning.
Systems where the same text flows between browsers, Java, Python, databases, and search engines.

Pure ASCII internal tools may rarely trigger issues, but as long as the output reaches an external customer interface, you should not rely on “we haven’t seen garbled text yet” as a design basis.

Common Misconceptions

Misconception 1: SSE Events Are Naturally Character Boundaries

SSE only defines the event transmission format; it does not guarantee that a business delta ends at an extended grapheme cluster boundary. The underlying HTTP byte stream can even split in the middle of a UTF-8 byte sequence.

Misconception 2: JavaScript’s `string.length` Is the Character Count

It counts UTF-16 code units. Emoji, supplementary plane characters, and combining sequences all cause it to deviate from the user-visible character count.

Misconception 3: Performing NFC on Each Delta Is Safe

Individually normalized fragments may not remain in the same normalization form when concatenated. The final message must be re-normalized, and boundary context must be preserved during streaming.

Misconception 4: NFKC Is More Thorough, So It’s Better

NFKC eliminates compatibility differences, which is useful for some search and identifier scenarios, but it can alter code, mathematical symbols, typographic characters, and business IDs. It is not a universal sanitization switch.

Misconception 5: Truncating by Token Is Equivalent to Safe Character Truncation

Model token boundaries serve tokenization and billing; they do not align with Unicode grapheme boundaries. Display truncation must be performed by extended grapheme cluster, while model budget is still calculated by token.

Pre-Launch Checklist

FAQ

Why can each delta print normally, but concatenation still leads to comparison failures?

Because visually identical Unicode text can have different code point sequences, and normalization results can change at fragment concatenation boundaries. Printing normally only proves the renderer can display it, not that the binary representation, indices, and hashes are consistent.

If I only do NFC before final persistence, can I skip grapheme cluster buffering?

No. Final NFC solves consistency for persistence and comparison, but it does not prevent emoji splitting, combining character jitter, incorrect truncation, and cursor misalignment in the streaming UI. The two address different stages.

If the server already returns a JSON string, do I still need incremental UTF-8 decoding?

It depends on the abstraction layer. If the SDK has reliably parsed the network bytes into a string, the business layer should not re-decode. If you are directly consuming Fetch, Socket, or proxy-forwarded binary chunks, you must use a stateful decoder. In either case, string deltas still require grapheme boundary governance.