KITFORMA · EDITORIAL ANSWER
Why do UTF-8 characters become corrupted when decoding network chunks separately?
KitForma editorial guide. Keep decoder state across UTF-8 chunks and flush it at end-of-input; also choose an explicit malformed-byte policy.
Step-by-step answer
import assert from "node:assert/strict";
import { StringDecoder } from "node:string_decoder";
const original = "İ€🙂";
const bytes = Buffer.from(original, "utf8");
let checked = 0;
for (let split = 1; split < bytes.length; split++) {
const decoder = new StringDecoder("utf8");
const text = decoder.write(bytes.subarray(0, split))
+ decoder.write(bytes.subarray(split)) + decoder.end();
assert.equal(text, original);
checked++;
}
console.log(checked + " split points preserved");
const strict = new TextDecoder("utf-8", { fatal: true });
strict.decode(new Uint8Array([0xE2]), { stream: true });
try {
strict.decode();
} catch (error) {
console.log("malformed UTF-8 rejected");
}8 split points preserved
malformed UTF-8 rejectedSources and verification
- Node.js — StringDecoder incremental character handling
- WHATWG Encoding — streaming and fatal TextDecoder modes
Sources checked:
This editorial answer was prepared by KitForma with AI assistance. It is not presented as a real member question or an independent user review. Check the sources and the result with your own file; report corrections in the discussion.