KKitForma.

Language

EnglishEnglishTürkçeTurkishDeutschGermanGuide unavailable in this language · open toolsEspañolSpanishGuide unavailable in this language · open toolsFrançaisFrenchGuide unavailable in this language · open toolsPortuguêsPortugueseGuide unavailable in this language · open toolsItalianoItalianGuide unavailable in this language · open toolsNederlandsDutchGuide unavailable in this language · open toolsPolskiPolishGuide unavailable in this language · open toolsРусскийRussianGuide unavailable in this language · open toolsУкраїнськаUkrainianGuide unavailable in this language · open toolsSvenskaSwedishGuide unavailable in this language · open toolsNorskNorwegianGuide unavailable in this language · open toolsDanskDanishGuide unavailable in this language · open toolsSuomiFinnishGuide unavailable in this language · open toolsČeštinaCzechGuide unavailable in this language · open toolsRomânăRomanianGuide unavailable in this language · open toolsΕλληνικάGreekGuide unavailable in this language · open toolsالعربيةArabicGuide unavailable in this language · open toolsעבריתHebrewGuide unavailable in this language · open toolsفارسیPersianGuide unavailable in this language · open toolsاردوUrduGuide unavailable in this language · open toolsहिन्दीHindiGuide unavailable in this language · open toolsবাংলাBengaliGuide unavailable in this language · open toolsதமிழ்TamilGuide unavailable in this language · open toolsతెలుగుTeluguGuide unavailable in this language · open toolsमराठीMarathiGuide unavailable in this language · open toolsગુજરાતીGujaratiGuide unavailable in this language · open tools简体中文Chinese SimplifiedGuide unavailable in this language · open tools繁體中文Chinese TraditionalGuide unavailable in this language · open tools日本語JapaneseGuide unavailable in this language · open tools한국어KoreanGuide unavailable in this language · open toolsTiếng ViệtVietnameseGuide unavailable in this language · open toolsไทยThaiGuide unavailable in this language · open toolsBahasa IndonesiaIndonesianGuide unavailable in this language · open toolsBahasa MelayuMalayGuide unavailable in this language · open toolsFilipinoFilipinoGuide unavailable in this language · open toolsKiswahiliSwahiliGuide unavailable in this language · open toolsAfrikaansAfrikaansGuide unavailable in this language · open toolsMagyarHungarianGuide unavailable in this language · open toolsБългарскиBulgarianGuide unavailable in this language · open toolsHrvatskiCroatianGuide unavailable in this language · open toolsSrpskiSerbianGuide unavailable in this language · open toolsSlovenčinaSlovakGuide unavailable in this language · open toolsSlovenščinaSlovenianGuide unavailable in this language · open toolsLietuviųLithuanianGuide unavailable in this language · open toolsLatviešuLatvianGuide unavailable in this language · open toolsEestiEstonianGuide unavailable in this language · open toolsCatalàCatalanGuide unavailable in this language · open toolsEuskaraBasqueGuide unavailable in this language · open tools
← Questions & answers

KITFORMA · EDITORIAL ANSWER

How do I fix a UnicodeDecodeError without silently losing characters?

KitForma editorial guide. Reproduce a decoding error with known bytes, select the documented source encoding, and handle a UTF-8 BOM explicitly.

Python →

Step-by-step answer

UnicodeDecodeError means the bytes are invalid for the chosen decoder; it does not identify the correct encoding. Keep the original bytes and use the producer’s specification. This fixture is explicitly Windows-1254. Runnable example:
python
raw = bytes([0xDD, 0x73, 0x74, 0x61, 0x6E, 0x62, 0x75, 0x6C])
try:
    raw.decode("utf-8", errors="strict")
except UnicodeDecodeError:
    print("UTF-8 rejected")
print(raw.decode("cp1254", errors="strict"))
utf8_with_bom = bytes.fromhex("efbbbf") + "İstanbul".encode("utf-8")
print(utf8_with_bom.decode("utf-8-sig") == "İstanbul")
Expected result:
text
UTF-8 rejected
İstanbul
True
Why it works: errors="ignore" would silently discard the problematic byte and change the name. utf-8-sig removes an initial UTF-8 byte-order mark; ordinary UTF-8 decoding retains it as U+FEFF. A BOM is not evidence that arbitrary non-UTF-8 bytes are valid UTF-8. Scope and checks: Python 3.12 or later; behavior checked against the Python 3.14 codec documentation. For a real file, use open(path, encoding="cp1254", errors="strict") only when that encoding is established. Unknown source encoding remains an investigation, not a reason to silently migrate replacement characters. Related tools and guides: Browser CSV encoding example: https://kitforma.com/en/community/questions/7aa10000-0000-4000-8000-000000003920/ Write CSV with explicit encoding and newline policy: https://kitforma.com/en/community/questions/7aa10000-0000-4000-8000-000000007340/ Primary sources (reviewed 2026-10-08): Python 3.14 — codec error handlers and utf-8-sig: https://docs.python.org/3.14/library/codecs.html

Sources and verification

Sources checked:

This editorial answer was prepared by KitForma with AI assistance. It is not presented as a real member question or an independent user review. Check the sources and the result with your own file; report corrections in the discussion.

Discussion and community answers

Loading community…
KitForma

What would you like to do?