bug/encoding-detection-returns-mojibake

[huuufu]

Describe the bug

detect_file_encoding() accepts whatever charset_normalizer.detect() returns as long as confidence clears ENCODE_REC_THRESHOLD, without checking it against COMMON_ENCODINGS (unstructured/file_utils/encoding.py:74-78). When detection lands on a codec outside that set that still decodes the bytes, the mojibake becomes the element text and nothing raises. validate_encoding() (encoding.py:53) exists for exactly this check but has no call sites in the package or the test suite. Affects everything reading text through read_txt_file(): partition_text, partition_md, partition_html.

To Reproduce

import os, tempfile
from unstructured.partition.text import partition_text

for text, encoding in [
    ("会議室予約", "shift_jis"),
    ("café très à Paris", "iso_8859_1"),
    ("Bonjour, ça va très bien. Le café est déjà prêt.", "iso_8859_1"),
]:
    fd, path = tempfile.mkstemp(suffix=".txt"); os.close(fd)
    with open(path, "wb") as f:
        f.write(text.encode(encoding))
    print(repr(partition_text(filename=path)[0].text), "  expected:", repr(text))
    os.unlink(path)

Output on main (d68ab0f):

'괦귺꿬듚뒴'   expected: '会議室予約'
'café trčs ŕ Paris'   expected: 'café très à Paris'
'Bonjour, ça va trčs bien. Le café est déjŕ pręt.'   expected: 'Bonjour, ça va très bien. Le café est déjà prêt.'

detect() returns johab at 0.8 for the first and windows-1250 at 1.0 for the third; both decode cleanly, so the UnprocessableEntityError guard at encoding.py:107-111 never fires.

Second defect, same function

The inconclusive-detection fallback walks COMMON_ENCODINGS in order and takes the first codec that decodes (encoding.py:80-90). iso_8859_1 sits at index 1 and maps all 256 byte values, so it never raises and the remaining 18 of 20 entries — including every CJK codec the list was added for — are unreachable. Reordering does not fix it: with iso_8859_1 moved last, the latin-1 French bytes above are claimed by iso_8859_6 as 'cafى trوs ـ Paris'. The only test covering this path, test_charset_detection_failure, mocks both detect and COMMON_ENCODINGS, so the real list is never exercised.

Expected behavior

Text partitions to its original content, or fails loudly — not silently to mojibake.

Environment Info

OS version:  Windows-11-10.0.26200-SP0
Python version:  3.12.13
unstructured version:  0.27.5
unstructured-inference version:  1.6.13
pytesseract is not installed
Torch version:  2.13.0
Detectron2 is not installed
PaddleOCR is not installed
Libmagic version: file-5.45
magic file from /usr/share/misc/magic

charset_normalizer 3.5.1. Which encodings get misreported varies by version; accepting an unchecked out-of-set result does not.

Additional context

I tried to fix this and backed out, which is why this is an issue and not a PR.

Wiring up validate_encoding() and re-running detection constrained to the supported set (from_bytes(..., cp_isolation=COMMON_ENCODINGS)) repairs the cases above, but regresses others that work today. Türkçe bir cümle: şu ğüzel çiçekler İstanbul. written as cp1254 is detected as Windows-1254 at confidence 1.0 — correct, but outside COMMON_ENCODINGS, so the constrained pass overrides it with cp1252 and returns þu ðüzel çiçekler Ýstanbul.. Letting the detected encoding compete inside the candidate set only swaps which languages break. Widening COMMON_ENCODINGS far enough to hold the right answers also admits the wrong ones: windows-1250 is the correct detection for Czech and the incorrect one for the French line above.

The French case looks unfixable by any rule over these bytes alone — cp1252 and windows-1250 agree on most positions and both yield plausible text — so the question is less "which encoding" than which way to be wrong, and whether to stay silent about it. That reads like a policy call for the maintainers rather than something an outside patch should decide. Happy to implement whichever direction you prefer.

Worth flagging alongside this: #4434 proposes routing file-like objects through this same detect_file_encoding() fallback, which would widen the blast radius.