github-featured / unstructured-io--unstructured

发布

0 发布 249 Git 标签

RSS 订阅源
  • 0.22.27 8daa154ae4
    比较

    fix: ndjson file type detection (#4349)

    frostbyte_neo 发布于 2026-05-05 13:46:11 +00:00

    This PR fixes a bug where njson detection misclassifies multiline single
    json files as ndjson.

    Note

    Medium Risk
    Changes JSON/NDJSON detection heuristics and detect_filetype
    routing, which can affect which partitioner is invoked for JSON-like
    inputs. The logic is more strict and covered by new edge-case tests, but
    misclassification could still impact downstream parsing behavior.

    Overview
    Fixes NDJSON file-type detection so multi-line single JSON objects
    (including .json and .ipynb notebook payloads) are no longer
    misrouted to partition_ndjson and crashing.

    Updates is_ndjson_processable to require the first line to
    independently parse as a JSON object (with special handling for
    potentially truncated long single-line records), adds a bounded read
    helper (json_disambiguation_text) for disambiguation beyond the 4KB
    text_head, and changes JSON/NDJSON disambiguation to default to
    FileType.JSON when NDJSON criteria aren’t met.

    Adds a focused test suite for these NDJSON edge cases, bumps version
    to 0.22.27, updates the changelog, and makes CI apt installs more
    resilient by wrapping them in a retry helper.

    Reviewed by Cursor Bugbot for commit
    e82dbd7d017e2a1f1d5406ad8ca887109b3c5553. Bugbot is set up for automated
    code reviews on this repo. Configure
    here.

    下载附件