github-featured / unstructured-io--unstructured

发布

0 发布 249 Git 标签

RSS 订阅源
  • 0.22.29 bfd78b2678
    比较

    fix: handle text too long for spacy issue (#4353)

    frostbyte_neo 发布于 2026-05-17 13:28:24 +00:00

    This PR addresses the issue that very long text can fail partition
    because its length exceeds spacy's character limit. spacy is used to
    classify text content. For text too long to fit under the limit we now
    truncate the text and use the truncated text to represent the full text
    for classification purposes.

    Summary by cubic

    Prevents tokenization failures on very long inputs by truncating text
    that exceeds spacy’s max_length, keeping partition/classification
    stable for large documents without affecting normal cases.

    • Bug Fixes
    • Guard _process for inputs over nlp.max_length; truncate at the
      last whitespace within budget, log a warning, and avoid spacy
      ValueError E088.
    • Add tests for truncation behavior and for normal processing within the
      limit.
      • Update version to 0.22.29 and changelog.

    Written for commit 1bfdd91b99789c5333f300f8e10481250a536add.
    Summary will update on new commits. Review
    in cubic

    Co-authored-by: cubic-dev-ai[bot] <191113872+cubic-dev-ai[bot]@users.noreply.github.com>

    下载附件