github-featured / unstructured-io--unstructured

发布

0 发布 249 Git 标签

RSS 订阅源
  • 0.21.2 4a77a8c995
    比较

    fix: self-install pinned spaCy model at runtime with SHA256 verification (#4258)

    frostbyte_neo 发布于 2026-02-23 00:36:20 +00:00

    Summary

    • Replace en-core-web-sm direct URL dependency in pyproject.toml
      with the installer library
    • spaCy model is now downloaded and installed on first use with SHA256
      hash verification
    • Removes [tool.uv.sources] section, making the install more portable
      across package managers

    Test plan

    • Verify tokenize.py downloads and installs the spaCy model on
      first use
    • Verify SHA256 hash check rejects tampered wheels
    • Verify existing NLP tokenization tests pass

    🤖 Generated with Claude Code

    Note

    Medium Risk
    Introduces runtime download-and-install behavior into the NLP path and
    writes into site-packages, which can fail under restricted
    networking/permissions or in unusual multi-process environments despite
    locking and hash checks.

    Overview
    Updates NLP tokenization to lazy-load and self-install the pinned
    en_core_web_sm spaCy model on first use, downloading the wheel from
    GitHub and verifying it via SHA256, with a cross-process FileLock
    to avoid concurrent installs.

    Removes the en-core-web-sm wheel URL dependency and
    [tool.uv.sources] override, adding installer (for wheel
    installation) and filelock (for install locking) to dependencies;
    uv.lock is updated accordingly and the version is bumped to 0.21.2.

    Adjusts the Dockerfile to trigger model installation during image
    build (via uv run importing _get_nlp) so the model is present before
    HF_HUB_OFFLINE=1 is set.

    Written by Cursor
    Bugbot     for commit
    df62a9cf08b07f7292104ea69bd0b9725948b4a8. This will update automatically
    on new commits. Configure
    here.

    Co-authored-by: Claude Opus 4.6 noreply@anthropic.com
    Co-authored-by: Yao You yao@unstructured.io

    下载附件