github-featured / unstructured-io--unstructured

发布

0 发布 249 Git 标签

RSS 订阅源
  • 0.14.10 7b25dfc337
    比较

    fix(CVE-2024-39705): remove nltk download (#3361)

    frostbyte_neo 发布于 2024-07-08 22:55:36 +00:00

    Summary

    Addresses
    CVE-2024-39705, which
    highlights the risk of remote code execution when running
    nltk.download . Removes nltk.download in favor of a .tgz file with
    the appropriate NLTK data files and checking the SHA256 hash to validate
    the download. An error now raises if nltk.download is invoked.

    The logic for determining the NLTK download directory is borrowed from
    nltk, so users can still set NLTK_DATA as they did previously.

    Testing

    1. Create a directory called ~/tmp/nltk_test. Set
      NLTK_DATA=${HOME}/tmp/nltk_test.
    2. From a python interactive session, run:
    from unstructured.nlp.tokenize import download_nltk_packages

download_nltk_packages()
    1. Run ls /tmp/nltk_test/nltk_data. You should see the downloaded
      data.

    Co-authored-by: Steve Canny stcanny@gmail.com

    下载附件