github-featured / unstructured-io--unstructured

发布

0 发布 249 Git 标签

RSS 订阅源
  • 0.24.0 445c95735c
    比较

    refactor: centralize URL fetching with host validation and default timeouts (#4388)

    frostbyte_neo 发布于 2026-07-06 20:36:49 +00:00

    Summary

    Introduces unstructured/safe_http.py as a single, shared entry point
    for
    outbound URL fetches, and routes the url= code paths in partition,
    partition_html, and partition_md through it. Previously each of
    these
    called requests.get directly with inconsistent (and in some cases
    absent)
    timeout and validation behavior; this centralizes that logic in one
    place so
    it stays consistent and testable.

    What changed

    • New unstructured/safe_http.py — a safe_get() helper that all
      three
      partitioners now use instead of calling requests.get directly.
    • The helper applies, consistently:
      • an http/https scheme allowlist
      • a hostname denylist with IDNA normalization
      • address validation performed at TCP connect time, so the address
        validated is the address actually connected to
      • manual redirect following with per-hop re-validation, dropping
        credential material (Authorization/Cookie/proxy auth, plus
        auth=/cookies=) on cross-origin hops via requests' own
        should_strip_auth
      • default (connect, read) timeouts, and refusal of proxied requests
    • an opt-out via allow_private= / the UNSTRUCTURED_ALLOW_PRIVATE_URL
      environment variable for controlled local usage

    Behavior changes (why this is a minor release)

    Fetches that resolve to non-routable, loopback, or link-local addresses
    are
    now rejected by default. Outbound fetches also now carry default
    timeouts
    where some previously had none. Callers that legitimately need to reach
    a
    private/internal host can opt out with
    UNSTRUCTURED_ALLOW_PRIVATE_URL=1 (or
    allow_private=True).

    Tests

    Adds test_unstructured/test_safe_http.py covering IP/hostname
    classification, connect-time validation, redirect re-validation,
    cross-origin
    credential stripping, and the opt-out.

    Version

    Bumps to 0.24.0 (see CHANGELOG) — minor, reflecting the behavior
    changes
    above.

    下载附件