github-featured / unstructured-io--unstructured

发布

0 发布 249 Git 标签

RSS 订阅源
  • 0.25.2 4c61d871d0
    比较

    feat: add lazy chunking entry points (#4423)

    frostbyte_neo 发布于 2026-08-03 19:04:52 +00:00

    Summary

    Adds iter_chunk_elements() and iter_chunks_by_title(), generator
    counterparts to the existing chunk_elements() and chunk_by_title().
    Same options, same chunks, same order — the only difference is that
    chunks are yielded as they are formed rather than accumulated into a
    list.

    Why

    Chunking has always been lazy internally. PreChunker.iter_pre_chunks()
    and PreChunk.iter_chunks() are generators, and the public functions
    only wrapped them in a list comprehension. There was no public way to
    reach that pipeline, so a caller that wanted to stream chunks had to
    either buffer the whole document or import private symbols
    (_ByTitleChunkingOptions, _BasicChunkingOptions) and reassemble the
    pipeline by hand.

    Combined with a lazy elements source, the new entry points keep peak
    memory proportional to the largest pre-chunk instead of the whole
    document. That matters when elements carry large metadata.image_base64
    payloads, where a document's element list can be far larger than the
    source file.

    Implementation

    This exposes the existing pipeline rather than adding a second one:

    • _iter_chunk_elements() / _iter_chunks_by_title() hold the
      generator logic.
    • _chunk_elements() / _chunk_by_title() become list() over those
      generators, so there is a single implementation to keep correct and the
      list and generator forms cannot drift.
    • Both public entry points keep their existing (elements, opts) call
      into the private implementation, so the option-plumbing unit tests are
      unchanged.

    Options are validated eagerly, when the function is called, not when
    the returned iterator is first advanced. iter_chunk_elements and
    iter_chunks_by_title are therefore plain functions returning an
    iterator, not generator functions — a generator function would have
    deferred the whole body, so an invalid option combination would surface
    at an unrelated point in the caller, or not at all for a document that
    yields no chunks.

    Tests

    test_unstructured/chunking/test_lazy.py:

    • Equivalence with the list form for both strategies, with
      include_orig_elements both on and off, comparing serialized chunks
      field-by-field (orig_elements included; element_id is a fresh UUID
      per chunk and is excluded).
    • Laziness — yielding the first chunk does not drain the source
      iterator.
    • Eager option validation — an invalid option raises at the call,
      not at first advance.
    • Signature parity — guards against the generator and list
      signatures drifting as options are added.

    The existing chunking suite (343 tests) passes unchanged.

    🤖 Generated with Claude Code

    Review in cubic

    Co-authored-by: Claude Opus 5 (1M context) noreply@anthropic.com

    下载附件