github-featured / unstructured-io--unstructured

发布

0 发布 249 Git 标签

RSS 订阅源
  • 0.23.0 dacae2cc5d
    比较

    feat: add enrichment origins metadata field (#4370)

    frostbyte_neo 发布于 2026-06-10 17:23:48 +00:00

    Summary

    Adds a new enrichment_origins field to ElementMetadata that tracks
    which model wrote (or contributed to) each enriched attribute.

    What's new

    • ElementMetadata.enrichment_origins — a serialized dict[str, list[dict[str, str]]] mapping a written attribute name (e.g. text,
      text_as_html, embeddings) to a list of {"type", "provider", "model"} records, in application order. Authoring enrichments overwrite
      the list; additive enrichments append, preserving the prior author.
    • ConsolidationStrategy.DICT_LIST_UNIQUE — a new chunking
      consolidation strategy for dict[str, list] fields. It unions keys
      across elements and, per key, concatenates then dedupes records while
      preserving first-seen order. Wired up as the strategy for
      enrichment_origins.

    Testing

    • Unit tests for the new metadata field (test_elements.py).
    • Unit tests for the DICT_LIST_UNIQUE merge behavior during chunking
      (test_base.py).

    Summary by cubic

    Adds per-attribute enrichment provenance via
    ElementMetadata.enrichment_origins and a dict-list merge strategy, and
    isolates per-chunk metadata so provenance updates can’t leak across
    split chunks.

    • New Features

    • ElementMetadata.enrichment_origins: maps an attribute to a list of
      {"type","provider","model"} in application order. Authoring
      overwrites; additive appends.

    • ConsolidationStrategy.DICT_LIST_UNIQUE: unions keys and per-key
      concat+dedupes while preserving order; used for enrichment_origins in
      chunking.

    • Bug Fixes

    • First and continuation split chunks each get a fresh metadata object.

    • Deep-copies enrichment_origins per chunk and sets is_continuation
      on later chunks, preventing any in-place updates from leaking to
      siblings or lazy continuations.

    Written for commit 7a03b8c9886b6cbdbfcfe647984482f5daaa6724.
    Summary will update on new commits.

    Review in cubic

    Co-authored-by: Claude Opus 4.8 (1M context) noreply@anthropic.com

    下载附件