github-featured / unstructured-io--unstructured

发布

0.25.0 c38745b32f

feat: partition arbitrary valid JSON and NDJSON files (#4391)

frostbyte_neo 发布于 2026-07-13 18:43:02 +00:00

Summary

partition_json() and partition_ndjson() currently accept only
serialized Unstructured element output - any other valid JSON is
rejected with "Schema does not match the Unstructured schema" (and an
array of scalars crashes with a raw AttributeError). This PR makes
both partitioners two-mode:

Rehydration (unchanged): a payload of serialized Unstructured
elements is rehydrated back into its elements, exactly as before.
Arbitrary JSON (new): any other valid JSON/NDJSON is converted to
Text elements containing the pretty-printed JSON, instead of raising.

It also fixes a file-type detection bug this feature exposed: a
compact single-line JSON object was misrouted to FileType.NDJSON
and could never reach partition_json at all.

Behavior change

Input	Before	After
Object `{"customer": "Acme", ...}`	`ValueError: JSON cannot be
partitioned. Schema does not match…`	1 `Text` element, pretty-printed

Array of objects `[{"id":"one"},{"id":"two"}]`	same `ValueError`
one `Text` per object, array order
Array of scalars `[1,2,3]`	crash: `AttributeError: 'int' object has
no attribute 'get'`	1 `Text` with the whole array
Mixed array / top-level scalar	`ValueError` / crash	1 `Text`
Mixed element-shaped + arbitrary array	partial rehydrate, arbitrary
items silently dropped	whole array as arbitrary JSON - nothing dropped

Compact single-line object, `.json` file	misrouted to NDJSON →
rejected	detected as `FileType.JSON` → 1 `Text`
Arbitrary NDJSON (one record per line)	`ValueError`	one `Text` per
line, line order
Serialized element JSON / NDJSON (arrays / line-per-element)
rehydrates	rehydrates (unchanged, full regression suite green)
Element-shaped payload with corrupt contents (bad
`metadata.coordinates`, non-gzip `orig_elements`)	raw
`ValueError`/`binascii.Error`/`zlib.error` leaks	chained
`ValueError("Payload resembles serialized Unstructured elements but
could not be reconstructed: …")`
Serialized `TableChunk` elements (chunked output with split tables)
silently dropped by `elements_from_dicts()`	rehydrate as `TableChunk`;
whole chunked payloads round-trip and can feed
`reconstruct_table_from_chunks()` (#4291)
`{}` (JSON route)	error	one `Text` containing `{}` (per the output
contract)
`[]` / empty string (JSON route)	`[]` / `[]`	`[]`
`{}` / `[]` as an NDJSON line	error	`Text("{}")` / `Text("[]")` (a
line is a record)
Deeply nested payload (any depth)	`RecursionError` escapes
`ValueError` (`"Not a valid json"` / `"…nested too deeply…"`)
Malformed JSON `[{"hi":"there"}]]`	`ValueError("Not a valid json")`
unchanged
Malformed NDJSON line	`ValueError("Not a valid ndjson")`	unchanged

Design notes

Mode selection: an explicit shape predicate
(is_element_shaped_dict in partition/common/json_partitioning.py) -
a list rehydrates only when every item is a dict with a recognized str
type, the type's required field (str text / bool checked for
CheckBox), and dict-or-absent metadata. Branches are exclusive; no
exception-based control flow. Prefix/schema pre-gates are removed from
partition_json, partition_ndjson, and auto.py;
is_json_processable/is_ndjson_processable are deprecated
(DeprecationWarning naming the replacement) but keep working for
downstream importers. unstructured.file_utils.ndjson.loads/load are
intentionally retained undeprecated as generic utilities.
Documented limitation (pinned by tests): an array whose items
all look like serialized elements rehydrates rather than being treated
as arbitrary JSON. An element-shaped payload whose field contents fail
rehydration raises a chained ValueError - loud, never a leaked
low-level error.
staging/base.py deliberately not modified: the shape predicate
rejects payloads like {"type": "Title"} (no str text) before
elements_from_dicts is ever called, so they partition as arbitrary
JSON; hardening item["text"] → .get() in staging instead would
silently rehydrate customer dicts as empty elements.
Filetype disambiguation: whole-payload json.loads success →
FileType.JSON; else ≥2 newline-delimited JSON values → NDJSON. The
probe is bounded to 1 MiB, distinguishes an exact-bound-size payload
from a truncated one, restores the file position (a
detect_filetype(file=f) → partition_json(file=f) sequence works on
the same handle), and treats RecursionError as a parse failure. For
payloads exceeding the bound, one or more complete parsing lines
classify as NDJSON (first-line semantics for oversized records); the
residual degradation is an NDJSON file whose first record has no
newline inside the bound, which classifies as JSON. One intentional
flip: a one-line serialized-element object previously rehydrated via
the NDJSON route; it now partitions as arbitrary JSON (rehydration
applies only to arrays).
Output contract: pretty_json_text() - json.dumps(value, indent=2, sort_keys=True); stable, diffable output; sort_keys
alphabetizes source field order (commented at the definition as the knob
to revisit). NDJSON is strictly one Text per line; the only
empty-container divergence is [] (an NDJSON line yields Text("[]"),
a JSON-mode [] document yields no elements).
Deferred by design: per-field metadata / JSONPath addressing and
structure-aware tree walking. elements_from_arbitrary_value()
(partition/common/json_partitioning.py) is the single swap-point for a
future walker.

TableChunk rehydration - intent

Review of this branch surfaced that serialized TableChunk dicts could
not be deserialized at all: TYPE_TO_TEXT_ELEMENT_MAP has no
TableChunk entry, so elements_from_dicts() silently dropped them on
main, and with this branch's shape predicate one TableChunk flipped an
entire serialized payload to arbitrary-JSON Text. The intent of the
fix commit is narrow and explicit:

Complete #4291's design, using only existing mechanisms.
reconstruct_table_from_chunks() (added in #4291) filters
isinstance(e, TableChunk) and its docstring states reconstruction "can
be called on user-provided/deserialized chunks" - but no deserializer
could produce a TableChunk until now. The reconstruction metadata
(table_id, chunk_index, is_continuation,
num_carried_over_header_rows) already round-trips via
ElementMetadata; only the element-class dispatch was missing.
Special-case, not a map entry - deliberately.
TYPE_TO_TEXT_ELEMENT_MAP also feeds the COCO category vocabulary
(convert_to_coco derives positional category ids from its keys), so
adding TableChunk there would renumber existing category ids on every
export, even for data with no TableChunks. Instead
elements_from_dicts() special-cases "TableChunk" exactly like the
existing CheckBox special case. Verified strictly additive: a 40-dict
all-type sweep shows byte-identical output vs the pre-fix function for
every non-TableChunk payload; documents/elements.py is untouched; COCO
ids unchanged.
No new downstream exposure. type: "TableChunk" records already
flow to destinations today whenever live chunking splits a table
(library output, ingest, hosted API responses); this changes only the
read-back side.

Testing

Combined suites (partition json/ndjson/shared-predicate, filetype,
staging serde, chunking dispatch/reconstruct): 691 passed, 11 skipped
(pre-existing optional-dep skips), 1 xfailed; test_auto.py -k "json or ndjson" 7 passed, 1 xfailed (the #3365 strict-xfail - unaffected
and kept).
Full rehydration regression set green (round-trips, chunking,
last_modified, metadata stamping), including end-to-end NDJSON
rehydration through partition().
Five pre-existing tests intentionally flip expected behavior ({} no
longer raises, empty NDJSON container lines emit Text, one-record
payloads route JSON) - each renamed to describe the new behavior.
Exact pretty-printed literals are confined to one canonical test per
output shape; other tests assert structurally (element type + content
containment) so a future formatting change doesn't invalidate dozens of
assertions.
TableChunk: end-to-end chunked-table round-trip (chunk -> serialize ->
partition_json -> byte-identical re-serialization ->
reconstruct_table_from_chunks returns the table), NDJSON TableChunk
lines, predicate accept/reject, and staging serde tests. The
exact-bound-size detection probe is pinned by a test proven to fail if
the probe is reverted.
Coverage includes text=/file=/filename= routes, corrupt-payload
errors via file=, deep-nesting through both partitioners,
boundary-size (1 MiB ± 1) disambiguation, detect-then-partition on one
file handle, and chunking over arbitrary JSON and NDJSON output.
New business-neutral fixtures: example-docs/arbitrary-records.json,
single-line-object.json, arbitrary-records.ndjson.
Hardened by two independent review passes (adversarial correctness +
convention/nit pass), each finding verified by reproduction before being
fixed.

下载附件

0 发布 249 Git 标签