## Summary
`partition_json()` and `partition_ndjson()` currently accept **only
serialized Unstructured element output** - any other valid JSON is
rejected with `"Schema does not match the Unstructured schema"` (and an
array of scalars crashes with a raw `AttributeError`). This PR makes
both partitioners **two-mode**:
- **Rehydration (unchanged):** a payload of serialized Unstructured
elements is rehydrated back into its elements, exactly as before.
- **Arbitrary JSON (new):** any other valid JSON/NDJSON is converted to
`Text` elements containing the pretty-printed JSON, instead of raising.
It also fixes a file-type detection bug this feature exposed: a
**compact single-line JSON object** was misrouted to `FileType.NDJSON`
and could never reach `partition_json` at all.
## Behavior change
| Input | Before | After |
|---|---|---|
| Object `{"customer": "Acme", ...}` | `ValueError: JSON cannot be
partitioned. Schema does not match…` | 1 `Text` element, pretty-printed
|
| Array of objects `[{"id":"one"},{"id":"two"}]` | same `ValueError` |
one `Text` per object, array order |
| Array of scalars `[1,2,3]` | crash: `AttributeError: 'int' object has
no attribute 'get'` | 1 `Text` with the whole array |
| Mixed array / top-level scalar | `ValueError` / crash | 1 `Text` |
| Mixed element-shaped + arbitrary array | partial rehydrate, arbitrary
items silently dropped | whole array as arbitrary JSON - nothing dropped
|
| Compact single-line object, `.json` file | misrouted to NDJSON →
rejected | detected as `FileType.JSON` → 1 `Text` |
| Arbitrary NDJSON (one record per line) | `ValueError` | one `Text` per
line, line order |
| Serialized element JSON / NDJSON (arrays / line-per-element) |
rehydrates | rehydrates (unchanged, full regression suite green) |
| Element-shaped payload with corrupt contents (bad
`metadata.coordinates`, non-gzip `orig_elements`) | raw
`ValueError`/`binascii.Error`/`zlib.error` leaks | chained
`ValueError("Payload resembles serialized Unstructured elements but
could not be reconstructed: …")` |
| Serialized `TableChunk` elements (chunked output with split tables) |
silently dropped by `elements_from_dicts()` | rehydrate as `TableChunk`;
whole chunked payloads round-trip and can feed
`reconstruct_table_from_chunks()` (#4291) |
| `{}` (JSON route) | error | one `Text` containing `{}` (per the output
contract) |
| `[]` / empty string (JSON route) | `[]` / `[]` | `[]` |
| `{}` / `[]` as an NDJSON line | error | `Text("{}")` / `Text("[]")` (a
line is a record) |
| Deeply nested payload (any depth) | `RecursionError` escapes |
`ValueError` (`"Not a valid json"` / `"…nested too deeply…"`) |
| Malformed JSON `[{"hi":"there"}]]` | `ValueError("Not a valid json")`
| unchanged |
| Malformed NDJSON line | `ValueError("Not a valid ndjson")` | unchanged
|
## Design notes
- **Mode selection:** an explicit shape predicate
(`is_element_shaped_dict` in `partition/common/json_partitioning.py`) -
a list rehydrates only when every item is a dict with a recognized `str`
`type`, the type's required field (`str` `text` / `bool` `checked` for
CheckBox), and dict-or-absent `metadata`. Branches are exclusive; no
exception-based control flow. Prefix/schema pre-gates are removed from
`partition_json`, `partition_ndjson`, and `auto.py`;
`is_json_processable`/`is_ndjson_processable` are **deprecated**
(DeprecationWarning naming the replacement) but keep working for
downstream importers. `unstructured.file_utils.ndjson.loads/load` are
intentionally retained undeprecated as generic utilities.
- **Documented limitation** (pinned by tests): an array whose items
*all* look like serialized elements rehydrates rather than being treated
as arbitrary JSON. An element-shaped payload whose field contents fail
rehydration raises a chained `ValueError` - loud, never a leaked
low-level error.
- **`staging/base.py` deliberately not modified:** the shape predicate
rejects payloads like `{"type": "Title"}` (no `str` text) before
`elements_from_dicts` is ever called, so they partition as arbitrary
JSON; hardening `item["text"]` → `.get()` in staging instead would
silently rehydrate customer dicts as empty elements.
- **Filetype disambiguation:** whole-payload `json.loads` success →
`FileType.JSON`; else ≥2 newline-delimited JSON values → `NDJSON`. The
probe is **bounded to 1 MiB**, distinguishes an exact-bound-size payload
from a truncated one, restores the file position (a
`detect_filetype(file=f)` → `partition_json(file=f)` sequence works on
the same handle), and treats `RecursionError` as a parse failure. For
payloads exceeding the bound, one or more complete parsing lines
classify as NDJSON (first-line semantics for oversized records); the
residual degradation is an NDJSON file whose *first* record has no
newline inside the bound, which classifies as JSON. One intentional
flip: a *one-line serialized-element object* previously rehydrated via
the NDJSON route; it now partitions as arbitrary JSON (rehydration
applies only to arrays).
- **Output contract:** `pretty_json_text()` - `json.dumps(value,
indent=2, sort_keys=True)`; stable, diffable output; `sort_keys`
alphabetizes source field order (commented at the definition as the knob
to revisit). NDJSON is strictly one `Text` per line; the only
empty-container divergence is `[]` (an NDJSON line yields `Text("[]")`,
a JSON-mode `[]` document yields no elements).
- **Deferred by design:** per-field metadata / JSONPath addressing and
structure-aware tree walking. `elements_from_arbitrary_value()`
(`partition/common/json_partitioning.py`) is the single swap-point for a
future walker.
## TableChunk rehydration - intent
Review of this branch surfaced that serialized `TableChunk` dicts could
not be deserialized at all: `TYPE_TO_TEXT_ELEMENT_MAP` has no
`TableChunk` entry, so `elements_from_dicts()` silently dropped them on
`main`, and with this branch's shape predicate one TableChunk flipped an
entire serialized payload to arbitrary-JSON `Text`. The intent of the
fix commit is narrow and explicit:
- **Complete #4291's design, using only existing mechanisms.**
`reconstruct_table_from_chunks()` (added in #4291) filters
`isinstance(e, TableChunk)` and its docstring states reconstruction "can
be called on user-provided/deserialized chunks" - but no deserializer
could produce a `TableChunk` until now. The reconstruction metadata
(`table_id`, `chunk_index`, `is_continuation`,
`num_carried_over_header_rows`) already round-trips via
`ElementMetadata`; only the element-class dispatch was missing.
- **Special-case, not a map entry - deliberately.**
`TYPE_TO_TEXT_ELEMENT_MAP` also feeds the COCO category vocabulary
(`convert_to_coco` derives positional category ids from its keys), so
adding TableChunk there would renumber existing category ids on every
export, even for data with no TableChunks. Instead
`elements_from_dicts()` special-cases `"TableChunk"` exactly like the
existing `CheckBox` special case. Verified strictly additive: a 40-dict
all-type sweep shows byte-identical output vs the pre-fix function for
every non-TableChunk payload; `documents/elements.py` is untouched; COCO
ids unchanged.
- **No new downstream exposure.** `type: "TableChunk"` records already
flow to destinations today whenever live chunking splits a table
(library output, ingest, hosted API responses); this changes only the
read-back side.
## Testing
- Combined suites (partition json/ndjson/shared-predicate, filetype,
staging serde, chunking dispatch/reconstruct): **691 passed, 11 skipped
(pre-existing optional-dep skips), 1 xfailed**; `test_auto.py -k "json
or ndjson"` **7 passed, 1 xfailed** (the #3365 strict-xfail - unaffected
and kept).
- Full rehydration regression set green (round-trips, chunking,
`last_modified`, metadata stamping), including end-to-end NDJSON
rehydration through `partition()`.
- Five pre-existing tests intentionally flip expected behavior (`{}` no
longer raises, empty NDJSON container lines emit `Text`, one-record
payloads route JSON) - each renamed to describe the new behavior.
- Exact pretty-printed literals are confined to one canonical test per
output shape; other tests assert structurally (element type + content
containment) so a future formatting change doesn't invalidate dozens of
assertions.
- TableChunk: end-to-end chunked-table round-trip (chunk -> serialize ->
`partition_json` -> byte-identical re-serialization ->
`reconstruct_table_from_chunks` returns the table), NDJSON TableChunk
lines, predicate accept/reject, and staging serde tests. The
exact-bound-size detection probe is pinned by a test proven to fail if
the probe is reverted.
- Coverage includes `text=`/`file=`/`filename=` routes, corrupt-payload
errors via `file=`, deep-nesting through both partitioners,
boundary-size (1 MiB ± 1) disambiguation, detect-then-partition on one
file handle, and chunking over arbitrary JSON **and** NDJSON output.
- New business-neutral fixtures: `example-docs/arbitrary-records.json`,
`single-line-object.json`, `arbitrary-records.ndjson`.
- Hardened by two independent review passes (adversarial correctness +
convention/nit pass), each finding verified by reproduction before being
fixed.
Simon
c38745b32f