Simon
c38745b32f
## Summary
`partition_json()` and `partition_ndjson()` currently accept **only
serialized Unstructured element output** - any other valid JSON is
rejected with `"Schema does not match the Unstructured schema"` (and an
array of scalars crashes with a raw `AttributeError`). This PR makes
both partitioners **two-mode**:
- **Rehydration (unchanged):** a payload of serialized Unstructured
elements is rehydrated back into its elements, exactly as before.
- **Arbitrary JSON (new):** any other valid JSON/NDJSON is converted to
`Text` elements containing the pretty-printed JSON, instead of raising.
It also fixes a file-type detection bug this feature exposed: a
**compact single-line JSON object** was misrouted to `FileType.NDJSON`
and could never reach `partition_json` at all.
## Behavior change
| Input | Before | After |
|---|---|---|
| Object `{"customer": "Acme", ...}` | `ValueError: JSON cannot be
partitioned. Schema does not match…` | 1 `Text` element, pretty-printed
|
| Array of objects `[{"id":"one"},{"id":"two"}]` | same `ValueError` |
one `Text` per object, array order |
| Array of scalars `[1,2,3]` | crash: `AttributeError: 'int' object has
no attribute 'get'` | 1 `Text` with the whole array |
| Mixed array / top-level scalar | `ValueError` / crash | 1 `Text` |
| Mixed element-shaped + arbitrary array | partial rehydrate, arbitrary
items silently dropped | whole array as arbitrary JSON - nothing dropped
|
| Compact single-line object, `.json` file | misrouted to NDJSON →
rejected | detected as `FileType.JSON` → 1 `Text` |
| Arbitrary NDJSON (one record per line) | `ValueError` | one `Text` per
line, line order |
| Serialized element JSON / NDJSON (arrays / line-per-element) |
rehydrates | rehydrates (unchanged, full regression suite green) |
| Element-shaped payload with corrupt contents (bad
`metadata.coordinates`, non-gzip `orig_elements`) | raw
`ValueError`/`binascii.Error`/`zlib.error` leaks | chained
`ValueError("Payload resembles serialized Unstructured elements but
could not be reconstructed: …")` |
| Serialized `TableChunk` elements (chunked output with split tables) |
silently dropped by `elements_from_dicts()` | rehydrate as `TableChunk`;
whole chunked payloads round-trip and can feed
`reconstruct_table_from_chunks()` (#4291) |
| `{}` (JSON route) | error | one `Text` containing `{}` (per the output
contract) |
| `[]` / empty string (JSON route) | `[]` / `[]` | `[]` |
| `{}` / `[]` as an NDJSON line | error | `Text("{}")` / `Text("[]")` (a
line is a record) |
| Deeply nested payload (any depth) | `RecursionError` escapes |
`ValueError` (`"Not a valid json"` / `"…nested too deeply…"`) |
| Malformed JSON `[{"hi":"there"}]]` | `ValueError("Not a valid json")`
| unchanged |
| Malformed NDJSON line | `ValueError("Not a valid ndjson")` | unchanged
|
## Design notes
- **Mode selection:** an explicit shape predicate
(`is_element_shaped_dict` in `partition/common/json_partitioning.py`) -
a list rehydrates only when every item is a dict with a recognized `str`
`type`, the type's required field (`str` `text` / `bool` `checked` for
CheckBox), and dict-or-absent `metadata`. Branches are exclusive; no
exception-based control flow. Prefix/schema pre-gates are removed from
`partition_json`, `partition_ndjson`, and `auto.py`;
`is_json_processable`/`is_ndjson_processable` are **deprecated**
(DeprecationWarning naming the replacement) but keep working for
downstream importers. `unstructured.file_utils.ndjson.loads/load` are
intentionally retained undeprecated as generic utilities.
- **Documented limitation** (pinned by tests): an array whose items
*all* look like serialized elements rehydrates rather than being treated
as arbitrary JSON. An element-shaped payload whose field contents fail
rehydration raises a chained `ValueError` - loud, never a leaked
low-level error.
- **`staging/base.py` deliberately not modified:** the shape predicate
rejects payloads like `{"type": "Title"}` (no `str` text) before
`elements_from_dicts` is ever called, so they partition as arbitrary
JSON; hardening `item["text"]` → `.get()` in staging instead would
silently rehydrate customer dicts as empty elements.
- **Filetype disambiguation:** whole-payload `json.loads` success →
`FileType.JSON`; else ≥2 newline-delimited JSON values → `NDJSON`. The
probe is **bounded to 1 MiB**, distinguishes an exact-bound-size payload
from a truncated one, restores the file position (a
`detect_filetype(file=f)` → `partition_json(file=f)` sequence works on
the same handle), and treats `RecursionError` as a parse failure. For
payloads exceeding the bound, one or more complete parsing lines
classify as NDJSON (first-line semantics for oversized records); the
residual degradation is an NDJSON file whose *first* record has no
newline inside the bound, which classifies as JSON. One intentional
flip: a *one-line serialized-element object* previously rehydrated via
the NDJSON route; it now partitions as arbitrary JSON (rehydration
applies only to arrays).
- **Output contract:** `pretty_json_text()` - `json.dumps(value,
indent=2, sort_keys=True)`; stable, diffable output; `sort_keys`
alphabetizes source field order (commented at the definition as the knob
to revisit). NDJSON is strictly one `Text` per line; the only
empty-container divergence is `[]` (an NDJSON line yields `Text("[]")`,
a JSON-mode `[]` document yields no elements).
- **Deferred by design:** per-field metadata / JSONPath addressing and
structure-aware tree walking. `elements_from_arbitrary_value()`
(`partition/common/json_partitioning.py`) is the single swap-point for a
future walker.
## TableChunk rehydration - intent
Review of this branch surfaced that serialized `TableChunk` dicts could
not be deserialized at all: `TYPE_TO_TEXT_ELEMENT_MAP` has no
`TableChunk` entry, so `elements_from_dicts()` silently dropped them on
`main`, and with this branch's shape predicate one TableChunk flipped an
entire serialized payload to arbitrary-JSON `Text`. The intent of the
fix commit is narrow and explicit:
- **Complete #4291's design, using only existing mechanisms.**
`reconstruct_table_from_chunks()` (added in #4291) filters
`isinstance(e, TableChunk)` and its docstring states reconstruction "can
be called on user-provided/deserialized chunks" - but no deserializer
could produce a `TableChunk` until now. The reconstruction metadata
(`table_id`, `chunk_index`, `is_continuation`,
`num_carried_over_header_rows`) already round-trips via
`ElementMetadata`; only the element-class dispatch was missing.
- **Special-case, not a map entry - deliberately.**
`TYPE_TO_TEXT_ELEMENT_MAP` also feeds the COCO category vocabulary
(`convert_to_coco` derives positional category ids from its keys), so
adding TableChunk there would renumber existing category ids on every
export, even for data with no TableChunks. Instead
`elements_from_dicts()` special-cases `"TableChunk"` exactly like the
existing `CheckBox` special case. Verified strictly additive: a 40-dict
all-type sweep shows byte-identical output vs the pre-fix function for
every non-TableChunk payload; `documents/elements.py` is untouched; COCO
ids unchanged.
- **No new downstream exposure.** `type: "TableChunk"` records already
flow to destinations today whenever live chunking splits a table
(library output, ingest, hosted API responses); this changes only the
read-back side.
## Testing
- Combined suites (partition json/ndjson/shared-predicate, filetype,
staging serde, chunking dispatch/reconstruct): **691 passed, 11 skipped
(pre-existing optional-dep skips), 1 xfailed**; `test_auto.py -k "json
or ndjson"` **7 passed, 1 xfailed** (the #3365 strict-xfail - unaffected
and kept).
- Full rehydration regression set green (round-trips, chunking,
`last_modified`, metadata stamping), including end-to-end NDJSON
rehydration through `partition()`.
- Five pre-existing tests intentionally flip expected behavior (`{}` no
longer raises, empty NDJSON container lines emit `Text`, one-record
payloads route JSON) - each renamed to describe the new behavior.
- Exact pretty-printed literals are confined to one canonical test per
output shape; other tests assert structurally (element type + content
containment) so a future formatting change doesn't invalidate dozens of
assertions.
- TableChunk: end-to-end chunked-table round-trip (chunk -> serialize ->
`partition_json` -> byte-identical re-serialization ->
`reconstruct_table_from_chunks` returns the table), NDJSON TableChunk
lines, predicate accept/reject, and staging serde tests. The
exact-bound-size detection probe is pinned by a test proven to fail if
the probe is reverted.
- Coverage includes `text=`/`file=`/`filename=` routes, corrupt-payload
errors via `file=`, deep-nesting through both partitioners,
boundary-size (1 MiB ± 1) disambiguation, detect-then-partition on one
file handle, and chunking over arbitrary JSON **and** NDJSON output.
- New business-neutral fixtures: `example-docs/arbitrary-records.json`,
`single-line-object.json`, `arbitrary-records.ndjson`.
- Hardened by two independent review passes (adversarial correctness +
convention/nit pass), each finding verified by reproduction before being
fixed.
Example Docs
The sample docs directory contains the following files:
example-10k.html- A 10-K SEC filing in HTML formatlayout-parser-paper.pdf- A PDF copy of the layout parser paperfactbook.xml/factbook.xsl- Example XML/XLS files that you can use to test stylesheets
These documents can be used to test out the parsers in the library. In addition, here are instructions for pulling in some sample docs that are too big to store in the repo.
XBRL 10-K
You can get an example 10-K in inline XBRL format using the following curl. Note, you need
to have the user agent set in the header or the SEC site will reject your request.
curl -O \
-A '${organization} ${email}'
https://www.sec.gov/Archives/edgar/data/311094/000117184321001344/0001171843-21-001344.txt
You can parse this document using the HTML parser.