-
feat(hybrid): split long documents across several layout requests
frostbyte_neo 发布于
2026-08-24 05:21:42 +00:00 The Hancom AI layout module takes a whole PDF and offers no page-selection
parameter, and rejects requests past roughly 30 pages. HancomAIClient never
read request.getPageNumbers(), so the 50-page chunking in
HybridDocumentProcessor was a no-op for this backend: every chunk re-uploaded
the entire document and got every page back. Long documents failed outright
and shorter ones did the same work twice.The client now slices the document into shorter PDFs of
hybrid.layoutPageChunk pages (default 20, leaving headroom under the limit)
and merges the replies. Slicing keeps the PDF form rather than rendering
pages to images, which would strip the text layer and push a digital
document down the OCR path.The correctness hinge is page numbering: the module numbers response pages
from 0 relative to the request, while the transformer reads page_number as an
absolute document index. Without translation, page 30's content lands on
page 0 — nothing throws and the output looks well formed. Requested pages are
filtered against the real page count before slicing so the slice list and the
uploaded PDF cannot disagree, records numbered outside the slice are dropped
rather than placed, and merged records are sorted by absolute page so the
evidence JSON is not left shuffled.Every requested page that comes back without a layout record is reported for
the Java pipeline to retry, on both the sliced and whole-document paths. Left
unreported it becomes an empty page in the output, which reads as a page that
genuinely had no content. That covers a slice that errors, one answered
(empty but well-formed, which an array-size check passes as success), one
that comes back short, and one whose page tree is damaged, since PDFBox
throws IllegalStateException there rather than IOException. Only a total
failure raises.The document goes up whole only when every page is wanted: 10 pages of a
100-page document fit under the limit, but sending the file would upload all
100 and make the backend process them. Turning slicing off means "do not
divide the pages", not "ignore the selection", and a selection naming no page
that exists raises rather than answering with pages nobody asked for.hancom-ai no longer chunks in HybridDocumentProcessor, since the client does
it: the outer split only re-uploaded the file and, because just the last
call's raw JSON is kept, dropped the earlier pages' evidence.The chunk size is settable as --hybrid-hancom-ai-layout-page-chunk. The
backend's real limit is not measured — the servers were unreachable while
this was built — so it needs to be adjustable without a code change, and
verifying the behaviour needs it from the command line.Verified against a page-aware probe that reports which source page each
record carried: 1, 20, 21, 30, 31, 82 and 255-page documents place every page
exactly once, zero misplaced, and no request exceeds the chunk size. A mutant
that ignores the slice mapping collapses a 32-page document to 20 pages, and
one that skips the missing-page accounting fails five tests. Core suite: 822
tests, no failures; CLI verification 61 passed.下载附件