-
feat(hybrid): retry transient hancom-ai failures and bound the call
frostbyte_neo 发布于
2026-08-25 09:44:14 +00:00 The client had no retry and no effective timeout: a backend restart lost
the layout for the pages in flight, and the default timeout of 0 meant a
hung request waited forever, which in a batch cannot be told apart from a
hung client.Timeouts. An explicit --hybrid-timeout keeps its exact meaning, 0
included, so a deliberate setting is unaffected. Only the unset case
gains a ceiling: connect 10s, and one whole call capped at 1 hour via
callTimeout. Not readTimeout — the backend computes for minutes and then
answers at once, so an idle-byte limit would kill healthy requests. The
cap sits far above real work (a 20-page OCR chunk measured 268s); its job
is to end a hang, not to pace normal calls. The health probes override
callTimeout rather than inherit it: an hour-long health check is not one.Retries. Three attempts, waiting 5s then 45s. The second wait clears a
restart rather than a blip — scan.sh records restarts taking 45-60s, so a
retry sooner than that just spends an attempt on a server still starting.What is retried is deliberately narrow:
- 5xx, unusable body, connect failure, timeout — transient
- 4xx, and a module the server does not have — permanent, one attempt
- SUCCESS=false — one retry only. It arrives with HTTP 500 and
RESULT: both from a restarting backend and from a document the
backend genuinely rejects, and the two are indistinguishable; measured,
one 2.2MB PDF returned it three times running. Each attempt costs GPU
time, so it does not get the full budget. Carried as a field on the
attempt rather than re-derived from the log wording.
Layout, page images and the per-region enrichment calls all share the
budget: the enrichment calls are the numerous ones, so a restart lands
hardest there — without a retry every table on the page comes back
structurally empty and nothing reports it.That breadth needs bounding, because the budget is per request and a
backend refusing everything makes every page pay it. Measured on a
document whose pdf2img calls all failed: 50s per page, past ten minutes
of pure waiting. Two streak limits now cap it — after 3 consecutive
exhausted enrichment calls, and after 3 layout slices that returned no
pages, the rest are sent once each. The same document finishes in 325s
with 6 retries instead of one per page.Every retry logs at WARNING with the attempt and the wait, so a retried
hang is visible rather than silent.865 tests pass, 11 new. Each retry decision is pinned by exact request
count, including one that fails at 37 requests instead of 19 if the
breaker is removed. Output on a healthy backend is unchanged and no retry
fires.Co-Authored-By: Claude Opus 5 (1M context) noreply@anthropic.com
下载附件