-
[OPIK-7118] [BE] perf: make attachment base64 detection linear (possessive match) (#7252)
frostbyte_neo 发布于
2026-07-08 09:18:53 +00:00 - [OPIK-7118] [BE] perf: make attachment base64 detection linear (possessive match)
The {minBase64Size,} regex kept the size threshold inside the pattern, so find() re-scanned
from every offset on large payloads (O(n*runlen)), saturating CPU and crash-looping
opik-backend in prod. Match each maximal base64 run possessively ([A-Za-z0-9+/]++) and apply
the size threshold in code instead: a possessive run is consumed once with no backtracking, so
detection is linear regardless of payload size. Behavior is identical to the old pattern --
maximal contiguous [A-Za-z0-9+/] run >= minBase64Size plus up to two trailing '=' (any other
char, incl. line-wrapping whitespace, ends a run; wrapped base64 left intact). Drops the
per-find() OpenTelemetry span; keeps the minBase64Size floor, no ceiling.Verified byte-for-byte identical to the old regex: AttachmentStripperServiceTest 15/15
(contiguous strip, threshold boundary, 0/1/2 padding capture, no-bridge-across-whitespace with
strict RFC 4648 round-trip, wrapped-left-intact, negatives, OOXML); a 1.8M-case fuzz across 6
thresholds; and a differential over 253 real prod payloads (45 workspaces) -- 0 divergences,
with the 79 fields that timed out the old regex (>5s) handled in <=234ms.Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com
- [OPIK-7118] [BE] Add attachment-stripping observability metrics
Add two OpenTelemetry instruments alongside the linear detection fix so a stuck
or saturated stripper is visible in real time (the throughput counters only move
when work completes, so a hang is invisible in them):- opik.attachments.strip.in_progress (UpDownCounter): payloads currently being
stripped; stays elevated while work is in flight. - opik.attachments.input.size.chars (histogram): length of candidate fields,
recorded before scanning, as a leading indicator for oversized payloads.
These back the saturation panels in comet-monitoring#179. Detection behaviour
is unchanged.Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
- [OPIK-7118] [BE] Restore detection latency span (base64.detect)
The original code wrapped the regex in trace spans; they were dropped with the
possessive rewrite. The spans were never the performance problem (the O(n*runlen)
re-scan was), and they provide the latency attribution that pinpoints which step
is slow — e.g. a stuck scan shows as base64.detect self-time with no child spans.Re-add a single span per text node (not per find(), avoiding span churn) around
the detection scan. The existing base64.decode / tika.detect / eventBus.post
spans nest under it, so per-method latency stays visible in traces.Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
- [OPIK-7118] [BE] Address review comments
- Restore the original regex.match / regex.find span names (replacing the single
base64.detect span) so detection latency stays comparable before/after the fix. - processTextNode: make
lengthfinal and use it instead of re-calling
text.length(); add a comment explaining the trailing-'=' padding loop. - BASE64_RUN: convert the field comment to Javadoc.
- Test: replace fully-qualified names with imports; use text blocks for the
OOXML XML fixtures.
Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
Co-authored-by: Claude Opus 4.8 noreply@anthropic.com
Co-authored-by: Thiago Hora thiagoh@comet.com
(cherry picked from commit1262bc9ad9)下载附件