-
[OPIK-6555] [BE] feat: online LLM-as-judge scorer loads and reasons over trace attachments (#7202)
frostbyte_neo 发布于
2026-06-26 08:03:05 +00:00 - [OPIK-6555] [BE] feat: online LLM-as-judge scorer loads and reasons over trace attachments
Extend the single-trace LLM-as-judge scorer to route to the agentic-tools
path when a trace has attachments, mirroring Phase-1 behaviour for thread
evaluations.- GetAttachmentTool: fetches attachment bytes from MinIO (base64) or
presigns a download URL from S3, then stages the media on TraceToolContext
for injection by ToolCallLoop as a multimodal UserMessage. - ReadTool: appends attachments:[{file_name,mime_type,media_type}] to every
trace read so the judge can discover files without a separate list call. - MediaCategory / MediaPayload / MediaMessageBuilder / AttachmentSummaries:
supporting types for the media side-channel. - TraceToolContext: stageMedia/drainPendingMedia/canInjectMedia with a
shared count cap (MAX_INJECTED_ATTACHMENTS=8) applied equally to MinIO
and S3 paths. - ToolCallLoop: drains staged media after each round and injects it as a
multimodal UserMessage; surfaces provider failures via the shared
surfaceInjectedMediaFailure handler. - OnlineScoringLlmAsJudgeScorer: fetches AttachmentInfo alongside spans;
routes to agentic tools when attachments are present; emits a user-facing
warning when the provider does not support tool-calling.
Implements OPIK-6555
-
improvements from local CR
-
fix misleading comment on attachment fetch — no {{trace}} substitution exists
Co-Authored-By: Claude Sonnet 4.6 noreply@anthropic.com
- fix BE formatting
Co-Authored-By: Claude Sonnet 4.6 noreply@anthropic.com
- fix unit test stubs: add charsPerToken stub and empty variables to EVALUATOR_JSON
Co-Authored-By: Claude Sonnet 4.6 noreply@anthropic.com
- replace count-based attachment cap with 20 MB size-based cap
Remove MAX_INJECTED_ATTACHMENTS=8 and replace with MAX_INJECTED_BYTES=20MB
across TraceToolContext, GetAttachmentTool, MediaPayload, and AttachmentInfo.
A byte cap prevents context blow-up regardless of file count; the 20 MB
budget accommodates several high-resolution images without overwhelming
the judge's context window. file_size is already stored in the DB and
propagated through AttachmentService → AttachmentInfo for use here.Co-Authored-By: Claude Sonnet 4.6 noreply@anthropic.com
- [OPIK-6555] test: add unit tests for 20 MB attachment injection cap
Tests cover TraceToolContext.canInjectMedia boundary conditions and
byte accumulation via stageMedia, plus GetAttachmentTool rejection on
both MinIO and S3 paths when fileSize exceeds MAX_INJECTED_BYTES.Co-Authored-By: Claude Sonnet 4.6 noreply@anthropic.com
- [OPIK-6555] feat: make attachment injection size cap configurable
MAX_INJECTED_BYTES is now driven by onlineScoring.agenticToolsMaxInjectedBytes
(env: ONLINE_SCORING_AGENTIC_TOOLS_MAX_INJECTED_BYTES) with a default of 50 MB.
Tests continue to use the DEFAULT_MAX_INJECTED_BYTES constant so no test
call sites need changing.Co-Authored-By: Claude Sonnet 4.6 noreply@anthropic.com
- [OPIK-6555] fix: base64 inflation in MinIO attachment size cap
canInjectMedia pre-check now uses ceiling base64 estimate ((n*4+2)/3)
instead of raw fileSize, and stageMedia accumulates base64.length() so
the injected-bytes counter reflects what is actually sent to the model.
S3 path keeps fileSize as a conservative proxy (no bytes are inlined).Also switch AttachmentInfo positional constructor to builder in
SpansResourceTest for resilience to field-order changes.Co-Authored-By: Claude Sonnet 4.6 noreply@anthropic.com
-
[OPIK-6555] [BE] fix: reject zero-size attachment file size in get_attachment tool
-
[OPIK-6555] perf: cache per-trace attachment list in TraceToolContext
Avoids repeated DB round-trips when the judge re-reads the same trace
(common in thread evals drilling through several traces) by memoising
the attachment list alongside the entity JSON already cached in fetched.Co-Authored-By: Claude Sonnet 4.6 noreply@anthropic.com
- [OPIK-6555] fix: add missing workspaceName arg to TraceToScoreLlmAsJudge in test
Co-Authored-By: Claude Sonnet 4.6 noreply@anthropic.com
-
limit pre-signed url duration to 2 min
-
[OPIK-6555] feat: make S3 presign TTL configurable via OnlineScoringConfig
Replaces the hardcoded Duration.ofSeconds(120) constant in GetAttachmentTool
with agenticToolsS3PresignTtlSeconds in OnlineScoringConfig (@Min(1) @Max(3600),
default 120 s). Tool description updated to say "time-limited" rather than a
specific duration to avoid drift when the config value changes.Co-Authored-By: Claude Sonnet 4.6 noreply@anthropic.com
- [OPIK-6555] fix: add agenticToolsS3PresignTtlSeconds to test config
Prevents validation failure when CI environment doesn't set the corresponding
env var (defaults to 0, which fails the @Min(1) constraint).Co-Authored-By: Claude Sonnet 4.6 noreply@anthropic.com
- [OPIK-6555] fix: add agenticToolsS3PresignTtlSeconds to config.yml
Root cause of the CI validation failure: config.yml is the authoritative
source Dropwizard deserializes — fields absent from it deserialize as 0
for primitives, which fails @Min(1). Added the field with the standard
${ENV_VAR:-120} pattern alongside the other agenticTools* entries.Co-Authored-By: Claude Sonnet 4.6 noreply@anthropic.com
- [OPIK-6555] test: verify configured S3 presign TTL is forwarded to attachmentService
Uses a non-default TTL (60 s) in fetchImageFromS3StagesPresignedUrl and
matches presignDownloadUrl with eq(Duration.ofSeconds(60)) instead of any(),
so a regression to a hardcoded duration would break the mock and fail the test.Co-Authored-By: Claude Sonnet 4.6 noreply@anthropic.com
Co-authored-by: Claude Sonnet 4.6 noreply@anthropic.com
下载附件