-
[OPIK-7103] feat: {{trace}} + {{span}} agentic attachment evaluation for online LLM-as-judge (#7277)
frostbyte_neo 发布于
2026-07-07 13:39:25 +00:00 - [OPIK-7103] feat: {{trace}} + {{span}} agentic attachment evaluation for online LLM-as-judge
Squashed feature + review fixes; see PR #7277. Reconciled with main: agentic eval
is triggered ONLY by the {{trace}}/{{span}} sentinels (not by attachment presence),
and main's OnlineEvaluationRecorder monitoring is threaded through the flow.Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com
- [OPIK-7103] perf: hop {{trace}} structure serialization onto parallel scheduler
buildTraceStructure ran buildFullJson + compress(FULL) — the most
CPU-/GC-expensive part of routing — inside a .map() on the Mono.zip of the
attachment lookups, so it executed on the R2DBC/attachment scheduler thread.
Wrap the body in Mono.fromCallable(...).subscribeOn(Schedulers.parallel()),
mirroring evaluate()'s existing prep hop, so large-trace serialization no
longer taxes the DB scheduler.Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com
- [OPIK-7103] refactor: address PR review feedback on online-eval attachment scorers
Addresses reviewer comments on #7277:
- AttachmentSummaries: serialize EntityType via its @JsonValue getValue() instead of name().toLowerCase()
- GetAttachmentTool: convert the tool/param description concatenations to text blocks (byte-identical strings via \ continuations)
- AttachmentService.getAttachmentInfoByEntityIds: null-safe entityIds via CollectionUtils.isEmpty (drop @NonNull)
- TraceCompressor: chain the 4-arg compress() through the attachments overload so the WITH_JQ_HINT default lives in one place
- OnlineScoringLlmAsJudgeScorer: give TraceStructure @Builder(toBuilder=true) + @NonNull and build it via the builder
- OnlineScoringBaseScorer: log the best-effort attachment re-read failures (previously swallowed silently) in both the single-entity and span upload-race helpers
- OnlineScoringSpanLlmAsJudgeScorer: extract the request-building if-branches into small aux methods returning an LlmRequests carrier
- OnlineScoringSpanLlmAsJudgeScorerTest: use imports over fully-qualified names, and Podam-manufacture span fixtures with toBuilder overrides
Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com
- [OPIK-7103] feat: make online-eval prompt cap + attachment upload-race retry configurable
Addresses reviewer comments K/L on #7277 (precedent: PR 6871):
- OnlineScoringConfig: add maxPromptFieldChars (@Min 500, default 4000), attachmentFetchMaxRetries
(@Min 0 @Max 20, default 5) and attachmentFetchRetryDelay (@MinDuration 50ms/@MaxDuration 5s, default 300ms) - OnlineScoringBaseScorer: read the retry count/delay from config; store config as a protected field
- Trace/Span scorers: read the prompt cap from config (getMaxPromptFieldChars)
- Remove the now-duplicate onlineScoringConfig field from the trace/span/thread scorers (use the base field)
- config.yml + config-test.yml: document the three new keys with env overrides
The attachment-fetch retry stays an interim guard; the pre-evaluation dispatch delay (OPIK-7224)
is intended to replace it, but the values are now tunable per environment in the meantime.Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com
- [OPIK-7103] fix: keep body-referenced transient attachments in {{trace}}/{{span}} structures
Addresses reviewer finding "Drops unrelated transient attachments" on #7277.
preferPersistentAttachments used an entity-wide gate: if any persistent attachment
existed, it dropped every auto-stripped attachment — so a legitimate transient-only
attachment (e.g. a REST-ingested image whose only copy is auto-stripped) was dropped
whenever an unrelated persistent attachment coexisted on the same entity.The reviewer's suggested "-sdk name pairing" is not implementable: the backend transient
name (input-attachment-{seqN}-{backendTs}.ext) and the SDK persistent name
(input-attachment-{random}-{clientTs}-sdk.ext) share no key. Use the body reference
instead — the reliable per-attachment signal:- AttachmentUtils.collectAttachmentReferences: gather the filenames referenced in the
entity body (mirrors the recursive walk of hasAttachmentReferences). - preferPersistentAttachments(attachments, referencedNames): keep every persistent
attachment plus any auto-stripped copy still referenced in the body; drop only orphaned
auto-stripped copies when a persistent copy coexists (unchanged when none does). - Thread a per-span referenced-name map through listSpanAttachmentsToleratingUploadRace /
groupBySpanPreferringPersistent; gatherSpanAttachments builds it in its existing pass.
A transient coexisting with its own -sdk twin is not a reachable state (backend-strip and
SDK-extract paths are mutually exclusive per upload), so erring toward keeping a referenced
attachment is safe. Updated the one test that encoded the old drop behavior; added retention
tests for the span + batched-trace paths and CollectAttachmentReferences unit tests.Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com
- [OPIK-7103] refactor: extract agentic-tools capability into AgenticScoringService
Addresses PR review feedback (comment on OnlineScoringBaseScorer#runToolCallLoop):
the online scorers' agentic-tools surface (tool-call loop orchestration,
upload-race-tolerant attachment listing, media-failure surfacing, tool-spec
attachment, provider capability check, size estimation) had grown too large to
keep propagating via inheritance from OnlineScoringBaseScorer. Extract it into
an injected AgenticScoringService; the trace/span/thread scorers are now users
of it instead of subclassing it, and the two Python-metric scorers no longer
inherit agentic-tools machinery they never use. Also collapses the thread
scorer's hand-rolled duplicate of the tool loop onto the shared implementation.Co-Authored-By: Claude Opus noreply@anthropic.com
- [OPIK-7103] refactor: address review nits on AgenticScoringService
- Drop @NonNull from AgenticScoringService interface method params (validation
belongs on the impl, not the interface contract). - surfaceInjectedMediaFailure: quote the attachments/detail log placeholders
and pass the throwable as the trailing SLF4J arg so the stack trace isn't
lost. - Extract resolveWithUploadRaceTolerance to de-duplicate the retry/best-effort
pipeline shared by listAttachmentsToleratingUploadRace and
listSpanAttachmentsToleratingUploadRace.
Co-Authored-By: Claude Opus noreply@anthropic.com
- [OPIK-7103] refactor: address andrescrz review nits on agentic-tools PR
Favours builders over constructors in the span scorer, narrows a swallowed
exception in AttachmentUtils with logging, caps maxPromptFieldChars, uses
lombok for the AgenticScoringServiceImpl constructor, avoids the exposed
Jackson mapper in favor of JsonUtils, and drops fully-qualified class names
in the newly added OnlineScoringEngineTest cases.Co-Authored-By: Claude Sonnet 5 noreply@anthropic.com
Co-authored-by: Claude Opus 4.8 noreply@anthropic.com
下载附件