-
[OPIK-6994] [BE][FE] feat: persist online-evaluation runs as hidden monitoring traces (#7244)
frostbyte_neo 发布于
2026-06-30 17:05:51 +00:00 - [OPIK-6994] [BE][FE] feat: persist online-evaluation runs as hidden monitoring traces
Backend:
- New OnlineScoringTracePersistence records each LLM-as-judge evaluation
(trace/span/thread) as one hidden, source=evaluator monitoring trace with a
prepare_evaluation span + one llm_call span per round + tool spans (agentic
mode). Spans carry token usage incl. provider prompt-cache tokens
(Anthropic/OpenAI/Gemini) so existing span->trace cost aggregation prices them. - Null-object EvaluationRecorder (NOOP) keeps the scoring path branch-free;
gated by TOGGLE_ONLINE_SCORING_TRACING_ENABLED. Errors are stamped on the
span and the trace; persistence is best-effort and never breaks scoring. - Migration 000092 adds 'evaluator' to the source Enum8 on traces and spans.
- The engine skips its own monitoring traces (Source.EVALUATOR is non-logging).
Frontend:
- Per-rule "Go to traces" action on the Online Evaluation page reuses the
experiment logs sidebar, configured for the evaluator view via a
TraceLogsViewConfig (KPI dashboard on top, hidden-visibility traces, isolated
column state, trimmed default columns). Shared sidebar/useTracesList stay
config-driven with defaults that leave the existing logs pages unchanged.
Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com
- feat(online-eval): preview evaluated input/output prettily on prepare span
The prepare_evaluation span now puts the evaluated entity's own input on the
span input and its output on the span output, so the UI renders them in "pretty"
mode (chat/text) exactly as on the source trace instead of an escaped JSON blob.
Evaluation bookkeeping (evaluated id, project id, name, model, fetched span
count, estimated tokens, mode) moves to the span metadata. Small payloads keep
their structure; payloads over the cap fall back to a truncated text node.Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com
- fix(online-eval): address review feedback on tracing persistence
- previewNode: measure and slice the same string representation per node kind;
the old code measured node.toString() but sliced node.asText(), which could
slice a string shorter than the cap and throw IndexOutOfBounds (begin() runs
outside the best-effort error chain, so this could surface during scoring). - EvaluatedSubject: add @Builder(toBuilder = true) and route the ofTrace/ofSpan/
ofThread factories through the builder, per the backend DTO convention. - create(): single-quote the entity placeholder in the warn log, per logging convention.
- migration 000092: make the rollback a documented no-op (narrowing the source
Enum8 to drop 'evaluator' fails for rows already written with that value).
Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com
- fix(fe): fix evaluation-traces sidebar pagination and controls
The online-evaluation rules table rendered one TraceLogsSidebar per row, all
sharing the same tls_* query params, and the sidebar's trace list never kept
previous data across page changes. Two consequences:- Per-row sidebars raced on the shared tls_* state. Replace them with a single
page-level EvaluationTracesSidebar driven by the new useTraceLogsSidebarControls
hook; each rule row now renders only a trigger (renderSidebar={false}). - Paging never advanced: without keepPreviousData,
totaldropped to 0 mid-fetch,
tripping DataTablePagination's out-of-range guard which reset back to page 1.
Add placeholderData: keepPreviousData (the sidebar already read isPlaceholderData),
so total stays stable and the guard only resets genuinely out-of-range pages.
Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com
- fix: address high-severity PR review findings (OPIK-6994)
- previewNode: measure the serialized form against the preview cap (heavily
escaped strings could otherwise exceed it once rendered as JSON) and clamp
the slice index to the raw text length, keeping the earlier IOOBE fix. - thread scorer: make the monitoring recorder best-effort with onErrorResume,
so a project-name lookup failure degrades to NOOP instead of aborting the
Mono.zip and stopping thread scoring. - openSidebar: always (re)write tls_filters, clearing it when no source filters
are given, so an unfiltered open can't reuse the previous open's filters.
Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com
- refactor(online-scoring): decouple eval observability into a fire-and-forget two-layer recorder
Addresses the main review feedback: a monitoring framework must never disrupt the
flow it observes. The recorder was woven into the scoring value/error chain
(flatMap+thenReturn, onErrorResume+Mono.error, span write gating the response);
it now hangs off the flow as a side effect, with the contract enforcing safety.- Split the OnlineScoringTracePersistence god-class into two layers:
- domain/observability: ObservabilityTraceRecorder + ObservabilityContext, a
domain-agnostic fire-and-forget writer + single tap combinator (no evaluation refs). - domain/evaluation: EvaluationRecorder (+NOOP), EvaluatedSubject, EvaluationContext,
OnlineEvaluationRecorder — pure span/trace mapping, no reactive control flow.
- domain/observability: ObservabilityTraceRecorder + ObservabilityContext, a
- Enforce the protocol structurally: complete/fail are no longer public; scorers
finalize via a single monitor(scoring) combinator that taps success/error and
returns the result unchanged. recordLlmCall/recordToolCall are transparent taps. - Build AND write run on a separate scheduler, guarded — a monitoring failure can
never break, block, or throw into scoring; business errors propagate unchanged. - Drop per-evaluation llm/tool counters (denormalized, could diverge from spans).
Verified: 70 unit tests, API/ClickHouse e2e, and UI (hidden source=evaluator traces
excluded from default views/cost; a monitoring trace renders with spans + cost).Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com
- refactor(online-scoring): clean up the evaluation adapter (immutability, @NonNull, records over hand-rolled JSON)
Addresses the implementation-detail review comments on the evaluation adapter:
- Replace hand-built ObjectNode/ArrayNode JSON with small records (messages/response/
scores) serialized via JsonUtils.valueToTree, and Map for the dynamic-key metadata.
The shared mapper already applies snake_case + non-null, so output is unchanged. - Make EvaluationContext immutable: drop the volatile agentic flag / markAgentic().
The parent-trace 'mode' metadata is dropped (it's already on the prepare_evaluation
span; nothing consumed the trace-level copy), removing the last mutable field. - Add @NonNull to the public EvaluatedSubject components (kind, id).
Verified: 70 unit tests + localhost e2e (JSON output unchanged, monitoring trace
well-formed with correct spans and cost).Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com
- fix(fe): lock the evaluation-traces sidebar to its evaluator
The per-rule "Evaluation traces" sidebar seeded the rule scope (metadata.rule_id)
into the editable filter bar, so it showed as a removable "Metadata = " chip
and could be cleared to reveal every evaluator's traces.Make the rule scope a locked constraint instead: it's applied to the list, export,
stats and metrics queries but lives in a separate tls_scope param (not the editable
tls_filters), so it can't be changed or removed via the filter bar. A read-only
"" indicator shows what the view is locked to. Opt-in via lockScope, so
the experiment/annotation-queue trigger sidebars keep their current behavior.Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com
- refactor(online-scoring): address remaining review comments
- Replace the anonymous EvaluationRecorder.NOOP with a named NoopEvaluationRecorder.
- EvaluationContext is now a Lombok @Builder built entirely in the begin() factory
(no hand constructor); EvaluatedSubject.Kind uses @Getter/@RequiredArgsConstructor. - Reuse the Opik-wide responseFormatting.truncationSize for the prepare-span preview
cap instead of a bespoke constant/config (same field-truncation limit as all traces). - Migration rollback -> "empty" (matches the repo convention).
- Tests: use the real time-ordered IdGenerator (TestIdGeneratorFactory) instead of a
v4 mock, final-field mocks, evaluation types via imports; document the unit test's
scope. Add onlineScoringTracingEnabled to config-test.yml.
Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com
- refactor(online-scoring): polymorphic evaluated subjects + extract entity factory
- Replace the single subject factory with a sealed EvaluatedSubject
(EvaluatedTrace / EvaluatedSpan / EvaluatedThread) and overloaded begin(...) - Extract all span/trace construction into EvaluationEntityFactory, slimming
OnlineEvaluationRecorder down to orchestration - Use @Builder for the subjects so construction is name-safe (no positional
multi-arg constructors) - Assert the full span metadata object in the recorder test
Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com
- test(online-scoring): use imports instead of fully-qualified names in thread scorer test
Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com
- fix(fe): derive rule project from projects[] so the Go to traces button renders
GET rule responses populate projects[] (multi-project), not the legacy
project_id/project_ids, so the cell's projectId gate was always undefined
and the button never rendered. Read projects[0].project_id first, keeping
the legacy fields as fallbacks.Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com
- refactor(online-scoring): address review nits (record, builders, lombok, test assertions)
- EvaluationContext is now a record (immutable, private fields)
- RealRecorder uses @RequiredArgsConstructor
- DTO records get @Builder(toBuilder); construction uses the builders
- Recorder test: class comment -> javadoc, stub returns use the real v7
IdGenerator, and the usage-map checks are full-object assertions
Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com
- fix(online-scoring): extend trace_threads.source enum to match traces/spans
Migration 000092 added 'evaluator' to the source Enum8 on traces and spans but
not trace_threads. Thread-resolution queries combine the source column across
traces and trace_threads, and ClickHouse rejects mixing two Enum8 columns with
different value sets (by type, not data), so thread creation failed and
threadModelId stayed null — surfacing as TracesResourceTest thread-test timeouts.
Extend trace_threads.source to the same 6 values to keep the enums in sync.Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com
Co-authored-by: Claude Opus 4.8 noreply@anthropic.com
下载附件