-
[OPIK-5768] [BE] fix: restrict online evaluation rules to SDK source traces/spans (#6217)
frostbyte_neo 发布于
2026-04-14 07:41:14 +00:00 - [OPIK-5768] [BE] fix: restrict online evaluation rules to SDK source traces/spans
Online evaluation rules were running on all traces regardless of source,
including playground and experiment traces. This caused confusing duplicate
scoring when using evaluation suites in the playground.Now traces/spans are only scored if their source is SDK (or null for legacy
backward compatibility). Playground traces with selected_rule_ids metadata
are still scored by the explicitly selected rules.- fix(online-scoring): scope selected_rule_ids per trace to prevent cross-trace leakage
Previously,
selected_rule_idswas merged project-wide and the union was used
to narrow the whole evaluator list. That narrowing then applied to every
scorable trace in the batch — including SDK traces — so a single Playground
trace's selection could silently restrict SDK scoring when the two sources
share a batch. This was also a latent issue in main (order-dependent, since
the original code read only the first trace's metadata) — OPIK-5768's
batch-wide union made the symptom deterministic rather than fix it.Fix: scope evaluator selection PER TRACE.
- Build
Map<UUID, Set<UUID>> selectedRuleIdsByTrace(traceId → selected rules). - Absent key = SDK trace = all evaluators apply.
- Present key = non-SDK trace = only that trace's selected evaluators apply.
- Apply the filter upstream of
shouldSampleTracein the evaluator loop. - Remove the now-unused
filterEvaluatorsBySelectedRuleIdshelper.
Tests updated:
scoresEachNonSdkTraceOnlyByItsOwnSelectedRuleIds(renamed from
unionsSelectedRuleIdsAcrossMultipleNonSdkTraces): asserts evalA scores
only traceA (not traceB), and evalB scores only traceB (not traceA).scoresSdkTraceByAllEvaluatorsEvenWhenBatchIncludesNonSdkSelection
(renamed fromscoresMixedBatchAndNarrowsEvaluators): asserts the SDK
trace is scored by BOTH selected and other evaluators, while the
playground trace is scored only by its own selection.
Addresses PR#6217 review comment from baz-reviewer[bot] on
OnlineScoringSampler.java:156.- test(automation-rules): pin Source.SDK on Podam traces in resource tests
AutomationRuleEvaluatorsResourceTest relied on factory.manufacturePojo(Trace.class)
to build traces, which meant Podam filled Trace.source with a random Source
enum (SDK/PLAYGROUND/EXPERIMENT/OPTIMIZATION). With OPIK-5768's new online-scoring
source filter rejecting non-SDK traces without selected_rule_ids, ~75% of runs
in DisabledRulesTest and GetLogs would skip scoring entirely, making the
Awaitility.await() assertions time out after 10s.Fix: explicitly set .source(Source.SDK) on all 11 Podam-built traces in this
file. The inline comment references OPIK-5768 so future readers understand
why this is load-bearing.Mirrors the equivalent pin applied to OnlineScoringEngineTest.onTracesUpdatedWhenEndTimeSetTriggersScoring.
- refactor(online-scoring): consolidate isLoggingSource into Source enum
Both OnlineScoringSampler and OnlineScoringSpanSampler carried an identical
one-line predicate (source == null || source == Source.SDK) as a private
helper. Consolidate into a single static method on the Source enum —
domain logic belongs with the data — so:- Source.isLoggingSource(Source) is the single source of truth for which
sources are eligible for automatic online-scoring evaluation - Both samplers call Source.isLoggingSource(entity.source()) directly
- The private helpers are removed from both samplers
- The Javadoc on Source.isLoggingSource documents the SDK/null contract
and the evaluation-eligibility semantics
Addresses PR#6217 review comment from baz-reviewer[bot] (comment 3074225227).
下载附件