-
[OPIK-7051] [BE] feat: per-workspace online-scoring sampling metric + populate error workspace (#7195)
frostbyte_neo 发布于
2026-06-23 13:35:09 +00:00 - feat(online-scoring): per-workspace sampling metric + populate error workspace
Adds the per-workspace observability the online-scoring funnel was missing:
-
online_scoring_sampler_decisions_total{workspace_id, evaluator_type, decision}
emitted from OnlineScoringSampler. Decisions: sampled / skipped_disabled /
skipped_filter / skipped_sampling. Gives "sampled-to-be-scored per workspace"
(the stage between ingestion and scoring) which had no metric before. -
OnlineScoringBaseScorer now overrides messageContext(), so
online_scoring_*_processing_errors_total carries the real workspace_id instead
of "unknown". Backed by a new WorkspaceScopedMessage marker interface on the six
ToScore records; the base scorer's type param is bounded to it.
No functional change to scoring behavior; metrics only.
Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
- refactor(online-scoring): record 'sampled' at enqueue + dedupe skip branches
Addresses PR review:
- 'sampled' is now recorded at the enqueue site (count = messages published),
so online_scoring_sampler_decisions_total{decision="sampled"} reflects work
actually sent to Redis — never the sampling roll alone (e.g. the
python-evaluator-disabled path no longer counts as sampled). - Extracted a skip() helper used by the disabled / filter-mismatch / sampling
branches of shouldSampleTrace, so a new skip reason is one call site.
Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
- feat(online-scoring): per-workspace processed counter
Adds online_scoring__processed_total{workspace_id}, incremented on the
scorer success path in OnlineScoringBaseScorer.processEvent. Combined with the
now workspace-tagged *_processing_errors_total (failures), this gives the
consumer-side processed-vs-failed split per workspace, completing the funnel:
ingested -> sampled -> processed/failed, all per workspace.Scoped to online scoring (not the shared BaseRedisSubscriber) so workspace_id
cardinality is not added to the webhook/dataset/experiment consumers.Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
- fix(online-scoring): count processed only after the full chain succeeds
PR review: the processed counter incremented when score() completed, but
OnlineScoringLlmAsJudgeScorer does post-scoring work (testSuiteAssertion
decrementAndFinishIfComplete) after super.processEvent; a failure there retries
the message, so a failed/retried attempt was counted as processed.Make processEvent a final template that wraps an overridable doScore() with the
processed-success counter, and have the LLM scorer override doScore (not
processEvent). The counter now fires only after the entire per-message chain
completes, so retried failures are not counted.Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
- feat(online-scoring): add workspace_name (from context) to per-workspace metrics
Per-workspace metrics now carry both workspace_id and a human-readable
workspace_name, so dashboards aren't stuck showing UUIDs.The name is sourced from the request's reactive context (RequestContext.
WORKSPACE_NAME, populated by auth) at trace-event publish time in TraceService,
carried on TracesCreated/TracesUpdated, then onto the ToScore messages by the
sampler — no WorkspaceNameService lookup on the consumer path. The base scorer
also puts workspace_id + workspace_name on the reactive context for the whole
scoring chain. All sites fall back to workspace_id via StringUtils.defaultIfBlank
when the name is absent (older in-flight events).Metrics updated: online_scoring_sampler_decisions_total,
online_scoring_processed_total, online_scoring_processing_errors_total.WorkspaceScopedMessage gains a workspaceName() default (returns id); the two
trace-level records carry a real workspaceName field.Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
- chore(online-scoring): mark optional workspaceName @Nullable on trace events
PR review: make the null contract explicit on TracesCreated/TracesUpdated
workspaceName (field + 4-arg constructor param), since it's intentionally
optional and consumers fall back to workspaceId.Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
- chore(online-scoring): address review nits — MessageContext builder + clearer workspaceName doc
- Add @Builder to MessageContext and use it in OnlineScoringBaseScorer.messageContext.
- Make WorkspaceScopedMessage.workspaceName() Javadoc explicit that it falls back to
the workspace id when no name is carried.
Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
Co-authored-by: Claude Opus 4.8 (1M context) noreply@anthropic.com
下载附件