-
[OPIK-6597] [QA] feat: Online Evaluation smoke (LLM-judge + Python rule) (#6874)
frostbyte_neo 发布于
2026-05-27 16:40:31 +00:00 - [OPIK-6597] [QA] feat: add backendClient helpers for online-eval polling + cleanup
Adds the four backendClient methods the Online Evaluation CUJ needs:
- getTrace(traceId): typed-down trace inspection with feedbackScores
- pollTraceForFeedbackScore(traceId, scoreName, opts): the async-state
polling helper that resolves when a feedback_score with the given name
appears on the trace, or throws with a verbose diagnostic listing the
last polled scores and elapsed time. Default 60s / 2s poll interval;
configurable per call. - listAutomationRulesForProject(projectId): inspection for cleanup.
- deleteAutomationRule(projectId, ruleId): explicit teardown because
project deletion does NOT cascade to automation rules (verified
against staging).
Phase 2 verification against staging confirmed:
- engine takes ~2s to score a trace once a rule is in place
- feedback_scores[*].source is the literal "online_scoring"
- feedback_scores[*].name is the metric's internal ScoreResult.name,
NOT the rule's UI-form name (POM will interpolate) - arguments: { output: "output.output" } correctly unwraps the
@opik.track-wrapped trace output (the engine uses JsonPath $.output
for dot-containing paths, raw $ for bare paths) - importing extra BaseMetric subclasses at module scope shadows the
user class (python_evaluator picks alphabetically-first); inline
equality is the safer pattern
- feat(fe): add data-testids for Online Evaluation page + AddEditRuleDialog
- online-evaluation-create-rule-button on the toolbar Create rule button
- add-edit-rule-dialog on the DialogContent
- add-edit-rule-dialog-submit on the dialog's Create/Update rule button
- add-edit-rule-dialog-type on the LLM-as-judge / Code metric toggle group
Used by the new tests/online-evaluation/online-evaluation-smoke.spec.ts.
Each testid replaces a load-bearing accessible-name selector that would
break under label refactors; the page is otherwise testid-free.- feat(global-setup): inject opik-version-override=v2 + dismiss welcome wizard
Two portability fixes that the suite needs to run reliably against any
deployment (cloud, OSS, self-hosted):-
The FE's WorkspaceVersionGate consults /workspaces/versions and mounts
the V1App for workspaces flagged "version_1" — which is most fresh
accounts on staging AND the OSS default workspace. V1App is missing
2.0 routes (e.g. /projects//online-evaluation) and renders
"Not Found". The Gate honors localStorage key opik-version-override
before the API check; injecting "v2" into the storage state file
side-steps the v1 fallback without depending on workspace-side flag
flips. Works for both auth-bearing deployments (extends existing
storage state) and OSS (creates a minimal file from scratch). -
Fresh OSS deploys show a "Welcome to Opik 🚀" first-run survey modal
that intercepts pointer events on every page. Dismiss it via the
/welcome-wizard API in global-setup so tests start against a clean
page. POST is idempotent — no-op on already-completed workspaces.
playwright.config.ts updated to always use the storage state file (OSS
previously had none).- fix(pom): wait for ConfigurationPage table to settle before counting rows
ConfigurationPage.listConfiguredProviders read the row count immediately
after the tabpanel became visible. The tabpanel renders before its row
list resolves, so counting before either a real row OR the empty-state
marker has rendered would report "no providers" for a populated table —
hasProvider then returned false and ensureProviderConfigured would open
the Add-provider dialog against a workspace that already had the key,
landing on the Edit-provider variant whose submit button has a different
label and not matching the POM's "Add provider" selector.Wait for whichever of (first row cell, "No AI providers yet" empty
state) resolves first, with a 10s timeout. If neither resolves, fall
through (the previous behavior).- feat(pom): extend TracePanelPage with Feedback scores tab helpers
Adds:
- feedbackScoresTab + feedbackScoresTabPanel locators
- openFeedbackScoresTab() — switches to the tab and waits for the panel
- feedbackScoreRow(scoreName) — filters by hasText (the Key cell
truncates long names via CSS ellipsis, so the cell's accessible name
reads "cuj-..." while the DOM textContent preserves the full string) - readFeedbackScoreValue(scoreName) — parses the Score column to a number
Used by the new online-evaluation smoke for the "see scores in the Logs
page" UI verification.- feat(pom): add OnlineEvaluationPage with LLM-judge + Python-code rule flows
Models /{workspace}/projects/{projectId}/online-evaluation and the
create-rule dialog. Two fill methods covering both rule types:-
fillAndSubmitCreateRuleDialogLLMJudge({name, template, modelDisplayName})
Selects a canned template (Moderation by default), picks an LLM model,
overrides the output variable-mapping to "output.output" so the
judge LLM sees the bare string instead of the @opik.track-wrapped
JSON node {"output": ""}. -
fillAndSubmitCreateRuleDialogPythonEquals({name, referenceValue})
Toggles to Code metric, injects a deterministic Equals snippet with
the rule's UI-form name interpolated as the metric's ScoreResult name
(so feedback_scores[].name === rule.name on the trace), then
overrides the output variable-mapping path same as the LLM-judge
flow. Notably the snippet does NOT import opik's heuristic Equals at
module scope — the python_evaluator backend's get_metric_class picks
the alphabetically-first BaseMetric subclass, which would silently
shadow the user's class with the imported one.
Falls back to the empty-state "Create your first rule" button when the
project has no rules yet (the toolbar "Create rule" button is hidden
in that state).- [OPIK-6597] [QA] feat: Online Evaluation smoke (LLM-judge + Python rule, 5 traces each)
The first eventually-consistent CUJ test in the 2.0 suite. Exercises the
dynamic evaluation engine — rule created via UI, trace lands, engine
asynchronously scores it, score appears as a feedback_score on the trace.Two test cases, both 5 traces, polling in parallel for engine fan-out:
Test A (LLM-as-judge — the headline journey):
- ensureModelAvailable() provisions Anthropic or OpenAI from env;
skips when neither is set - creates a Moderation-template rule via the UI dialog
- seeds 5 traces: 3 safe outputs + 2 unsafe (illegal-activity prompts
that the Moderation rubric reliably flags) - polls all 5 for the Moderation feedback_score (90s/trace, parallel)
- asserts engine fan-out (5/5 scored), bimodal verdict
(safe ≤ 0.3, unsafe > 0.3), and class separation
(max(safe) < min(unsafe)) — robust to LLM variance - UI spot-check: opens 1 safe + 1 unsafe trace, asserts the score
renders in the Feedback scores tab of the trace panel
Test B (Python Equals — deterministic anchor):
- creates a Python-code rule via the UI dialog using a hand-built
inline equality snippet (see the POM for the no-import rationale) - seeds 5 traces: 3 matching + 2 non-matching outputs
- polls all 5 for the rule's score (60s/trace)
- asserts exact deterministic values (3 × 1.0, 2 × 0.0)
- UI spot-check: opens 1 matching + 1 non-matching trace, asserts
exact values in the Feedback scores table
Each test deletes its rule before the project fixture's teardown because
project deletion does NOT cascade to automation rules (verified during
Phase 2 staging scratch).Tags: @t1-smoke @online-evaluation
下载附件