-
[OPIK-6596] [QA] feat: Experiments smoke (SDK-create + UI-verify, deterministic evaluator) (#6859)
frostbyte_neo 发布于
2026-05-26 10:23:22 +00:00 - feat(bridge): add POST /experiments/evaluate route
Wraps opik.evaluation.evaluate(...) against a freshly-seeded dataset with a deterministic Equals metric. Returns per-item scores so callers can verify the 2-pass-1-fail seed shape end-to-end without polling experiment items via the NDJSON stream API.
- feat(sdk): add evaluateExperiment on PythonSdkClient
Wraps POST /experiments/evaluate; returns experimentId, datasetId, item counts, and per-item scores.
- feat(backend-client): add findExperimentByName, listExperimentsWithPrefix, deleteExperiment
Inspection-only wrappers over opik.api.experiments.findExperiments and deleteExperimentsById. Used by the experiment fixture for teardown and by the test for SDK-side score verification.
- feat(test-infra): sweep experiments before datasets in global setup/teardown
Experiments reference datasets which reference projects. Sweep order: experiments -> datasets -> projects so each delete sees a clean cascade. Mirrors the dataset-before-project ordering added in OPIK-6595.
- feat(fixtures): add experiment fixture extending dataset
Seeds its own 2-pass-1-fail dataset shape inline (does not reuse dataset.fixture's happy-path-only seed) so the smoke test can assert on both pass and fail surfaces. Per retro lesson #12: a fixture's seed shape must let the test see the failure surface of the CUJ.
- chore(e2e): gitignore .env.local and .env.cloud
The .example templates are checked in; the real-values files need to be ignored so credentials can't be committed accidentally.
- feat(pom): add ExperimentsPage and ExperimentDetailPage
Both POMs key off the comet-table data-row-id / data-cell-id composite selectors that already ship in production — no new FE testids needed. ExperimentDetailPage.readItemScore takes (datasetItemId, metricName) because the items table sorts by expected_output, not insertion order.
- test(experiments): add @t1-smoke experiments CUJ
Seeds an experiment via the bridge with a 2-pass-1-fail dataset shape, asserts (1) row appears in the list page, (2) detail page renders all items, (3) per-item scores match the bridge response keyed on datasetItemId, (4) aggregate score chip ~0.67, (5) sanity-check the seed shape. test.step boundaries make each phase failure-readable.
- refactor(bridge): tighten experiment items schema (baz-reviewer)
Replace list[dict[str, Any]] with list[ExperimentItemSeed] on ExperimentEvaluateRequest. Malformed payloads now return FastAPI's standard 422 with a precise field-level error instead of a generic 500 from a downstream KeyError. Dataset items elsewhere stay dict-typed because user datasets are intentionally schema-free; experiment seeds aren't — they require exactly these three keys.
下载附件