-
[OPIK-5793] [FE][BE] feat: playground experiment UX improvements for test suites (#6361)
frostbyte_neo 发布于
2026-04-23 17:09:29 +00:00 - [OPIK-5793] [FE][BE] feat: playground experiment UX improvements for test suites
- Add test suite pass/fail status display in playground output columns
- Show assertion breakdown tooltips with pass rates per prompt
- Add winner/loser badge coloring for prompt headers based on pass rate
- Fix regression: resolve null selectedRuleIds to actual IDs for online scoring
- Add progress indicator polling for test suite experiment completion
- Extract useTestSuitePromptResults hook with tests for scoring logic
- Extract usePromptBadgeColor hook for clean badge color computation
- Remove dead stale prop from PlaygroundOutputAssertionStatus
- Add dark theme support for --click-blue CSS variable
- Return assertion_results and status in experiment item compare API
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- fix: address code review feedback
- Extract isItemScored helper, reuse in polling and per-cell scoring
- Extract usePromptResultStatus hook for shared winner/loser/default state
- Move animate-spin off SVG onto wrapper span for GPU acceleration
- Fix BE test assertions for empty evaluators list ([] vs null)
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- fix(tests): update dataset and experiment test assertions
- Set evaluators default to List.of() in DatasetItemTypeManufacturer
so PODAM-generated items match new API behavior - Fix explicit .evaluators(null) -> .evaluators(List.of()) in DatasetsResourceTest
- Add executionPolicy to ignored fields in ExperimentTestAssertions
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- fix(tests): add executionPolicy to ignored fields and fix evaluator assertions
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- fix(sdk): access assertion_results directly instead of via model_extra
The BE now returns executionPolicy in experiment item Compare responses,
which Pydantic stores in model_extra. The _extract_extra_field helper
checked model_extra first, so it looked for assertion_results there
(returning None) instead of reading the proper model field. Remove the
helper and access value.assertion_results directly, matching the pattern
already used for feedback_scores.Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- WIP: add SKIPPED status for experiment items without assertions
Backend: add RunStatus.SKIPPED, set status to SKIPPED for items with
empty assertions in enrichWithAssertions, handle SKIPPED in
computeRunSummaries. Frontend: simplify isItemScored, handle SKIPPED
in polling and display logic. Revert winner logic change. Still needs
backend fix for timing issue (items get SKIPPED before evaluation starts).Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- [OPIK-5793] [BE][FE] feat: delay experiment completion until assertions finish
BE: Add per-experiment assertion counter via Redis so test suite
experiments only complete after all assertion evaluations finish.
Pass experimentId through the scoring pipeline. Revert BE SKIPPED
status logic from AssertionResultMapper.FE: Check all experiment statuses with .every() for multi-prompt
runs. Show SKIPPED immediately for items without evaluators. Remove
dead SKIPPED status check since BE no longer returns it.Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- refactor(backend): extract assertion counter logic into AssertionCounterService
Centralizes all assertion counter operations (set, decrement, adjust,
delete, finish-on-complete) into a single service, removing duplicated
Redis counter logic from OnlineScoringBaseScorer, ExperimentItemPublisher,
ExperimentItemProcessingSubscriber, and TestSuiteAssertionSampler.Fixes race condition where failure-path decrement reaching 0 could mark
an experiment COMPLETED while the batch failure path marks it CANCELLED,
by deleting assertion counter keys before marking experiments as failed.Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- Revert "refactor(backend): extract assertion counter logic into AssertionCounterService"
This reverts commit
18b422113d.- refactor(backend): extract AssertionCounterService, scope to LLM scorer only
Extract Redis assertion counter operations into a dedicated
AssertionCounterService. Keep ExperimentService and counter logic only in
OnlineScoringLlmAsJudgeScorer — the sole scorer that processes test suite
assertion messages — instead of polluting the base class and all 6 subclasses.Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- fix(backend): finish experiment when no evaluators, extract metadata keys
- Fix stuck experiments when no assertions exist: TestSuiteAssertionSampler
now checks counter return value and finishes the experiment when it
reaches zero - Extract TestSuiteMetadataKeys shared constants to prevent contract drift
between ExperimentTracePersistence (writer) and TestSuiteAssertionSampler
(reader) - Revert formatting-only changes in 5 scorer files
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- fix(fe+be): progress bar skipped items, faster polling, fix test assertions
- Fix evaluators undefined check in progress bar (was always false for null)
- Poll assertion status immediately on step 2 start, reduce interval to 1s
- Remove invalid SKIPPED-status tests (backend never returns SKIPPED)
- Add executionPolicy to expected ExperimentItem objects in integration tests
- Refactor assertion counter: rename to TestSuiteAssertionCounterService,
scope to LLM scorer only, extract metadata keys
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- fix(backend): harden assertion sampler error handling and fix test executionPolicy assertions
Remove error-swallowing in TestSuiteAssertionSampler (fetchDatasetEvaluators/fetchItemEvaluators),
add per-trace try/catch to ensure assertion counter always decrements on failure, extract
decrementAssertionCounterForTrace helper. Fix DatasetsResourceTest executionPolicy mismatches
by adding ExecutionPolicy.DEFAULT to all expected ExperimentItem builders in Compare view tests.
Add unit tests for TestSuiteAssertionCounterService.Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- fix(test): add executionPolicy to StreamExperimentItems expected items
The stream endpoint has no @JsonView so executionPolicy is always
returned. Add ExecutionPolicy.DEFAULT to the two expected-item
builders in streamByExperimentNameAndProjectName test.Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- fix(backend): demote evaluator fetch log to DEBUG
The log inside computeIfAbsent fires once per cache miss but
can still be noisy at INFO when many datasets are involved.Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- refactor(backend): scope assertion counter Redis key by workspace with opik prefix
Add workspaceId to all TestSuiteAssertionCounterService methods and include
it in the Redis key to enforce workspace isolation. Update key prefix from
"experiment:assertion:" to "opik:experiment:assertion:{workspaceId}:{experimentId}".Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- refactor(backend): harden assertion counter service
- Use Optional chain in fetchItemEvaluators instead of null checks
- Renew TTL on decrement and adjust via withExpiryRenewal helper
- Guard decrementAndFinishIfComplete with exists check to prevent
phantom key creation on expired counters - Propagate finishExperiment errors to callers instead of swallowing
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- refactor(backend): remove unused deleteCounters and narrow catch blocks
Remove unused deleteCounters method from TestSuiteAssertionCounterService
and narrow catch(Exception) to catch(RuntimeException) in
TestSuiteAssertionSampler to avoid silently swallowing checked exceptions.Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- test(backend): add tests for TracesCreated trace partition logic
Document that TracesCreated fires only on insert (not update) so each
trace is processed exactly once. Add tests verifying incomplete traces
decrement the assertion counter immediately while complete traces
proceed to scoring.Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- refactor(backend): convert TestSuiteAssertionSampler to reactive chain
Eliminate scattered .block() calls inside helper methods and stream
lambdas by composing a proper Flux/Mono pipeline with a single .block()
at the EventBus entry point. Extract per-trace processing into a
reactive processTrace() method. Use Mono.cache() for lazy dataset
evaluator caching. Add TODO comments for finishExperiment deduplication.Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- refactor(backend): address PR review — null-safe collections, whole-object assertions, static imports
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- fix(backend): decrement assertion counter when dataset_id is missing from trace metadata
The missing-dataset-id early-return path was not decrementing the assertion
counter, which could cause experiments to hang indefinitely.Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- fix(fe): address PR review comments - DRY PassedCell, tighten types, fix column labels
- Extract resolveSkippedStatus() to DRY repeated hasEvaluators/experimentFinished logic
- Remove dead
status = undefinedno-op assignments - Drop
| stringfrom isExperimentTerminal, cast at call sites - Revert PlaygroundOutputTable column label to "Prompt" (not "Result"/"Output")
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- fix(backend,fe): null-safe assertion counter grouping and accurate skipped message
Filter out null experimentIds before grouping in setAssertionCounters to
prevent NPE for non-test-suite items. Show "Scoring failed" instead of
"No assertions defined" when evaluators exist but scoring produced no result.Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
Co-authored-by: Claude Opus 4.6 noreply@anthropic.com
下载附件