-
[OPIK-5650] [BE/FE] feat: support evaluation suites in playground (#6092)
frostbyte_neo 发布于
2026-04-09 13:25:47 +00:00 - [OPIK-5650] [BE/FE] feat: support evaluation suites in playground
Add backend experiment execution endpoint and frontend eval suite
flow so playground can run evaluation suite datasets with server-side
assertion processing and poll-based progress tracking.Made-with: Cursor
- fix: address PR review comments for eval suite playground support
- Switch to @RequiredArgsConstructor convention in EvalSuiteAssertionSampler and ExperimentItemProcessor
- Remove SDK references from comments, rename methods (fetchDatasetEvaluators, getMetadataString, toLangChain4jMessage, etc.)
- Fix log patterns: pass exception as last param instead of e.getMessage()
- Split catch: UncheckedIOException for deserialization, Exception for other errors
- Replace generateDeterministicId with IdGenerator.generateId() (UUID v7)
- Pre-process evaluators outside trace loop via PreparedEvaluator record
- Add dataset version filtering to DatasetItemStreamRequest
- Add null validation for datasetId with BadRequestException
- Extract buildMessagesInput/buildLlmOutput helpers to deduplicate trace/span creation
- Simplify buildTemplateContext using forEach
- Add backward-compatibility comment on OnlineScoringLlmAsJudgeScorer
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- fix: batch-prefetch item evaluators to avoid N blocking DB calls per trace
Collect unique dataset item IDs upfront, fetch and prepare evaluators
once per item, then look up from a map inside the trace loop.Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- fix: use actual userName instead of hardcoded "system" in fetchItemEvaluators
Pass the userName from TracesCreated event through to the reactive
context instead of hardcoding "system".Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- fix: rename eval_suite_version_hash to eval_suite_dataset_version_hash and fix unit tests
- Rename metadata key to eval_suite_dataset_version_hash across BE, FE, and tests
- Fix ExperimentExecutionServiceTest: add datasetId to test requests to
match the null-safety validation added earlier - Update test for missing datasetId to assert BadRequestException
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- fix: eliminate N+1 item evaluator queries and update dialog text
- Add getItemEvaluatorsByDatasetId to DAO/service for single-query
batch fetch of all item evaluators in a dataset version - Refactor EvalSuiteAssertionSampler to use batch fetch instead of
per-item reactive calls - Update RunOnDatasetDialog to reflect dataset/evaluation suite choice
with dynamic button text and labels
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- fix: remove dead eval-suite code from FE-orchestrated playground path
The FE-orchestrated path (createLogPlaygroundProcessor) is only used for
regular datasets. Eval suites use the BE-orchestrated path exclusively,
so evalSuiteDatasetId, evalSuiteVersionHash, and evaluationMethod fields
on LogQueueParams were never set and the related trace metadata / experiment
blocks were unreachable.Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- fix: add missing USER_NAME to reactive context for item-level evaluators
The prefetchItemEvaluators method was missing USER_NAME in contextWrite,
causing makeFluxContextAware to throw NoSuchElementException (silently
caught), which meant item-level assertions were never calculated.Also fixes test config construction to use getJsonNodeFromString instead
of readTree to properly parse evaluator config JSON.Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- fix: address PR review comments and add eval suite polling support
- Add UUID validation for eval_suite_dataset_item_id in EvalSuiteAssertionSampler
- Fix ClickHouse dedup ordering in DatasetItemVersionDAO (filter after LIMIT 1 BY)
- Add EXPERIMENT_STATUS enum and use constants instead of string literals
- Add two-phase polling (running → evaluating) for eval suite experiments
- Extract nested ternary into helper function in RunOnDatasetDialog
- Add progress indicator with phase-aware display (running/evaluating)
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- fix: add @Builder and @NonNull to EvalSuiteAssertionSampler DTOs
Address baz reviewer comment: use Lombok @Builder(toBuilder = true) and
@NonNull on DatasetEvaluatorsResult and PreparedEvaluator records per
project DTO conventions.Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- fix: address 5 new baz reviewer comments
- Assert evaluator config (name, model, schema) in includesItemLevelEvaluators test
- Extract areAllExperimentItemsScored() helper in PlaygroundOutputAssertionStatus
- Add SETTINGS log_comment to SELECT_ITEM_EVALUATORS_BY_DATASET ClickHouse query
- Fix snake_case bind params (:dataset_id, :version_id) in DatasetItemVersionDAO
- Use backend total from response instead of only counting page 1 in polling
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- fix: add @NotNull to scoreNameMapping and datasetId, remove manual null checks
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- fix: externalize hardcoded constants to Dropwizard config
- EvalSuiteConfig: defaultModelName, defaultRunsPerItem
- ExperimentExecutionConfig: maxConcurrentItems, defaultProjectName
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- fix: refactor EvalSuiteAssertionSampler to stream API and add block timeout
- Refactor main trace loop to stream pipeline with flatMap and parseUUID helper
- Add .timeout(Duration.ofSeconds(10)) before .block() in prefetchItemEvaluators
- Refactor prepareEvaluators from for-loop to stream with filter/flatMap
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- fix: make service layer reactive, stream dataset items, remove bulk evaluator fetch
- Make createAndExecute return Mono and stream dataset items via Flux
instead of collecting into memory - Make fetchDatasetEvaluators/prefetchItemEvaluators reactive with
workspace ID carried via reactive context - Move .block() to controller boundary in ExperimentsResource
- Compose finishExperiments into single reactive chain
- Remove custom getItemEvaluatorsByDatasetId query, revert to existing
datasetItemService.get() for per-item evaluator fetch
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- fix: add PythonTemplateParser and route PromptType through scoring pipeline
Add PromptType.PYTHON support to TemplateParseUtils with a proper
PythonTemplateParser instead of converting templates at preparation
time. The prompt type flows from EvalSuiteAssertionSampler through
the Redis message to OnlineScoringEngine, which uses the appropriate
parser at render time. Existing callers are unchanged (default MUSTACHE).Also fixes test compilation for ExperimentExecutionServiceTest and
EvalSuiteAssertionSamplerTest to match updated service signatures.Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- refactor: migrate experiment execution from in-process ExecutorService to Redis Streams
Replace the in-process ExecutorService with a durable Redis Streams pipeline
for experiment item processing. This ensures jobs survive restarts, provides
backpressure via consumer batch size, and enables distributed processing
across multiple instances.- Add ExperimentItemToProcess message record with batchId for completion tracking
- Add ExperimentItemPublisher that sets an atomic counter before publishing
- Add ExperimentItemProcessingSubscriber extending BaseRedisSubscriber
- Rewrite ExperimentExecutionService to collect messages and delegate to publisher
- Extend ExperimentExecutionConfig to implement StreamConfiguration
- Refactor EvalSuiteAssertionSampler.resolveModelName with Optional + StringUtils
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- fix: use proper imports and mark experiments CANCELLED on finish failure
- Replace fully-qualified reactor.util.context.Context and IntStream with imports
- Add onErrorResume in finishExperiments to transition experiments to CANCELLED
when the completion flow throws, preventing the FE poller from deadlocking
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- refactor: split ExperimentItemProcessor into focused components
Extract message rendering and trace persistence into dedicated classes:
- ExperimentMessageRenderer: template context, message rendering, LLM request building
- ExperimentTracePersistence: trace/span/experiment-item construction and persistence
- ExperimentItemProcessor: thin orchestrator delegating to the above
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- fix: address PR review comments — DI, dead code, test quality
- Use Guice DI for MustacheParser instead of inline instantiation
- Replace hand-rolled JSON string building with JsonUtils.valueToTree
- Populate trace/span errorInfo from errorMessage parameter
- Deduplicate buildMessagesInput/buildLlmOutput calls
- Wire AbortController signals for poll functions in playground FE
- Remove self-fulfilling tests from EvalSuiteAssertionSamplerTest,
replace with integration tests through onTracesCreated production path - Add edge case tests for missing dataset_item_id and version_hash
- Update test mocks to match refactored constructors
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- fix: address FE PR review comments for playground eval suite support
- Add defensive onError toast to useRunExperimentExecution hook
- Exclude ephemeral state (progress, running, datasetType) from PlaygroundStore persistence
- Use EVALUATION_METHOD enum instead of raw string for evaluationMethod type
- Move pollingStartTimeRef initialization from render body to useEffect
- Reuse getCompareExperimentsList instead of duplicated api.get() call
- Widen getExperimentById/getCompareExperimentsList param types to avoid
as nevercasts - Extract magic numbers to constants/experiments.ts (poll intervals, max page size)
- Remove redundant clearCreatedExperiments() calls (already in resetState)
- Simplify phaseLabel fallback in PlaygroundProgressIndicator
- Add error toasts to polling catch blocks
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- fix: rename schema fields to stable assertion_N keys for LLM structured output
Schema field names (which are often full assertion descriptions like
"The response must contain at least 10 top-level sections.") were being
used as JSON property keys in the LLM structured output schema. Some LLM
calls ignored these long keys and outputted generic "Assertion 1" through
"Assertion N" names, which bypassed the scoreNameMapping.Now renameSchemaToAssertionKeys() converts schema names to assertion_1,
assertion_2, etc. before injection. The scoreNameMapping maps these stable
keys back to the original assertion text for display.Also adds tests verifying assertions don't leak between items and that
dataset-level assertions remain consistent across traces.Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- fix: address PR review comments, add PromptType.PYTHON migration, fix playground refresh bug
- Add DB migration for PromptType.PYTHON in prompt_versions ENUM column
- Add configurable fetchTimeoutSeconds to EvalSuiteConfig and config.yml
- Fix playground refresh losing datasetType by persisting it in Zustand store
- Add @RequiredPermissions to /execute endpoint
- Improve experiment failure tracking with Redis counters in ExperimentItemProcessingSubscriber
- Handle empty datasets by marking experiments COMPLETED immediately
- Improve error fallback in useRunExperimentExecution
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- fix(test): stub finishExperiments and update in EmptyDataset test
The EmptyDataset test was missing stubs for experimentService.update()
and finishExperiments() called by markExperimentsCompleted(), causing NPE.Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- fix: add --comment and trailing newline to migration per convention
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- fix(test): set explicit DatasetType.DATASET in DatasetsResourceCreateFromSpansTest
PODAM randomly assigns DatasetType, which can be EVALUATION_SUITE.
Commit33c276e6fadded filterDataForDatasetType() that unwraps input
for evaluation suite datasets, causing "input" key assertion failures
when PODAM picks EVALUATION_SUITE type.Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- fix(fe): add 5-minute timeout to poll functions in useActionButtonActions
Extract shared handlePollTimeout callback used by both
pollExperimentCompletion and pollAssertionEvaluation to prevent
indefinite polling if BE has issues or evaluator crashes.Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- [OPIK-5650] [BE] fix: address PR review comments from thiagohora
- Move promptType fallback from DTO to publishers (OnlineScoringSampler, ManualEvaluationService), make field @NotNull
- Replace @RequiredArgsConstructor with manual constructor in EvalSuiteAssertionSampler (@Config + Lombok JAR issue)
- Extract evaluator preparation chain to EvalSuiteEvaluatorMapper
- Use setRequestContext() in ExperimentsResource instead of manual context setting
- Remove WORKSPACE_NAME propagation in ExperimentItemProcessingSubscriber
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- [OPIK-5650] [BE] refactor: make experiment execution pipeline fully reactive
Address PR review comments: convert ExperimentTracePersistence,
ExperimentItemProcessor, ExperimentItemPublisher and
ExperimentItemProcessingSubscriber to reactive patterns. Remove .block()
calls from domain layer, propagate reactor context instead of passing
workspaceId/userName as parameters, use setRequestContext in resource
layer, wrap blocking ops with Schedulers.boundedElastic(), and make
DatasetItemService.getItems read workspaceId from reactive context.Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- [OPIK-5650] [BE] fix: reactor context propagation, prompt ordering, and new tests
Add contextWrite to processEvent so reactor context (workspaceId, userName)
propagates to trace/span/item creation. Fix prompt-to-experiment mapping
race condition by using collectSortedList on promptIndex instead of
collectList, ensuring correct ordering despite concurrent Flux.merge.
Include EvalSuiteEvaluatorMapper, EvalSuitePromptConstants, and new unit
tests for ExperimentMessageRenderer and PythonTemplateParser.Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- [OPIK-5650] [BE] refactor: address PR review - reduce Redis payload, per-trace evaluator resolution
- Replace full DatasetItem with datasetItemId in Redis messages to reduce memory pressure
- Fetch dataset item from DB in consumer (ExperimentItemProcessor) instead of serializing into message
- Use actual exception class name instead of hardcoded "LlmProviderError" for error info
- Extract datasetId/versionHash per-trace in EvalSuiteAssertionSampler with cache for dedup
- Wrap blocking DB calls in Mono.fromCallable + Schedulers.boundedElastic
- Add version hash to execution policy warning log
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- [OPIK-5650] [FE] fix: assertion progress count and status flickering
- Fix totalItems to multiply dataset items by experiment count
- Count individual scored experiment items instead of scored rows
- Cache last valid assertion status to prevent flickering between refetches
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- [OPIK-5650] [BE] fix: update test assertion to use OpikUserMessage instead of UserMessage
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- [OPIK-5650] [BE] fix: narrow exception catch in EvalSuiteEvaluatorMapper
Remove broad catch(Exception) that silently swallowed runtime errors.
Keep only UncheckedIOException for deserialization failures and add
evaluator config to the log for debuggability.Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- [OPIK-5650] [BE] fix: remove config from error log in EvalSuiteEvaluatorMapper
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- [OPIK-5650] [BE] refactor: introduce ScoreDestination enum for type-safe score routing
Replace implicit string-based categoryName check with a ScoreDestination enum
to make score routing (feedback_scores vs assertion_results) explicit and type-safe.Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- [OPIK-5650] [BE] refactor: address PR review - reactive patterns and code quality
- Wrap blocking LLM call in Mono.fromCallable with boundedElastic scheduler
- Remove redundant Mono.defer and subscribeOn from subscriber
- Add TTL to Redis failure counter to prevent memory leaks
- Extract PersistenceContext record to reduce parameter count
- Add projectName to ExperimentItem creation
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- [OPIK-5650] [BE] fix: let NotFoundException propagate for invalid versionHash, parallelize trace+span creation
- Distinguish user errors (invalid versionHash) from transient DB failures
in fetchDatasetExecutionPolicy - Run trace and span creation in parallel via Mono.when()
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- [OPIK-5650] [BE] refactor: derive scoreDestination from categoryName instead of storing it
Make scoreDestination() a derived method on FeedbackScoreItem that computes routing
from categoryName, eliminating the stored field. This ensures correct routing for all
entry points (JSON API, internal scorer, builder) with a single source of truth.Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
Co-authored-by: Claude Opus 4.6 noreply@anthropic.com
下载附件