-
[OPIK-4581] [OPIK-4582] [BE] Add assertion_results and status to compare endpoint (#5634)
frostbyte_neo 发布于
2026-03-16 11:13:57 +00:00 - [OPIK-4581] [OPIK-4582] [BE] Add assertion_results and status to compare endpoint
Split suite_assertion feedback scores into a dedicated assertion_results
array on ExperimentItem. Compute per-item passed/failed status and
multi-run passedRuns/totalRuns for the compare and stream endpoints.- New AssertionResult DTO (value, passed, reason)
- New AssertionResultMapper for splitting and status computation
- Add execution_policy to SQL tuples in DatasetItemDAO, DatasetVersionDAO,
ExperimentAggregatesDAO, and ExperimentItemDAO STREAM query - Update DatasetItemResultMapper and ExperimentItemMapper to enrich items
- Unit tests (8) and integration tests (3) for stream endpoint
Implements OPIK-4581 and OPIK-4582
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- [OPIK-4581] [OPIK-4582] [BE] Use passThreshold for item-level status, hide executionPolicy from response
- enrichWithMultiRunStatus now uses executionPolicy.passThreshold to
determine item-level status (passedRuns >= passThreshold) - Mark executionPolicy as @JsonIgnore to avoid sending it to FE on
every experiment item (only needed internally for threshold computation) - Add unit tests for threshold met/not-met scenarios
- Fix DatasetsResourceTest to ignore executionPolicy in assertions
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- [OPIK-4581] [OPIK-4582] [BE] Fix ExperimentAggregatesDAO: placeholder for execution_policy
experiment_item_aggregates table doesn't have execution_policy column yet.
Use empty string placeholder to keep tuple index alignment.
OPIK-4666 will add the column to this table.Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- [OPIK-4581] [OPIK-4582] [BE] Option B: Move run summaries to DatasetItem-level aggregate
Instead of stamping passedRuns/totalRuns on every ExperimentItem (wasteful
for multi-run scenarios), introduce ExperimentRunSummary as a separate
map on DatasetItem keyed by experiment ID. This avoids duplicating
aggregate data across potentially thousands of run items.- New DTO: ExperimentRunSummary (passedRuns, totalRuns, status)
- DatasetItem gets experimentRunSummaries map
- ExperimentItem loses passedRuns/totalRuns fields
- Refactored AssertionResultMapper.computeRunSummaries returns map
- Deduped parseExecutionPolicy to ExecutionPolicyMapper.fromJson
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- [OPIK-4581] [OPIK-4582] [BE] Fix ClickHouse alias collision in ExperimentAggregatesDAO
Remove AS alias from execution_policy placeholder inside groupArray tuple
to avoid conflicting with di.execution_policy in the outer SELECT scope.Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- [OPIK-4581] [OPIK-4582] [BE] Rename experimentRunSummaries to runSummariesByExperiment
Improves clarity: the field on DatasetItem contains run summaries
keyed by experiment, not experiment-level summaries.Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- [OPIK-4581] [OPIK-4582] [BE] Update diagram: rename to runSummariesByExperiment
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- [OPIK-4581] [OPIK-4582] [BE] Remove accidentally committed diagram from tracking
The diagrams/ directory is gitignored — diagram was force-added by mistake.
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- [OPIK-4581] [SDK] Extract assertion_results from compare endpoint in ExperimentItemContent
The backend now splits suite_assertion feedback scores into a separate
assertion_results field. Since the Fern SDK isn't regenerated yet, extract
assertion_results from the Pydantic model's extra fields. Update the E2E
test verifier to count both feedback_scores and assertion_results.Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- [OPIK-4581] [SDK] Fix formatting for SDK linter
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- [OPIK-4581] [SDK] Fix E2E tests to use assertion_results instead of feedback_scores
Two tests directly accessed feedback_scores for suite assertions, which
are now returned in the assertion_results field by the backend.Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- [OPIK-4581] [SDK] Add bounds checks before indexing assertion_results
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- [OPIK-4581] [SDK] Apply ruff formatting to test files
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- [OPIK-4581] [BE] Add compare endpoint tests for assertion_results, status, and runSummaries
Add 6 integration tests in DatasetsResourceTest covering:
- Suite experiment assertion_results and status in compare response
- Failing assertions produce status=failed
- Suite vs regular experiment mix (only suite has assertionResults)
- Multi-run with passThreshold met → runSummariesByExperiment passed
- Multi-run with passThreshold not met → runSummariesByExperiment failed
- Two suite experiments with independent status per item
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- [OPIK-4503] [BE] Add execution_policy to experiment_item_aggregates table
Add execution_policy column to experiment_item_aggregates via migration 000067,
wire it through the aggregation pipeline (INSERT, SELECT, mapper, binding),
and replace the OPIK-4666 placeholder in ExperimentAggregatesDAO.Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- [OPIK-4503] [BE] Address PR review: type safety, null checks, and test improvements
- Add @NonNull annotations to AssertionResultMapper utility methods
- Use CollectionUtils.isEmpty/isNotEmpty for null-safe collection checks
- Replace String status with RunStatus enum across API and tests
- Extract magic tuple index 18 to named EXECUTION_POLICY_INDEX constant
- Add StringUtils.isNotBlank filter for execution_policy binding
- Use Podam factory for FeedbackScoreBatchItem in test helpers
- Add structural ID assertions in compare endpoint tests
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- [OPIK-4503] [BE] Add missing RunStatus enum file
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- [OPIK-4503] [BE] Remove @NonNull from computeRunSummaries to allow null input
The method already handles null via CollectionUtils.isEmpty(), but @NonNull
causes NPE before the null check when callers pass null experimentItems.Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
Co-authored-by: Claude Opus 4.6 noreply@anthropic.com
下载附件