-
[OPIK-4942] [BE] Option E — Separate assertion_results table (#5716)
frostbyte_neo 发布于
2026-03-19 17:19:13 +00:00 - [OPIK-4942] [BE] POC: Separate assertion_results table (Option E)
Demonstrates the architecture for storing assertion results in a dedicated
ClickHouse table instead of piggybacking on feedback_scores with
category_name='suite_assertion'.Changes:
- New assertion_results ClickHouse table (migration 000070)
- AssertionResultDAO for writing assertion data to the new table
- FeedbackScoreDAO splits writes: assertions -> assertion_results, regular -> feedback_scores
- ExperimentItemDAO STREAM query adds assertion_results_per_trace CTE
- ExperimentItemMapper passes assertions_array to enrichWithAssertions
- AssertionResultMapper reads from dedicated column instead of partitioning feedback scores
Not included in this POC (would be needed for production):
- DatasetItemDAO/DatasetItemVersionDAO assertion CTE changes
- ExperimentAggregatesDAO pass rate aggregation from new table
- REST endpoint exclude_category_names cleanup
- Data migration for existing installations
- SDK changes
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- [OPIK-4942] [BE] Fix test compilation: inline suite_assertion constant
The SUITE_ASSERTION_CATEGORY constant was removed from AssertionResultMapper
in the Option E refactor, but the test still referenced it.Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- [OPIK-4942] [BE] Add assertion_results_per_trace CTE to compare endpoint
- Add assertion_results_per_trace CTE to DatasetItemVersionDAO (both
has_aggregated and has_raw branches) — compare endpoint was using
DatasetItemVersionDAO, not DatasetItemDAO which had the CTE - Add arp.assertions_array at tuple index 19 in both branches
- Remove group.size() <= 1 guard in AssertionResultMapper.computeRunSummaries()
so run summaries are emitted when a dataset item has 1 run per experiment - Add assertion_scores_avg Map column to experiment_aggregates (migration 000071)
- Add AssertionScoreAverage API record
Co-Authored-By: Claude Sonnet 4.6 noreply@anthropic.com
- [OPIK-4942] [BE] Revert package-lock.json — unintentional change from lint hook
Co-Authored-By: Claude Sonnet 4.6 noreply@anthropic.com
- [OPIK-4942] [BE] Fix test compilation errors
- AssertionResultMapperTest: update enrichWithAssertions calls to use
(item, jsonString) signature; rewrite tests for assertion_results
table approach (no longer reads from feedbackScores); update
computeRunSummaries_singleRun test to reflect removed group.size()<=1 guard - ExperimentsResourceTest: remove extra null arg from getFeedbackScoreNames
calls (leftover from older branch version of the method)
Co-Authored-By: Claude Sonnet 4.6 noreply@anthropic.com
- [OPIK-4942] [BE] Bump migration numbers to 071 and 072
000070 conflicts with 000070_add_project_id_to_experiments.sql from main.
Co-Authored-By: Claude Sonnet 4.6 noreply@anthropic.com
- [OPIK-4942] [BE] Use JsonUtils import in AssertionResultMapper
Co-Authored-By: Claude Sonnet 4.6 noreply@anthropic.com
- [OPIK-4942] [BE] Update test: runSummaries emitted for single-run suite experiments
Co-Authored-By: Claude Sonnet 4.6 noreply@anthropic.com
- [OPIK-4942] [BE] Remove misleading comment from runSummaries test
Co-Authored-By: Claude Sonnet 4.6 noreply@anthropic.com
- [OPIK-4942] [BE] Narrow catch to JsonProcessingException in AssertionResultMapper
Co-Authored-By: Claude Sonnet 4.6 noreply@anthropic.com
- [OPIK-4942] [BE] Add assertionScores to EXPERIMENT_IGNORED_FIELDS in test
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- [OPIK-4942] [BE] Add assertionScores assertions to PassRate tests + fix score routing
- Add .categoryName("suite_assertion") to PassRate.score() helper so
scores route to assertion_results table (required for pass rate SQL) - Fix itemThreshold test: set per-item executionPolicy in createDatasetItems
instead of applyDatasetItemChanges to avoid version-2 row-ID mismatch - Add assertionScores assertions to 4 tests: thenReturnPassRate (2/3),
multipleAssertions (scoreName1=1.0, scoreName2=0.5), passThresholdNotMet
(1/3), and itemThreshold (4/6)
Co-Authored-By: Claude Sonnet 4.6 noreply@anthropic.com
- [OPIK-4942] [BE] Address PR review comments: AssertionStatus enum, JsonUtils, SQL cleanup
- Use JsonUtils.readValue() instead of getMapper().readValue() (comment #1)
- Replace explicit CAST with tuple() in SQL for type flexibility (comments #2, #3)
- Change passed column from UInt8 to Enum8('passed'=0,'failed'=1) (comment #4)
- Add AssertionStatus enum used end-to-end from DB to API response
- Update all SQL queries using toFloat64(passed) to toFloat64(passed = 'passed')
- Add project_id filter to assertion_results query in DatasetItemVersionDAO (comment #6)
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- [OPIK-4942] [BE] Fix test compilation: use AssertionStatus enum instead of boolean assertions
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- [OPIK-4942] [BE] Add AssertionResultService + fix toJSONString tuple serialization
Move assertion score routing from FeedbackScoreDAO to service layer via
dedicated AssertionResultService. Fix assertion_results query where
toJSONString(tuple(...)) produced arrays instead of objects — use CAST
with named Tuple type so toJSONString emits JSON objects matching
AssertionResultRow record.Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- [OPIK-4942] [BE] Remove suite_assertion exclusion tests from Traces and Projects
suite_assertion scores now go to the separate assertion_results table,
so exclude_category_names filtering is no longer needed.Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- [OPIK-4942] [BE] Return boolean passed field in AssertionResult API response
Map AssertionStatus enum to boolean in AssertionResultMapper so
SDK/FE consumers receive passed: true/false instead of passed/failed.Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
Co-authored-by: Claude Opus 4.6 noreply@anthropic.com
Co-authored-by: Andres Cruz andresc@comet.com下载附件