-
[OPIK-4503] [BE] Option B: Denormalize execution_policy to experiment_items for pass_rate (#5606)
frostbyte_neo 发布于
2026-03-11 20:53:05 +00:00 - [OPIK-4503] [BE] WIP: Add pass_rate, passed_count, total_count to Experiment API response
Compute pass rate at query time for evaluation_suite experiments using CTEs.
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- [OPIK-4503] [BE] Use execution_policy thresholds in pass_rate CTE
Pre-fetch suite-level execution_policy from MySQL dataset_versions
and pass as bind parameter arrays to ClickHouse CTEs. Threshold
resolution: item-level > suite-level > default(1).- Add getDatasetVersionIds() + fetchSuiteLevelThresholds() to
pre-fetch pass_threshold from dataset versions - Update pass_rate CTEs (FIND + FIND_GROUPS_AGGREGATIONS) to join
dataset_item_versions for item-level thresholds and use arrayElement
for suite-level thresholds - Fix ClickHouse UInt16 type mismatch and column alias issues
- Add 2 integration tests for suite-level pass_threshold behavior
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- [OPIK-4503] [BE] Fix ExperimentAggregatesDAO constructor for new fields
Add missing passRate, passedCount, totalCount arguments to the
Experiment record constructor in ExperimentAggregatesDAO.Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- [OPIK-4503] [BE] Fix bindSuiteThresholds called on FIND_GROUPS query
The FIND_GROUPS query doesn't have :suite_thresholds / :suite_version_ids
bind parameters, but executeQueryWithTargetProjects was unconditionally
calling bindSuiteThresholds for all queries including FIND_GROUPS. This
caused ClickHouse errors when trying to bind non-existent parameters.Fix: pass null for the default overload and only bind when non-null.
Null means "not applicable" (FIND_GROUPS), while empty map means
"applicable but no eval suite experiments" (FIND_GROUPS_AGGREGATIONS).Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- [OPIK-4503] [BE] Address PR review feedback: fix passedCountSum, simplify SQL casts, add DISTINCT
- Fix passedCountSum returning null when 0 items passed (use totalTotalCount > 0 as gate)
- Replace toUInt16OrDefault(JSON_VALUE(...)) with JSONExtractUInt for cleaner JSON extraction
- Remove unnecessary toUInt16 wrapper from arrayElement (Int32 from bind params is sufficient)
- Add DISTINCT to dataset_version_id subquery in IN clause for both FIND and FIND_GROUPS_AGGREGATIONS
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- [OPIK-4503] [BE] Fix pass_rate_agg CTE to return null instead of 1.0 for 0-items case
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- [OPIK-4503] [BE] Extract experiments_eval_suite CTE to consolidate repeated evaluation_method filters
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- [OPIK-4503] [BE] Fix missing PercentageValues import after cherry-pick
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- [OPIK-4503] [BE] Fix missing IntStream import after cherry-pick
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- [OPIK-4503] [BE] Address PR review: feature toggle, CTE optimization, code cleanup
- Add evaluationSuiteStatsEnabled feature toggle (OFF by default in prod, ON in tests)
- Gate all fetchSuiteLevelThresholds calls behind toggle via getSuiteThresholds helper
- Add threshold-aware overloads for getByIds/get(stream) in ExperimentDAO
- Wire thresholds through finishExperiments and get(stream) in ExperimentService
- Collapse 5 pass_rate CTEs to 3 to reduce ClickHouse CTE inlining overhead
- Remove duplicate getCostValue/getDuration/convertToBigDecimal from ExperimentDAO, reuse ExperimentGroupMappers
- Refactor bindSuiteThresholds to use Map.forEach with ArrayList builders
- Add unit tests for toggle on/off behavior in ExperimentServiceTest
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- [OPIK-4503] [BE] Fix ClickHouse CTE column aliasing for pass_rate_runs
Add explicit AS aliases for table-prefixed columns in pass_rate_runs CTE
to prevent ClickHouse from losing column names during CTE inlining.Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- [OPIK-4503] [BE] Add item-level pass_threshold integration test
Test that item-level execution_policy.pass_threshold overrides the
suite default: itemA with threshold=3 fails (2<3), itemB with
default threshold=1 passes (2>=1), resulting in passRate=0.5.Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- [OPIK-4503] [BE] Option C: Store execution_policy on experiments table in ClickHouse
Replace MySQL threshold lookups with experiments.execution_policy column in CH.
At experiment creation, resolve execution_policy from DatasetVersion and persist
it on the experiments table. Pass_rate CTEs now use JSONExtractUInt on the stored
policy instead of array bind params from MySQL. Remove feature toggle, MySQL
threshold plumbing, and related test code.Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- [OPIK-4503] [BE] Address PR review: CTE fixes and query optimization
- Add execution_policy to FIND_GROUPS_AGGREGATIONS experiments_final SELECT
- Add execution_policy to UPDATE query column list to prevent loss on ReplacingMergeTree re-insert
- Add dataset_item_id to experiment_items_final CTE, eliminating redundant subquery in pass_rate_runs
- Apply same fixes to both FIND and FIND_GROUPS_AGGREGATIONS query copies
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- [OPIK-4503] [BE] Option B: Denormalize execution_policy to experiment_items at write time
Store execution_policy on experiment_items at creation time, resolving
from dataset_item_version > experiment > DEFAULT. This eliminates the
dataset_item_versions LEFT JOIN from pass_rate CTEs, making the read
path self-contained (experiments + experiment_items only).Write path: populateExecutionPolicy() in ExperimentItemService resolves
the 3-tier policy and persists it per experiment_item row.Read path: pass_rate_runs CTE reads execution_policy directly from
experiment_items instead of joining dataset_item_versions, reducing
table accesses from 4 to 2.Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- [OPIK-4503] [BE] Address PR review: workspace scoping and code dedup
- Add workspace_id filter to DatasetItemVersionDAO.getExecutionPoliciesByRowIds
- Add workspace_id filter to ExperimentDAO.getExecutionPoliciesByIds using makeFluxContextAware
- Replace duplicate serializeExecutionPolicy in ExperimentService with ExecutionPolicy.serialize()
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- [OPIK-4503] [BE] Fix item-level policy lookup and dedup JSON deserialization
- Fix: query dataset_item_versions by dataset_item_id (stable ID) instead of
id (row-level UUID) — fixes item-level pass_threshold not being resolved - Add ExecutionPolicy.fromJson() static helper and use it in both DAOs to
eliminate duplicated JSON deserialization logic
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- [OPIK-4503] [BE] Version-specific execution_policy lookup and method rename
Item-level policy lookup now filters by the experiment's dataset_version_id,
matching Option C's query-time behavior. This prevents a race condition where
a later dataset version could change an item's execution_policy after the
experiment was created.Also renames getExecutionPoliciesByRowIds -> getExecutionPoliciesByDatasetItemIds
since the method queries by stable dataset_item_id, not row-level UUID.Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- [OPIK-4503] [BE] Key item-level policies by (versionId, datasetItemId) to prevent collision
DAO now returns Map<versionId, Map<datasetItemId, ExecutionPolicy>> so the
caller picks the correct version's policy per item. Prevents silent overwrite
when a batch spans multiple experiments pinned to different dataset versions.Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- [OPIK-4503] [BE] Address PR review: extract mapper, DTO contracts, DB-level precision
- Move serialize/fromJson off ExecutionPolicy DTO into ExecutionPolicyMapper
- Replace nested Map return types with DatasetItemPolicyEntry and ExperimentPolicyInfo DTOs
- Move pass_rate precision to DB level (toDecimal64 instead of Java BigDecimal rounding)
- Extract 3-tier policy resolution into ExecutionPolicyMapper.resolvePolicy()
- Use random values and PodamFactory in ExperimentResponseBuilderTest
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- [OPIK-4503] [BE] Address PR review: whole-object test assertions for pass rate
Remove passRate, passedCount, totalCount from EXPERIMENT_IGNORED_FIELDS
and refactor PassRate integration tests to use getAndAssert() with full
Experiment comparison instead of cherry-picking individual fields.Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- [OPIK-4503] [BE] Fix PassRate tests: copy server-computed fields before getAndAssert
Fetch actual experiment first to copy duration, datasetVersionId, and
datasetVersionSummary into the expected object before whole-object
comparison via getAndAssert(). These fields are server-computed and
can't be predicted from the test setup.Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- [OPIK-4503] [BE] Address PR review: inline CTEs, extract helpers, recursive comparison
- Inline experiments_eval_suite and pass_rate_runs CTEs into pass_rate_agg subqueries
- Extract fetchItemPolicies() private method in ExperimentItemService
- Move execution policy serialization to query binding time in ExperimentService
- Replace hardcoded score names with random UUIDs in PassRate tests
- Extract score() helper to reduce test boilerplate
- Use assertJ recursive comparison in ExperimentResponseBuilderTest
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- [OPIK-4503] [BE] Address PR review: stream pattern, null-safe Mono, duration fix
- DatasetItemVersionDAO: Use asyncTemplate.stream() instead of
nonTransaction()+collectList()+flatMapMany() per reviewer feedback - ExperimentService: Fix Mono.fromCallable returning null (Reactor
forbids nulls) by using flatMap+Mono.empty() so switchIfEmpty works - ExperimentResponseBuilder: Return null duration when no children have
duration data instead of PercentageValues(0,0,0)
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- [OPIK-4503] [BE] Add integration test for items-with-no-scores edge case
Verifies that evaluation_suite experiments with items but no feedback
scores return pass_rate=1.0 (no assertions = not failing).Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
Co-authored-by: Claude Opus 4.6 noreply@anthropic.com
下载附件