github-featured / comet-ml--opik

发布

0 发布 8,340 Git 标签

RSS 订阅源
  • 1.10.24-4357 182b94ad4b
    比较

    [OPIK-4380] [BE] Add experiment aggregates for denormalized metrics (#5338)

    frostbyte_neo 发布于 2026-03-02 14:08:26 +00:00

    • [OPIK-4380] [BE] Add experiment aggregates for denormalized metrics
    • Add experiment_aggregates and experiment_item_aggregates tables
    • Implement ExperimentAggregatesDAO with population and query methods
    • Add ExperimentAggregatesService for aggregation management
    • Refactor DTOs into organized model classes:
      • ExperimentAggregatesModel: aggregation results
      • ExperimentEntityData: entity models
      • ExperimentSourceData: raw source data
      • ExperimentAggregatesUtils: utilities
    • Add FEEDBACK_SCORES_AGGREGATED filter strategy for map-based filtering
    • Add comprehensive integration tests (10/10 passing)
    • Configure batch size and parallelism settings
    • [OPIK-4380] [BE] Add MySQL deadlock retry mechanism for concurrent dataset operations

    Problem:

    • MySQL deadlock on dataset_version_tags composite PRIMARY KEY (workspace_id, dataset_id, tag)
    • Occurred during parallel dataset creation in same workspace
    • Multiple threads inserting "latest" tag for different datasets caused lock contention
    • Experiments with parallel execution were failing with MySQLTransactionRollbackException

    Solution:

    • Add handleOnDeadLocks() method in RetryUtils with:
      • 5 retry attempts with exponential backoff (250ms to 2s)
      • 0.5 jitter to reduce thundering herd effect
      • Recursive isDatabaseDeadlock() detection for MySQLTransactionRollbackException
    • Apply retry logic in DatasetItemService.setDatasetItemVersion()
    • Enables concurrent dataset creation for same workspace

    Impact:

    • Supports parallel experiment execution with proper deadlock handling
    • Test success rate improved from 0/10 to 10/10 in ExperimentAggregatesIntegrationTest

    • Fix visibility

    • [OPIK-4380] [BE] Address PR review comments for experiment aggregates

    Fixed 11 automated review comments from baz-reviewer:

    CRITICAL fixes:

    • Prevent NPE on null span aggregations by adding coalesce() in SQL
    • Handle multi-project experiments with LIMIT 1 in GET_PROJECT_ID
    • Handle zero-item experiments with empty aggregation helpers
    • Bind feedback_scores_percentiles map instead of empty CAST

    HIGH priority fixes:

    • Use toDecimal128(12) instead of toDecimal64(9) for cost percentiles
    • Add null-safe tags handling with Optional.ofNullable()
    • Include exception objects in retry logging for stack traces

    MEDIUM priority fixes:

    • Add missing log_comment to SELECT_EXPERIMENT_BY_ID query
    • Add missing log_comment to GET_PROJECT_ID query

    LOW priority fixes:

    • Remove duplicate "id" binding in bindItemsParameters
    • Enhance batchSize config documentation with details

    All 11 integration tests passing.

    • [OPIK-4380] [BE] Fix table definition

    • [OPIK-4380] [BE] Address PR comments and consolidate DatasetItemService methods

    • Fix tags NPE in ExperimentAggregatesDAO with defaultIfNull
    • Remove unnecessary FINAL clause from GET_EXPERIMENT_DATA query
    • Fix test naming in ExperimentAggregatesIntegrationTest
    • Consolidate 7 duplicate createVersionFromDelta methods into single canonical implementation
    • Remove debug logger from config-test.yml
    • [OPIK-4380] [BE] Fix missing log_comment and centralize search criteria binding

    • Fix SELECT_EXPERIMENT_BY_ID to properly render log_comment metadata

      • Use getSTWithLogComment pattern in getExperimentFromAggregates
      • Ensures ClickHouse query logging populates workspace/experiment IDs

    • Centralize ExperimentSearchCriteria binding logic

      • Create ExperimentSearchCriteriaBinder utility class
      • Parameterize filter strategies to support both DAO variants
      • Eliminate 29-line duplication between ExperimentDAO and ExperimentAggregatesDAO
      • Single source of truth prevents DAOs from getting out of sync
    • [OPIK-4380] [BE] Fix createVersionFromDelta consolidation after rebase

    • Update canonical method signature to include new parameters:

      • List evaluators
      • ExecutionPolicy executionPolicy
      • boolean clearExecutionPolicy

    • Update all 5 caller sites to pass new parameters:

      • Use changes.evaluators(), changes.executionPolicy(), changes.clearExecutionPolicy() when available
      • Pass null/false for auto-generated versions that inherit from base

    • Add imports for EvaluatorItem and ExecutionPolicy

    Fixes compilation errors introduced by rebase with upstream changes to DatasetVersionService

    • [OPIK-4380] [BE] Address PR review comments - fix type mismatch, extract constants, remove DAO logging
    • Fixed BigDecimal[] to Double[] conversion for experiment_scores (matches ClickHouse Float64)
    • Extracted FilterStrategy lists to static final constants to avoid repeated allocations
    • Added @NonNull validation to populateExperimentAggregate parameter
    • Removed DAO layer logging, keeping service-level logging only

    • [OPIK-4380] [BE] Address PR review comments - fix TYPE_REFERENCE visibility, redundant IN subquery, hardcoded context keys, Instant.now in loop, and inline defaultIfNull

    • Revision 4: Address remaining JetoPistola review comments (#7, #8, #10)

    • #7: Remove "Used for testing and verification" from getExperimentFromAggregates javadoc
    • #8: Replace recursive flatMap with Mono.expand() in populateExperimentItemsInBatches
    • #10: Remove unrelated subscribeOn addition from DatasetItemService.createVersionFromDelta

    • Revision 3: Add switchIfEmpty fallback for deleted traces in populateExperimentAggregate

    • Fix tests

    • Revision 6: Move countTotal log from DAO to service layer

    Operational logs belong in the service layer, not the DAO.

    • Revision 7: Apply Spotless formatting

    • Revision 8: Make populateAggregations(UUID, int) private

    Removes the uncapped public batch size entry point. All callers now go
    through the public no-arg overload which reads batchSize safely from config.

    • [OPIK-4380] [BE] Add evaluation_method support to experiment_aggregates pipeline
    • Add ClickHouse migration (000062) to add evaluation_method column to experiment_aggregates table
    • Add evaluationMethod field to ExperimentData record
    • Update GET_EXPERIMENT_DATA query to read evaluation_method from experiments
    • Update INSERT_EXPERIMENT_AGGREGATE to write evaluation_method to experiment_aggregates
    • Update SELECT_EXPERIMENT_BY_ID to read evaluation_method from experiment_aggregates
    • Fix Experiment record constructor call: insert EvaluationMethod at correct position (10)
    • [OPIK-4380] [BE] Extract shared helper for experiment aggregation queries

    Reduce copy-paste in getTraceAggregations, getSpanAggregations, and
    getFeedbackScoreAggregations by extracting queryExperimentAggregation,
    which centralises the context-aware execution, workspace/experiment/project
    parameter binding, and singleOrEmpty pattern shared by all three methods.

    • [OPIK-4380] [BE] Enforce non-null contract on countTotal criteria parameter

    Add @NonNull to ExperimentSearchCriteria in the interface and implementation
    so that a null argument fails fast with an explicit NullPointerException at
    the DAO boundary instead of crashing deep inside buildCountTemplate.

    • [OPIK-4380] [BE] Fix countTotal ignoring target project IDs in normal path

    target_project_ids was only applied inside the project_deleted LEFT JOIN
    subquery; the main WHERE had no project restriction, so counts were
    workspace-wide. Reuse has_target_projects in the main WHERE so
    project_id IN :target_project_ids always takes effect. Also replace
    manual null/empty checks with CollectionUtils.isNotEmpty.

    • [OPIK-4380] [BE] Apply Spotless formatting
    下载附件