github-featured / comet-ml--opik

发布

0 发布 8,340 Git 标签

RSS 订阅源
  • 2.0.34-5328 4b2960f6f6
    比较

    [OPIK-6390] [BE] fix: prevent silent dataset-item loss when items_total drifts (#6678)

    frostbyte_neo 发布于 2026-05-13 13:41:33 +00:00

    • [OPIK-6390] [BE] fix: prevent silent dataset-item loss when items_total drifts

    When dataset_versions.items_total drifts below the actual ClickHouse row
    count, COPY_VERSION_ITEMS sized its :uuids array off the stale value and
    arrayElement returned empty strings for overflowing rows. The NUL-padded
    ids then collapsed under ReplacingMergeTree and items disappeared (e.g.
    v16 -> v17 dropped from 15 to 2 visible items in prod).

    • New countRowsInVersion DAO method returns a live count(DISTINCT
      dataset_item_id) with the same exclusion semantics as the COPY.
    • COPY_VERSION_ITEMS now materialises row_number() once and falls back
      to generateUUIDv7() when the pool is undersized, eliminating the
      silent NUL-id collisions. Source dedup switched to LIMIT 1 BY
      dataset_item_id for consistency with the rest of the file.
    • Six service-layer call sites switched from *.itemsTotal() to the
      live count.

    Root cause of the items_total drift itself (additive arithmetic in
    updateVersionCountsForInsert/Delete) is deferred to a follow-up.

    • Revision 2: address baz-reviewer comments on PR #6678
    • COPY_VERSION_ITEMS: row_number() now runs on the post-LIMIT-1-BY-dataset_item_id
      result via an inner deduped subquery. Before, numbering before dedup left
      sparse ranks on unmerged ReplacingMergeTree duplicates and pushed valid rows
      onto the generateUUIDv7 fallback even when the UUID pool size was correct,
      breaking the sort-order invariant.
    • DatasetItemService: extract applyEditDeleteWithLiveCount helper used by
      patchItemWithVersion, createVersionWithDeletion, and
      batchUpdateByIdsWithVersioning. Consolidates the duplicated
      countRowsInVersion -> unchangedUuids -> applyDelta wiring.
    • Revision 3: address andrescrz and BorisTkachenko comments on PR #6678
    • DatasetItemService: merge split log format string at createVersionFromDelta call site (andrescrz nit).
    • DatasetVersionResourceTest: add drift-tolerance reproducer for the createVersionWithDeletion path so the same items_total=1 corruption is now guarded against on a second fixed flow (BorisTkachenko suggestion).
    下载附件