-
[OPIK-7705] [BE] perf: bound the existing-item lookup in insertItemsIntoVersion (#7699)
frostbyte_neo 发布于
2026-08-04 12:59:34 +00:00 - [OPIK-7705] [BE] perf: bound the existing-item lookup in insertItemsIntoVersion
Every batch written into an existing dataset version read the entire version
out of ClickHouse purely to classify incoming items as new vs. updates,
decoding data_hash, tags, and three more hashes only to discard them. The cost
was quadratic across a multi-batch upload, and it ran inside
withDatasetVersionLock, so the batches could not overlap.Add a narrow countExistingItemIds DAO method that computes the intersection
server-side, and dedupe the incoming batch by stable id before classifying.
getItemIdsAndHashes is left intact for its two callers that need the hashes.Measured on a 100k-item version with a 1000-item batch:
before 112,990 rows / 22.0 MB / 419ms
after 6,437 rows / 0.4 MB / 57ms
EXPLAIN indexes=1 confirms both dataset_item_id skip indexes from migration
000074 are used: primary key 1405 -> 73 granules, minmax 73 -> 4, bloom 4/4.Deduping by stable id also corrects a latent miscount: a batch repeating the
same id previously counted it twice as new.Co-Authored-By: Claude Opus 5 (1M context) noreply@anthropic.com
- refactor(datasets): apply review feedback on countExistingItemIds
- Use CollectionUtils.isEmpty and drop @NonNull on itemIds so a null set
yields 0 rather than failing, matching the existing empty-set handling. - Move the counted value to the end of the debug log line for grep-ability
during production debugging. - Hoist the test IdGenerator to a top-level private static final constant
and drop the duplicate declaration in ExperimentDatasetVersionLinking.
Co-Authored-By: Claude Opus 5 (1M context) noreply@anthropic.com
Co-authored-by: Claude Opus 5 (1M context) noreply@anthropic.com
下载附件