github-featured / comet-ml--opik

发布

0 发布 8,340 Git 标签

RSS 订阅源
  • 2.1.22-5982 ff7931d138
    比较

    [OPIK-6884] [BE] perf: project/trace-id scope dataset-item + optimization enrichment scans (#7412)

    frostbyte_neo 发布于 2026-07-09 15:33:29 +00:00

    • [OPIK-6884] [BE] perf: bound dataset-item filter-gate traces scans to the experiment trace-id set

    Bucket-1 R3 residual from the OPIK-6883 full-scan audit. The DatasetItemDAO
    experiment-item/feedback filter gates scanned the whole workspace's traces
    (workspace_id only, all projects) to resolve the experiment items' trace ids.

    • Bound both gates (count and stats) by the referenced experiment trace-id set:
      AND id IN (SELECT trace_id FROM experiment_items WHERE workspace_id = :workspace_id AND experiment_id IN :experimentIds). The set is implied by the existing
      ei.trace_id = tfs.id join, so results are unchanged; it adds an id
      primary-key condition. Prod EXPLAIN (large workspace): traces read drops
      98,507 -> 1,309 granules (~75x).
    • Use traces FINAL in both gates: <experiment_item_filters> can filter
      mutable trace columns, which must be matched against the deduplicated latest
      version, not stale ReplacingMergeTree versions. FINAL is now affordable
      because it is scoped to the same ~1,309 bounded granules (same I/O as the
      prior LIMIT 1 BY, only a small bounded merge added); the now-redundant
      LIMIT 1 BY id on the count gate is dropped.

    OptimizationDAO.FIND (also flagged) was investigated and needs no change: it
    scans only the small workspace-scoped optimizations entity table (not
    traces/spans, so not a tiering concern), and project_id is not in its sorting
    key (workspace_id, dataset_id, id), so a project predicate would not prune.

    Validated by DatasetsResourceTest$FindDatasetItemsWithExperimentItems (56 tests).

    • perf(optimizations): project-scope the FIND duration/cost trace+span scans

    OptimizationDAO.FIND's experiment_durations CTE enriches optimizations with
    per-experiment duration/cost by scanning traces and spans for the experiments'
    trace ids. Both were scoped by workspace_id + (trace_)id IN only, with no
    project_id, so the (workspace_id, project_id, ...) sorting-key prefix couldn't
    prune - the flagged "no target-project pruning" (R3/row 12, OPIK-6883).

    Add AND project_id IN (SELECT DISTINCT project_id FROM traces WHERE workspace_id = :workspace_id AND id IN (SELECT trace_id FROM experiment_items_final)) to both scans. A span/trace shares its trace's
    project, so the resolved project set is a superset filter - results unchanged -
    and it restores the PK-prefix prune. Prod EXPLAIN (large workspace): spans
    4,842 -> 163 granules (~30x); traces also project-pruned.

    Validated by OptimizationsResourceTest GetOptimizerById + FindOptimizations
    (13 tests).

    • perf(dataset-items): project-scope the filter-gate traces scans

    Follow-up to the id-set bound: the DatasetItemDAO count/stats filter gates were
    bounded by id IN (experiment trace-id set) but not by project, so id (the
    3rd sort-key column) only did generic-exclusion pruning across all projects.
    Add AND project_id IN (SELECT DISTINCT project_id FROM traces WHERE workspace_id = :workspace_id AND id IN (...)) to both gates, enabling the full
    (workspace_id, project_id, id) prefix (result-preserving: the traces are in the
    resolved projects; also makes FINAL cheaper). Prod EXPLAIN: the same id set
    drops from ~1,305 to ~54 granules (~24x further).

    Validated by DatasetsResourceTest$FindDatasetItemsWithExperimentItems (56 tests).

    下载附件