-
[OPIK-7193] [BE] Page-key enrichment aggregate CTEs in trace search (#7323)
frostbyte_neo 发布于
2026-07-06 08:30:37 +00:00 - [OPIK-7193] [BE] Page-key enrichment aggregate CTEs in trace search
The trace list query keyed its enrichment aggregate CTEs (feedback scores,
spans, comments, guardrails, annotation queues, experiments) on the full
filtered trace-id set, then LEFT JOINed them to the single page of rows.
For large projects this scanned whole tables (millions of spans/scores)
only to discard everything outside the page, and because ClickHouse inlines
CTE references the id set was re-evaluated at every reference site.When no filter or sort reads those aggregates (the common enrichment-only
case), key them on the page ids instead, consumed via
IN (SELECT arrayJoin((SELECT groupArray(id) FROM page_ids))): the inner
scalar subquery is evaluated once and cached for the whole query, and the
materialized constant array is primary-key-prunable. This turns the
aggregate scans into page-sized index lookups. The existing prefilter path
is kept for the filtered/sorted cases where aggregates drive page selection.Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
- [OPIK-7193] [BE] Structured experiment-sort detection + CH upgrade note
Review feedback: sortHasExperiment now gates page-keyed aggregates, so
detect the experiment sort from the requested sorting fields
(SortableFields.EXPERIMENT_ID) instead of substring-matching the rendered
ORDER BY SQL, which was coupled to EXPERIMENT_FIELD_MAPPING's output.
Also document in the template javadoc that a ClickHouse upgrade must
re-verify the three behaviors the page-keyed mode relies on (forward CTE
resolution, scalar-subquery caching, PK pruning of the materialized
IN-set) — all verified on 24.3, 24.8, 25.3, 25.8 (upstream + Altinity),
and 26.5.Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
- [OPIK-7193] [BE] Plan-shape regression test for page-keyed aggregates
Review feedback: functional tests cover result correctness, but the goal
of page_keyed_aggregates is a plan property that degrades silently —
results stay correct while aggregate CTEs regress to whole-project scans.
Add a container test that renders the real SELECT_BY_PROJECT_ID template
in page-keyed mode against the real migrated schema (small trace set as
pagination source, 300k spans as the aggregate population) and asserts
from system.query_log that reads stay page-sized (primary-key pruning
applied) and the page-id scalar subquery is served from the cache at the
remaining reference sites (evaluated once, not per reference).Mirrors the SpanDAO plan-shape test on OPIK_7196.
Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
- [OPIK-7193] [BE] Harden trace plan-shape test per review
- Assert the page-keyed predicate count at all 9 aggregate CTE sites
instead of a bare contains(). - Seed trace-level feedback scores (legacy + authored) and comments so
those aggregate paths are exercised by the plan-shape assertions. - Switch the query_log wait to untilAsserted so a missing row reports the
query-specific assertion instead of a generic Awaitility timeout.
Mirrors the SpanDAO test hardening on OPIK_7196.
Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
- [OPIK-7193] [BE] Assert page-keyed predicate per CTE block
Review feedback: a global countMatches == 9 passes if one CTE drops the
predicate while another duplicates it. Slice the rendered query into named
CTE blocks and assert the expected site count inside each aggregate CTE,
keeping the global total as a backstop.Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
- [OPIK-7193] [BE] Extract shared aggregate-keying helper in TraceDAO
Review feedback: getTracesByProjectId and findTraceStream duplicated the
page_keyed_aggregates / trace_id_prefilter decision tree; factor it into
addAggregateKeyingFlags so the gating stays in sync. The stream passes
false for all sort flags, reproducing its previous unguarded branch.Mirrors the SpanDAO helper on OPIK_7196.
Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
- [OPIK-7193] [BE] Remove plan-shape test per review
Review agreement: the team standard is high-value black-box tests through
public APIs; the low-level template-rendering test with synthetic data is
not maintainable. Contract behavior is covered by the existing find
suites; the plan/cost property is validated operationally. Also reverts
the @VisibleForTesting visibility widening of SELECT_BY_PROJECT_ID.Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
Co-authored-by: Claude Opus 4.8 (1M context) noreply@anthropic.com
下载附件