-
[OPIK-7196] [BE] Page-key enrichment aggregate CTEs in span search (#7325)
frostbyte_neo 发布于
2026-07-06 08:41:51 +00:00 - [OPIK-7196] [BE] Page-key enrichment aggregate CTEs in span search
The span list query keyed its enrichment aggregate CTEs (feedback scores,
comments) on the full filtered span-id set, then LEFT JOINed them to the
single page of rows. For large projects this scanned the whole project's
feedback scores / comments only to discard everything outside the page, and
because ClickHouse inlines CTE references the id set was re-evaluated at each
of its 3 reference sites.When no filter or sort reads those aggregates (the common enrichment-only
case), key them on the page ids instead, consumed via
IN (SELECT arrayJoin((SELECT groupArray(id) FROM page_ids))): the inner
scalar subquery is evaluated once and cached for the whole query, and the
materialized constant array is primary-key-prunable. This turns the
aggregate scans into page-sized index lookups. The existing prefilter path
is kept for the filtered/sorted cases where aggregates drive page selection.Follows the same approach as OPIK-7193 for the trace list query.
Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
- [OPIK-7196] [BE] Extract shared aggregate-keying helper in SpanDAO
Review feedback: find and findSpanStream duplicated the
page_keyed_aggregates / span_id_prefilter decision tree; factor it into
addAggregateKeyingFlags so the gating stays in sync.Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
- [OPIK-7196] [BE] Make span_id_prefilter reachable for feedback-score-sorted queries
Review feedback: after introducing page_keyed_aggregates, the
span_id_prefilter else-branch was unreachable (its guard required
!hasFeedbackScoreFilters && !sortHasFeedbackScores, but the branch is only
entered when one of those holds). Drop the redundant !sortHasFeedbackScores
guard: page-keying stays disabled for feedback-score sorts (page selection
reads feedback_scores_agg), while the prefilter — a superset of any page —
now narrows the feedback-score and comment scans for sorted+filtered
queries instead of falling back to whole-project scans.Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
- [OPIK-7196] [BE] Plan-shape regression test for page-keyed aggregates
Review feedback: functional tests cover result correctness, but the goal
of page_keyed_aggregates is a plan property that degrades silently —
results stay correct while aggregate CTEs regress to whole-project scans.
Add a container test that renders the real SELECT_BY_PROJECT_ID template
in page-keyed mode against the real migrated schema (small span set as
pagination source, 300k feedback scores as the aggregate population) and
asserts from system.query_log that reads stay page-sized (primary-key
pruning applied) and the page-id scalar subquery is served from the cache
at the remaining reference sites (evaluated once, not per reference).Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
- [OPIK-7196] [BE] Harden plan-shape test + black-box coverage for prefilter branch
Review feedback, two items:
- Plan-shape test: assert the page-keyed predicate count (all 3 aggregate
CTE sites) instead of a bare contains(), and seed comments and
authored_feedback_scores alongside feedback_scores so every aggregate
path is exercised by the read_rows / scalar-cache assertions. - Black-box test for the newly reachable span_id_prefilter branch:
feedback-score sort combined with a narrowing filter returns a correctly
sorted, fully enriched page (FindSpansResourceTest, both directions).
Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
- [OPIK-7196] [BE] Use untilAsserted for query_log wait in plan-shape test
Review feedback: polling with a null-return predicate ends in a generic
Awaitility timeout; untilAsserted reports the query-specific assertion
context when the query_log row is missing.Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
- [OPIK-7196] [BE] Assert page-keyed predicate per CTE block
Mirrors the trace-side review feedback: a global countMatches passes if
one CTE drops the predicate while another duplicates it. Assert the
expected site count inside each aggregate CTE block, keeping the global
total as a backstop.Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
- [OPIK-7196] [BE] Remove plan-shape test per review
Review agreement: the team standard is high-value black-box tests through
public APIs; the low-level template-rendering test with synthetic data is
not maintainable. The new prefilter branch keeps its black-box coverage
(FindSpansResourceTest feedback-score sort + narrowing filter); the
plan/cost property is validated operationally. Also reverts the
@VisibleForTesting visibility widening of SELECT_BY_PROJECT_ID.Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
Co-authored-by: Claude Opus 4.8 (1M context) noreply@anthropic.com
下载附件