-
[OPIK-5023] [BE] feat: add source column to traces and spans with filtering support (#5798)
frostbyte_neo 发布于
2026-03-25 10:55:44 +00:00 - [OPIK-5023] [BE] feat: add source column to traces and spans with filtering support
- Add TraceSource enum (SDK, EXPERIMENT, PLAYGROUND, OPTIMIZATION)
- Add source Enum8 column to ClickHouse traces and spans (migration 000074)
- Accept and store source on trace/span ingestion; defaults to null (unknown) when omitted
- ExperimentItemBulkIngestionService sets source=EXPERIMENT on synthetic traces
- OpenTelemetryService sets source=SDK on OTel-ingested traces/spans
- Add SOURCE to TraceField and SpanField filter enums (EQUAL/NOT_EQUAL support)
- Filtering by source=SDK also returns legacy rows with source='unknown'
- Integration tests: TracesResourceSourceTest and SpansResourceSourceTest (22 tests)
Implements OPIK-5023, OPIK-5024, OPIK-5029
-
fix(migration): rename migration prefix 000074 → 000075 to avoid conflict
-
chore: remove stale migration 000074 (renamed to 000075)
-
fix(test): add SOURCE to EXCLUDE_FUNCTIONS maps in SpanAssertions and TraceAssertions
Adding SpanField.SOURCE and TraceField.SOURCE to the EXCLUDE_FUNCTIONS maps
so the parameterized exclude-field tests don't NPE when iterating over all
enum values.- fix: bind :source param in partialInsert for traces and spans, fix filter aliasing, and improve source tests
- TraceDAO.partialInsert: bindNull("source") so INSERT_UPDATE query can resolve :source
- SpanDAO.partialInsert: same fix for PARTIAL_INSERT query
- FilterQueryBuilder: use dbField alias instead of SOURCE_DB constant for SDK filter special case
- TracesResourceSourceTest / SpansResourceSourceTest: use full-object comparison pattern
- Migration 000075: add rationale comments above index statements
-
feat(source): add source field to SpanUpdate and TraceUpdate, wire UPDATE SQL path
-
Fix source
-
[OPIK-5023] [BE] fix: add @JsonCreator to TraceSource.fromString and switch source indexes to bloom_filter(0.01)
-
[OPIK-5023] [BE] refactor: use TraceSource constants, remove redundant null guards, merge source tests
- Fix TraceSource.java: move UNKNOWN_VALUE constant after enum values (valid Java)
- FilterQueryBuilder: replace local SOURCE_SDK_VALUE/SOURCE_UNKNOWN_VALUE with TraceSource.SDK.getValue() and TraceSource.UNKNOWN_VALUE
- SpanDAO/TraceDAO: remove if(:source IS NULL, 'unknown', :source) wrappers — ClickHouse DEFAULT 'unknown' handles null binds
- Merge SpansResourceSourceTest and TracesResourceSourceTest nested classes into SpansResourceTest and TracesResourceTest respectively, delete standalone source test files
-
fix: use imported HttpHeaders instead of fully qualified name in SpansResourceTest
-
refactor(api): rename TraceSource enum to Source for shared use across traces and spans
-
refactor(source): move legacy-unknown fallback rule from FilterQueryBuilder into Source enum
Source.legacyFallbackDbValue() now owns the domain rule that sdk equality
filters should also match unknown rows. FilterQueryBuilder delegates to it,
keeping value-specific knowledge out of the generic query builder.-
chore: add trailing blank line to migration 000075
-
refactor(filter): push legacy-unknown fallback into Field interface, make FilterQueryBuilder generic
Move the source-field OR expansion out of FilterQueryBuilder into a default
method on the Field interface. TraceField.SOURCE and SpanField.SOURCE override
it to delegate to Source.legacyFallbackDbValue(). The query builder calls
field.legacyFallbackDbValue() without any field-specific checks, and picks
AND vs OR combinator based on the operator (EQUAL → OR, NOT_EQUAL → AND NOT).This makes the pattern extensible: new entities only override the interface
method, and new operators are handled in one place.- fix(migration): replace bloom_filter with set(0) for source index
source has only 5 distinct values; bloom_filter returns 'maybe' for
almost every granule on such low-cardinality columns, so no granules
are skipped. set(0) stores the exact set of values per granule and
enables precise skipping on equality filters.下载附件