-
[OPIK-7142] [BE] feat: ingest cost intelligence into dedicated cipx tables (#7302)
frostbyte_neo 发布于
2026-07-01 14:03:11 +00:00 - [OPIK-7142] [BE] feat: ingest cost intelligence into dedicated cipx tables
Populate cipx_spend and cipx_trace_identity asynchronously off span/trace
create and update events, so AI-spend reads no longer scan the spans/traces
metadata blob. Create reuses SpansCreated/TracesCreated; update consumes the
new *CostIntelligenceChanged events (the *Updated events carry neither ids nor
metadata). cipx rows are gated and built in Java, then written via a plain
INSERT that ReplacingMergeTree merges by sorting key. start_time is the real
entity start on create and the UUIDv7-embedded time on update.Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
- test(backend): cover cipx ingestion via direct ClickHouse queries
Drive the real span/trace create and update path (API -> event bus ->
listener -> DAO) and assert against cipx_spend / cipx_trace_identity with raw
ClickHouse reads, no new production code. Covers create landing, non-cipx
siblings skipped, cipx-first-on-update inserts, and update merging to a single
row via FINAL.Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
-
refactor
-
bump migration versions
-
fix(cost-intelligence): resolve per-span trace on batch updates; rename cipx tables plural
Address PR review:
- Batch span updates reuse one SpanUpdate across spans that may belong to
different traces, so using update.traceId() for every cipx_spends row wrote
the wrong trace_id. Resolve project_id and trace_id per span from the
persisted spans (SpanDAO.getProjectAndTraceIdsBySpanIds). - Rename tables to plural: cipx_spends, cipx_trace_identities.
- SpanRow / TraceIdentityRow now use @Builder(toBuilder=true) + @NonNull.
- Correct migration comments that claimed the ReplacingMergeTree version came
from the source entity's last_updated_at; it uses the now64() default, same
as spans/traces.
Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
- fix(cost-intelligence): resolve cipx start_time from source span/trace on update
Address PR review: the update path wrote cipx_spends/cipx_trace_identities
start_time from the UUIDv7 timestamp (RetentionUtils.extractInstant), so on
FINAL the update row overwrote the create row's real start_time — wrong for
backfilled/imported entities.- SpanDAO.getSpanRefsBySpanIds now reads FROM spans FINAL and returns
SpanRef(project_id, trace_id, start_time); the listener builds each
cipx_spends row with the stored start_time. - TraceDAO.getStartTimesByTraceIds (FROM traces FINAL) resolves the stored
trace start_time off the request path; the trace listener uses it and skips
traces it can't resolve. - Correct the DAO comments that claimed the UUIDv7 time is used on update.
- Update the two update-path test assertions to expect the stored start_time.
Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
- refactor(cost-intelligence): dedup cipx ref lookups with LIMIT 1 BY id, not FINAL
Address PR review: FINAL on the per-event span/trace resolution reads pays for
a collapse scan on the ingestion hot path. Switch both getSpanRefsBySpanIds and
getStartTimesByTraceIds to the codebase-standard ORDER BY id, last_updated_at
DESC + LIMIT 1 BY id (as BULK_UPDATE and other reads do). Also add
@Builder(toBuilder=true) to SpanRef and construct it via the builder.Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
下载附件