-
[OPIK-7398] [BE] feat: create spans_local_v2 shadow table with weekly partitioning and traces-informed codecs (#7628)
frostbyte_neo 发布于
2026-08-04 08:30:58 +00:00 - [OPIK-7398] [BE] feat: create spans_local_v2 shadow table with weekly partitioning and traces-informed codecs
Creates spans_local_v2 empty next to the live spans table: weekly
PARTITION BY toMonday(id_at), microsecond timestamps, epoch/NaN
sentinels replacing the three Nullable columns, the is_deleted
ReplacingMergeTree meta-column, an id_at minmax skip index and raised
index_granularity_bytes.Every column the traces equivalent also has carries the exact codec the
traces tests pin after their three refinement passes. The spans-only
columns follow the same rules: FixedString(36) ids on ZSTD(1), Enum8 on
ZSTD(1), and small repetitive text (model, provider,
total_estimated_cost_version) on the 26.3-safe ZSTD(3). Only usage and
total_estimated_cost, which no rule covers, stay best guesses for the
spans codec benchmark to settle.No cutover, no backfill and no storage_policy here.
Counterpart of migration 000101 (OPIK_6900).
Co-Authored-By: Claude Opus 5 (1M context) noreply@anthropic.com
- [OPIK-7398] [BE] fix: correct the parent_span_id sentinel note and pin both halves of its contract
The cutover note claimed the DAO's presence checks become always-true on
a FixedString(36) parent_span_id. Verified against the table on 26.3,
that is wrong: ClickHouse trims the NUL padding when casting FixedString
to String, so LENGTH(CAST(parent_span_id AS Nullable(String))) still
reads 0 for the sentinel and the DAO predicates behave exactly as they
do on the live String column. A bare LENGTH would be 36, so the cast has
to stay.What is affected is the read path: the driver hands Java the padded form
as 36 NUL characters, which is not blank, so a !isBlank() guard lets it
through and UUID.fromString throws. Reads must go through
SentinelTranslation.emptyUuidToNull.Both facts are now pinned by a test so the cutover story can rely on
them instead of re-deriving them.Co-Authored-By: Claude Opus 5 (1M context) noreply@anthropic.com
- [OPIK-7398] [BE] fix: drop the false lossless claim from the timestamp narrowing note
created_at defaults to now64(9) and the ingest path binds nanoseconds,
so sub-microsecond digits do reach the DB: 1,998,018 of a 2M-row prod
sample carry them on created_at. The narrowing is still intended, but it
truncates rather than being a no-op, so say that instead.Co-Authored-By: Claude Opus 5 (1M context) noreply@anthropic.com
- [OPIK-7398] [BE] refactor: store id_at as DateTime64(3) to keep the millisecond the id encodes
UUIDv7ToDateTime returns DateTime64(3), so declaring id_at as DateTime
made the materialized expression a narrowing cast that dropped the
millisecond UUIDv7 carries. DateTime64(3) keeps it and matches the
function's own type, at 8 bytes instead of 4.toMonday still yields a Date, so the weekly partition expression and its
partition ids are unchanged.Co-Authored-By: Claude Opus 5 (1M context) noreply@anthropic.com
- [OPIK-7398] [BE] docs: note id_at as the exception to the microsecond timestamps
The header claimed every timestamp is DateTime64(6); id_at is
DateTime64(3) since the last change.Co-Authored-By: Claude Opus 5 (1M context) noreply@anthropic.com
- [OPIK-7398] [BE] refactor: apply review feedback on the spans_local_v2 schema and tests
Schema:
- model, provider and total_estimated_cost_version become
LowCardinality(String); a 4M-row prod sample holds 53, 14 and 2
distinct values. Their codec follows environment's ZSTD(1). - id_at drops to DateTime64(0): a weekly partition and the id-range
filters need whole seconds only. DateTime64 rather than DateTime is
about range, not precision, since DateTime wraps past 2106-02-07 and
prod holds ids dated 2199. - Add idx_spans_id_bf, a bloom filter on id, for traces parity: minmax
serves id ranges, the filter serves exact-match lookups. The minmax
index is renamed idx_spans_id_minmax to match.
SentinelTranslation gains the two UUID helpers the cutover read and
write paths will need, with coverage: emptyUuidToNullableUuid, which
folds the sentinel check into the parse, and a nullToEmptyUuid(UUID)
overload.Tests:
- Drop showCreateTableMatchesSpec, which asserted ClickHouse and
Liquibase internals rather than a contract. - getById uses asterisk_include_materialized_columns instead of
repeating the DDL's materialized column list. - randomCost varies the whole-unit part and all twelve fractional
digits, not just the trailing nine. - Cover an id past the DateTime ceiling, and id_at's second precision.
- Cover the plain CAST AS String, not only the Nullable form.
Co-Authored-By: Claude Opus 5 (1M context) noreply@anthropic.com
- [OPIK-7398] [BE] refactor: drop parent_span_id from the sort key, index it instead
The query-pattern audit settles it: parent_span_id is not a SpanField,
so it cannot be filtered through the API, and 0 of 162,172 span queries
in a 7-day prod sample carry a predicate on it, against 94,238 that
filter trace_id. With ~1.7 distinct parents per trace it barely
subdivided a trace's ~4.5 spans. trace_id stays for the same reason it
was measured: it is the filter that actually gets used.Dedup identity is unchanged, since id alone is unique per span. Dropping
it also removes a hazard: parent_span_id is mutable, and a mutable
sort-key column breaks ReplacingMergeTree dedup, which is what forces
the deliberate poison-pill CAST in SpanDAO's upsert.The children-of-a-span path moves to a bloom filter on the column, since
equality is the only useful predicate on a UUID-or-sentinel field.SpanDAO's LIMIT 1 BY tiebreakers still target the live table and are
untouched; swapping them is cutover work.Co-Authored-By: Claude Opus 5 (1M context) noreply@anthropic.com
- [OPIK-7398] [BE] docs: correct the stale id_at precision comment on the column
The header bullet was updated when id_at moved to DateTime64(0) but the
inline comment above the column still claimed DateTime64(3) kept the
millisecond.Co-Authored-By: Claude Opus 5 (1M context) noreply@anthropic.com
- [OPIK-7398] [BE] refactor: express SentinelTranslation through Optional
Every translation is the same shape — take the value, reject the absent
forms, fall back — so Optional states it directly instead of each method
spelling out its own null and sentinel checks.Signatures are unchanged, so the ~40 call sites across the DAOs are
untouched, and the existing unit tests pin the behaviour as identical.Co-Authored-By: Claude Opus 5 (1M context) noreply@anthropic.com
- [OPIK-7398] [BE] chore: renumber the spans_local_v2 migration to 000114
main has since taken 000112 (add_subcategory_to_cipx_spend_blocks) and 000113
(add_id_bloom_filter_index_to_traces), and the migration-prefix CI check rejects a duplicate
prefix. Renamed the file and its changeset id to 000114, the next free number. Liquibase
discovers this directory with includeAll and executes in filename order, so no changelog
edit is needed, and nothing has run this changeset anywhere yet.Co-Authored-By: Claude Opus 5 (1M context) noreply@anthropic.com
- [OPIK-7398] [BE] fix: partition spans_local_v2 on the honest Date32 week
Replicates the traces_local_v2 fix from OPIK_7456 (#7690), which landed
after this table was written. toMonday / toStartOfWeek return a 16-bit
Date that wraps at both ends: a far-future id (litellm mints ~2201, and
prod already holds ids dated 2199) folds forward into a plausible recent
week, and an epoch id_at — what UUIDv7ToDateTime returns for any non-v7
id, which is commoner — underflows backward to ~2149. Those rows are
legitimate customer data, so mixing them into a real week means a
per-week DROP PARTITION, retention or tiering pass touches them and real
rows together.The Date32 arithmetic computes the same Monday across the in-range
calendar, never wraps, needs no server setting, and through toYYYYMMDD
stays a UInt32 so the partition id remains a readable YYYYMMDD.
enable_extended_results_for_datetime_functions would fix toMonday
instead, but it also changes toStartOfInterval and breaks the metrics
API, so it is not an option.id_at was already DateTime64(0) here, so unlike traces this table needs
no recreate: the column type and the key expression are both right at
creation, which is the only moment either can be set.SpanDAO's toMonday(id_at) read predicates stay as they are. They prune
via the id_at minmax ClickHouse keeps for the partition-key columns,
independent of the key expression.The migration also moves to 000115, since #7690 took 000114 on main.
Co-Authored-By: Claude Opus 5 (1M context) noreply@anthropic.com
- [OPIK-7398] [BE] test: name the far-future fixture for what it is
badId read as invalid input, but the row is legitimate customer data
that merely carries a future timestamp — the whole point of the guard.Also records why the id_at assertions are not redundant with
farFutureIdBeyondDateTimeCeilingKeepsItsInstant: the expected Monday
comes from the value read back, so without anchoring that value the
partition assertion would pass just as happily if id_at and the
partition wrapped together.Co-Authored-By: Claude Opus 5 (1M context) noreply@anthropic.com
- [OPIK-7398] [BE] test: assert raw sentinels and key on the full sort key
keyColumnsOnlyRowTakesEveryDefault is a raw-column test, so it now
asserts the stored epoch and NaN directly instead of routing them
through SentinelTranslation to null. allAbsentColumnsRoundTripAsNull
already pins the outside view, so the translation assertion here was
restating that test rather than checking what the columns hold.The two read helpers also filter on trace_id, so the WHERE matches the
full (workspace_id, project_id, trace_id, id) sort key instead of
skipping a prefix column.Co-Authored-By: Claude Opus 5 (1M context) noreply@anthropic.com
Co-authored-by: Claude Opus 5 (1M context) noreply@anthropic.com
下载附件