-
[OPIK-7400] [BE] Benchmark the spans_local_v2 per-column codecs and apply the real-data refinements (#7691)
frostbyte_neo 发布于
2026-08-04 13:07:07 +00:00 - [OPIK-7400] [BE] test: benchmark the spans_local_v2 per-column codecs
Adds SpansLocalV2BenchmarkTest, the spans counterpart of TracesLocalV2BenchmarkTest: a
codec drift-guard that pins every spans_local_v2 column, plus a side-by-side codec
bake-off on a deterministic synthetic slice shaped from a read-only measurement of the
production spans table (1.26 B rows, 19.25 TiB, ClickHouse 26.3.16).The shared columns are re-confirmed to inherit the traces codec set correctly; the work
concentrates on the columns with no traces evidence — usage, total_estimated_cost and
_version, model, provider, type, trace_id and parent_span_id.Co-Authored-By: Claude Opus 5 (1M context) noreply@anthropic.com
- [OPIK-7400] [BE] feat: apply the real-data codec refinements to spans_local_v2
Migration 000114, a metadata-only ALTER on the still-empty table, applying the five
refinements a byte-weighted sample of real production spans identified (115,925 deduped
rows, ~94% of the table's payload, measured on the deployed 26.3 LTS):- usage: ZSTD(1) -> ZSTD(3), 19.3% smaller. The one spans-only column 000112 had to
guess at; its Map keys are long repetitive dotted strings, the exact shape 26.3
regressed ZSTD level 1 on. Also decodes faster. - error_info: ZSTD(1) -> ZSTD(3), 8.0% smaller on real stack traces.
- start_time, created_at, id_at: drop Delta, 2.7% / 18.2% / 5.6% smaller. Raw
microsecond values inside a weekly partition share their high-order bytes, and
created_at is flat across 46.7% of adjacent rows under batch ingest.
end_time and last_updated_at keep Delta: they are a wash on spans, so there is no reason
to diverge from traces_local_v2 there. The benchmark test's codec pin map and its
whole-row comparison move in lockstep, and its two synthetic-vs-real divergences (T64 on
the *_length counters, and the storage effect of de-nullifying) are documented in place so
neither gets re-decided on synthetic evidence.Co-Authored-By: Claude Opus 5 (1M context) noreply@anthropic.com
- [OPIK-7400] [BE] docs: justify the timestamp codecs per ISO week, not over the whole sample
A weekly partition is what one partition of this table actually holds, and Delta's
economics are set by the adjacent-row deltas, so a comparison run over a multi-month
sample misjudges it. Re-measured per week over the 10 densest weeks of the real sample:
plain ZSTD(1) is smaller in 10/10 weeks for created_at (median 19%), 9/10 for start_time
(median 3.8%) and 8/10 for id_at (median 13%) — the three columns 000114 changes, now on
stronger evidence than the whole-sample numbers gave.end_time and last_updated_at still keep Delta, matching traces_local_v2: end_time is a
genuine wash (7/10 weeks, every margin under 8%, both signs), and while last_updated_at
does lean plain ZSTD(1) (9/10, median 5.0%), a ~5% swing on a column that is ~0.03% of the
table does not warrant diverging from a merged real-data-validated traces decision. Also
records that 000107 only ever contested those two columns, never these three.Co-Authored-By: Claude Opus 5 (1M context) noreply@anthropic.com
- [OPIK-7400] [BE] chore: renumber the codec-refinement migration to 000115
Follows the create-table migration's move to 000114 (main took 000112 and 000113). This ALTER
has to sort after the CREATE, since Liquibase discovers the directory with includeAll and
executes in filename order, so it becomes 000115. The changeset id, every reference in the
migration header, and the benchmark test's javadoc and comments move with it.Co-Authored-By: Claude Opus 5 (1M context) noreply@anthropic.com
- [OPIK-7400] [BE] docs: document error_info's codec change and make the rollback note explicit
Review found the header claimed five columns but only documented four: error_info's move to
ZSTD(3) was in the ALTER with no rationale. Added its bullet — mostly-empty column, but a
long repetitive JSON traceback when present, so it belongs with the other structured text —
and corrected "two groups" to three.Also spells out what forward-only means here rather than only asserting it: recovery is a new
forward ALTER, never a rollback or a restore, and nothing is at risk because the table is
empty and unread when this runs. Notes the asymmetry that motivates the ordering requirement —
applied after the backfill instead, both this change and any correction would be a full
re-compress mutation rather than a metadata edit.Co-Authored-By: Claude Opus 5 (1M context) noreply@anthropic.com
- [OPIK-7400] [BE] chore: renumber the codec refinements to 000116 after the DDL merged
The create-table migration landed on main as 000115 (it was 000114 on the pre-merge branch,
and main had meanwhile taken 000112 and 000113), so this ALTER moves to 000116 to keep sorting
after it — Liquibase discovers the directory with includeAll and executes in filename order.
The changeset id and every reference in the migration header and the test move with it.Also corrects the test's javadoc: spans_local_v2 no longer partitions by toMonday(id_at) but by
toYYYYMMDD(toDate32(id_at) - toIntervalDay(toDayOfWeek(id_at, 1))) since OPIK-7456, so the
borrowed partition-invariance result is now justified by what actually transfers — that
partitioning splits data into independently-compressed weekly parts, whichever expression names
the week. Notes that id_at is part of that expression, so the Delta removal here is a codec
change on a partition-key column, which the codec pin is what demonstrates ClickHouse accepts.Co-Authored-By: Claude Opus 5 (1M context) noreply@anthropic.com
- [OPIK-7400] [BE] docs: trim the codec migration header to the traces counterpart's scale
The header had grown to roughly twice 000107's length, restating evidence that belongs on the
ticket: per-week tables, the LowCardinality cardinality correction, and a paragraph each on the
rollback and IF EXISTS. Cut to what a reader of the migration needs — the three groups with
their headline numbers, why end_time and last_updated_at stay on Delta, and one line each for
IF EXISTS and the rollback. 32 lines against 000107's 31.Co-Authored-By: Claude Opus 5 (1M context) noreply@anthropic.com
Co-authored-by: Claude Opus 5 (1M context) noreply@anthropic.com
下载附件