-
[OPIK-7456] [BE] fix: honest Date32 weekly partition for traces_local_v2 so far-future ids don't wrap (#7690)
frostbyte_neo 发布于
2026-08-03 14:03:39 +00:00 - [OPIK-7456] [BE] fix: partition traces_local_v2 by toYYYYMMDD(toMonday(id_at)) so far-future bad ids partition honestly
traces_local_v2 (migration 000101) declared id_at as a 32-bit DateTime (max 2106). A litellm bug
(BerriAI/litellm#31294) mints UUIDv7 ids whose embedded timestamp is ~2201; UUIDv7ToDateTime overflowed
those into a plausible recent year, so the bad rows hid in a real-looking weekly partition instead of an
obviously-bogus far-future one.Migration 000114 recreates the (empty, pre-cutover) traces_local_v2 with id_at as DateTime64(0) — second
resolution, range extended to 2299 — and PARTITION BY toYYYYMMDD(toMonday(id_at)). The paired
enable_extended_results_for_datetime_functions=1, added to every deployment's default profile, makes
toMonday return Date32 so far-future ids land in an honest YYYYMMDD partition. The toYYYYMMDD wrapper keeps
the partition id human-readable and, crucially, keeps the key type UInt32 regardless of the setting, so
toggling it never detaches existing parts (a bare Date/Date32 key would). Unchanged toMonday(id_at) read
predicates still prune.Tests (black-box against real ClickHouse): far-future id lands in an honest 2201 partition (guarding both
the DateTime64 and the setting at once), present-day weekly placement, partition pruning preserved for the
DAO predicate shapes, the detach-safe UInt32 key type, upsert partition stability, and round-trip/codec
fidelity. The cutover fingerprint normalizes id_at across the source DateTime and successor DateTime64.Docs (cutover README, load-test seeder) and config (test users.xml, docker + helm default profiles) updated
for the honest-partition behavior and the setting-at-ingest requirement.- fix(traces_local_v2): partition by an honest Date32 weekly Monday, drop the metrics-breaking setting
The toYYYYMMDD(toMonday(id_at)) + enable_extended_results_for_datetime_functions approach is not viable:
that global setting also changes toStartOfInterval's return type, which the metrics DAOs return as their
time buckets, so it broke the metrics API (366+ CI failures across ProjectMetrics/Workspaces/RecentActivity
in groups 9/15/16). Session-scoping it does not work (placement follows the server default), and it created
a mixed write/read false-negative hazard (Baz).Instead partition by toYYYYMMDD(toDate32(id_at) - toIntervalDay(toDayOfWeek(id_at, 1))): Date32 arithmetic
that computes the identical Monday as toMonday across the calendar but never wraps (Date32 reaches 2299),
needs no setting, stays a UInt32 so the partition id is a readable YYYYMMDD, and is proven independent of the
setting. Far-future-timestamp rows — legitimate customer data carrying a future UUIDv7 timestamp — land in
their own honest weekly partition, isolated from real recent weeks. id_at stays DateTime64(0), which also
makes the id_at > now() audit reliable. The ~80 toMonday(id_at) read/retention predicates are unchanged:
each is paired with an authoritative id-range and prunes via the id_at minmax, independent of the key
expression. The global setting is removed from all three deployment profiles.Tests: honest far-future isolation into a distinct 2201 partition; the week expression equals toMonday
across a week/year boundary and stays an honest far-future Monday (setting-independent; verified vs toMonday
over the full 1970-2299 range incl. leap years); pruning intact via id_at minmax; getById of a far-future
row. Cutover README + load-test seeder reframed to the honest, no-setting model.- test(traces_local_v2): exact far-future week assertion, bound weekProbe date, honest source audit (Baz review)
- weekProbe binds the date (a value) instead of interpolating it; the expr stays interpolated because it is a
SQL fragment the test supplies, not a bindable value. - The far-future week test now asserts the exact expected Monday (as YYYYMMDD) via a Java oracle, not just the
year plus Monday-alignment, so an off-by-one-week regression that still lands on a Monday in the right year
is caught (Baz: "incorrect weekly partition boundaries go undetected"). - The cutover README far-future audit derives the timestamp from id via UUIDv7ToDateTime — the source
traces.id_at is a 32-bit DateTime (000091) that wraps far-future values (a 2140 id wraps to 2004 and evaded
the old id_at > now() filter) — and counts distinct weeks with the honest partition expression, not the
wrapping toMonday (Baz: "runbook claims traces.id_at is DateTime64" + "far-future audit undercounts").
- test(traces_local_v2): far-future cutover fidelity coverage + partition-probe cleanup
- Add farFutureRowSurvivesCutoverWithMatchingFingerprint: a ~2201-id row rides the created_at-sliced
backfill and is pinned to land in its own honest ~2201 weekly partition (exact YYYYMMDD via a Java
oracle) and to leave the source/destination derived fingerprint identical — the successor's DateTime64
id_at must wrap under toDateTime to the same value the 32-bit source stores. - partitionKeyTypeFor / partitionIdFor share one Pair-returning partitionInfoFor query.
- Tighten PR javadocs, comments and docstrings: timeless phrasing, no redundant setting echoes.
- test(traces_local_v2): benchmark id_at matches production DateTime64(0); fix comment grammar
- compressionIsInvariantToWeeklyPartitioning built its synthetic table with id_at DateTime('UTC'); make it
DateTime64(0, 'UTC') (and the insert toDateTime64) so it exercises the production traces_local_v2 type the
test claims its sizes transfer to (Baz). - "not the ~2021 a 16-bit toMonday would wrap it into" -> add "that" for readability, in all three
occurrences (Partitioning + Table tests) (Baz).
- test(traces_local_v2): address review nits — epoch/non-v7 wrap, IF EMPTY drop guard, doc fixes
- Migration header: the toMonday wrap hits BOTH ends — a non-v7 id (v4/nil) materializes id_at at the epoch,
which toMonday underflows to ~2149 (commoner in real data than litellm ~2201 ids); note the four epoch-week
days where the Date32 expression is more correct, and that out-of-range ids saturate rather than wrap. - DROP: use IF EMPTY (row-exact, matching 000023/000039) instead of the max_table_size_to_drop=1 bytes proxy,
and document that IF EMPTY is a per-replica check — sufficient here, since traces_local_v2 is empty on every
replica pre-cutover; the cluster-aware live-empty check is finalize.sh's job at the irreversible swap. - honestWeekExpressionStaysHonestForFarFuture -> ...WhereToMondayWraps: add the epoch case
"1970-01-01" -> 19691229, pinning the low end of the Date32 window (reachable by ordinary non-v7 ids). - README: the stale "always toMonday(id_at)" placement note now names the honest weekly Monday.
下载附件