-
[OPIK-6901] [BE] test: buffered traces cutover deletion-bridge E2E gate + runbook (#7572)
frostbyte_neo 发布于
2026-07-27 13:23:32 +00:00 -
[OPIK-6901] [BE] test: buffered cutover deletion-bridge E2E gate + runbook
-
test(cutover): address Baz review — safe wrap default, input validation, sim fixes
Make the Distributed wrap opt-in in exchange_and_wrap.sh (default EXCHANGE-only,
--with-wrap to combine; --skip-wrap kept as an explicit alias) so a bare run
can't apply the wrap and break trace deletes before the DAO retarget. Validate
--database as a ClickHouse identifier across all drivers and the timestamp args
(--backfill-start/--cutover-start) and verify's table args, so operator inputs
interpolated into the reference SQL fail cleanly instead of altering the query.
Load-sim fixes: end() the live-traffic update so it flushes as completed traffic;
fill only the missing one of --workspace-id/--project-id; exclude in-flight pool
ids from delete-traffic refills; correct the seeding-order comment. Runbook +
README updated for the new wrap default and the replay-wall-time wording.- test(cutover): validate the resumed backfill_start anchor content
On resume, backfill.sh read the operator-owned --state-file and reused its
content as backfill_start without checking it. A corrupted or wrong file would
feed a garbage anchor forward to step 2. Validate the content is a well-formed
timestamp and fail fast otherwise. This addresses the spirit of the --state-file
finding by constraining the anchor's value rather than allow-listing its path
(the path is operator-owned; allow-listing would break legitimate absolute
paths and crosses no privilege boundary the operator lacks).- test(cutover): address second Baz review — wrap-only split-state guard + fixes
- exchange_and_wrap.sh --wrap-only now refuses when traces_local_v2 still exists
(EXCHANGE done but post-swap RENAME not): wrapping then would orphan the old
data under the wrong name. Prints the rename to finish first. - seed_history: build BAD_ID_INSTANT from a fixed date so it can't raise
ValueError at import on a Feb-29 run (2201 is not a leap year). - delete_traffic: a transient search_traces failure now backs off and retries
instead of being mistaken for "no more traces" and ending the run. - README: document the mutually-exclusive wrap flags and EXCHANGE-only default;
reword the awkward "EXCHANGEd" to "already-swapped".
- test(cutover): address third Baz review — numeric arg validation + comment fixes
- Validate numeric args that flow into the reference SQL / window arithmetic
across verify.sh, estimate.sh, delta_replay.sh and backfill.sh (--sample-mod,
--max-rows-per-insert, --probe-rows, --max-insert-block-size, week/stride/
pause/factor args) so they fail cleanly instead of altering the query. - Fix the stage-C rollback comment: the reverse-replay runs but matches zero ids
here (no deletes bridged in the window), rather than being skipped. - Correct the durationMismatches Javadoc to state the actual 1.5 us tolerance
and why (both endpoints truncate ns->us); the threshold is unchanged.
- test(cutover): self-heal leaked topology in resetTables
@BeforeEach reset assumed the prior test left
tracesa regular table. A test
that failed mid-cutover (after a swap/wrap, before its rollback) would leave
tracesa Distributed wrapper (or the successor) over dropped/renamed tables,
cascading into every later test. Reverse whichever of the two completed
non-canonical topologies (post-wrap, post-EXCHANGE) is present before truncating,
so one mid-cutover failure no longer corrupts the rest of the run.- test(cutover): make resetTables normalize from any leaked topology
The prior self-heal reversed only the two completed topologies and ran DDL
without confirming every referenced table existed, so a partial swap/wrap (only
the first of a two-statement helper) could make the reset itself throw and
cascade. Rewrite it to normalize to canonical from every reachable state —
canonical, partial/complete EXCHANGE, partial/complete wrap — using the
Nullable-end_time schema check to tell the original table from the successor,
and guarding every DDL on the tables it touches so no leaked state can throw.
Verified: all 8 tests in the class pass.- test(cutover): address Thiago review — drop-size guard, UTC bounds, log_comment, doc fixes
Substantive:
- Add SETTINGS max_table_size_to_drop = 0 to the stage-A TRUNCATE and the
finalize DROP so rollback/finalize don't throw on prod-scale (>50 GB) tables. - Force 'UTC' on the toDateTime64 window bounds (000001 INSERT + backfill.sh
reconciliation) so the window invariant is enforced, not assumed to match the
UTC-forced settled/split checks. - Log both counts in the reconciliation "OK" line; reserve "equal" for exact match.
- Add log_comment to the heavy backfill/delta INSERTs and the deletion-replay
DELETE for query_log/mutations attribution during the cutover.
Docs/comments:
- Fix the false "id skip index" claim on source traces (000088 indexes only
created_at/last_updated_at); reconcile the README. - Note the deferred --wrap-only path briefly makes traces unavailable and needs
the buffer re-raised (driver reminder + README). - Reword the Go/No-Go replay item to gate on the final-delta->EXCHANGE gap.
- Restate prereq 1 without a never-running retention cycle.
- Correct the 000001 "read in destination order" comment (no such guarantee).
- Document the workspace-scoped replay residual (retired by OPIK-7483; caught by
000005), stage-B non-atomic recovery, and stage-C brief traces absence.
Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com
- test(cutover): make stage-C rollback gapless; enforce --wrap-only maintenance gate
Stage C previously DROP-then-RENAMEd, leaving
tracesbriefly absent. Promote
the original back with an atomic EXCHANGE instead (traces is never absent on a
node), then drop the now data-less wrapper and park the shard — same end state,
gaplessly. Updated the gate test's rollbackAfterWrap() to match; 8/8 green.exchange_and_wrap.sh: the deferred --wrap-only path is a non-atomic RENAME->CREATE
that briefly makestracesunavailable against live, unbuffered ingestion. Turn
the documented precondition into an enforced gate — --wrap-only now requires
--confirm-maintenance (checked up front, before touching ClickHouse) asserting the
buffer is re-raised / ingestion quiesced / a maintenance window is in effect.True per-node gaplessness for the wrap itself (pre-created Distributed + atomic
multi-RENAME rotate) is deferred to prod-clone validation (OPIK-7404/7405).Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com
- test(cutover): make stage-B rollback atomic; clarify buffer prereq is operator-verified
Stage B was EXCHANGE + RENAME (two non-atomic statements) — a failed RENAME could
strand the successor under traces_pre_cutover_backup. Replace with a single atomic
multi-target RENAME rotate (traces -> traces_local_v2, traces_pre_cutover_backup ->
traces, the name freed by the first clause), so there is no orphan window. The
freed-name-reuse rename-rotate is validated against ClickHouse 26.3 by the gate
test (8/8 green); updated rollbackExchangeBack() to match.README prereq 6: state explicitly that the async-insert ceiling is operator-verified
(confirmed on the prod-clone/staging load test + Go/No-Go), not script-enforced —
it's a backend per-query setting the migration's own clickhouse-client session
cannot see or gate on.Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com
- test(cutover): make the Distributed wrap gapless per node
The wrap was RENAME traces->traces_local then CREATE Distributed traces, leaving
tracesabsent in between. Replace with the validated rename-rotate: pre-build the
Distributed wrapper under a temp name (traces_dist; its traces_local target need
not exist yet), then one atomic multi-target RENAME rotates the data to
traces_local and the wrapper into traces (the name freed by the first clause). So
tracestransitions MergeTree->Distributed with no absent window on a node.A cross-node ON CLUSTER propagation skew still exists, so --wrap-only keeps
requiring --confirm-maintenance (comments/README updated to reflect per-node
gapless + the remaining cross-node window). resetTables drops a leaked traces_dist
(a test dying between the CREATE and the RENAME). Gate test 8/8 green on CH 26.3.Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com
- test(cutover): stage-C rollback drops the ex-wrapper by an unambiguous temp name
Stage C did EXCHANGE traces<->backup then DROP traces_pre_cutover_backup — a DROP
of a name that a data-bearing table had just occupied, which reads as unsafe even
though ON CLUSTER DDL ordering makes it correct. Replace with a single atomic
3-way RENAME rotate (traces -> traces_dist_old, traces_pre_cutover_backup ->
traces, traces_local -> traces_local_v2), then DROP traces_dist_old — a fresh name
only the data-less Distributed wrapper ever held, so the DROP structurally cannot
hit the original data regardless of replica timing. Still gapless. resetTables also
clears a leaked traces_dist_old; gate test 8/8 green on CH 26.3.Also append --confirm-maintenance to the deferred-wrap guidance echo so the printed
command is copy-paste complete.Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com
- test(cutover): enforce wrap prerequisites, tag DDL, correct buffer-coverage claim
- exchange_and_wrap.sh: require --confirm-daos-retargeted whenever the wrap is
applied (--with-wrap or --wrap-only) — the OPIK-7455 hard prerequisite, else
DELETE FROM the Distributed traces returns 500. Fail fast, before ClickHouse. - --wrap-only now also asserts traces_pre_cutover_backup exists, so it can't wrap
into a state where stage C rollback has nothing to restore (one-way cutover). - Tag the exchange/wrap DDL (000003) and all rollback stages (000004 A/B/C +
reverse-replay) with log_comment via a leading SET (reliable for DDL), for
cutover attribution in system.query_log. - README: correct the "already covered" claim — the same-run --with-wrap path
shares the brief cross-node ON CLUSTER window; the buffer reduces but does not
eliminate INSERT exposure, and SELECTs are unbuffered.
Driver-only guards + reference-SQL tags (not exercised by the gate test's embedded
SQL); to be exercised in the post-merge local rehearsal.Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com
- test(cutover): enforce the async-insert buffer precondition on the EXCHANGE
The buffer (asyncInsertBusyTimeoutMaxMs) is the zero-loss mechanism for the
forward cutover — it holds writes across the final-delta->EXCHANGE gap so they
land on the new table; at the default, a write in that window can commit to the
old table and be lost after the swap. It was only a documented/echoed reminder.Add a fail-fast --confirm-buffer-raised gate, required on every EXCHANGE path
(default and --with-wrap; not --wrap-only, which does no EXCHANGE), matching the
--confirm-maintenance / --confirm-daos-retargeted pattern. It's an operator
assertion, not verification (the buffer is a backend per-query setting the
migration's clickhouse-client session can't read), so the Go/No-Go load-test
item remains the actual "took effect" check. README + prereq #6 + Go/No-Go and
the EXCHANGE example updated to match.Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com
- test(cutover): address Baz review — arg guards, query tags, verify skip-index, rollback restart, replay coverage
- Drivers: guard value-taking flags with ${2:?...} so a bare flag prints a clear
"requires a value" error instead of a set -u unbound/shift crash; tag every
ch() read (and estimate probe, exchange cutover_start, verify compare/drill-down)
with --log_comment for system.query_log attribution. - 000005 verify: add use_skip_indexes_if_final=1 to the compare + drill-down FINAL
queries so the created_at skip index prunes the scan on the 4 TB table. - README rollback: reverting traceColumnsNonNullable requires a rolling restart
(config is a startup snapshot via toInstance); until then the app mixes sentinel
and null semantics against the Nullable column. - Test: add workspaceScopedReplaySparesLiveCrossProjectRow — the empty-project
replay branch must spare a live cross-project reused id (guard coverage); rename
weekInstant's misleading minuteOffset param to secondOffset. 9/9 green.
Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com
- test(cutover): address JetoPistola review (clear wins) — align test SQL, fix flakes/footguns
- Gate test: replace parseDateTime64BestEffort with toDateTime64(..., N[, 'UTC'])
matching the reference SQL exactly (also drops the OPIK-5694 anti-pattern that
pollutes system.errors); derive the during-window delta instants from the CH
server clock (via a later now64(6)) instead of the JVM host clock, so host<->
container skew can't push last_updated_at/created_at below backfillStart and
flake the delta arms. - delete_traffic.py: tolerate a few consecutive empty refills (delete-mask
visibility lag) before concluding the project is drained, so a blip doesn't end
the run early. - requirements.txt: drop
opik(README installs it editable; listing it here lets
pip shadow the local unreleased SDK) and pin a clickhouse-connect floor. - verify.sh / estimate.sh: capture the clickhouse-client output before
read <<<
so set -e catches an infra failure instead of it being swallowed and mislabeled
(verify: a false MISMATCH; estimate: a false "table is empty").
Gate test 9/9 green on CH 26.3.
Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com
- test(cutover): address JetoPistola review (remaining) — replay fail-safe, tiered gate, version-check, gate-scope note
- Deletion replay + reverse-replay: guard the deletion-events subqueries with
length(deleted_id)=36 (and project_id) so a malformed bridge row is skipped
rather than aborting the whole replay on toFixedString — a >36-char id can't
match a real trace id, so skipping loses nothing. Mirrored in the gate test's
inline replay to keep it aligned with the reference SQL. - backfill.sh: require --confirm-tiered-headroom when the destination
storage_policy is tiered (multi-volume) or differs from the source — the
whole-node free-space gate can't see per-volume hot headroom (an accurate
auto-check isn't feasible), so force an explicit operator acknowledgment. - 000005 + verify.sh: add a version-collapse check — count old-table keys whose
last_updated_at has more ns-distinct values than us-distinct, surfacing where
the ns->us truncation could pick a different ReplacingMergeTree version than
source FINAL (invisible to the us fingerprint). Reported, not pass/fail. - Gate test Javadoc: document that the inline SQL reimplementation is deliberate
(an independent logic check); the drivers are the single-source reference-SQL
path, validated end-to-end by the prod-clone QA. Kept inline aligned to the ref.
Gate test 9/9 green on CH 26.3.
Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com
- docs(cutover): cross-reference the gate test from the reference SQL it mirrors
Add a one-line pointer to each reference .sql the gate reimplements inline
(000001-000004 family) noting that TracesLocalV2CutoverTest mirrors it and the
two must be kept in step. Closes the drift-awareness gap in the direction the
test's Javadoc doesn't cover — a maintainer editing the SQL now sees the link
without opening the test. 000005 is not pointed at (its fingerprint is validated
independently, not reimplemented inline). Comment-only.Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com
- test(cutover): address Baz — bad-id partition msg, liveCount FINAL, empty-old-table guard
- seed_history.py: the bad-id log said id_at ~2201, but the successor's 32-bit
DateTime wraps to ~2065 (OPIK-7456) — point operators at the ~2065 partition. - liveCount / liveCountScoped: SELECT ... FROM %s FINAL so leak/survivor assertions
count ReplacingMergeTree winners (mask-applied), aligning with 000005; 9/9 green. - verify.sh: an empty OLD_TABLE now fails if NEW_TABLE has rows (unexplained
successor data) instead of exiting 0 "nothing to verify"; both-empty still exits 0.
Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com
- test(cutover): final deletion replay before EXCHANGE; version-collapse fails REVIEW-REQUIRED
#687 — close the delete coverage hole: delta_replay.sh replays deletes only up to
when it runs, but cutover_start is captured later in exchange_and_wrap.sh, so
deletes bridged in [final delta_replay, cutover_start) were covered by neither the
forward replay nor the rollback reverse-replay and leaked live across the swap.
exchange_and_wrap.sh now takes --backfill-start (required for every EXCHANGE path)
and, right after capturing cutover_start, runs a final deletion replay — reusing
the single-source 000002 deletion-replay block (now marked so it can be extracted
without re-running the delta INSERT; writes in the gap are carried by the async
buffer). Runbook + a gate-test case added; residual is the inherent in-flight
micro-window, documented.#717 — verify.sh no longer only warns on version-collapse: a non-zero count is now
a REVIEW-REQUIRED failure (exit 3, distinct from a fidelity mismatch exit 1) unless
the operator reviews and passes --allow-version-collapse; --drill-down lists the
offending keys (new 000005 version-check-drill block).Gate test 10/10 green on CH 26.3.
Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com
- test(cutover): server-clock resurrection instant + explicit container teardown
- deleteThenResurrectSurvivesTheReplay: source the recreated row's last_updated_at
from the CH server clock (now64(6) -> Instant) instead of JVM Instant.now(), so
host<->container skew can't put it below backfillStart and make the delta miss
the resurrection path (same fix already applied to the main test). - Add an @AfterAll (PER_CLASS, non-static) that stops the ClickHouse + ZooKeeper
containers and closes the network, so the dedicated non-reused containers don't
rely solely on the Ryuk reaper across reruns / a shared JVM.
Gate test 10/10 green.
Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com
- fix(cutover): gate EXCHANGE/rollback on retention-paused; make delete_traffic best-effort
Address Baz round:
- exchange_and_wrap.sh / rollback.sh: require --confirm-retention-paused on every
EXCHANGE path and on rollback stages B/C. Retention deletes bypass the deletion
bridge, so a sweep in the cutover/rollback window leaks live across the swap
(forward) or resurrects a deleted row from the backup (reverse-replay). It is an
operator assertion — a backend setting the script can't read — so the README
Go/No-Go keeps the real "actually paused on every backend" verification. - delete_traffic.py (#851): larger newest-page refill and more generous empty-refill
tolerance so mask-visibility lag doesn't end the run early; docstring states it is a
best-effort traffic generator, not a guaranteed full-drain.
Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com
- fix(cutover): resume only on dst>=src; residual full-compare gate; test hygiene
Address Baz round (items 1-5):
- backfill.sh (#916, high): resume skips a window only when dst >= src. DIVERGENCE is
no longer a resume criterion — a partially-copied window within tolerance would be
skipped and its pre-anchor rows lost forever (delta only re-copies at/after
backfill_start). The INSERT is idempotent, so re-copying a short window is safe;
DIVERGENCE still governs the post-copy abort. - README (#666, high): the pre-EXCHANGE fidelity gate must be a full compare
(--sample-mod 1 --weeks-stride 1, no week narrowing); a single cross-project
workspace-scoped residual row can be hashed out or week-skipped by sampling. - delete_traffic.py (#890): comment referenced newest 500; now references REFILL_FETCH.
- TracesLocalV2CutoverTest (#901): wrapInDistributed() uses a text block + .formatted().
- _common.py (#922): delete the cutover-anchor trace in a finally block so discovery
leaves no extra live trace skewing seeded counts.
Gate test 10/10.
Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com
- fix(cutover): gate rollback stages B/C on --accept-post-cutover-write-loss
Address Baz #919 (high, rollback safety). Promoting the frozen pre-cutover backup
makes traces the successor accepted after cutover_start non-live. They are not
destroyed — the successor is parked as traces_local_v2 until finalize.sh, recoverable
during the soak — but the live table stops serving them. This is inherent to promoting
a point-in-time backup and is deliberately not auto-repaired (merging the successor's
post-cutover writes back would re-import the data the rollback exists to discard).- rollback.sh: stages B/C now require --accept-post-cutover-write-loss and print the
recovery pointer before the promote. Named --accept-* (not --confirm-*) because it
acknowledges an unavoidable consequence rather than a precondition the operator fixes. - README: scope the "no data loss" claim to "no data-bearing table dropped"; add the
post-cutover-write caveat with the recovery path; show the flag in the B/C examples.
Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com
- fix(cutover): server-clock resurrection stamp; scope loss claim; confirm anchor cleanup
Address Baz round:
- TracesLocalV2CutoverTest (#808): the workspace-scoped resurrection row stamped
last_updated_at from Instant.now() (JVM clock) while backfillStart uses server
now64(6); host/container skew could put it below backfillStart and flake the delta.
Now parses nowMicros() like the other two window inserts. Swept the file: this was
the last host-clock stamp feeding a server-clock comparison. Gate test 10/10. - README (#792): the opening claimed "zero deletion loss" while the cutover-tail
section documents a bounded residual micro-window; reworded to "near-zero deletion
loss" with a pointer to that section's mitigation. - _common.py (#789): delete_traces returns before ClickHouse applies the mask, so the
#922 cleanup could still race the seeder; poll for the anchor's disappearance
(bounded, best-effort) before returning.
Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com
- fix(cutover): add --reverse-replay-only restart path; guide partial-EXCHANGE rollback
Address Baz round (rollback robustness):
- #801: a stage B/C promote and its reverse-replay run as two statements; if the replay
is interrupted, the promote has already restored the canonical shape, so re-running
the stage is refused by the topology guard — stranding post-cutover deletes
unreplayed (they resurrect). This hole is symmetric across B and C. Add
--reverse-replay-only: re-applies just 000004_rollback_reverse_replay.sql against the
live traces (asserts non-Distributed MergeTree; requires --cutover-start and
--confirm-retention-paused). Idempotent, safe to re-run, closes both stages. - #798: if the forward EXCHANGE succeeded but its post-swap RENAME did not, the parked
original is still under traces_local_v2 and stage B aborts "backup not found". Stage B
now detects that state and prints the one-line RENAME to finish it (same remediation
exchange_and_wrap.sh gives), then proceeds. Not auto-completed: rollback does exactly
one thing per run. - README: document both recovery paths in the rollback section.
Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com
- fix(cutover): drop flaky replay-time gate; disclose workspace-replay residual
Address Baz round:
- TracesLocalV2CutoverTest (#444): the replay wall-time assertion (< 60s) gated a
non-correctness, environment-sensitive property (container startup, CI contention),
making it a flaky gate; correctness is already asserted (the mask is applied). Drop
the hard bound and the unused REPLAY_BUDGET_SECONDS; keep the measurement as a log. - README (#447): the workspace-scoped replay branch was called a "faithful mirror" that
"ends matching the source" with no caveat, but 000002 documents a known residual (the
(workspace_id, id) resurrection guard lets a live copy in one project shield the same
reused id's delete in another). Disclose it, note 000005 flags it (ok=0), and point at
OPIK-7483 which retires the arm.
Gate test 10/10.
Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com
- docs(cutover): fix grammar in workspace-scoped replay explanation
Address Baz #3657021887: "the source delete it replays already removed" ->
"the source deletion that it replays already removed".Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com
Co-authored-by: Claude Opus 4.8 noreply@anthropic.com
下载附件
-