-
[OPIK-4891] [BE] Add data retention policy enforcement (#5647)
frostbyte_neo 发布于
2026-03-23 16:36:04 +00:00 - [OPIK-4891] [BE] Data retention policy enforcement job
- Add retention rule CRUD endpoints and scheduled enforcement job
- Delete only traces and spans (children first); feedback scores and
comments are left as lightweight orphans (~3% of storage) - 3-day sliding window [cutoff-3d, cutoff) for incremental processing
- Experiment exclusion: traces/spans linked to experiments are protected
via NOT IN subquery with allow_nondeterministic_mutations=1 - Cutoff normalized to start-of-day UTC using InstantToUUIDMapper
- Two delete patterns: applyToPast=true (simple IN) and applyToPast=false
(per-workspace OR conditions with max(minId, cutoff-3d)) - UUID v7 range partitioning splits workspace space across N fractions/day
- Distributed locking via LockService for multi-instance safety
- lightweight_deletes_sync=1 ensures mutations complete before returning
-
Make sliding window days configurable via retention.slidingWindowDays
-
Bump migration to 000059, fix slidingWindowDays @Min(1)
-
Fix test failures and address PR review comments
- Fix RetentionRulesResourceTest: applyToPast default is now true
- Fix RetentionPolicyServiceTest: remove feedback_scores/comments
assertions (not part of retention deletion), add Awaitility waits
for ClickHouse async write consistency - Make organizationLevel write-only in RetentionRule (excluded from
read responses since it's only used on create) - Wrap log placeholders in single quotes per codebase convention
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
Co-authored-by: Claude Opus 4.6 noreply@anthropic.com
下载附件