-
[OPIK-4842] [BE] Refactor retention jobs to Quartz, extract estimation (#5898)
frostbyte_neo 发布于
2026-03-27 15:52:48 +00:00 - Refactor retention from Managed+Flux.interval to Quartz jobs
Split the single RetentionPolicyJob (Managed pattern) into two
independent Quartz jobs:- RetentionSlidingWindowJob: regular sliding-window cycle, runs every
(24*60)/executionsPerDay minutes (default 30min), fraction-based
workspace sharding - RetentionCatchUpJob: progressive historical deletion, runs on its
own schedule (default 60min), separate distributed lock
Both follow the established Quartz pattern (Job + InterruptableJob,
@DisallowConcurrentExecution, bestEffortLock) and are registered via
OpikGuiceyLifecycleEventListener.Benefits:
- Separate locks: catch-up never blocks regular retention
- Independent schedules: catch-up can run less frequently
- Either can be disabled independently
- Follows codebase conventions (TraceThreadsClosingJob pattern)
-
Change catch-up interval default from 60 to 45 minutes
-
Extract velocity estimation into background job
Move velocity estimation from the synchronous HTTP rule creation
endpoint into a dedicated RetentionEstimationJob (3rd Quartz job):- RetentionEstimationJob: runs every 5 min (configurable), finds
rules with catchUpDone=false and no velocity, estimates velocity- cursor for each, updates the rule in MySQL
- RetentionRuleService.create(): no longer calls ClickHouse during
rule creation. Saves rule with velocity=null, cursor=null,
catchUpDone=false. The estimation job picks it up within minutes. - RetentionEstimationService: extracted from RetentionRuleServiceImpl,
contains estimateVelocity, scoutFirstDataCursor, isTooManyRowsException
This fixes the blocking HTTP thread concern raised in PR #5820 review:
the scouting loop could make up to ~18 sequential ClickHouse queries
for huge workspaces, now it runs in a background job instead.- Use holdUntilExpiry=true to prevent redundant runs across instances
After rebase onto main, use the new bestEffortLock overload with
holdUntilExpiry=true on all three retention jobs. The lock is held
for the full interval (30min/5min/45min) so that with N instances,
only one execution happens per interval, not N sequential ones.- Address PR review: fix catch-up toggle edge case, demote DAO logs to DEBUG
- Remove catchUp.enabled check from rule creation so rules are always
marked for catch-up when applyToPast=true, surviving temporary job disablement - Convert findUnestimatedCatchUpRules SQL to text block
- Demote SpanDAO.estimateVelocityForRetention log to DEBUG
- Demote TraceDAO.scoutFirstDayWithData log to DEBUG
- Address review: remove fake graceful shutdown, unused lockTimeoutSeconds, fix reactor thread blocking
- Remove InterruptableJob, AtomicBoolean interrupted, and @DisallowConcurrentExecution
from all 3 retention jobs. Concurrency is guarded by Redis lock (holdUntilExpiry),
not Quartz. doJob() returns immediately via subscribe(), freeing the Quartz thread. - Remove unused lockTimeoutSeconds from RetentionConfig and CatchUpConfig.
- Add subscribeOn(boundedElastic) to estimation job to avoid blocking a reactor thread
(estimatePendingRules calls .block() on DAO chains internally). - Retention deletes and estimation are idempotent — incomplete work during shutdown
is safely retried on the next cycle.
- Address review feedback: jobs, DAO, config, and index improvements
- Add @DisallowConcurrentExecution, InterruptableJob, .block() to all 3 jobs
- Add LIMIT 10 to findUnestimatedCatchUpRules to bound work per cycle
- Move isTooManyRowsException to RetentionUtils
- Rename CatchUpConfig.getInterval() to getCatchUpInterval() for clarity
- Expand idx_catch_up_pending to cover all equality + ORDER BY columns
- Standardize lock-not-acquired logs to DEBUG across all jobs
- Consolidate 3 retention setup methods into single setRetentionJobs()
- Add OpenTelemetry metrics and pre-delete counts for retention observability
- Add run counter and duration histogram to all 3 retention jobs
- Add domain-specific counters to services (workspaces processed, rules
processed/completed, rules estimated, velocity values) - Add lightweight pre-delete row counts (SELECT count) in SpanDAO/TraceDAO
for observability — upper-bound ceiling with >99% precision, excludes
expensive experiment_items exclusion subquery - Bump migration from 000061 to 000062 (collision with workspace rule
project index migration on main)
- Add ORDER BY to findUnestimatedCatchUpRules and document count-before-delete ordering
- FIFO ordering (ORDER BY created_at ASC) prevents a consistently failing
rule from starving other pending rules - Document why counts run sequentially before deletes (not in parallel):
metric must reflect what's about to be removed, cost is minimal via
ClickHouse primary key index
- Improve pre-delete count comments: document sequential execution rationale
Sequential counts avoid overloading ClickHouse and the collected metrics
help assess query cost over time.- Add log_comment SETTINGS to COUNT_FOR_RETENTION queries
Without the SETTINGS clause in the SQL template, the log_comment
placeholder from getSTWithLogComment was never injected into the
query, making these counts invisible in ClickHouse query logs.- Drop catch_up_velocity from index — B-tree range break makes it dead weight
catch_up_velocity is always a range predicate (<, >=, BETWEEN) in finder
queries, so MySQL's B-tree can't use columns after it for ORDER BY. The
new index (catch_up_done, enabled, apply_to_past, catch_up_cursor)
satisfies both filter and sort from a single index scan. Velocity
filtering happens post-index with negligible cost on a small table.下载附件