-
[OPIK-2770] [BE] Add optimization log sync service with Redis to S3 streaming (#4539)
frostbyte_neo 发布于
2025-12-23 16:20:20 +00:00 - [OPIK-2770] [BE] Add optimization log sync service with Redis to S3 streaming
- Add OptimizationLogSyncService for syncing logs from Redis to S3
- Use Flux.interval() with Managed lifecycle (no Quartz dependency)
- Implement per-optimization distributed locking to prevent duplicates
- Gzip compress logs before S3 upload
- S3 key pattern: logs/optimization-studio/{workspace_id}/{id}.log.gz
- Use KeysScanOptions for Redis key scanning (non-deprecated API)
- Add syncConcurrency config for parallel S3 uploads
- Trigger final sync on optimization completion via finalizeLogsOnCompletion
- Reduce Redis TTL to 1 hour after completion for cleanup
-
Fix test to use getLockTimeout() instead of getLockTtlSeconds()
-
Fix: Only preserve name on upsert if incoming name is blank
The SDK sends blank names during status updates, but explicit name
updates via upsert should be honored.- Address Baz review feedback
- Fix: Update last_flush_ts in doSyncToS3AndReduceTTL to prevent repeated uploads
- Fix: Log full exception instead of just message in finalizeLogsAsync
- Fix: Log full exception instead of just message in OptimizationLogFlusherJob
- Remove unused deleteRedisKeys method
- Test: Add verification for last_flush_ts update in finalization test
- Revision 3: Address Ido's review - DRY refactor and shared META_KEY_PATTERN
下载附件