-
[OPIK-2618] [BE] Add isolated subprocess executor with HTTP-based log streaming (#3702)
frostbyte_neo 发布于
2025-10-23 15:38:45 +00:00 -
[OPIK-2618] [BE] Java Python integration queue optimization studio
-
Fix implementation
-
OPIK-2618: Address PR review comments (cleanup and docs sync)
-
OPIK-2618: Address unresolved PR comments (import, docs, javadoc)
-
OPIK-2618: Apply @andrescrz review items (docs move, TTL 14d, snake_case, fields, typeref, worker cleanup, 5s timeout)
-
OPIK-2618: Replace process_hello_world with process_optimizer_job across Python and Java tests
-
OPIK-2618: Align TTL defaults to config (no in-code defaults), rely on JsonUtils null exclusion
-
OPIK-2618: Address remaining review items (UUID v7 IDs, simplify Redis connect, rely on JsonUtils for nulls)
-
OPIK-2618: Begin splitting rq_worker into smaller modules; re-export process_optimizer_job
-
OPIK-2618: Extract MetricsWorker and NoOpDeathPenalty to opik_backend.workers; keep rq_worker re-exports
-
OPIK-2618: Use Instant for QueueMessage timestamps; honor provided Instants in RqJobUtils
-
OPIK-2618: Document immutable defaults on JobBuilder (args/kwargs)
-
OPIK-2618: Add queues.useJacksonSerialization flag and surface in config.yml
-
OPIK-2618: Introduce MapStruct RqJobMapper and use in RqJobUtils
-
OPIK-2618: Add fallback short join in RqWorkerManager.stop for forceful shutdown
-
OPIK-2618: Make Redis health_check_interval configurable via REDIS_HEALTH_CHECK_INTERVAL_SECONDS (default 60)
-
OPIK-2618: Let client decode Redis responses (decode_responses=True) and remove internal assumption
-
OPIK-2618: Use Lombok on JobStatus (@Getter, @RequiredArgsConstructor)
-
OPIK-2618: Simplify RQ worker manager - single ping at startup, no reconnect loop; delegate to client
-
OPIK-2618: Remove unused _connect_with_backoff and backoff settings; rely on client
-
OPIK-2618: Add /health/liveness and /health/readiness endpoints; Redis ping for readiness
-
OPIK-2618: Introduce shared Redis client singleton; reuse in worker manager and health endpoints
-
OPIK-2618: Delegate Redis connection to shared singleton; remove worker-managed close and startup ping logic from run loop
-
OPIK-2618: DRY RQ_WORKER_ENABLED access via is_rq_worker_enabled(); gate readiness and startup ping
-
Address comments
-
OPIK-2618: Add utils package marker to fix imports
-
OPIK-2618: Simplify RqJobMapper - rely on same-name mapping; only convert Instants
-
Fix tests
-
Fix tests
-
Fix tests
-
[OPIK-2618] Implement IsolatedSubprocessExecutor with lifecycle management and comprehensive tests
-
Revision 2: Add stack memory limit (20MB) to prevent infinite recursion and update documentation
-
Revision 3: Fix Copilot review comments - add exception variables and initialize process variable
-
Revision 4: Address additional Copilot comments - use 'raise' instead of 'raise e' and clarify process initialization comment
-
Revision 5: Address nitpick comments - shorten inline comment and extract duplicate cleanup logic to helper method
-
Revision 6: Improve readability by using newline variable instead of chr(10) in _create_wrapper_script
-
[OPIK-2618] [BE] Add isolated subprocess executor with HTTP-based log streaming
- Implement IsolatedSubprocessExecutor for creating fresh subprocesses with isolated environment variables
- Add BatchLogCollector for streaming subprocess logs to HTTP backend with gzip compression
- Add SubprocessLogConfig for centralized configuration management
- Implement 20MB stack memory limiting via RLIMIT_STACK
- Add process lifecycle management with teardown callbacks and context manager support
- Include comprehensive test suite (25 tests, 100% pass rate)
- Add complete documentation with architecture diagrams and usage examples
- Support for tenant-aware logging with optimization_id and job_id correlation
-
Revision 2: Fix BatchLogCollector shutdown with guaranteed finally block
-
Revision 3: Refactor BatchLogCollector lifecycle with SubprocessLogManager class
-
Revision 4: Simplify SubprocessLogManager usage with immediate finally cleanup
-
Revision 5: Remove SubprocessLogManager, use BatchLogCollector directly
-
Revision 6: Real-time log streaming - process logs after communicate() completes
-
Revision 7: Initialize log_collector BEFORE communicate() for proper setup timing
-
Revision 8: Restore _close_log_collector method for proper cleanup
-
Revision 9: Implement real-time log streaming with proper threading and manual pipe handling
-
Revision 10: Fix log filtering to properly handle result JSON and plain text output
-
Refactor: Move read_stream logic to BatchLogCollector.start_stream_from_process - separates log concerns from process management
-
Revision 11: Complete refactoring - use BatchLogCollector.start_stream_from_process for both logging and output collection
-
Refactor: Extract read_stream as private _read_stream method for better encapsulation
-
OPIK-2618: Implement real-time subprocess log streaming with thread-based architecture
Features:
- Real-time log streaming via background reader threads
- Memory-efficient O(1) space complexity (only keeps last line)
- Proper thread lifecycle management with timeouts
- Clean separation of concerns (BatchLogCollector handles logging)
- Thread-safe concurrent execution support
- Automatic cleanup and resource management
Implementation Details:
- Removed dead code (59+ lines, unused imports)
- Moved thread management to BatchLogCollector
- Added wait_for_reader_threads() for proper synchronization
- Improved test isolation with autouse fixture
- Fixed test flakiness with proper state clearing
Testing:
- 4 integration tests with 100% pass rate
- 5+ consecutive runs verified for consistency
- Proper test isolation prevents pollution
Memory Optimization:
- Before: O(n) memory growth with output size
- After: O(1) constant memory regardless of output
- Safe for 10+ hour long-running processes
-
Revision 6: Address Copilot PR review comments - fix misplaced comment and remove redundant code
-
Revision 6: Address all Copilot PR review comments - remove obsolete comments and fix code quality
-
[OPIK-2618] [PY-BE] Add requirements-test.txt for python backend tests
-
[OPIK-2618] [PY-BE] Remove duplicate requirements-test.txt; rely on tests/test_requirements.txt
-
[OPIK-2618] [PY-BE] Fix NameError in isolated executor latency metrics (use creation_start)
-
[OPIK-2618] [PY-BE] Support inline code execution in IsolatedSubprocessExecutor.execute and pass logging context
-
[OPIK-2618] [PY-BE] Update subprocess logging tests to use file-based execution only
-
Revision 2: Address PR #3702 review comments - refactor logging initialization, add timeout config, and thread safety
-
Revision 2: Address PR #3702 review comments - refactor logging initialization, add timeout config, and thread safety
-
Revision 2: Update documentation - per-process log collectors, concurrent execution support, and zero log loss guarantee
-
Revision 3: Fix kill_all_processes timeout distribution - use total timeout across all processes instead of per-process
-
Revision 4: Make kill_all_processes thread-safe with parallel execution using ThreadPoolExecutor
-
Revision 5: Simplify timeout logic - remove distribution since parallel execution handles it naturally
-
Revision 6: Address Copilot PR review comments - proper error logging, thread initialization, and code cleanup
-
Revision 7: Refactor subprocess_logger to use ThreadPoolExecutor for reader threads instead of storing individual thread references
-
Revision 8: Implement proper resource lifecycle management with thread-safe executor, proper exception handling, and thread pool reuse
-
Revision 9: Improve clarity - add comment explaining why futures are cleared before new stream
-
Revision 10: Add improved comment to finally block explaining per-execution log collector cleanup
下载附件
-