项目文件夹 工单

github-featured / comet-ml--opik

项目文件夹 工单

发布

0 发布 8,340 Git 标签

RSS 订阅源

• 1.10.19-4271 5317785e9e

  比较

  [OPIK-3511] Implement replay manager framework (#5031)

  frostbyte_neo 发布于 2026-02-20 14:37:15 +00:00

  • [OPIK-3837] Add Connection Probe and Monitor with Unit Tests

  • Introduced ConnectionProbe to evaluate server health via lightweight probes.
  • Added OpikConnectionMonitor for monitoring server connectivity, handling state transitions (e.g., disconnect, reconnect).
  • Implemented comprehensive unit tests for both modules, covering edge cases and integration scenarios.

  • [OPIK-3511] Refactor unit test names for improved clarity and consistency

  • Updated test function names in test_connection_monitor.py and test_connection_probe.py to follow consistent and descriptive naming conventions.
  • Consolidated repetitive test cases in test_connection_probe.py using parameterization to reduce redundancy.

  • [OPIK-3511] Refactor OpikConnectionMonitor to simplify state management

  • Replaced _last_beat with last_beat as a public instance variable.
  • Updated unit tests in test_connection_monitor.py to reflect the changes.
  • Improved naming consistency across ConnectionMonitor implementation and tests.

  • [OPIK-3837] Refactor ConnectionProbe error handling and update related tests

  • Consolidated httpx.ConnectError and httpx.TimeoutException into a single handler with updated error messaging for clarity.
  • Simplified test cases by reusing error variables and ensuring consistency in assertions for error messages.
  • Added log verification for unexpected exceptions in test_check_connection__unexpected_exception.

  • [OPIK-3837] Refactor message_processing module and add ReplayManager

  • Added ReplayManager for handling message lifecycle (registering, updating, replaying failed messages).
  • Introduced message_type attribute to all message classes for consistent identification.
  • Added from_db_message_dict for easier conversion of dictionary data to message objects.
  • Refactored BaseMessage serialization methods to support additional attributes.

  • [OPIK-3837] Enhance OpikConnectionMonitor connection handling and update tick docstring

  • Added _has_server_connection reset logic in reset method to ensure clean state initialization.
  • Updated tick method docstring for improved clarity around connection monitoring behavior and return values.

  • [OPIK-3837] Add comprehensive unit tests and enhance serialization for message_processing module

  • Added extensive unit tests covering serialization/deserialization for all core message types, including nested objects.
  • Improved from_db_message_dict to handle fields with init=False.
  • Enhanced message classes (AddFeedbackScoresBatchMessage, CreateTraceBatchMessage, etc.) with _deserialize and _serialize methods for batch processing.
  • Updated ReplayManager to register additional message types.

  • [OPIK-3837] Add JSON-based serialization/deserialization for messages and update tests

  • Refactored serialization logic to serialize messages to JSON strings and deserialize them back via message_serialization module.
  • Replaced AttachmentSupportingMessage with CreateAttachmentMessage for streamlined handling.
  • Updated ReplayManager to utilize serialize_message and deserialize_message functions.
  • Renamed and expanded test functions for detailed coverage of JSON-based round-trip serialization.

  • [OPIK-3837] Enhance ReplayManager with batch processing and improve error handling

  • Introduced a batch_size parameter with a default value of 1000 for efficient batch processing of messages.
  • Added _fetch_failed_messages_batched for cursor-based pagination of failed messages to avoid OOM issues.
  • Updated register_messages and replay_failed_messages to support batch operations.
  • Improved logging by including exception details for better debugging.
  • Ensured database schema creation uses IF NOT EXISTS for idempotency.

  • [OPIK-3837] Add batch_replay_delay to ReplayManager and improve batch processing

  • Introduced batch_replay_delay parameter to control delays between batch replays for better memory management.
  • Renamed ReplayManager._fetch_failed_messages_batched to fetch_failed_messages_batched for public access.
  • Fixed SQL typo in CREATE TABLE statement.
  • Added tests for ReplayManager to validate new batch replay logic, including delay handling and batch processing behavior.
  • Enhanced error handling for file cleanup in _clean_message_leftovers.

  • [OPIK-3837] Add tests for datetime_object_hook and improve datetime handling

  • Implemented unit tests for datetime_object_hook to ensure proper conversion of known datetime fields while preserving non-datetime fields.
  • Introduced DATETIME_FIELD_NAMES to restrict datetime conversions to specific fields, preventing accidental conversions of ISO-like strings.
  • Updated related deserialization logic to use the new datetime_object_hook.

  • [OPIK-3837] Add detailed docstrings to ReplayManager methods for clarity

  • Included comprehensive docstrings for all public and critical methods in ReplayManager to improve readability and maintainability.
  • Documented arguments, return values, and method functionality, ensuring clear usage guidelines for developers.

  • [OPIK-3837] Refactor file upload flow and remove FileUploadPreprocessor

  • Replaced FileUploadPreprocessor with FileUploadManager for improved attachment handling.
  • Updated OpikClient and Streamer to directly utilize FileUploadManager without the intermediary preprocessor.
  • Refactored related unit tests to mock and test FileUploadManager.
  • Simplified message processing by removing redundant file upload preprocessor logic.

  • [OPIK-3837] Updated design docs

  • [OPIK-3837] Fixed linter errors

  • [OPIK-3838] Implement integration of replay manager with Opik message processing (#5195)

  • [OPIK-3838] Refactor ReplayManager to DBManager and update dependencies

  • Renamed ReplayManager to DBManager for better alignment with its database management responsibilities.
  • Updated all references, imports, and tests to reflect the new class name.
  • Standardized usage of DBManagerStatus instead of ManagerStatus for state tracking.

  • [OPIK-3838] Add ReplayManager for handling offline message replay

  • Introduced ReplayManager to manage the replay of failed messages when the connection is restored.
  • Added ReplayCallback type definition for defining replay logic.
  • Enhanced DBManager to integrate with ReplayManager via callback handling.

  • [OPIK-3838] Add ReplayManager for handling offline message replay

  • Introduced ReplayManager to manage the replay of failed messages when the connection is restored.
  • Added ReplayCallback type definition for defining replay logic.
  • Enhanced DBManager to integrate with ReplayManager via callback handling.

  • [OPIK-3838] Refactor ReplayManager initialization and optimize tick loop

  • Updated ReplayManager to accept DBManager as a constructor parameter, improving dependency injection.
  • Refined _loop method to include sleep logic, reducing CPU usage during idle periods.
  • Enhanced DBManager to correctly handle MessageStatus when processing messages.

  • [OPIK-3895] Refactor ReplayManager for improved synchronization and reliability

  • Added new defaults for batch_size, batch_replay_delay, and tick_interval_seconds in ReplayManager.
  • Introduced threading.RLock for synchronized message replay and integrated it with DBManager.
  • Enhanced _loop method with interruptible sleep via threading.Event for clean shutdown.
  • Improved error handling and message replay logic, ensuring failed messages are properly updated.

  • [OPIK-4321] Add unit tests for ReplayManager

  • Introduced comprehensive test suite for ReplayManager covering initialization, message registration, reconciliation, and lifecycle management.
  • Validated functionality across scenarios including failed messages, connection restoration, and concurrency handling.

  • [OPIK-3838] Add db_manager property to ReplayManager and update tests

  • Introduced db_manager property for cleaner access to the database manager.
  • Refactored test cases to use the new property, improving code readability.

  • [OPIK-3838] Refine test to validate loop behavior with replay callback errors

  • Updated test name for better clarity.
  • Adjusted test to simulate connection restoration and ensure thread resilience to replay callback errors.

  • [OPIK-3838] Add failed_messages_count method to DBManager and integrate with Streamer

  • Implemented failed_messages_count in DBManager to retrieve the count of failed messages.
  • Added unit tests to verify behavior across various scenarios, including initialization, mixed statuses, and database errors.
  • Integrated ReplayManager with Streamer, enabling replay callback handling and lifecycle management.
  • Updated Streamer to close ReplayManager gracefully and handle failed message replay during shutdown.

  • [OPIK-3838] Integrate ReplayManager into Streamer and add lifecycle unit tests

  • Connected ReplayManager to Streamer for fallback processing of failed messages.
  • Added unit and integration tests to validate ReplayManager interaction with Streamer lifecycle events (init, flush, close).
  • Introduced configuration options for replay-related parameters such as batch size, delay, and tick interval.
  • Refactored constructors and fixtures to support ReplayManager injection.

  • [OPIK-3838] Add unit tests for OpikMessageProcessor with ReplayManager integration

  • Added test coverage for OpikMessageProcessor methods interacting with ReplayManager, including message registration, unregistration, and error handling.
  • Enhanced error handling in ReplayManager and DBManager to log failures and raise appropriate exceptions.
  • Updated process lifecycle for better resilience under connection failures and server errors.
  • Adjusted ReplayManager to handle failed messages gracefully, ensuring unprocessed messages remain registered for retries.

  • [OPIK-3838] Fixed typo

  • Added test coverage for OpikMessageProcessor methods interacting with ReplayManager, including message registration, unregistration, and error handling.
  • Enhanced error handling in ReplayManager and DBManager to log failures and raise appropriate exceptions.
  • Updated process lifecycle for better resilience under connection failures and server errors.
  • Adjusted ReplayManager to handle failed messages gracefully, ensuring unprocessed messages remain registered for retries.

  • [OPIK-3838] Add message ID validation in ReplayManager and update references

  • Introduced _check_message_id method in ReplayManager to validate message IDs in critical methods.
  • Updated OpikMessageProcessor to include type ignores for MyPy compliance.
  • Ensured robust error handling by verifying message ID presence at registration, unregistration, and failure points.

  • [OPIK-3838] Automatically assign message IDs in ReplayManager if missing

  • Added logic to auto-assign unique message IDs when registering messages with missing IDs.
  • Extended unit tests to confirm correct ID assignment and database registration behavior.

  • [OPIK-3838] Add offline fallback handling and refactor ReplayManager interactions

  • Introduced offline handling in OpikMessageProcessor to register messages as failed when there is no server connection.
  • Added unit tests for offline scenarios to validate message registration and ensure handlers are not invoked without connectivity.
  • Renamed db_manager to database_manager in ReplayManager for improved clarity.
  • Updated related methods and test cases to reflect the renamed property.
  • Enhanced ReplayManager to support message registration with specific statuses, including MessageStatus.failed.

  • [OPIK-3838] Add upload success and failure callbacks across file upload flow

  • Enhanced file upload system to support on_upload_success and on_upload_failed callbacks.
  • Updated related classes and methods, including BaseFileUploadManager, DBManager, and OpikMessageProcessor.
  • Introduced callback types and modified upload method signatures.
  • Updated S3 error handling to distinguish connection errors.
  • Added unit tests for callback integration and error scenarios.

  • [OPIK-3838] Add error handling and unit tests for message replay and attachment uploads

  • Enhanced OpikMessageProcessor with additional error handling for API errors, validation errors, retry errors, and generic exceptions during message processing.
  • Added robust unit tests to validate ReplayManager callbacks and message state changes across various error scenarios.
  • Introduced support for attachment upload callbacks (on_upload_success and on_upload_failed), validating their interactions with the replay lifecycle.
  • Updated file uploader methods to accept upload callbacks and modified related test cases to align with the changes.

  • [OPIK-3838] Reduced unnecessary large sleep times to reduce test execution time

  • [OPIK-3838] Refactor message registration and processing logic in OpikMessageProcessor

  • Simplified ReplayManager interaction for message registration based on server connection status.
  • Consolidated logic to ensure proper handling of CreateAttachmentMessage and other message types.
  • Updated file upload callbacks with type ignores for MyPy compliance.

  • [OPIK-3838] Remove unnecessary sleep in UUID generation for unit tests

  • Replaced time.sleep with timestamp generation using datetime.fromtimestamp to improve test execution speed.

  • [OPIK-3511] Remove default parameter values from ReplayManager constructor

  • Eliminated default values for batch_size, batch_replay_delay, and tick_interval_seconds in ReplayManager to enforce explicit configuration.

  • [OPIK-3511] Add upsert behavior for message registration in DBManager

  • Modified register_message and register_messages to use ON CONFLICT for updating existing records.
  • Enhanced unit tests to validate upsert functionality for single and batch message operations.

  • [OPIK-3511] Add ignored message types handling in OpikMessageProcessor

  • Introduced _ignored_message_types_for_replay to bypass replay registration for specific message types.
  • Added _should_ignore_replay_for_message_type helper method for streamlined type checking.
  • Updated processing logic to skip replay manager interaction for ignored types while ensuring handler execution.
  • Enhanced unit tests to verify correct behavior for ignored types under online and offline conditions.
  • Fixed typo in replay log message ("was" to "were").
  • Added explicit configuration options to ReplayManager in tests for batch_size, batch_replay_delay, and tick_interval_seconds.

  • [OPIK-3511] Ensure DBManager is properly closed after assertion in unit test

  • Added a finally block to close DBManager after the assert statement to ensure resource cleanup.

  • [OPIK-4132] Enhance thread-safety and concurrency handling in DBManager

  • Introduced _replay_mutex to prevent concurrent replay_failed_messages calls from duplicating message fetches.
  • Modified lock logic to ensure self.__lock__ is only held for critical sections, avoiding blockage of producers during replay callbacks or delays.
  • Added unit tests to verify concurrency behavior, ensuring producers remain unblocked during sleep and callbacks.
  • Improved documentation in replay_failed_messages to clarify locking strategy and concurrency guarantees.
  • Updated unit tests to validate lock behavior during multi-threaded operations.

  • [OPIK-3511] Refactor locking and synchronization in ReplayManager

  • Replaced _replay_lock with _message_id_lock for improved thread-safety during message registration.
  • Removed unnecessary locking around _replay_failed_messages to simplify synchronization and improve efficiency.
  • Updated DBManager initialization to remove dependency on shared locks.

  • [OPIK-3511] Improve message replay handling and add E2E test coverage

  • Fixed strip() logic in feedback reason handling to avoid errors when reason is None.
  • Adjusted replay log message for proper singular/plural grammar.
  • Added comprehensive E2E tests for offline fallback and failed message replay functionalities.

  • [OPIK-3511] Improve thread-safety in fetch_failed_messages_batched and fix test fixture docstrings

  • Wrapped fetch_failed_messages_batched logic with self.__lock__ to ensure proper synchronization during database operations.
  • Fixed incorrect docstrings in test fixtures to refer to DBManager instead of ReplayManager.

  • [OPIK-3511] Fixed docstring for batch_size

  • [OPIK-3511] Fixed docstring for DBManager.closed property

  • [OPIK-3511] Fixed docstring for DBManager

  • [OPIK-3511] Update e2e tests to include batching and ensure correct replay behavior

  • Refactored non_batching_opik_client to not_batching_opik_client for naming consistency.
  • Added comprehensive e2e test cases for batching mode: replaying CreateTraceBatchMessage, CreateSpansBatchMessage, feedback score batches, and simultaneous offline operations.
  • Verified attachment-related operations in both batching and non-batching modes.
  • Updated and clarified docstrings across tests.

  • [OPIK-3511] Add E2E test for CreateExperimentItemsBatchMessage replay and refactor attachment usage

  • Introduced a new E2E test to verify the successful replay of CreateExperimentItemsBatchMessage stored during offline mode.
  • Refactored Attachment references to use attachment.Attachment for consistency and improved readability.
  • Updated imports to accommodate the new test case enhancements.

  • [OPIK-3511] Added additional debug logging for DBManager for clarity

  • [OPIK-3839] Write documentation about offline fallback (#5316)

  • [OPIK-3839] Add documentation for offline fallback and message replay feature

  • Introduced new section on offline fallback and message replay in docs.yml.
  • Added detailed documentation outlining the offline fallback mechanism, supported message types, configuration options, and troubleshooting steps.

  • [OPIK-3839] Update tracing documentation to include log_threads_feedback_scores

  • Added client.log_threads_feedback_scores() mapping to AddThreadsFeedbackScoresBatchMessage in supported message types table.

  • [OPIK-3838] Added commentary to explain the need for OpikMessageProcessor._ignored_message_types_for_replay field

  • [OPIK-3838] Ensure UTC timezone is explicitly set for datetime fields in serialization tests and update JSON encoding logic

  • Updated unit tests to include tzinfo=datetime.timezone.utc for datetime fields to ensure consistent handling of timezone information.
  • Replaced MessageJSONEncoder with jsonable_encoder.encode for improved JSON serialization logic.

  • [OPIK-3838] Add tests for non-JSON-serializable types and cyclic references in message serialization

  • Introduced TestNonJsonSerializableTypesInMessageFields with cases for handling non-serializable fields like bytes, sets, tuples, and numpy arrays.
  • Verified proper serialization of datetime, custom classes, and cyclic references without crashes.
  • Updated serialize_message logic to leverage jsonable_encoder.encode for robust encoding of special field types.

  • [OPIK-3838] Ensure datetime fields in DBManager tests explicitly include UTC timezone

  • Updated test_db_manager.py to set tzinfo=datetime.timezone.utc for start_time and end_time datetime fields in unit tests.

  下载附件

  • 源代码 (ZIP)
  • 源代码 (TAR.GZ)