github-featured / comet-ml--opik

发布

0 发布 8,340 Git 标签

RSS 订阅源
  • 1.8.84-3044 d9b93aafb3
    比较

    [OPIK-2590] [P SDK] Provide trace object to scoring metrics during evaluation (#3646)

    frostbyte_neo 发布于 2025-10-21 13:00:19 +00:00

    • [OPIK-2590] [P SDK] Add in-memory emulator for message processing and refactor backend emulator
    • Introduced a new in-memory emulator to handle local message processing in tests, replacing real backend interactions.
    • Refactored BackendEmulatorMessageProcessor to extend the modular EmulatorMessageProcessor for consistency and reusability.
    • Added models, local_emulator_message_processor, and emulator_message_processor to structure the new emulation logic.
    • Updated test fixtures and imports to reflect the new in-memory emulator integration.
    • [OPIK-2640] [P SDK] Remove optionality from feedback_scores and update error_info initialization
    • Updated feedback_scores in models.py to use non-optional List with default factory for consistency.
    • Refactored error_info initialization in emulator_message_processor to use explicit attribute mapping instead of dict(by_alias=True).
    • [OPIK-2650] [P SDK] Update spans and feedback_scores handling with optional initialization
    • Changed spans and feedback_scores parameters to be optional in TraceModel and SpanModel.
    • Added default empty list initialization to prevent NoneType errors.
    • Updated usage in emulator_message_processor and related tests for consistency.
    • [OPIK-2590] [P SDK] Add message field initialization to error_info in LangChain OpenAI tests
    • Included message field with None as default in error_info initialization.
    • Updated LangChain OpenAI test cases for consistency with the new field addition.
    • [OPIK-2590] [P SDK] Add unit tests for LocalEmulatorMessageProcessor
    • Added comprehensive test suite for LocalEmulatorMessageProcessor covering initialization, trace/span creation, message processing, exception handling, and property validation.
    • Validated trace_trees behavior under various scenarios, including nested spans, feedback scores, and batch message processing.
    • Ensured correctness of data updates and hierarchy handling in traces and spans.
    • [OPIK-2590] [P SDK] Add delivery attempt tracking and message retry skipping
    • Introduced delivery_attempts field to BaseMessage for tracking delivery retries.
    • Enhanced LocalEmulatorMessageProcessor to skip processing messages marked as retries.
    • Updated queue_consumer to increment delivery_attempts on message requeue.
    • Added unit tests to validate retry logic and message skipping functionality.
    • [OPIK-2655] [P SDK] Add active flag to message processors and update tests
    • Introduced active parameter in message processors to enable or disable message handling.
    • Updated LocalEmulatorMessageProcessor and BackendEmulatorMessageProcessor constructors to accept the active flag.
    • Modified processing logic to respect the active state, skipping inactive processors.
    • Added relevant test cases for active functionality and adjusted existing tests for initialization changes.
    • Refactored queue_consumer to encapsulate message processing in _process_message.

    • [OPIK-2655] [P SDK] Fixed unit test

    • [OPIK-2590] [P SDK] Extract OpikMessageProcessor into a dedicated module

    • Moved OpikMessageProcessor implementation to online_message_processor module for improved segregation.
    • Updated streamer_constructors to use the relocated OpikMessageProcessor.
    • Cleaned up message_processors by removing redundant imports and outdated code.
    • [OPIK-2590] [P SDK] Introduce ChainedMessageProcessor for modular message handling
    • Added ChainedMessageProcessor to allow sequential message processing with multiple processors.
    • Updated OpikClient to use the new chain creation logic via message_processors_chain.
    • Refactored streamer_constructors to integrate ChainedMessageProcessor.
    • Enhanced LocalEmulatorMessageProcessor with a set_active method for dynamic activation state changes.
    • [OPIK-2700] [P SDK] Enhance message processors for improved emulator control and modular handling
    • Added toggle_local_emulator_message_processor and reset_local_emulator_message_processor functions to manage the state of the Local Emulator Message Processor dynamically.
    • Introduced get_processor_by_type method in ChainedMessageProcessor to retrieve processors by type.
    • Enhanced LocalEmulatorMessageProcessor with a reset method and initialization updates.
    • Improved constructor documentation and inline comments for clarity and modularity.
    • [OPIK-2590] [P SDK] Add task span evaluation support and enhance emulator message processing
    • Introduced task span evaluation capability with dedicated scoring logic.
    • Enhanced LocalEmulatorMessageProcessor with thread safety using locks and added functionality to manage spans and traces.
    • Refactored _evaluate_test_case for modular metric scoring and introduced _scores_by_metrics.
    • Added utility to detect evaluation span parameters in scoring functions.
    • Improved message processing APIs with better reset and retrieval methods for the local emulator.
    • [OPIK-2590] [P SDK] Refactor task span evaluation logic and improve error handling
    • Renamed _evaluate_task_result_span to _evaluate_llm_task_result_span for clarity.
    • Added context manager in helpers.py to handle LLM task span evaluations with improved trace data management.
    • Moved score accumulation logic into a dedicated _score_llm_task_result_span method for modularity.
    • Enhanced error handling with better trace finalization and error_info tracking.

    • [OPIK-2590] [P SDK] Simplify LocalEmulatorMessageProcessor tests by removing redundant assertions

    • [OPIK-2590] [P SDK] Rename evaluation_span to task_span in task evaluation logic

    • Updated variable and parameter names to improve consistency and clarity in task evaluation methods.
    • Adjusted related annotations, decorators, and call sites to reflect the renaming.
    • [OPIK-2590] [P SDK] Add extended task span evaluation logic and tests
    • Enhanced task evaluation span handling with clearer logging and improved error handling for missing spans.
    • Updated data models to use more specific types for input and output fields (Optional[Dict[str, Any]]).
    • Added extensive end-to-end tests for task span evaluation, covering various scenarios with multiple and mixed metrics.
    • [OPIK-2590] [P SDK] Extend task span evaluation tests with multiple trials
    • Updated test_evaluate_with_task_span_metrics to include trial_count parameter for comprehensive testing with multiple trials.
    • Adjusted related assertions to account for trial multiplicity in traces and experiment items.
    • Enhanced clarity of test function name to reflect trial-based evaluation.
    • [OPIK-2590] [P SDK] Add docstrings for data models in models.py
    • Added comprehensive docstrings for FeedbackScoreModel, SpanModel, and TraceModel to improve code documentation and clarity.
    • [OPIK-2590] [P SDK] Add RST documentation for message processing emulation models
    • Introduced RST files for FeedbackScoreModel, SpanModel, and TraceModel to provide detailed documentation.
    • Added examples, attributes, and usage guidelines for all mentioned classes.
    • Updated index.rst to include references to the new model documentation.
    • [OPIK-2590] [Docs] Add documentation for task span evaluation metrics
    • Introduced a new section detailing the usage and creation of task span metrics in the Opik documentation.
    • Added example code snippets for creating, scoring, and analyzing task spans.
    • Included best practices and tips for integrating task span metrics into evaluations.
    • [OPIK-2590] [Docs] Add dedicated page for task span evaluation metrics
    • Added a new documentation page explaining task span evaluation metrics in detail, including setup, creation, and usage.
    • Updated navigation to include the new "Task span metrics" section under the evaluation metrics category.
    • Refined older references to task span evaluation to align with the newly added documentation.

    • [OPIK-2590] [Docs] Updated SDK reference documentation

    • [OPIK-2590] [P SDK] Refine span evaluation error messaging and enhance docstrings

    • Improved error messaging for missing spans in task trace during span evaluation.
    • Added detailed explanations to docstrings for reset parameter in message processing chain methods to clarify its usage.
    • [OPIK-2590] [P SDK] Simplify docstrings in models.py for attributes
    • Removed redundant type annotations from attribute docstrings in FeedbackScoreModel, SpanModel, and TraceModel for conciseness.
    • Retained descriptive explanations for improved readability.

    • [OPIK-2590] [P SDK] Fixed typo in the method name

    • [OPIK-2590] [DOC] Removed metric example

    • [OPIK-2590] [DOC] Removed HierarchyComplexityMetric metric example

    • [OPIK-2590] [P SDK] Refactor message processing with a handler-based dispatch mechanism

    • Replaced large conditional blocks in _dispatch_message with a handler-based registry for improved readability and scalability.
    • Introduced _register_handlers method to consolidate message type handlers.
    • Simplified message handling by delegating processing to dedicated handler methods.
    • [OPIK-2590] [Docs] Add an example for nested span hierarchy in task evaluation documentation
    • Introduced a new example showcasing a tracked function with nested spans for a clearer understanding of task span hierarchies.
    • Provided detailed code snippets with explanations for key operations: research_topic, gather_context, analyze_information, and generate_summary.
    • [OPIK-2590] [Docs] Enhance task span metric examples with detailed hierarchy and recursive analysis
    • Improved examples for task span metrics by including comprehensive recursive analysis methods for span trees (_analyze_hierarchy_recursively, _check_execution_success_recursively).
    • Updated scoring logic to provide clearer, context-aware outputs based on depth, span usage, and error analysis.
    • Added explanations and inline comments in code for improved readability.
    • Included new insights into task evaluation workflows, emphasizing caching, trajectory analysis, and usage tracking in nested spans.
    • [OPIK-2590] Pinned crewai version to fix failing tests.
    下载附件