github-featured / comet-ml--opik

发布

0 发布 8,340 Git 标签

RSS 订阅源
  • 1.7.44-2132 f9b90dbb55
    比较

    [OPIK-1833]: [P SDK] Implement evaluation engine for threads (#2514)

    frostbyte_neo 发布于 2025-07-07 14:08:20 +00:00

    • [OPIK-1833]: Added threads evaluation engine stub

    • [OPIK-1833]: Added ThreadsClient stub with core methods for thread search and feedback logging

    • [OPIK-1833]: Added ThreadsClient stub with core methods for thread search and feedback logging

    • [OPIK-1833]: Implemented threads evaluation engine with multi-threading support

    • Added ThreadsEvaluationEngine for evaluating threads and logging feedback scores.
    • Integrated ThreadsEvaluationResult model for structured evaluation outputs.
    • Developed execute function with ThreadPoolExecutor for parallel task processing.
    • Added types and utilities to support thread evaluation tasks.

    • [OPIK-1833]: Implemented proper handling of score results

    • [OPIK-1833]: Refactored ThreadsEvaluationEngine constructor to accept an instance of threads_client.ThreadsClient to allow easy mocking during testing.

    • [OPIK-1833]: Refactored conversation imports to use a centralized types module

    Streamlined Conversation and related type imports by using the newly centralized types module. Updated method signatures and replaced inline type definitions for better maintainability.

    • [OPIK-1833]: Enhanced error handling and logging in ThreadsEvaluationEngine
    • Added a logging framework for better traceability.
    • Introduced specific exceptions to handle empty metrics and thread data scenarios.
    • Improved conversation trace validation to skip evaluation for threads without conversation traces.
    • Enhanced span and trace outputs to include detailed evaluation results.
    • [OPIK-1833]: Added unit tests for ThreadsEvaluationEngine
    • Implemented comprehensive test coverage for ThreadsEvaluationEngine methods.
    • Added tests for conversation trace retrieval, feedback score logging, and thread evaluation.
    • Verified concurrency behavior with multiple threads and workers.
    • Ensured proper exception handling and logging for edge cases.
    • [OPIK-1833]: Added evaluate_threads function and updated __init__.py exports
    • Implemented evaluate_threads to enable conversation thread evaluation with custom metrics.
    • Updated __all__ in __init__.py for proper module export of the new function.

    • [OPIK-1833]: Wrapped threads evaluation by async_http_connections_expire_immediately context manager

    • [OPIK-1833]: Replaced MetricComputationError with EvaluationError in thread evaluation logic

    • Introduced EvaluationError for better exception clarity.
    • Updated exception handling in ThreadsEvaluationEngine and corresponding unit tests.
    • [OPIK-1833]: Fixed typo in _get_conversation_thread method name and updated logic to handle scoring failures
    • Renamed _get_conversation_tread to _get_conversation_thread across implementation and tests.
    • Enhanced scoring logic to exclude failed scores from feedback logging in ThreadsEvaluationEngine.
    • [OPIK-1833]: Refactored ThreadsEvaluationEngine methods and added context management for evaluations
    • Renamed _evaluate_thread to evaluate_thread for better clarity.
    • Introduced evaluate_llm_conversation_context to manage context and trace data during evaluations.
    • Moved trace and span logic into context_helper for cleaner code organization.
    • Updated unit tests to reflect method renaming and added context mock handling.
    • General improvement to exception handling and error logging mechanisms across evaluation flows.
    • [OPIK-1833]: Refactored threads evaluation to align with the shared engine framework
    • Replaced _types.ThreadTestResult with evaluation_result.ThreadEvaluationResult for consistency.
    • Merged evaluation logic into evaluation_tasks_executor from evaluation_executor.
    • Updated thread evaluation tasks to use generic EvaluationTask with TypeVar to improve type safety.
    • Simplified ThreadsEvaluationResult structure for clearer results handling.
    • Refactored tests to reflect the updated structure and improve validation of evaluation results.
    • [OPIK-1833]: Added better type annotations and comments
    • Introduced Literal for type hinting conversation keys, improving clarity and type safety.
    • Included comments to enhance code readability and maintainability.
    • [OPIK-1833]: Refactored threading helpers and removed redundant methods
    • Extracted _get_conversation_thread and _log_feedback_scores into helpers module, improving modularity and reusability.
    • Updated ThreadsEvaluationEngine to use new helper methods and streamlined logic.
    • Added unit tests for helpers.py to ensure correctness of extracted functionality.

    • [OPIK-1833]: Fixed coment

    • [OPIK-1834] [P SDK] ConversationalCoherence conversation metric (#2579)

    • [OPIK-1834]: Fixed imports

    • [OPIK-1834]: Added synchronous Conversational Coherence Metric implementation.

    • Introduced ConversationalCoherenceMetric for evaluating coherence of conversation exchanges within sliding windows.
    • Implemented templates and schema for constructing evaluation queries and parsing responses.
    • Added helper methods to generate sliding windows of conversation turns.
    • Included Pydantic models for validating and handling evaluation responses.
    • Integrated scoring calculation and irrelevancy extraction logic for detailed analysis.
    • [OPIK-1834]: Enhanced Conversational Coherence and Session Completeness metrics.
    • Added ConversationTurn class for structured representation of conversation exchanges.
    • Implemented build_conversation_turns and merge_turns utilities for processing conversational data.
    • Refactored window-based scoring logic to use conversation turns, improving clarity and modularity.
    • Introduced comprehensive tests for conversational coherence and session completeness metrics, covering error scenarios and ensuring robustness.
    • Improved error handling and logging across metric computation functions.
    • [OPIK-1834]: Added and tested conversation turn utilities
    • Introduced test_build_conversation_turns__happy_path for validating the build_conversation_turns utility.
    • Rewrote test_window_size as test_get_turns_in_sliding_window to align with the updated sliding window logic.
    • Enhanced test coverage for sliding window generation and conversation turn processing functions.
    • [OPIK-1834]: Added asynchronous support for Conversational Coherence Metric
    • Introduced ascore methods for asynchronous scoring and evaluation.
    • Added _a_calculate_score, _a_reason_from_verdicts, and _a_evaluate_conversation to handle async operations seamlessly.
    • Extracted common logic into _extract_turns_windows_from_conversation to reduce redundancy and improve readability.
    • Extended test suite with async tests to ensure correctness of async metric computation and error handling.
    • Updated logging and exception handling for async workflows.
    • [OPIK-1834]: Updated imports and added tests for Conversation Coherence Metric.
    • Refactored imports for Session Completeness Metric to improve maintainability.
    • Introduced tests for Conversation Coherence Metric, including synchronous and asynchronous scenarios.
    • Adjusted track parameter in Session Completeness Metric tests to avoid delays while trying to log metric to not running server.
    • Verified integration with real model outputs through expanded test coverage.

    • [OPIK-1834]: Fixed imports

    • [OPIK-1834]: Added another example of irrelevant response reason

    • [OPIK-1834]: Simplified conversation turn extraction logic.

    • [OPIK-1834]: Added documentation to the ConversationalCoherenceMetric and exposed it as export from opik.evaluation.metrics

    • [P SDK] UserFrustration conversation metric (#2603)

    • [OPIK-1835]: Implemented User Frustration Score Metric - synchronous mode.

    • Added UserFrustrationMetric to evaluate user frustration in conversations based on sliding window analysis.
    • Introduced templates and schema for constructing LLM queries and parsing frustration-related responses.
    • Implemented scoring and reasoning generation, leveraging context and per-message analysis.
    • Refactored common sliding window logic to helpers.py for reusability.
    • Updated ConversationTurn utilities to handle cases where output is absent.
    • [OPIK-1835]: Added comprehensive tests for User Frustration Metric
    • Implemented unit tests for UserFrustrationMetric covering various scenarios: no frustration, mixed responses, all frustration, and error handling.
    • Added support for asynchronous test cases (test_score__async) ensuring correctness of async methods.
    • Updated logic in metric.py to fix the calculation of frustrated counts.
    • Ensured detailed validation of edge cases such as empty conversations and invalid response formats.
    • [OPIK-1835]: Added asynchronous support for User Frustration Metric
    • Introduced ascore method for asynchronous frustration score evaluation.
    • Added _a_calculate_score, _a_reason_from_verdicts, and _a_evaluate_conversation methods to handle async operations.
    • Leveraged asyncio.gather for efficient asynchronous processing of conversation sliding windows.
    • Enhanced logging and error handling for async workflows, ensuring robustness.
    • [OPIK-1835]: Updated User Frustration Metric with documentation and integration tests.
    • Detailed class documentation for UserFrustrationMetric, explaining arguments, behavior, and scoring.
    • Integrated metric into the library's exports for seamless access.
    • Added integration tests to validate real model outputs for both synchronous and asynchronous methods.

    • [OPIK-1772]: [P SDK] Implement API class for threads (#2633)

    • [OPIK-1772]: Implement thread operations and add tests.

    • Extended thread client functionality with support for searching threads using filter expressions.
    • Added parse_filter_expressions utility for translating filter strings to filter objects.
    • Introduced end-to-end tests for threads, validating search functionality and active thread handling.
    • Updated filter and truncation handling logic across related API calls for consistency.
    • [OPIK-1772]: Refactor ThreadsClient instantiation and add utility to OpikClient.
    • Replaced direct ThreadsClient instantiation in tests with get_threads_client utility method in OpikClient.
    • Added get_threads_client to facilitate cleaner and context-respective creation of ThreadsClient.
    • Updated related test cases and adjusted import statements for consistency.
    • [OPIK-1772]: Add support for thread feedback score processing.
    • Introduced ThreadsFeedbackScoreMessage and AddThreadsFeedbackScoresBatchMessage for handling thread-specific feedback scores.
    • Added _process_add_threads_feedback_scores_batch_message in MessageProcessor to process batch messages related to thread feedback scores.
    • Implemented log_threads_feedback_scores in ThreadsClient for logging thread feedback scores.
    • Refactored feedback score handling logic into parse_feedback_score_messages utility.

    • [OPIK-1772]: Added unit test cases for AddThreadsFeedbackScoresBatchMessageBatcher

    • [OPIK-1772]: Add warning for rejected AddThreadsFeedbackScoresBatchMessage and update supported query operators.

    • Log warning when AddThreadsFeedbackScoresBatchMessage is rejected due to unclosed threads.
    • Extend SUPPORTED_OPERATORS to include "id" and "status".
    • [OPIK-1772]: Add verification of thread feedback scores and end-to-end test for logging threads feedback scores.
    • Added verify_thread and _assert_feedback_scores utilities to validate feedback scores.
    • Introduced E2E test for logging and verifying logging thread feedback scores.
    • Refactored duplicated feedback score verification logic for reuse.

    • [OPIK-1772]: Updated search_threads documentation to include description of filter string format.

    • [OPIK-1772]: Update error handling for AddThreadsFeedbackScoresBatchMessage rejection.

    • Moved the warning log for rejected AddThreadsFeedbackScoresBatchMessage to _process_add_threads_feedback_scores_batch_message.

    • [OPIK-1772]: Increased timeout to avoid flaky test failures

    • [OPIK-1772]: Modified unit tests workflow to list installed dependencies

    • [OPIK-1833]: Replace evaluate_llm_conversation_context with trace_context for thread evaluations.

    • Removed context_helper and migrated context management logic into trace_context in opik_context.py.
    • Updated evaluation_engine.py to utilize trace_context, ensuring consistent trace handling.
    • Refactored tracing-related imports for cleanup and better organization.

    • [OPIK-1833]: Added additional columns to the OpikQueryLanguage to support filtering of threads.

    • [OPIK-1943]: [P SDK] Integrate ThreadsClient into evaluation engine (#2660)

    • [OPIK-1833]: Add E2E tests for threads evaluation.

    • Introduced E2E test cases for thread evaluation in test_threads_evaluate.py to validate evaluation metrics and feedback score processing.
    • Enhanced create_conversation_from_traces to sort traces by start time, ensuring a proper sequence of messages.
    • Updated OpikQueryLanguage to support filtering by thread_id.
    • Fixed filter_string usage in evaluation helpers to quote thread IDs correctly.
    • Refactored and adjusted imports in tests for consistency and structure.

    • [OPIK-1943]: Fixed failing unit tests.

    • [OPIK-1948]: Handle missing input/output in conversation traces after transformation, which allows filtering.

    • Updated create_conversation_from_traces to skip traces with missing or invalid input/output.
    • Added unit tests to validate trace filtering logic and ensure robustness.
    • [OPIK-1955]: Standardized score formatting in conversation metric templates.
    • Updated templates for session_completeness, user_frustration, and conversational_coherence to format scores with 1 decimal place.
    • [OPIK-1943]: Refactored thread evaluation E2E tests for better thread status handling.
    • Introduced _all_threads_closed and _one_thread_is_active helper functions for thread status verification.
    • Updated imports to remove unused datetime_helpers and include threads_client.
    • Improved thread closing and evaluation workflows in test_evaluate_threads__happy_path to ensure proper propagation and error handling.
    • [OPIK-1943]: Enhanced thread evaluation to handle active vs inactive thread filtering.
    • Fixed to evaluate only inactive (closed) threads.
    • Added logic to verify and log warnings for active threads during evaluation.
    • Introduced _mock_evaluate_thread_two_test_results for test consistency.
    • Updated tests to handle inactive threads and exceptions for all-active scenarios.

    • [OPIK-1943]: Removed obsolete unit test. Added comments for clarity.

    • [OPIK-1933]: Fixed prompt for better clarity.

    • [OPIK-1833]: Updated documentation to use the correct filter string.

    • [OPIK-1833]: Fixed linter errors

    下载附件