github-featured / comet-ml--opik

发布

0 发布 8,340 Git 标签

RSS 订阅源
  • 1.7.44-2131 d31bdd5b8e
    比较

    [OPIK-1872]: [Docs] Add documentation page for P SDK thread evaluation (#2615)

    frostbyte_neo 发布于 2025-07-07 13:08:53 +00:00

    • [OPIK-1833]: Added threads evaluation engine stub

    • [OPIK-1833]: Added ThreadsClient stub with core methods for thread search and feedback logging

    • [OPIK-1833]: Added ThreadsClient stub with core methods for thread search and feedback logging

    • [OPIK-1833]: Implemented threads evaluation engine with multi-threading support

    • Added ThreadsEvaluationEngine for evaluating threads and logging feedback scores.
    • Integrated ThreadsEvaluationResult model for structured evaluation outputs.
    • Developed execute function with ThreadPoolExecutor for parallel task processing.
    • Added types and utilities to support thread evaluation tasks.

    • [OPIK-1833]: Implemented proper handling of score results

    • [OPIK-1833]: Refactored ThreadsEvaluationEngine constructor to accept an instance of threads_client.ThreadsClient to allow easy mocking during testing.

    • [OPIK-1833]: Refactored conversation imports to use a centralized types module

    Streamlined Conversation and related type imports by using the newly centralized types module. Updated method signatures and replaced inline type definitions for better maintainability.

    • [OPIK-1833]: Enhanced error handling and logging in ThreadsEvaluationEngine
    • Added a logging framework for better traceability.
    • Introduced specific exceptions to handle empty metrics and thread data scenarios.
    • Improved conversation trace validation to skip evaluation for threads without conversation traces.
    • Enhanced span and trace outputs to include detailed evaluation results.
    • [OPIK-1833]: Added unit tests for ThreadsEvaluationEngine
    • Implemented comprehensive test coverage for ThreadsEvaluationEngine methods.
    • Added tests for conversation trace retrieval, feedback score logging, and thread evaluation.
    • Verified concurrency behavior with multiple threads and workers.
    • Ensured proper exception handling and logging for edge cases.
    • [OPIK-1833]: Added evaluate_threads function and updated __init__.py exports
    • Implemented evaluate_threads to enable conversation thread evaluation with custom metrics.
    • Updated __all__ in __init__.py for proper module export of the new function.

    • [OPIK-1833]: Wrapped threads evaluation by async_http_connections_expire_immediately context manager

    • [OPIK-1833]: Replaced MetricComputationError with EvaluationError in thread evaluation logic

    • Introduced EvaluationError for better exception clarity.
    • Updated exception handling in ThreadsEvaluationEngine and corresponding unit tests.
    • [OPIK-1833]: Fixed typo in _get_conversation_thread method name and updated logic to handle scoring failures
    • Renamed _get_conversation_tread to _get_conversation_thread across implementation and tests.
    • Enhanced scoring logic to exclude failed scores from feedback logging in ThreadsEvaluationEngine.
    • [OPIK-1833]: Refactored ThreadsEvaluationEngine methods and added context management for evaluations
    • Renamed _evaluate_thread to evaluate_thread for better clarity.
    • Introduced evaluate_llm_conversation_context to manage context and trace data during evaluations.
    • Moved trace and span logic into context_helper for cleaner code organization.
    • Updated unit tests to reflect method renaming and added context mock handling.
    • General improvement to exception handling and error logging mechanisms across evaluation flows.
    • [OPIK-1833]: Refactored threads evaluation to align with the shared engine framework
    • Replaced _types.ThreadTestResult with evaluation_result.ThreadEvaluationResult for consistency.
    • Merged evaluation logic into evaluation_tasks_executor from evaluation_executor.
    • Updated thread evaluation tasks to use generic EvaluationTask with TypeVar to improve type safety.
    • Simplified ThreadsEvaluationResult structure for clearer results handling.
    • Refactored tests to reflect the updated structure and improve validation of evaluation results.
    • [OPIK-1833]: Added better type annotations and comments
    • Introduced Literal for type hinting conversation keys, improving clarity and type safety.
    • Included comments to enhance code readability and maintainability.
    • [OPIK-1833]: Refactored threading helpers and removed redundant methods
    • Extracted _get_conversation_thread and _log_feedback_scores into helpers module, improving modularity and reusability.
    • Updated ThreadsEvaluationEngine to use new helper methods and streamlined logic.
    • Added unit tests for helpers.py to ensure correctness of extracted functionality.

    • [OPIK-1833]: Fixed coment

    • [OPIK-1834] [P SDK] ConversationalCoherence conversation metric (#2579)

    • [OPIK-1834]: Fixed imports

    • [OPIK-1834]: Added synchronous Conversational Coherence Metric implementation.

    • Introduced ConversationalCoherenceMetric for evaluating coherence of conversation exchanges within sliding windows.
    • Implemented templates and schema for constructing evaluation queries and parsing responses.
    • Added helper methods to generate sliding windows of conversation turns.
    • Included Pydantic models for validating and handling evaluation responses.
    • Integrated scoring calculation and irrelevancy extraction logic for detailed analysis.
    • [OPIK-1834]: Enhanced Conversational Coherence and Session Completeness metrics.
    • Added ConversationTurn class for structured representation of conversation exchanges.
    • Implemented build_conversation_turns and merge_turns utilities for processing conversational data.
    • Refactored window-based scoring logic to use conversation turns, improving clarity and modularity.
    • Introduced comprehensive tests for conversational coherence and session completeness metrics, covering error scenarios and ensuring robustness.
    • Improved error handling and logging across metric computation functions.
    • [OPIK-1834]: Added and tested conversation turn utilities
    • Introduced test_build_conversation_turns__happy_path for validating the build_conversation_turns utility.
    • Rewrote test_window_size as test_get_turns_in_sliding_window to align with the updated sliding window logic.
    • Enhanced test coverage for sliding window generation and conversation turn processing functions.
    • [OPIK-1834]: Added asynchronous support for Conversational Coherence Metric
    • Introduced ascore methods for asynchronous scoring and evaluation.
    • Added _a_calculate_score, _a_reason_from_verdicts, and _a_evaluate_conversation to handle async operations seamlessly.
    • Extracted common logic into _extract_turns_windows_from_conversation to reduce redundancy and improve readability.
    • Extended test suite with async tests to ensure correctness of async metric computation and error handling.
    • Updated logging and exception handling for async workflows.
    • [OPIK-1834]: Updated imports and added tests for Conversation Coherence Metric.
    • Refactored imports for Session Completeness Metric to improve maintainability.
    • Introduced tests for Conversation Coherence Metric, including synchronous and asynchronous scenarios.
    • Adjusted track parameter in Session Completeness Metric tests to avoid delays while trying to log metric to not running server.
    • Verified integration with real model outputs through expanded test coverage.

    • [OPIK-1834]: Fixed imports

    • [OPIK-1834]: Added another example of irrelevant response reason

    • [OPIK-1834]: Simplified conversation turn extraction logic.

    • [OPIK-1834]: Added documentation to the ConversationalCoherenceMetric and exposed it as export from opik.evaluation.metrics

    • [P SDK] UserFrustration conversation metric (#2603)

    • [OPIK-1835]: Implemented User Frustration Score Metric - synchronous mode.

    • Added UserFrustrationMetric to evaluate user frustration in conversations based on sliding window analysis.
    • Introduced templates and schema for constructing LLM queries and parsing frustration-related responses.
    • Implemented scoring and reasoning generation, leveraging context and per-message analysis.
    • Refactored common sliding window logic to helpers.py for reusability.
    • Updated ConversationTurn utilities to handle cases where output is absent.
    • [OPIK-1835]: Added comprehensive tests for User Frustration Metric
    • Implemented unit tests for UserFrustrationMetric covering various scenarios: no frustration, mixed responses, all frustration, and error handling.
    • Added support for asynchronous test cases (test_score__async) ensuring correctness of async methods.
    • Updated logic in metric.py to fix the calculation of frustrated counts.
    • Ensured detailed validation of edge cases such as empty conversations and invalid response formats.
    • [OPIK-1835]: Added asynchronous support for User Frustration Metric
    • Introduced ascore method for asynchronous frustration score evaluation.
    • Added _a_calculate_score, _a_reason_from_verdicts, and _a_evaluate_conversation methods to handle async operations.
    • Leveraged asyncio.gather for efficient asynchronous processing of conversation sliding windows.
    • Enhanced logging and error handling for async workflows, ensuring robustness.
    • [OPIK-1835]: Updated User Frustration Metric with documentation and integration tests.
    • Detailed class documentation for UserFrustrationMetric, explaining arguments, behavior, and scoring.
    • Integrated metric into the library's exports for seamless access.
    • Added integration tests to validate real model outputs for both synchronous and asynchronous methods.

    • [OPIK-1872]: Updated documentation to include threads evaluation.

    • [OPIK-1872]: Updated threads evaluation documentation

    • [OPIK-1872]: Enhanced documentation for evaluate_threads to detail supported fields and operators.

    • [OPIK-1872]: Fixed feedback_scores filter example

    • [OPIK-1872]: Added feedback_scores filter parameters description.

    • [OPIK-1872]: Fixed broken link

    • [OPIK-1872]: Changed screenshot for threads evaluation

    • Optimised images with calibre/image-actions

    • Optimised images with calibre/image-actions

    Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

    下载附件