github-featured / comet-ml--opik

发布

0 发布 8,340 Git 标签

RSS 订阅源
  • 1.7.37-2003 4d5655ea2d
    比较

    [OPIK-1832]: [P SDK] Implement SessionCompletenessQuality conversation metric and update class hierarchy (#2494)

    frostbyte_neo 发布于 2025-06-18 16:40:48 +00:00

    • [OPIK-1832]: Rename Conversation to ConversationThread for clarity and update references in factory.

    • [OPIK-1832]: Rename module conversation to conversation_thread for clarity and update references in factory.

    • [OPIK-1903]: Implement Session Completeness Quality metric for evaluating conversation threads

    Added SessionCompletenessQuality metric to evaluate how well user goals are fulfilled within a conversation thread. Introduced template generation for extract_user_goals and evaluate_user_goal using LLMs. Integrated UserGoalsResponse and EvaluateUserGoalResponse schemas for structured LLM outputs.

    • [OPIK-1903]: Fixed typo test package name. Refactored to turn off metrics tracking in unit tests to avoid unnecessary networking.

    • [OPIK-1903]: Refactored to have a separate workflow for metrics evaluation tests.

    • [OPIK-1832]: Refactored SessionCompletenessQuality metric to use conversation structures instead of ConversationThread.

    Updated method signatures to accept list-based conversation structures. Removed ConversationThread dependencies. Added reason generation for scoring results. Extended test coverage with both unit and integration tests for varied conversation scenarios.

    • [OPIK-1832]: Added async support for the SessionCompletenessQuality metric and refactored scoring logic.

    Introduced asynchronous methods for extraction, evaluation, and scoring using LLMs in SessionCompletenessQuality. Refactored shared logic into reusable helper functions for verdict processing and scoring. Updated integration tests to include async scenarios and enhanced SSL certificate handling for test environments.

    • [OPIK-1832]: Fixed parameter types

    • [OPIK-1832]: Added permissions to github workflows

    • [OPIK-1832]: Enhanced SessionCompletenessQuality metric with stricter type annotations, improved error handling, and modularized parsing logic.

    Updated conversation typing to use Literal for stricter schema compatibility. Refactored redundant parsing code into reusable helper functions for improved maintainability and reduced duplication. Enhanced logging and error handling for metric scoring logic. Refactored tests by moving mock-based tests into unit tests.

    • [OPIK-1832]: Moved common assertion method into testlib. Fixed imports to use a new location.

    • [OPIK-1832]: Fixed linter complains

    • [OPIK-1832]: Modularized conversation types and removed inline definitions.

    Extracted Conversation and ConversationDict definitions into a new types module for cleaner separation of concerns and reusability. Updated imports across relevant modules to use the new definitions.

    下载附件