github-featured / comet-ml--opik

发布

0 发布 8,340 Git 标签

RSS 订阅源
  • 1.8.101-3204 9536eb2ee4
    比较

    [OPIK-2632] [P SDK] Alternative way to configure evaluation metrics (#3893)

    frostbyte_neo 发布于 2025-11-05 12:11:00 +00:00

    • [OPIK-2632] [P SDK] Add Scorer Function and Wrapper Metric for Scoring Evaluation
    • Introduced ScorerFunction and AsyncScorerFunction to standardize scoring interface for evaluation tasks.
    • Added ScorerWrapperMetric to integrate scoring functions into the BaseMetric framework.
    • Implemented validate_scorer_function to ensure proper scorer function signature.
    • Created unit tests to validate scorer function behavior and adherence to expected parameters.

    • [OPIK-2632] [P SDK] Fixed error messages.

    • [OPIK-2632] [P SDK] Fixed linter errors

    • [OPIK-2632] [P SDK] Extend scorer function support for evaluation metrics

    • Added task_span-related parameter handling to scorer functions and updated ScorerWrapperMetric.
    • Introduced ScorerWrapperMetricTaskSpan for metrics requiring task span inputs.
    • Updated evaluation logic in evaluate_task to support dynamic wrapping of scorer functions into metrics.
    • Refactored internal validation and scoring logic to streamline metric calculations with task span support.
    • [OPIK-2632] [P SDK] Expand scorer function output flexibility and enhance evaluation logic
    • Updated scorer function interfaces to support multiple results (ScoreResult or List[ScoreResult]).
    • Modified wrap_scorer_functions to return a list of BaseMetric objects for extended metric integration.
    • Extended evaluate_experiment to handle scorer functions alongside scoring metrics, enhancing evaluation flexibility and supporting dual input mechanisms.
    • [OPIK-2632] [P SDK] Enhance scorer function validation and extend evaluation tests
    • Updated validate_scorer_function to support more flexible parameter validation, allowing functions with task_span and/or scoring_inputs and task_outputs.
    • Enhanced error messages for improved clarity.
    • Added extensive unit and end-to-end tests to validate new scorer function behaviors, including combinations of parameters with task_span.

    • [OPIK-2632] [P SDK] Relaxed LangChain usage check

    • [OPIK-2632] [P SDK] Added documentation with examples of how to use scoring functions

    • [OPIK-2632] [P SDK] Fixed typo

    • [OPIK-2632] [P SDK] Fixed typo

    • [OPIK-2632] [P SDK] Added scoring_functions parameter to evaluate_prompt method.

    • [OPIK-2632] [P SDK] Replaced scoring_inputs with dataset_item for consistency and extended evaluation logic

    • Standardized parameter naming across scorer functions by replacing scoring_inputs with dataset_item.
    • Updated scorer function validation, associated tests, and documentation to reflect the parameter change.
    • Refactored evaluation logic to simplify scoring function wrapping with a new _wrap_scoring_functions utility.
    • [OPIK-2632] [P SDK] Replaced scoring_inputs with dataset_item in docstrings
    下载附件