-
[NA] [SDK] Evaluation suite API improvements and feedback fixes (#5626)
frostbyte_neo 发布于
2026-03-12 14:02:55 +00:00 - [NA] [SDK] fix: get_or_create_evaluation_suite updates existing suite with new assertions
- get_or_create_evaluation_suite now calls update() when assertions,
evaluators, or execution_policy are provided for an existing suite - update() supports partial updates: omitted params retain current values
- Add e2e tests for get_or_create with updates and partial update scenarios
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- [NA] [SDK] refactor: hide evaluator concept from evaluation suite public API
Replace all user-facing evaluator/LLMJudge references with assertions:
- get_evaluators() -> get_assertions() returning List[str]
- get_items() returns "assertions": List[str] instead of "evaluators"
- Remove evaluators param from update(), add_item(), create/get_or_create
- Rename evaluator_model -> model in run()
- Update all docstrings to use assertions terminology
- Update e2e tests to match new API
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- [NA] [SDK] feat: add task return value validation in evaluation suite run
Wrap the task function with validation that checks it returns a dict
with 'input' and 'output' keys, providing clear error messages with
examples when the format is wrong. Uses functools.wraps to preserve
the original function name for traces/spans.Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- [NA] [SDK] feat: add tags support to evaluation suite API
Add tags parameter to create_evaluation_suite(),
get_or_create_evaluation_suite(), and suite.update(). Tags are a
dataset-level field updated separately from version-level
assertions/execution_policy. Add get_tags() to both Dataset and
EvaluationSuite.Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- [NA] [SDK] feat: add suite-specific result display with pass rates
Replace avg score display with pass rate view for evaluation suites.
Categorize assertions into critical (caused item failures), non-critical
(items still passed due to execution_policy), and fully passing.Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- [NA] [SDK] feat: add batch add_items() and strict input validation
Add add_items() for batch item insertion (single dataset version).
Add EvaluationSuiteItem TypedDict for type hints.
Add validate_execution_policy() and validate_suite_items() with strict
checks for unknown keys, types, and required fields. Validation applied
to add_item, add_items, update, create, and get_or_create.Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- fix(sdk): export public types and fix linter errors
Export EvaluationSuite and EvaluationSuiteResult from opik.init.
Fix mypy (shadowed variable in report.py, Optional None check,
TypedDict/dict mismatch) and ruff (unused assignments in e2e tests).
Mark EvaluationSuite constructor as internal API.Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- fix(sdk): simplify suite result display and add verbosity control
Remove critical/non-critical assertion categorization. Show a flat
list sorted by pass rate (most failed first). Assertions are shown
at verbose>=2 (new default), hidden at verbose=1 (summary only).Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- fix(sdk): address PR review comments and handle 0-items edge case
- Fix pass_threshold=0 being coerced to 1 via
oroperator - Guard ZeroDivisionError in assertion sort when total is 0
- Gate experiment URL print behind verbose >= 1 check
- Replace bare assert with descriptive RuntimeError for None version_info
- Extract validate_task_result() and move validation tests from e2e to unit
- Return None for pass_rate when suite has 0 items
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- feat(sdk): add per-item timing to evaluation suite report
Measure task execution time and scoring time per run, display
avg task time, avg scoring time, and avg total time in the
suite results panel.Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- perf(sdk): merge suite and item-level LLM judges into single API call
Add LLMJudge.merged() factory method that combines assertions from
multiple judges, and use it in build_metrics_evaluator() to reduce
sequential LLM API calls from N to 1 per item.Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- fix(sdk): address review feedback on LLM judge merging
- LLMJudge.merged() returns None when settings differ (model,
temperature, seed, track) instead of silently dropping them - _merge_llm_judges preserves evaluation order by replacing the first
judge in-place instead of pushing merged judge to the end
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- refactor(sdk): extract merge_llm_judges to helpers module and harden APIs
- Move merge_llm_judges from private _merge_llm_judges in
metrics_evaluator.py to public helpers.py in llm_judge package - Add empty-list guard in LLMJudge.merged() to prevent IndexError
- Refactor tests to use only public APIs (build_metrics_evaluator
and LLMJudge.merged) instead of importing private helpers
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- refactor(sdk): simplify LLM judge merging, remove helpers module
- Delete helpers.py; use LLMJudge.merged() classmethod directly
- Simplify build_metrics_evaluator: split judges/non-judges, merge, reassemble
- LLMJudge.merged() now returns None for fewer than 2 judges
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- refactor(sdk): deduplicate pass/fail logic, return EvaluationSuiteResult from evaluate_suite
- Remove duplicated _compute_item_passed_map from report.py, reuse EvaluationSuiteResult
- Make evaluate_suite return EvaluationSuiteResult directly
- Simplify EvaluationSuite.run() to delegate to evaluate_suite
- Move suite_result_constructor import to top-level in evaluator.py
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
Co-authored-by: Claude Opus 4.6 noreply@anthropic.com
下载附件