-
[OPIK-4957] [SDK] feat: improve evaluation suite run experience and performance (#5677)
frostbyte_neo 发布于
2026-03-17 10:01:40 +00:00 - feat(sdk): add reasoning_effort support to LLMJudge
Defaults to "low" to reduce latency. Silently dropped for
models that don't support it (e.g. gpt-4o-mini).Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- feat(sdk): improve suite evaluation UX with progress hint and experiment link
Add a hint message to the progress bar while waiting for the first item
to complete, and show a clickable experiment link at the top of the
suite results panel.Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- refactor(sdk): clarify LLM judge prompt with explicit delimiters
Use ---BEGIN/END--- delimiters around input and output sections so
short agent responses don't blend into the description text.Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
-
Mention the default model gpt-5-nano
-
feat(sdk): show clickable dashboard link before suite evaluation starts
Print "Running evaluation suite, results will be available in Opik
dashboard" with a bold cyan clickable link before the progress bar
appears, so users can follow along in the UI.Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- test(sdk): update LLM judge test for new prompt delimiters
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- fix(sdk): serialize reasoning_effort in LLMJudge to_config/from_config
Store reasoning_effort in model.customParameters so persisted
online-evaluation configs preserve it. Old configs without it
fall back to DEFAULT_REASONING_EFFORT.Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- fix(sdk): restore detailed INPUT/OUTPUT section descriptions in LLM judge prompt
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
Co-authored-by: Claude Opus 4.6 noreply@anthropic.com
下载附件