-
[OPIK-5759] [BE/FE] feat: multi-provider LLM-as-judge support for eval suite assertions (#6167)
frostbyte_neo 发布于
2026-04-10 11:27:48 +00:00 - [OPIK-5759] [BE/FE] feat: multi-provider LLM-as-judge support for eval suite assertions
Support OpenAI, Anthropic, and Gemini as LLM-as-judge providers for eval suite
assertions. Previously only OpenAI was implicitly supported. The model is resolved
from connected providers using a priority order (OpenAI > Anthropic > Gemini).Backend:
- Add SupportedJudgeProvider enum with provider-to-model mapping
- Move provider resolution from config into EvalSuiteAssertionSampler
- Keep EvalSuiteEvaluatorMapper as a pure data transformer
- Remove unused defaultModelName from EvalSuiteConfig
- Fix: don't send snakeCased config keys to backend (camelCase expected)
Frontend:
- Add provider validation: disable run button when no supported provider is connected
- Pass pre-computed boolean to RunOnDatasetDialog instead of raw provider keys
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- refactor(eval-suite): improve evaluator mapper clarity and add unit tests
- Merge deserialization and model assignment into deserializeScoringCode()
- Remove separate setModel/withModel method
- Add parameterized unit tests for resolveModel() provider priority
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- refactor: remove FE provider blocking, add backend model fallback from trace metadata
- Remove frontend logic that blocked eval suite runs without a supported
LLM provider (revert llm.ts, PlaygroundHeader, RunOnDatasetDialog) - Resolve model once per batch on backend: try connected providers
(OpenAI > Anthropic > Gemini > Vertex AI), fall back to completion
task model from trace metadata - Add SupportedJudgeProvider enum referencing model name enums for
compile-time safety - Store eval_suite_model in trace metadata for fallback resolution
- Add Vertex AI as supported eval suite judge provider
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- fix: address PR review comments for eval suite model resolution
- Extract SupportedJudgeProvider to its own file
- Use streams instead of for-loop in resolveModel
- Preserve model parameters (temperature, seed, customParameters) when overriding name
- Guard against null modelName with early return and warning log
- Replace hardcoded model strings in tests with enum constants
- Use imported Context instead of qualified reactor.util.context.Context
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- fix: add proper import for reactor.util.context.Context
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- fix: update EvalSuiteAssertionSamplerTest for null model guard
- Mock connected OpenAI provider in setUp so existing tests pass
- Add test for eval_suite_model metadata fallback when no provider connected
- Add test for early return when neither provider nor metadata model available
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
Co-authored-by: Claude Opus 4.6 noreply@anthropic.com
下载附件