-
[OPIK-3211] [BE/FE] Add span-level Python metrics scorer (#4385)
frostbyte_neo 发布于
2025-12-10 09:50:47 +00:00 - feat(backend): Add span user-defined Python metric evaluator (OPIK-3211)
- Created AutomationRuleEvaluatorSpanUserDefinedMetricPython API class
- Created SpanUserDefinedMetricPythonAutomationRuleEvaluatorModel domain model
- Added SpanToScoreUserDefinedMetricPython event class
- Updated service layer CRUD operations to handle span Python scorers
- Added database migration for span_user_defined_metric_python type
- Added SpanFilter serialization in mapper using generic mapFilters() method
- Integrated with ManualEvaluationService to fetch spans via spanService.getByTraceIds()
- Updated OnlineScoringSpanSampler with necessary imports (placeholder for future implementation)
The implementation uses unified base classes with List<? extends Filter> for type safety.
Span Python evaluators can now be created, updated, and retrieved via CRUD operations.
Manual evaluation properly fetches spans and enqueues them for Python scoring.- fix(backend): Refactor span Python evaluator to use span IDs directly (OPIK-3211)
- Added SPAN entity type to ManualEvaluationEntityType enum
- Created evaluateSpans() method that works with span IDs directly, not trace IDs
- Fetch spans using spanService.getByIds() instead of extracting from traces
- Added convertSpanToTrace() helper for span LLM-as-Judge compatibility
- Fixed ManualEvaluationService to properly handle span-level rules with span IDs
- Span Python evaluators now correctly use SpanToScoreUserDefinedMetricPython events
- Added proper validation to only allow span-level rules when evaluating spans
This fixes the incorrect implementation that was trying to use trace IDs for span evaluation.
- fix(backend): Correct span evaluation - no trace conversion, separate entity types (OPIK-3211)
- TRACE evaluation now ONLY processes trace-level rules (LlmAsJudge, UserDefinedMetricPython, TraceThread*)
- SPAN evaluation now ONLY processes span-level rules (SpanLlmAsJudge, SpanUserDefinedMetricPython)
- Removed incorrect span→trace conversion logic
- Span LLM-as-Judge uses SpanToScoreLlmAsJudge events with Span objects (not traces)
- Span Python uses SpanToScoreUserDefinedMetricPython events with Span objects
- Added proper validation to reject invalid rule types for each entity type
- evaluateSpans() fetches spans by ID using spanService.getByIds()
- evaluateTraces() fetches traces by ID using traceService.getByIds()
This ensures clean separation: trace rules operate on traces, span rules operate on spans.
- test(backend): Add span Python evaluator to test data providers (OPIK-3211)
- Added AutomationRuleEvaluatorSpanUserDefinedMetricPython to createAndGet test
- Test currently failing with 500 errors (7 failures total)
- Need to debug server-side stacktrace to identify root cause
- Implementation is architecturally correct, likely a runtime configuration issue
- [OPIK-3211] [BE] [FE] Add span-level Python metrics scorer
-
Backend: Implement OnlineScoringSpanUserDefinedMetricPythonScorer
- Add span-level Python scorer service that consumes Redis stream messages
- Extend OnlineScoringEngine to support span context (input, output, metadata)
- Fix OnlineScoringSpanSampler to fetch both LLM and Python span evaluators
- Add service toggle for span Python evaluator
- Add stream configuration for span Python scoring
- Add comprehensive unit tests and integration tests
-
Frontend: Add UI support for span-level Python code rules
- Add span_python_code rule type to types and schemas
- Update AddEditRuleDialog to support span Python code rules
- Add feature toggle for span Python code metrics
- Update RunEvaluationDialog to include span Python code rules
- Add default Python code template for span scope
- Update helper functions for rule type/scope mapping
- [OPIK-3211] Move toggle checks before fetching evaluators in OnlineScoringSpanSampler
- Check both SPAN_LLM_AS_JUDGE and SPAN_USER_DEFINED_METRIC_PYTHON toggles before fetching evaluators
- Prevents unnecessary database queries when features are disabled
- Update tests to explicitly verify toggle behavior for both evaluator types
- Add test to verify Python evaluators are not fetched when toggle is disabled
- [OPIK-3211] Address PR review comments
- Rename misleading variable spanLevelLlmAsJudgeRules to traceLevelLlmAsJudgeRules in ManualEvaluationService
- Fix incorrect getStream() method signature in OnlineScoringSpanUserDefinedMetricPythonScorerTest
- Remove duplicate key in AddEditRuleDialog DEFAULT_PYTHON_CODE_DATA object
- [OPIK-3211] Add migration 000039 for span_user_defined_metric_python type
- Restore 000037 to match main (only contains span_llm_as_judge)
- Create new migration 000039 to add span_user_defined_metric_python type
- This follows proper migration practices by not modifying existing migrations
- [OPIK-3211] Fix compilation errors after cherry-pick
- Fix AutomationRuleEvaluatorSpanUserDefinedMetricPython: correct super() call parameter order and use getFilters() instead of accessing filters field directly
- Fix AutomationRuleEvaluatorUpdateSpanUserDefinedMetricPython: use getFilters() instead of filters field
- Fix ManualEvaluationService: restore correct variable names for trace evaluation (spanLevelLlmAsJudgeRules, spanLevelPythonRules) and add BadRequestException for invalid span-level rules in trace evaluation
- Add missing toUserDefinedMetricPythonMessage() method to OnlineScoringSpanSampler
- Add missing import for SpanToScoreUserDefinedMetricPython
-
fix config
-
address comments
-
add manual span evaluation button and endpoint
-
Fix linting error
-
Address comments
Co-authored-by: Thiago thiago.santos.hora@gmail.com
Co-authored-by: Thiago Hora thiagoh@comet.com
Co-authored-by: Borys Tkachenko borystkachenko@ip-192-168-0-122.eu-west-1.compute.internal
Co-authored-by: Borys Tkachenko borystkachenko@Boryss-MacBook-Pro.local下载附件