-
[OPIK-3210] [FE] Add span-level online scoring rules support (#4269)
frostbyte_neo 发布于
2025-12-02 14:04:05 +00:00 - [OPIK-3208] [BE] Add span feedback scores aggregation to trace detail view
- Add span_feedback_scores field to Trace API model
- Add span score aggregation logic to TraceDAO query
- Update TraceEnrichmentMapper to map aggregated span scores
- Add tests for span score aggregation
- Update testcontainers version to 2.0.2
- Fix getValue method to handle missing columns gracefully
[OPIK-3208] [FE] Display aggregated span feedback scores in trace detail view
- Add span_feedback_scores to Trace type definition
- Display aggregated span feedback scores in TraceDataViewer
- Add span feedback score chips in trace tree view
- Update TreeDetailsStore to support span_feedback_scores
- Remove individual span scores drill-down section
- [OPIK-3208] [FE] Fix feedback scores title for span view
- Show 'Trace Feedback Scores' only when viewing a trace
- Show 'Feedback Scores' when viewing a span
- [OPIK-3208] [FE] Disable deletion for span feedback scores at trace level
- Make onDeleteFeedbackScore optional in FeedbackScoreTable
- Hide delete actions column when deletion is disabled
- Prevent deletion of aggregated span scores shown at trace level
-
[OPIK-3209] Add spans feedback scores filters
-
Fix remaining issues
-
[OPIK-3209] Address PR review comments: add type guard, fix method spelling, add JavaDoc
-
[OPIK-3209] [BE] Add span feedback scores statistics to trace table headers
-
[OPIK-3209] [BE] Fix StatsUtils to only add span feedback scores stats when values exist
-
[OPIK-3209] Add tests for span feedback scores statistics
-
refactor: Remove code duplication in span feedback scores display
- Extract helper functions for value parsing and display logic in ValueCell
- Consolidate child row creation logic into reusable createChildRow function
- Export extractAuthorName helper and use it consistently in AuthorCell
- Simplify formatParentRowWithCounts by merging categorical/non-categorical logic
- Add spanId field to ValueEntry API model for proper span identification
This refactoring reduces code duplication by ~60 lines and improves maintainability
without changing functionality.- feat: Add Type column cell component for span feedback scores
- Create TypeCell component to display span type with icon and label
- Component hides type for parent/aggregated rows
- Uses BaseTraceDataTypeIcon for consistent iconography
- Part of span feedback scores feature implementation
- feat: Add aggregated span feedback scores display in trace detail view
Backend changes:
- Add span feedback scores aggregation in TraceDAO using CTEs
- Include span_id, span_type, and category_name in value_by_author map
- Use composite keys (author_spanId) to preserve individual span scores
- Filter out '' from aggregated reasons
- Add spanId field to ValueEntry API model
Frontend changes:
- Display aggregated span feedback scores in trace detail view
- Add hierarchical view with parent/child rows for multiple spans
- Add Type column to show span type (LLM, Tool, General, Guardrail)
- Support deletion of individual span scores using span_id
- Update table titles to 'Trace scores' and 'Span scores' (Sentence case)
- Fix author name display to handle composite keys correctly
- Update value formatting to show counts and averages for parent rows
- Remove span feedback scores tags from trace header and tree view
- Update tooltips to include 'span' suffix (e.g., 'LLM span')
- Fix 'All scores' summary to display actual values
- Handle empty reasons gracefully (don't show '')
This feature allows users to see aggregated feedback scores from multiple spans
at the trace level while maintaining the ability to view and manage individual
span scores.- fix: Update annotation sidebar titles and score filtering
- Change sidebar title to 'Trace feedback scores' / 'Span feedback scores'
- Change 'All scores' section to show 'Trace scores' / 'Span scores' (remove 'All scores' prefix)
- Filter scores correctly: only show trace scores when viewing trace, only span scores when viewing span
- Update 'Your scores' header to include entity type (Trace/Span/Thread scores)
- Update Thread annotations to match the same pattern
- [FE] Refactor feedback score tables to use isAggregatedSpanScores prop
- Replace 'span-detail' entityType with isAggregatedSpanScores prop for cleaner logic
- Extract helper functions to reduce code duplication (getStorageKeyType, getConfigurableColumnsWithoutType)
- Fix all linting errors and remove unused imports/variables
- Improve code organization and maintainability
-
Fix issues
-
[OPIK-3209] [FE] Fix linting errors
-
Fix tests
-
[OPIK-3209] [BE] Fix thread annotation queue IDs alias inconsistency
- Added THREAD_ANNOTATION_QUEUE_IDS_ANALYTICS_DB constant for thread annotation queue IDs (ttaqi alias)
- Updated TRACE_THREAD_FIELDS_MAP to use ttaqi.annotation_queue_ids for TraceThreadField.ANNOTATION_QUEUE_IDS
- Updated SELECT_COUNT_TRACES_THREADS_BY_PROJECT_IDS to use ttaqi alias to match SELECT_TRACES_THREADS_BY_PROJECT_IDS
- Fixes FindTraceThreads.whenFilterByAnnotationQueueId__thenReturnThreadsWithMatchingTags test failure
- [OPIK-3210] [BE] Add span LLM-as-Judge automation rule evaluator
- Add AutomationRuleEvaluatorSpanLlmAsJudge and related models
- Add SpansCreated event and publish it from SpanService
- Add OnlineScoringSpanSampler to sample spans for scoring
- Add OnlineScoringSpanLlmAsJudgeScorer to score spans using LLM
- Add SpanFilterEvaluationService for evaluating span filters
- Refactor FilterEvaluationServiceBase to reduce duplication
- Update AutomationRuleEvaluator to use List instead of List
- Add AutomationRuleEvaluatorFiltersDeserializer to handle polymorphic filter deserialization
- Add comprehensive tests for span filter evaluation and deserializer
- Add service toggle for span LLM as Judge feature
- Update OnlineScoringEngine to support span scoring
- Add migration to extend automation_rule_evaluators type enum
-
Fix tests
-
Fix tests
-
[OPIK-3210] [FE] Add span-level online scoring rules support
- Add span scope option to rule creation UI
- Implement span filter builder with all span fields including duration, usage, cost, errors
- Add span field-binding UI with autocomplete for input/output/metadata paths
- Support is_empty and is_not_empty operators for dictionary filters in rule context
- Add comprehensive tests for IS_EMPTY and IS_NOT_EMPTY operators on span filters
- Fix OUTPUT_JSON field extraction to properly handle nested keys
- Ensure only custom LLM-as-judge template is available for span scope
- Add feature toggle support for span LLM-as-judge functionality
-
Revision: Fix prettier formatting issues
-
Fix merge
-
[OPIK-3210] [BE] Address PR review comments
- Move config check to beginning of OnlineScoringSpanSampler.onSpansCreated()
- Add type filter to database query in AutomationRuleEvaluatorService.findAll()
- Replace fully qualified class names with imports in OnlineScoringEngine
- Rename chatResponse to score in OnlineScoringSpanLlmAsJudgeScorer
- Fix variable shadowing in TraceAnnotateViewer (allFeedbackScores -> filteredFeedbackScores)
- Add JavaDoc comment for FEEDBACK_SCORE_FIELDS constant
- [OPIK-3210] [BE] Optimize JSON operations and address remaining PR comments
- Use JsonUtils.getMapper() instead of static ObjectMapper reference
- Add comments about reactive JSON approach for future optimization
- Minor formatting improvements
- [OPIK-3210] Remove filter normalization and add CUSTOM field evaluation
- Remove input_json/output_json normalization logic from frontend
- Backend now stores filters in same format as sent by frontend
- Add CUSTOM field handling in filter evaluation services
- Convert CUSTOM fields only when evaluating filters, not when storing
- Add comprehensive tests for CUSTOM field evaluation
- [OPIK-3210] Add JSON path support with array indices for CUSTOM filters
- Update extractNestedValue to handle JSON paths with array indices
- Support both bracket notation (messages[0].content) and dot notation (messages.0.content)
- Add tests for array index path navigation
- Fixes issue where filters like 'input.messages[0].content' were not working
- [OPIK-3210] Simplify navigateJsonPath method
- Extract array element access into helper method getArrayElement
- Simplify path parsing logic
- Improve code readability while maintaining same functionality
- All tests passing
下载附件