github-featured / comet-ml--opik

发布

0 发布 8,340 Git 标签

RSS 订阅源
  • 1.9.75-3799 6a3a6f432c
    比较

    [OPIK-3211] [BE] Replace manual JSON traversal with JSONPath library (#4313)

    frostbyte_neo 发布于 2026-01-08 09:03:17 +00:00

    • [OPIK-3208] [BE] Add span feedback scores aggregation to trace detail view
    • Add span_feedback_scores field to Trace API model
    • Add span score aggregation logic to TraceDAO query
    • Update TraceEnrichmentMapper to map aggregated span scores
    • Add tests for span score aggregation
    • Update testcontainers version to 2.0.2
    • Fix getValue method to handle missing columns gracefully

    [OPIK-3208] [FE] Display aggregated span feedback scores in trace detail view

    • Add span_feedback_scores to Trace type definition
    • Display aggregated span feedback scores in TraceDataViewer
    • Add span feedback score chips in trace tree view
    • Update TreeDetailsStore to support span_feedback_scores
    • Remove individual span scores drill-down section
    • [OPIK-3208] [FE] Fix feedback scores title for span view
    • Show 'Trace Feedback Scores' only when viewing a trace
    • Show 'Feedback Scores' when viewing a span
    • [OPIK-3208] [FE] Disable deletion for span feedback scores at trace level
    • Make onDeleteFeedbackScore optional in FeedbackScoreTable
    • Hide delete actions column when deletion is disabled
    • Prevent deletion of aggregated span scores shown at trace level

    • [OPIK-3209] Add spans feedback scores filters

    • Fix remaining issues

    • [OPIK-3209] Address PR review comments: add type guard, fix method spelling, add JavaDoc

    • [OPIK-3209] [BE] Add span feedback scores statistics to trace table headers

    • [OPIK-3209] [BE] Fix StatsUtils to only add span feedback scores stats when values exist

    • [OPIK-3209] Add tests for span feedback scores statistics

    • refactor: Remove code duplication in span feedback scores display

    • Extract helper functions for value parsing and display logic in ValueCell
    • Consolidate child row creation logic into reusable createChildRow function
    • Export extractAuthorName helper and use it consistently in AuthorCell
    • Simplify formatParentRowWithCounts by merging categorical/non-categorical logic
    • Add spanId field to ValueEntry API model for proper span identification

    This refactoring reduces code duplication by ~60 lines and improves maintainability
    without changing functionality.

    • feat: Add Type column cell component for span feedback scores
    • Create TypeCell component to display span type with icon and label
    • Component hides type for parent/aggregated rows
    • Uses BaseTraceDataTypeIcon for consistent iconography
    • Part of span feedback scores feature implementation
    • feat: Add aggregated span feedback scores display in trace detail view

    Backend changes:

    • Add span feedback scores aggregation in TraceDAO using CTEs
    • Include span_id, span_type, and category_name in value_by_author map
    • Use composite keys (author_spanId) to preserve individual span scores
    • Filter out '' from aggregated reasons
    • Add spanId field to ValueEntry API model

    Frontend changes:

    • Display aggregated span feedback scores in trace detail view
    • Add hierarchical view with parent/child rows for multiple spans
    • Add Type column to show span type (LLM, Tool, General, Guardrail)
    • Support deletion of individual span scores using span_id
    • Update table titles to 'Trace scores' and 'Span scores' (Sentence case)
    • Fix author name display to handle composite keys correctly
    • Update value formatting to show counts and averages for parent rows
    • Remove span feedback scores tags from trace header and tree view
    • Update tooltips to include 'span' suffix (e.g., 'LLM span')
    • Fix 'All scores' summary to display actual values
    • Handle empty reasons gracefully (don't show '')

    This feature allows users to see aggregated feedback scores from multiple spans
    at the trace level while maintaining the ability to view and manage individual
    span scores.

    • fix: Update annotation sidebar titles and score filtering
    • Change sidebar title to 'Trace feedback scores' / 'Span feedback scores'
    • Change 'All scores' section to show 'Trace scores' / 'Span scores' (remove 'All scores' prefix)
    • Filter scores correctly: only show trace scores when viewing trace, only span scores when viewing span
    • Update 'Your scores' header to include entity type (Trace/Span/Thread scores)
    • Update Thread annotations to match the same pattern
    • [FE] Refactor feedback score tables to use isAggregatedSpanScores prop
    • Replace 'span-detail' entityType with isAggregatedSpanScores prop for cleaner logic
    • Extract helper functions to reduce code duplication (getStorageKeyType, getConfigurableColumnsWithoutType)
    • Fix all linting errors and remove unused imports/variables
    • Improve code organization and maintainability

    • Fix issues

    • [OPIK-3209] [FE] Fix linting errors

    • Fix tests

    • [OPIK-3209] [BE] Fix thread annotation queue IDs alias inconsistency

    • Added THREAD_ANNOTATION_QUEUE_IDS_ANALYTICS_DB constant for thread annotation queue IDs (ttaqi alias)
    • Updated TRACE_THREAD_FIELDS_MAP to use ttaqi.annotation_queue_ids for TraceThreadField.ANNOTATION_QUEUE_IDS
    • Updated SELECT_COUNT_TRACES_THREADS_BY_PROJECT_IDS to use ttaqi alias to match SELECT_TRACES_THREADS_BY_PROJECT_IDS
    • Fixes FindTraceThreads.whenFilterByAnnotationQueueId__thenReturnThreadsWithMatchingTags test failure
    • [OPIK-3210] [BE] Add span LLM-as-Judge automation rule evaluator
    • Add AutomationRuleEvaluatorSpanLlmAsJudge and related models
    • Add SpansCreated event and publish it from SpanService
    • Add OnlineScoringSpanSampler to sample spans for scoring
    • Add OnlineScoringSpanLlmAsJudgeScorer to score spans using LLM
    • Add SpanFilterEvaluationService for evaluating span filters
    • Refactor FilterEvaluationServiceBase to reduce duplication
    • Update AutomationRuleEvaluator to use List instead of List
    • Add AutomationRuleEvaluatorFiltersDeserializer to handle polymorphic filter deserialization
    • Add comprehensive tests for span filter evaluation and deserializer
    • Add service toggle for span LLM as Judge feature
    • Update OnlineScoringEngine to support span scoring
    • Add migration to extend automation_rule_evaluators type enum

    • Fix tests

    • Fix tests

    • [OPIK-3210] [FE] Add span-level online scoring rules support

    • Add span scope option to rule creation UI
    • Implement span filter builder with all span fields including duration, usage, cost, errors
    • Add span field-binding UI with autocomplete for input/output/metadata paths
    • Support is_empty and is_not_empty operators for dictionary filters in rule context
    • Add comprehensive tests for IS_EMPTY and IS_NOT_EMPTY operators on span filters
    • Fix OUTPUT_JSON field extraction to properly handle nested keys
    • Ensure only custom LLM-as-judge template is available for span scope
    • Add feature toggle support for span LLM-as-judge functionality

    • Revision: Fix prettier formatting issues

    • Fix merge

    • feat(backend): Add span user-defined Python metric evaluator (OPIK-3211)

    • Created AutomationRuleEvaluatorSpanUserDefinedMetricPython API class
    • Created SpanUserDefinedMetricPythonAutomationRuleEvaluatorModel domain model
    • Added SpanToScoreUserDefinedMetricPython event class
    • Updated service layer CRUD operations to handle span Python scorers
    • Added database migration for span_user_defined_metric_python type
    • Added SpanFilter serialization in mapper using generic mapFilters() method
    • Integrated with ManualEvaluationService to fetch spans via spanService.getByTraceIds()
    • Updated OnlineScoringSpanSampler with necessary imports (placeholder for future implementation)

    The implementation uses unified base classes with List<? extends Filter> for type safety.
    Span Python evaluators can now be created, updated, and retrieved via CRUD operations.
    Manual evaluation properly fetches spans and enqueues them for Python scoring.

    • fix(backend): Refactor span Python evaluator to use span IDs directly (OPIK-3211)
    • Added SPAN entity type to ManualEvaluationEntityType enum
    • Created evaluateSpans() method that works with span IDs directly, not trace IDs
    • Fetch spans using spanService.getByIds() instead of extracting from traces
    • Added convertSpanToTrace() helper for span LLM-as-Judge compatibility
    • Fixed ManualEvaluationService to properly handle span-level rules with span IDs
    • Span Python evaluators now correctly use SpanToScoreUserDefinedMetricPython events
    • Added proper validation to only allow span-level rules when evaluating spans

    This fixes the incorrect implementation that was trying to use trace IDs for span evaluation.

    • fix(backend): Correct span evaluation - no trace conversion, separate entity types (OPIK-3211)
    • TRACE evaluation now ONLY processes trace-level rules (LlmAsJudge, UserDefinedMetricPython, TraceThread*)
    • SPAN evaluation now ONLY processes span-level rules (SpanLlmAsJudge, SpanUserDefinedMetricPython)
    • Removed incorrect span→trace conversion logic
    • Span LLM-as-Judge uses SpanToScoreLlmAsJudge events with Span objects (not traces)
    • Span Python uses SpanToScoreUserDefinedMetricPython events with Span objects
    • Added proper validation to reject invalid rule types for each entity type
    • evaluateSpans() fetches spans by ID using spanService.getByIds()
    • evaluateTraces() fetches traces by ID using traceService.getByIds()

    This ensures clean separation: trace rules operate on traces, span rules operate on spans.

    • feat(backend): Add OnlineScoringSpanSampler support for span Python evaluators
    • Updated filter to include SPAN_USER_DEFINED_METRIC_PYTHON evaluator type
    • Added case to handle span Python evaluator sampling and enqueueing
    • Created toUserDefinedMetricPythonMessage helper method for SpanToScoreUserDefinedMetricPython event creation
    • Added import for SpanToScoreUserDefinedMetricPython event class
    • Python evaluator respects serviceToggles.isPythonEvaluatorEnabled flag
    • Messages are enqueued with SPAN_USER_DEFINED_METRIC_PYTHON type for downstream processing
    • fix(backend): Enforce strict entity type validation in manual evaluation
    • evaluateTraces() now rejects span-level rules (AutomationRuleEvaluatorSpanLlmAsJudge, AutomationRuleEvaluatorSpanUserDefinedMetricPython)
    • evaluateSpans() now rejects trace and thread-level rules (AutomationRuleEvaluatorLlmAsJudge, AutomationRuleEvaluatorUserDefinedMetricPython, AutomationRuleEvaluatorTraceThread*)
    • evaluateThreads() now rejects trace and span-level rules
    • Changed from logging warnings to throwing BadRequestException for invalid rule types
    • Each entity type (TRACE/SPAN/THREAD) now only accepts its appropriate evaluator types
    • Prevents accidental rule misapplication during manual evaluation
    • [OPIK-3210] [BE] Address PR review comments
    • Move config check to beginning of OnlineScoringSpanSampler.onSpansCreated()
    • Add type filter to database query in AutomationRuleEvaluatorService.findAll()
    • Replace fully qualified class names with imports in OnlineScoringEngine
    • Rename chatResponse to score in OnlineScoringSpanLlmAsJudgeScorer
    • Fix variable shadowing in TraceAnnotateViewer (allFeedbackScores -> filteredFeedbackScores)
    • Add JavaDoc comment for FEEDBACK_SCORE_FIELDS constant
    • [OPIK-3210] [BE] Optimize JSON operations and address remaining PR comments
    • Use JsonUtils.getMapper() instead of static ObjectMapper reference
    • Add comments about reactive JSON approach for future optimization
    • Minor formatting improvements
    • test(backend): Add span Python evaluator to test data providers (OPIK-3211)
    • Added AutomationRuleEvaluatorSpanUserDefinedMetricPython to createAndGet test
    • Test currently failing with 500 errors (7 failures total)
    • Need to debug server-side stacktrace to identify root cause
    • Implementation is architecturally correct, likely a runtime configuration issue
    • [OPIK-3211] [BE] [FE] Add span-level Python metrics scorer

    • Backend: Implement OnlineScoringSpanUserDefinedMetricPythonScorer

      • Add span-level Python scorer service that consumes Redis stream messages
      • Extend OnlineScoringEngine to support span context (input, output, metadata)
      • Fix OnlineScoringSpanSampler to fetch both LLM and Python span evaluators
      • Add service toggle for span Python evaluator
      • Add stream configuration for span Python scoring
      • Add comprehensive unit tests and integration tests

    • Frontend: Add UI support for span-level Python code rules

      • Add span_python_code rule type to types and schemas
      • Update AddEditRuleDialog to support span Python code rules
      • Add feature toggle for span Python code metrics
      • Update RunEvaluationDialog to include span Python code rules
      • Add default Python code template for span scope
      • Update helper functions for rule type/scope mapping
    • [OPIK-3211] Fix migration files: restore 000037 and update 000038
    • Restore migration 000037_add_scope_to_automation_rules.sql from main branch
      (it adds span_llm_as_judge evaluator type)
    • Update migration 000038 to only add span_user_defined_metric_python type
      (span_llm_as_judge is already added by 000037)
    • [OPIK-3211] Move toggle checks before fetching evaluators in OnlineScoringSpanSampler
    • Check both SPAN_LLM_AS_JUDGE and SPAN_USER_DEFINED_METRIC_PYTHON toggles before fetching evaluators
    • Prevents unnecessary database queries when features are disabled
    • Update tests to explicitly verify toggle behavior for both evaluator types
    • Add test to verify Python evaluators are not fetched when toggle is disabled
    • [OPIK-3210] Remove filter normalization and add CUSTOM field evaluation
    • Remove input_json/output_json normalization logic from frontend
    • Backend now stores filters in same format as sent by frontend
    • Add CUSTOM field handling in filter evaluation services
    • Convert CUSTOM fields only when evaluating filters, not when storing
    • Add comprehensive tests for CUSTOM field evaluation
    • [OPIK-3210] Add JSON path support with array indices for CUSTOM filters
    • Update extractNestedValue to handle JSON paths with array indices
    • Support both bracket notation (messages[0].content) and dot notation (messages.0.content)
    • Add tests for array index path navigation
    • Fixes issue where filters like 'input.messages[0].content' were not working

    • [OPIK-3210] Fix frontend linting errors in schema.ts

    • refactor: Replace manual JSON traversal with JSONPath library

    • Replace custom navigateJsonPath() method with Jayway JSONPath library
    • Simplify extractNestedValue() to use JsonPath.read() with configuration
    • Add JSON_PATH_CONFIG with options for null handling and exception suppression
    • Remove ~115 lines of manual path parsing code
    • Maintains backward compatibility with existing path formats
    • All 110 AutomationRuleEvaluatorsResourceTest tests passing
    • fix: Correct constructor parameter order and getFilters() access in span Python evaluator
    • Fix constructor in AutomationRuleEvaluatorSpanUserDefinedMetricPython to match parent signature (filters before code)
    • Change super.filters to super.getFilters() in both evaluator and update classes
    • All 110 AutomationRuleEvaluatorsResourceTest tests passing

    • Fix it

    • [OPIK-3211] Address PR review comments

    • Rename misleading variable spanLevelLlmAsJudgeRules to traceLevelLlmAsJudgeRules in ManualEvaluationService
    • Fix incorrect getStream() method signature in OnlineScoringSpanUserDefinedMetricPythonScorerTest
    • Remove duplicate key in AddEditRuleDialog DEFAULT_PYTHON_CODE_DATA object

    • [OPIK-3211] [BE] Fix JSONPath numeric dot notation support for array indices

    • Fix logs

    • [OPIK-3211] [BE] Address PR review comments: optimize JSONPath usage and improve error handling

    • [OPIK-3211] [BE] Fix blank key handling in JSONPath normalization

    Co-authored-by: Andres Cruz andresc@comet.com

    下载附件