github-featured / comet-ml--opik

发布

0 发布 8,340 Git 标签

RSS 订阅源
  • 2.0.27-5285 34cf7a1d9a
    比较

    [OPIK-5745] [BE] feat: add LLM tools for full trace context in eval suite assertions (#6511)

    frostbyte_neo 发布于 2026-05-08 12:16:08 +00:00

    • [OPIK-5745] [BE] feat: add LLM tools for full trace context in eval suite assertions

    Eval suite assertions can now inspect sub-span behavior (tool calls, LLM model,
    intermediate steps) via two on-demand LLM tools: get_trace_spans and
    get_span_details. Tools are only offered when experimentId is present (test suite
    assertions); regular online scoring is unchanged.

    Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com

    • [OPIK-5745] [BE] refactor: improve system prompt tool usage guidelines for eval suite assertions

    Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com

    • fix: preserve ChatRequest parameters when adding tool specs

    Removing .parameters(null) from addToolSpecs() — it was stripping
    maxCompletionTokens which Anthropic requires, causing
    "maxCompletionTokens cannot be null" errors for Claude models.

    Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com

    • [OPIK-5745] [BE] fix: address PR review - use InstructionStrategy for assertions, fix ChatRequest builder conflict
    • Force InstructionStrategy for test suite assertions so format instructions
      survive responseFormat stripping when adding tool specs
    • Fix "Cannot set both parameters and toolSpecifications" by building fresh
      ChatRequest instead of using toBuilder()
    • Guard handleToolCalls with experimentId != null check
    • Use constants for tool names in system prompt
    • Reduce tool call logging to debug level

    Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com

    • [OPIK-5745][BE] Refactored trace span tools and introduced modular execution framework.
    • Removed TraceSpanToolDefinition and transitioned to a modular setup for LLM-as-judge tools.
    • Introduced ToolRegistry, ToolExecutor interface, and dedicated tool implementations (GetTraceSpansTool, GetSpanDetailsTool, ReadTool) for improved extensibility and maintainability.
    • Added infrastructure for tool execution contexts (TraceToolContext) and entity compression (GenericCompressor, PathAwareTruncator, Tokens, etc.).
    • Refined logic for trace/span serialization with truncation markers and jq path annotations.
    • Updated core application wiring and tests to support the new tool-based architecture.
    • [OPIK-5745][BE] Introduced test coverage and input validation for LLM-as-Judge scoring.
    • Added OnlineScoringLlmAsJudgeScorerTest to validate tool-enabled scoring logic and regression guard for structured requests.
    • Introduced OnlineScoringEngineCapReplacementsTest to test variable substitution with input capping logic.
    • Enhanced OnlineScoringEngine with a capped variable substitution path for large trace inputs, improving token efficiency and leveraging tool-based retrieval for full data.
    • [OPIK-5745][BE] Add JqTool for evaluating jq expressions on cached entities.
    • Introduced JqTool to execute jq expressions on cached JSON entities (e.g., spans, traces, datasets) using jackson-jq.
    • Registered JqTool in ToolsModule and updated TestSuitePromptConstants to include usage guidance.
    • Implemented argument validation, output capping, and error handling for malformed or runtime errors in expressions.
    • Added comprehensive unit tests for JqTool to validate spec adherence, expression handling, and caching behavior.
    • Updated pom.xml to include jackson-jq dependency.
    • [OPIK-5745][BE] Add SearchTool for regex-based searches in cached entities.
    • Implemented SearchTool to perform case-insensitive regex searches on cached JSON entities (e.g., spans, traces).
    • Registered SearchTool in ToolsModule and TestSuitePromptConstants with detailed usage guidance.
    • Added argument parsing, output capping at 50 matches, value truncation, and robust error handling for malformed inputs.
    • Developed comprehensive unit tests covering pattern matching, path scoping, and edge cases.
    • Introduced functionality to return matched paths and truncated values for follow-up extraction with JqTool.
    • [OPIK-5745][BE] Introduce shouldUseTools utility to centralize tool path enablement logic.
    • Replaced repetitive experimentId checks with shouldUseTools for improved code readability and maintainability.
    • Added JavaDoc to clarify tool path scope and future extensibility.
    • Updated OnlineScoringLlmAsJudgeScorer to leverage the utility in tool-specific conditions.
    • [OPIK-5745][BE] Add builder pattern to CompressionResult and refactor compressors.
    • Annotated CompressionResult with @Builder to enable builder pattern and immutability.
    • Refactored GenericCompressor, TraceCompressor, and DatasetCompressor to use CompressionResult.builder() for object instantiation.
    • [OPIK-5745][BE] Extract and centralize span hierarchy logic into SpanHierarchy.
    • Consolidated span tree-building logic from SpanTreeSerializer and TraceCompressor into a shared SpanHierarchy utility for reusability and consistency.
    • Added unit tests in SpanHierarchyTest to validate root promotion, child attachment, and edge cases.
    • Refactored SpanTreeSerializer and TraceCompressor to leverage SpanHierarchy.toTree for span tree generation.
    • [OPIK-5745][BE] Add sticky cache truncation warnings and downgrade tier for capped entries.
    • Modified ReadTool to persist cache_warning on subsequent reads of previously truncated entities.
    • Updated applyCacheCap to handle ref and ctx, marking truncated entries for consistent behavior.
    • Introduced effectiveTier to downgrade compression tier to MEDIUM when cache_warning is present.
    • Extended TraceToolContext with tracking for capped entries (markTruncated, isTruncated).
    • Added unit tests to validate persistent warnings and tier downgrades for capped entities.
    • [OPIK-5745][BE] Enhance truncation logic with configurable suffix styles.
    • Introduced PathAwareTruncator.SuffixStyle to support WITH_JQ_HINT and BARE options for truncation markers.
    • Updated GenericCompressor and TraceCompressor to accept custom suffix styles for truncated output.
    • Modified ReadTool to dynamically select suffix styles based on cache state, ensuring consistent output for capped entries.
    • Extended unit tests to validate behavior with both suffix styles and scenarios involving cache capping.
    • Refactored and documented truncation logic to clarify suffix behaviors and usage contexts.
    • [OPIK-5745][BE] Improve cache capping with fallback thresholds and enhanced truncation logic.
    • Added multi-tier fallback thresholds (CAP_FALLBACK_LIMITS) to gracefully handle oversized cached entities.
    • Replaced fixed truncation with dynamic fallback logic in fitWithinCap to prioritize caching highest-fidelity data within memory limits.
    • Simplified redundant cache pre-checks to ensure consistent behavior when updating cached entries.
    • Extended unit tests to validate multi-tier fallback scenarios and ensure capped entities respect CACHE_CAP_CHARS.
    • [OPIK-5745][BE] Remove thread support and improve validation in tools.
    • Removed type=thread support in JqTool, SearchTool, and ReadTool due to unsupported caching.
    • Enhanced error messages for invalid thread-type usage and updated argument validation.
    • Fixed logging messages for clarity and consistency.
    • Updated ReadToolTest to reflect thread rejection with improved error text.
    • [OPIK-5745][BE] Extract shared argument handling and cache utilities to ToolArgs.
    • Moved common argument parsing and cache utilities from JqTool, SearchTool, and ReadTool to a new ToolArgs utility class for reusability and consistency.
    • Removed redundant methods like textOrNull, errorJson, and capWithHint from individual tools and replaced them with centralized counterparts in ToolArgs.
    • Updated tools to leverage ToolArgs for cleaner and more maintainable code.
    • Simplified unit tests by aligning with shared utility behaviors.
    • [OPIK-5745][BE] Add withTools support to test suite prompts and scoring logic.
    • Introduced withTools flag to conditionally include tool-teaching addendum in system prompts.
    • Refactored TestSuitePromptConstants with methods to handle dynamic prompt generation.
    • Updated EvaluatorMapper, TestSuiteAssertionSampler, and scorers to propagate withTools configuration.
    • Added LlmAsJudgeToolsMode utility to centralize tool path enablement logic.
    • Ensured consistent application of tool-teaching logic across test suite evaluation flows.
    • [OPIK-5745][BE] Centralize truncation logic and enhance tooling support.
    • Extracted shared truncation logic into a new utility class TruncationMarker for consistency across the codebase.
    • Updated OnlineScoringEngine, PathAwareTruncator, and related tools to use TruncationMarker for unified string truncations.
    • Improved logging and error handling in ReadTool with correlation IDs for secure debugging.
    • Applied builder pattern to ParsedArgs and CacheOutcome for cleaner object construction and immutability.
    • Refactored suffix-handling logic in PathAwareTruncator for modularity.
    • [OPIK-5745][BE] Rename TruncationMarker to StringTruncator and refactor truncation logic.
    • Renamed TruncationMarker to StringTruncator for clearer intent and consistency.
    • Refactored shared truncation methods (apply to truncate) for improved readability and usage.
    • Updated SpanTreeSerializer, DatasetCompressor, and related tools to use StringTruncator.truncate for centralization.
    • Removed redundant local truncation implementations in favor of the utility method.
    • [OPIK-5745][BE] Improve exception logging in ReadTool deserialization.
    • Updated log.warn to include full exception details when deserialization fails for better debugging.
    • [OPIK-5745][BE] Replace per-tool truncation with centralized StringTruncator.
    • Updated JqTool and SearchTool to use StringTruncator.truncate for string truncation.
    • Removed capWithHint from ToolArgs in favor of centralized truncation logic.
    • Added a new separator option to StringTruncator.truncate for improved flexibility.
    • [OPIK-5745][BE] Add debug logging for tool execution and cache behavior.
    • Enhanced debugging in JqTool, SearchTool, ReadTool, and GetTraceSpansTool with detailed logs for valid arguments, cache misses, and entity truncation.
    • Added @Slf4j annotation to GetTraceSpansTool to support logging.
    • [OPIK-5745][BE] Add detailed debug logging for tool execution summaries.
    • Enhanced JqTool, SearchTool, and ReadTool with additional debug logs to capture key execution details.
    • Logged arguments, output size, truncation status, and other relevant summary metrics for improved traceability.
    • [OPIK-5745][BE] Refactor OnlineScoringLlmAsJudgeScorer to enforce structured LLM outputs.
    • Introduced explicit strategy to force JSON-only responses after tool usage.
    • Added UserMessage to clarify response expectations and prevent prose in final output.
    • Improved handling of tool-driven and structured LLM requests for consistency.
    • [OPIK-5745][BE] Enhance workspace name resolution and update prompt path conventions.
    • Added resolveWorkspaceName method to OnlineScoringLlmAsJudgeScorer for workspace name lookup with fallback behavior.
    • Updated TestSuitePromptConstants with detailed path conventions for tool usage and jq expressions.

    • [OPIK-5745][BE] Prioritize Anthropic for tool-driven LLM judge assertions; update fallback provider and documentation.

    • [OPIK-5745][BE] Update judge provider ordering rationale and enhance debug logging.

    • Refined SupportedJudgeProvider ordering explanation with additional empirical details and caveats.
    • Added detailed debug logs in GetTraceSpansTool to capture trace ID, span count, and output size for improved logging granularity.
    • [OPIK-5745][BE] Force initial tool engagement for OpenAI judges; adjust fallback behavior.
    • Updated addToolSpecs to use ToolChoice.REQUIRED on the first call, ensuring at least one tool interaction for OpenAI judges with tool-loop bias.
    • Refined follow-up behavior with ToolChoice.AUTO to allow models to decide when to stop investigating.
    • Enhanced test coverage to verify tool engagement and output propagation under forced and automatic modes.
    • Updated rationale in SupportedJudgeProvider comments to reflect empirical results and fallback adjustments.

    • [OPIK-5745][BE] Update model usage to GPT-4O-MINI and refine logging for LLM requests and responses.

    • [OPIK-5745][BE] Refine logging for debug and user-facing messages.

    • Updated debug logging in GetTraceSpansTool and ReadTool to improve clarity and uniformity.
    • Guarded userFacingLogger invocations in OnlineScoringLlmAsJudgeScorer with isInfoEnabled to prevent unnecessary computation for deferred logging.

    • [OPIK-5745][BE] Update log assertions in AutomationRuleEvaluatorsResourceTest to validate both refactored and legacy log formats.

    • [OPIK-5745][BE] Enforce safe jq path expressions in SearchTool and add corresponding test coverage.

    • Introduced regex validation to restrict path to simple jq scope expressions.
    • Updated ParsedArgs to reject unsafe path inputs.
    • Added tests to validate safe and unsafe path behaviors.
    • [OPIK-5745][BE] Handle tool execution exceptions gracefully; propagate fatal errors.
    • Updated ToolRegistry to return error JSON for exceptions while allowing fatal errors to propagate.
    • Added tests to verify behavior for runtime exceptions and fatal errors.
    • [OPIK-5745][BE] Refine debug logging in ReadTool for capped entity size.
    • Replaced inline toString().length() with a variable for improved clarity in log statements.
    • [OPIK-5745][BE] Decouple truncation warnings from capping logic in ReadTool.
    • Separated truncation flag handling from size-based capping decisions to ensure independence.
    • Updated cache warning behavior to surface only when truncation is flagged.
    • [OPIK-5745][BE] Extract common header generation logic into headerPrefix in SearchTool.
    • Refactored successHeader and errorHeader to share a common headerPrefix method for consistent header construction.
    • [OPIK-5745][BE] Add tests to validate handleToolCalls behavior for failure propagation and max round handling.
    • Verified error propagation for transient failures without masking issues.
    • Ensured max round cap triggers final structured call with proper contract adherence.
    • [OPIK-5745][BE] Avoid field loss regression in addToolSpecs by switching to toBuilder.
    • Replaced the manual builder with toBuilder to ensure all top-level fields in ChatRequest are preserved.
    • Prevents silent field omissions in addToolSpecs during tool specification updates.
    • [OPIK-5745][BE] Refactor error JSON generation in ToolRegistry.
    • Replaced inline formatted strings with ToolArgs.errorJson for consistency and maintainability in error responses.

    Co-authored-by: Claude Opus 4.6 noreply@anthropic.com
    Co-authored-by: Iaroslav Omelianenko yaric_mail@yahoo.com

    下载附件