github-featured / comet-ml--opik

发布

0 发布 8,340 Git 标签

RSS 订阅源
  • 1.9.102-4017 46317ae567
    比较

    [OPIK-3897] [BE] Add structure-preserving JSON truncation for traces and spans (#4916)

    frostbyte_neo 发布于 2026-01-27 15:29:52 +00:00

    • [OPIK-3897] [BE] Add structure-preserving JSON truncation for traces and spans

    Add input_slim/output_slim columns that preserve JSON structure while truncating
    long string values. This improves frontend display of large traces/spans by
    maintaining key-value pairs instead of cutting off mid-JSON.

    Changes:

    • Add migration 000053 for input_slim/output_slim columns on traces and spans
    • Add TruncationUtils.createSlimJson() for recursive JSON structure preservation
    • Update TraceDAO and SpanDAO to populate slim columns during INSERT
    • Update ExperimentItemDAO, DatasetItemDAO, DatasetItemVersionDAO to use slim
      columns when truncate=true, with fallback to truncated_input/truncated_output
    • Add comprehensive tests for TruncationUtils
    • Revision 2: Address reviewer feedback on TruncationUtils
    • Return NullNode.getInstance() instead of raw null for null-safety
    • Pass exception to log.debug() instead of just the message for better observability
    • Revision 3: Update test to match NullNode return behavior

    Test now verifies that createSlimJson(null) returns NullNode
    instead of raw null, matching the code change from Revision 2.

    • Revision 4: Deduplicate null input tests with parameterized test

    • Revision 5: Address Thiago's review feedback

    • Use notEmpty() instead of length() > 0 for cleaner ClickHouse SQL
    • Remove truncated_input/output fallback in ingestion queries to avoid invalid JSON
    • Keep truncated fallback in retrieval queries for backward compatibility
    • Revision 6: Fix slim fallback for legacy rows without input_slim

    Only preserve old_span/trace.input_slim when both input AND input_slim
    are non-empty. This allows legacy rows with input but no input_slim to
    receive the new slim version on subsequent updates.

    • Revision 7: Consolidate JsonNode to String conversion in TruncationUtils
    • Add toJsonString() utility method for consistent null handling
    • Add createSlimJsonStringFromNode() convenience method
    • Replace Objects.toString(), ternary, and getOrDefault() with toJsonString()
    • Remove duplicate getOrDefault() method from TraceDAO
    • Revision 8: Extract bindInputOutputAndSlim helper in TraceDAO

    Centralizes input/output/metadata conversion and binding logic:

    • Computes JSON strings via TruncationUtils.toJsonString()
    • Creates slim versions via TruncationUtils.createSlimJsonString()
    • Handles both single insert and batch operations with index suffix
    • Removes unused Objects import
    • Revision 9: Rename bindInputOutputAndSlim to bindInputOutputMetadataAndSlim

    Method name now reflects that it also binds metadata. Updated Javadoc
    to clarify slim versions and index suffix usage.

    下载附件