github-featured / comet-ml--opik

发布

0 发布 8,340 Git 标签

RSS 订阅源
  • 1.8.60-2828 25bbc4682e
    比较

    [OPIK-2431] Removing embedded attachments from traces at ingestion time (#3363)

    frostbyte_neo 发布于 2025-09-30 15:05:28 +00:00

    • Add proper configuration support for AttachmentStripperService
    • Add stripAttachmentsMinSize to config.yml with S3_STRIP_ATTACHMENTS_MIN_SIZE env var (default: 1000)
    • Replace lazy initialization with proper constructor-based pattern compilation
    • Regex pattern now compiled once during service construction, not on every use
    • Service logs initialization with configured minBase64Length for debugging
    • Update tests to use constructor injection properly
    • All tests passing with efficient configuration-driven approach
    • [OPIK-2431] Fix AttachmentStripperService multipart upload and context-aware filenames
    • Implement AttachmentStripperService to autodetect (and strip) base64 attachments in Traces
    • Checks for input/output/metadata
    • Add context parameter to prevent filename collisions (input-attachment-1.png vs output-attachment-1.png)
    • Add proper unit tests and integration

    • feat: Add attachment stripping service for base64 attachments in traces

    • refactor: Move batch attachment test from CreateTrace to BatchInsert class

    • style: Ensure all Java files end with empty lines

    • Minor optimization using StringBuilder

    • Spotless

    • Add OpenTelemetry metrics and enhanced error handling to AttachmentStripperService

    Key improvements:

    • Add comprehensive OpenTelemetry metrics for monitoring attachment processing

      • opik.attachments.processed: Track successful uploads
      • opik.attachments.skipped: Track skipped base64 strings
      • opik.attachments.errors: Track processing errors
      • opik.attachments.processing.duration.ms: Track processing time

    • Enhance error handling with specific exception types

      • Graceful degradation: Continue processing even when individual attachments fail
      • Differentiate between network errors and unexpected errors
      • Improved logging with context information

    • Update tests to support OpenTelemetry metrics integration

      • Add mock OpenTelemetry builders for testing
      • Maintain all existing test coverage

    • Simplify OpenTelemetry test mocking with noop() implementation

    • Updating otel support

    • Updating threshold to 5000 base64 chars

    • Add attachment management support for updates and performance optimization

    • Add span attachment support and fix AttachmentStripperServiceTest

    • Add AttachmentStripperService integration to SpanService for attachment stripping
    • Add attachment stripping to span creation and update methods
    • Add comprehensive tests for span attachment functionality in SpansResourceTest
    • Fix AttachmentStripperServiceTest to use OpikConfiguration mock properly
    • Add real-world base64 attachment test case

    • Updating configuration for easier local development (still misses minio keys configuration for local development)

    • Removing debugging

    • Cleaning up

    • Fixing situation where we could be deleting attachments before an update that fails.

    • Fix AttachmentStripperService tests after timestamped filename changes

    • Fix AttachmentStripperServiceTest to with the proper test and incorrectly commited code from next branch

    • Make attachment stripping async with Mono and boundedElastic scheduler

    下载附件