github-featured / comet-ml--opik

发布

0 发布 8,340 Git 标签

RSS 订阅源
  • 1.9.13-3298 d201906b8b
    比较

    [OPIK-3049] [P SDK] Add support for PII information filtering (#4055)

    frostbyte_neo 发布于 2025-11-13 18:56:30 +00:00

    • [OPIK-3049] [P SDK] Introduce a text anonymization framework with rules-based abstraction
    • Added a framework for text anonymization, providing generic Rule, RegexRule, and FunctionRule implementations for customizable anonymization logic.
    • Introduced RulesAnonymizer to sequentially apply defined rules to anonymize text data.
    • Implemented a core Anonymizer class with support for recursive handling of nested structures (e.g., dicts, lists).
    • Added unit tests for various scenarios, including regex-based, function-based, and complex anonymization workflows.
    • Ensured safe and efficient recursion with a customizable depth limit and robust handling of edge cases like circular references.
    • [OPIK-3049] [P SDK] Add create_anonymizer factory function for flexible anonymization rule creation
    • Introduced create_anonymizer to handle multiple rule formats including dictionaries, tuples, functions, and lists.
    • Added robust validation for rule inputs with custom exceptions for invalid formats.
    • Included a comprehensive test suite covering various rule types, edge cases, and error handling scenarios.
    • [OPIK-3049] [P SDK] Refactor imports and extend anonymizer functionality
    • Consolidated imports in anonymizer modules to improve readability and maintain modularity.
    • Introduced anonymizer_hook for registering, clearing, and applying global anonymizers.
    • Refactored RulesAnonymizer and related factory code to align with the new import structure and naming conventions.
    • [OPIK-3049] [P SDK] Add has_anonymizers utility function
    • Introduced has_anonymizers to check if any anonymizers are registered.
    • [OPIK-3049] [P SDK] Refactor anonymizer imports and extend functionality with message anonymization
    • Updated import structure in anonymizer modules for consistency (factory, rules_anonymizer, rules).
    • Added message_anonymizer for anonymizing CreateSpanMessage and CreateTraceMessage inputs, outputs, and metadata fields.
    • Introduced unit tests for message_anonymizer covering diverse scenarios such as nested data, unsupported message types, and custom anonymization rules.
    • Refactored test cases in test_factory.py and test_rules_anonymizer.py to align with updated import paths and naming conventions.
    • [OPIK-3049] [P SDK] Refactor anonymizer structure
    • Extracted recursive anonymization logic to a new RecursiveAnonymizer base class for reusability.
    • Updated RulesAnonymizer to inherit from RecursiveAnonymizer, simplifying the inheritance structure.
    • Refactored anonymizer module to consolidate recursive handling and improve maintainability.
    • [OPIK-3049] [P SDK] Add encode_and_anonymize function with comprehensive tests
    • Introduced encode_and_anonymize function in jsonable_encoder to support field-level anonymization during encoding.
    • Added extensive unit tests to validate functionality, including scenarios with nested objects, various data types, and integration with custom anonymizers.
    • Ensured compatibility with existing encoding logic while enhancing flexibility for sensitive data handling.
    • [OPIK-3049] [P SDK] Extend anonymizer functionality to support field-specific metadata
    • Updated Anonymizer interface to accept additional kwargs, enabling more context-aware anonymization.
    • Enhanced jsonable_encoder to pass field names to anonymizers for improved control over field-specific anonymization.
    • Added has_anonymizers, apply_anonymizers, and add_anonymizer exports to hooks for streamlined integration of anonymizers.
    • Introduced new unit test scenarios to validate field-specific anonymization behavior within encode_and_anonymize.
    • [OPIK-3049] [P SDK] Enhance anonymization with fields_to_anonymize integration and refactor encode_and_anonymize
    • Introduced fields_to_anonymize static methods in message classes for centralized anonymization field definitions.
    • Refactored encode_and_anonymize to support multiple anonymizers and add context during anonymization.
    • Replaced jsonable_encoder calls with the new encode_and_anonymize function across message processing modules.
    • Added support for get_anonymizers in anonymizer hooks for streamlined anonymizer retrieval.
    • Extended unit tests in test_jsonable_encoder to cover multiple anonymizers and new context scenarios.
    • [OPIK-3049] [P SDK] Enhance anonymization with fields_to_anonymize integration and refactor encode_and_anonymize
    • Introduced fields_to_anonymize static methods in message classes for centralized anonymization field definitions.
    • Refactored encode_and_anonymize to support multiple anonymizers and add context during anonymization.
    • Replaced jsonable_encoder calls with the new encode_and_anonymize function across message processing modules.
    • Added support for get_anonymizers in anonymizer hooks for streamlined anonymizer retrieval.
    • Extended unit tests in test_jsonable_encoder to cover multiple anonymizers and new context scenarios.
    • [OPIK-3049] [P SDK] Added documentation describing anonymizers use in OPIK.
    下载附件