github-featured / comet-ml--opik

发布

0 发布 8,340 Git 标签

RSS 订阅源
  • 1.9.38-3559 98fb4af064
    比较

    [OPIK-2982] [P SDK] [BE] [FE] ChatPrompt support (#3987)

    frostbyte_neo 发布于 2025-12-04 16:48:31 +00:00

    • [OPIK-2982] Implement ChatPrompt class across SDK, Backend, and Frontend
    • SDK: Add ChatPrompt class using ChatPromptTemplate, serialize to JSON with template_structure='chat'
    • Backend: Add template_structure field to PromptVersion model, create migration script
    • Frontend: Add template_structure column to prompts table, implement chat prompt creation UI with message-by-message editor, add ChatPromptView component for displaying chat prompts
    • Tests: Add unit and e2e tests for ChatPrompt functionality

    • Revision 2: Update Fern-generated OpenAPI client code for template_structure field

    • Revision 3: Enhance chat prompt UI and add example script

    • Improve ChatPromptView with modern chat interface design
    • Add circular avatar icons with role-specific colors
    • Enhance hover effects and visual hierarchy
    • Update PromptVersion type to include template_structure field
    • Add create_chat_prompt_example.py script demonstrating ChatPrompt usage
    • Revision 4: Add verify_chat_prompt_version verifier and improve backend property consistency
    • Add verify_chat_prompt_version() function to verify ChatPrompt properties
    • Verify template_structure field is set to 'chat' for ChatPrompt instances
    • Update test_chat_prompt.py to use the new verifier
    • Initialize internal backend properties to None in both Prompt and ChatPrompt init
    • Import ChatPrompt in verifiers module
    • Revision 5: Use verify_chat_prompt_version consistently across all tests
    • Remove template_structure parameter requirement from verify_chat_prompt_version
    • Update all 10 chat prompt tests to use verify_chat_prompt_version
    • template_structure is now always verified to be 'chat' (no need to pass it)
    • Replace manual assertions with verifier calls for consistency

    • Refactor e2e tests for chat prompt

    • Revision 6: Remove temporary workaround and simplify prompt deduplication logic

    • Remove temporary inference logic for template_structure field
    • Simplify comparison logic to directly use template_structure from API response
    • Backend now properly returns template_structure after regenerating OpenAPI code

    • Revision 3: Fix frontend display of chat prompts - add @JsonView(Prompt.View.Public.class) to PromptVersion fields

    • Revision 2: Optimize prompt queries by removing latest_version complexity

    • Remove latest_version subqueries from PromptDAO find() and findById()
    • Remove conditional logic from PromptVersionColumnMapper
    • Update retrievePromptVersion to fetch latest version directly from prompt_versions table
    • Update frontend to use versions[0] instead of latest_version.id

    This simplifies the backend and improves query performance by eliminating
    unnecessary JSON object construction in SQL queries.

    • Revision 3: Update OpenAPI spec and regenerate REST API clients
    • Add template_structure field to Prompt model in OpenAPI spec
    • Regenerate Python SDK REST API client code with Fern
    • Regenerate TypeScript SDK REST API client code with Fern

    This includes all the type definitions, serialization, and client methods
    needed to support the template_structure field in both SDKs.

    • Revision 4: Add template_structure field to backend Prompt model
    • Add TemplateStructure enum (string/chat) in backend API
    • Add template_structure field to Prompt model with proper @JsonView annotations
    • Update PromptVersion to hide template_structure from Public view (internal field)
    • Add TemplateStructureArgumentFactory for JDBI enum handling
    • Add Liquibase migration to add template_structure column to prompts table

    This establishes the core backend support for differentiating between
    string and chat prompts at the Prompt level (not PromptVersion level).

    • Revision 5: Implement ChatPrompt class and template_structure in Python SDK
    • Add template_structure property to Prompt class (always 'string')
    • Implement ChatPrompt class with messages array support
    • Update PromptClient to send template_structure when creating prompts
    • Add JSON comparison for chat prompts to avoid formatting differences
    • Add verify_chat_prompt_version() to E2E test verifiers

    The ChatPrompt class serializes messages to JSON for backend storage
    and sends template_structure='chat' during creation.

    • Revision 6: Add template_structure display in frontend prompt list
    • Add template_structure field to Prompt TypeScript interface
    • Display template_structure column in prompts list (Chat/String)
    • Add type column with proper formatting and styling

    The frontend now displays whether each prompt is a 'Chat' or 'String'
    type based on the template_structure field from the backend.

    • Revision 7: Remove supported_modalities parameter from ChatPrompt.format()
    • Remove supported_modalities parameter from ChatPrompt.format() method
    • Update unit tests to remove supported_modalities usage
    • Update e2e tests to remove supported_modalities usage
    • Rename test functions to reflect they test multimodal content, not modalities

    This simplifies the ChatPrompt API by removing the unused
    supported_modalities parameter.

    • Revision 8: Remove internal_sandbox file from git tracking
    • Remove create_chat_prompt_example.py from git (should not be committed)
    • File remains locally but is no longer tracked by git

    • Refactor ChatPrompt and tests for it, fix python linter

    • Add import to prompt playground

    • Remove obsolete local migration

    • Revision 6: Fix save chat prompt icon and add template structure filtering to save dialog

    • Revision 7: Add dynamic search placeholders for prompt dropdowns

    • Revision 8: Improve template structure validation error handling in backend

    • Revision 9: Add backend validation for template structure immutability and e2e tests

    • Revision 10: Move imports to module top in test_prompt.py

    • Revision 11: Use module import for ApiError in test_prompt.py

    • Revision 12: Refactor prompt module structure

    • Create string/ and chat/ subnamespaces for better organization
    • Move Prompt and PromptTemplate to string/ subnamespace
    • Move ChatPrompt, ChatPromptTemplate, and content renderer registry to chat/ subnamespace
    • Rename chat_content_renderer_registry.py to content_renderer_registry.py
    • Update all imports to use module imports instead of direct name imports
    • Maintain backward compatibility by re-exporting classes from main __init__.py
    • Update evaluator.py to use new import structure
    • Revision 13: Simplify prompt module structure
    • Make string/__init__.py and chat/__init__.py empty
    • Only api_objects.prompt.__init__.py has content
    • Import names directly (not modules) in main __init__.py
    • Update all to include: PromptType, Prompt, ChatPrompt, PromptTemplate, ChatPromptTemplate
    • Revision 14: Fix unit test imports for PromptTemplate
    • Update test_prompt_template.py to import PromptTemplate directly
    • Remove module-level imports in favor of class imports
    • All unit tests now pass with simplified import structure
    • Revision 15: Add comprehensive unit tests for ChatPromptTemplate
    • Test basic text message formatting with variables
    • Test multiple messages with different roles
    • Test multimodal content (text + image)
    • Test Mustache and Jinja2 template types
    • Test Jinja2 control flow and loops
    • Test required_modalities detection
    • Test supported/unsupported modalities handling
    • Test placeholder replacement for unsupported modalities
    • Test edge cases: empty content, missing roles, empty messages
    • Test template variable overrides
    • 20 comprehensive tests covering all functionality
    • Revision 16: Improve test assertions with structure comparison
    • Replace multiple individual assertions with direct structure comparison
    • Use assert result == expected_structure pattern for cleaner tests
    • Makes tests more maintainable and easier to read
    • All 20 tests still pass
    • Revision 17: Fix imports in test_tracker_outputs after prompt module refactoring
    • Update prompt.prompt.to_info_dict to prompt.to_info_dict
    • Aligns with the new prompt module structure where to_info_dict is exported at the top level
    • Revision 18: Add helpers.py with to_info_dict function
    • Extract to_info_dict helper function to separate module
    • Supports both Prompt and ChatPrompt objects
    • Exports from main __init__.py for backward compatibility
    • Revision 19: Add placeholder validation to ChatPromptTemplate

    Implementation:

    • Add validate_placeholders parameter to ChatPromptTemplate.init
    • Implement _extract_placeholders method to extract placeholders from all messages
    • Add _extract_placeholders_from_string helper function
    • Validate placeholders against format arguments in format() method
    • Only validate for Mustache templates (not Jinja2)

    Tests:

    • test_chat_prompt_template__format__passed_arguments_not_in_template__error_raised
    • test_chat_prompt_template__format__some_placeholders_missing__error_raised
    • test_chat_prompt_template__format__placeholders_mismatch_both_ways__error_raised
    • test_chat_prompt_template__format__multimodal_placeholders__validates_all
    • test_chat_prompt_template__format__multiple_messages_placeholders__validates_all
    • test_chat_prompt_template__format__validation_disabled__no_error
    • test_chat_prompt_template__format__jinja2_no_validation

    All 27 tests pass (20 original + 7 new validation tests)

    • Revision 20: Implement base classes for prompts and templates

    Added BasePromptTemplate:

    • Abstract base class for PromptTemplate and ChatPromptTemplate
    • Defines common format() interface

    Added BasePrompt:

    • Abstract base class for Prompt and ChatPrompt
    • Defines common properties: name, commit, metadata, type, template_structure
    • Defines common format() method interface

    Updates:

    • PromptTemplate now inherits from BasePromptTemplate
    • ChatPromptTemplate now inherits from BasePromptTemplate
    • Prompt now inherits from BasePrompt
    • ChatPrompt now inherits from BasePrompt
    • helpers.py updated to use BasePrompt type hint
    • Both base classes exported from main __init__.py

    Benefits:

    • Improved type safety and IDE support
    • Clear interface contracts for prompt implementations
    • Enables polymorphic code that works with any prompt type
    • Better documentation of expected behavior

    All 36 unit tests pass and inheritance verified

    • Revision 21: Split base class and add to_info_dict() to classes
    • Split base.py into base_prompt.py and base_prompt_template.py
    • Removed helpers.py module
    • Added to_info_dict() method directly to Prompt and ChatPrompt classes
    • Removed template_structure from BasePrompt abstract properties
    • Added internal API fields to BasePrompt: internal_api__prompt_id and internal_api__version_id
    • Updated imports in all files to use new base_prompt and base_prompt_template modules
    • Updated opik_context.py and test_tracker_outputs.py to call to_info_dict() as instance method

    All 36 unit tests pass

    • Revision 22: Add @override decorators to all overridden methods
    • Added typing.override import to all prompt classes
    • Added @override decorator to all methods that override base class methods:
      • Prompt: name, commit, metadata, type, format, to_info_dict
      • ChatPrompt: name, commit, metadata, type, format, to_info_dict
      • PromptTemplate: format
      • ChatPromptTemplate: format

    All 36 unit tests pass

    • Clean up the prompt related classes

    • Revision 23: Update Experiment and evaluate functions to use BasePrompt

    • Changed Experiment.init to accept List[BasePrompt] instead of List[Prompt]
    • Updated evaluate() to accept BasePrompt for prompt and prompts parameters
    • Updated evaluate_prompt() to accept BasePrompt for prompt parameter
    • Updated evaluate_optimization_trial() to accept BasePrompt for prompt and prompts parameters
    • Updated experiment helpers to use BasePrompt:
      • build_metadata_and_prompt_versions() now accepts List[BasePrompt]
      • handle_prompt_args() now accepts and returns BasePrompt types
      • Updated logic to handle both Prompt (template) and ChatPrompt (messages) using to_info_dict()
    • Changed imports from 'from .. import prompt' to 'from ..prompt import base_prompt'
    • Changed imports from 'from .. import Prompt' to 'from ..api_objects.prompt import base_prompt'

    This allows both Prompt and ChatPrompt to be used interchangeably in experiments and evaluations.

    All 36 unit tests pass

    • Revision 24: Update Opik.create_experiment to use BasePrompt
    • Changed create_experiment() parameters from:
      • prompt: Optional[prompt_module.Prompt]
      • prompts: Optional[List[prompt_module.Prompt]]
        To:
      • prompt: Optional[prompt_module.base_prompt.BasePrompt]
      • prompts: Optional[List[prompt_module.base_prompt.BasePrompt]]

    This allows both Prompt and ChatPrompt to be passed to create_experiment().

    All 36 unit tests pass

    • Disable prompt placeholders validation in ChatPromptTemplate

    • Propage validate_placeholders variable

    • Revision 25: Fix experiment helper tests for BasePrompt

    • Updated fake_prompt() in test_helpers.py to include to_info_dict() method
    • The method returns a dict with 'name' and 'version' containing 'template'
    • This matches the structure expected by build_metadata_and_prompt_versions()

    All 11 experiment helper tests pass

    • Remove redundant fields

    • Revision 26: Fix TypeScript build error and remove unused fields

    • Fixed missing LLM_MESSAGE_ROLE import in useLoadPlayground.ts
    • Removed unused _template_structure fields from Prompt and ChatPrompt classes
    • All tests passing (36 prompt unit tests)
    • Frontend builds successfully in production mode

    • Rename to_info_dict to internal api notation

    • Revision 27: Add backend template_structure parameter for prompt filtering

    • Added template_structure parameter to PromptVersionRetrieve model
    • Updated PromptService to accept and validate template_structure parameter
    • Defaults to 'string' for backward compatibility when not specified
    • Backend returns 400 error when template_structure doesn't match
    • Updated Python SDK to pass template_structure to backend API
    • Simplified get_prompt/get_chat_prompt to rely on backend validation
    • Regenerated OpenAPI client code for Python and TypeScript SDKs
    • Revision 28: Optimize prompt history methods to use backend validation
    • Updated get_prompt_history() to validate prompt type using backend first
    • Updated get_chat_prompt_history() to validate prompt type using backend first
    • Removed client-side template_structure checks after fetching all versions
    • Removed ValueError raises - methods now return empty list for wrong type
    • Backend validates template_structure and returns None for mismatches
    • Improves performance by failing fast on wrong prompt type
    • Revision 29: Add backend OQL filter support for template_structure
    • Added TEMPLATE_STRUCTURE_QUERY_PARAM to Field interface
    • Added TEMPLATE_STRUCTURE field to PromptField enum
    • Updated search_prompts() to use OQL filter 'template_structure = "string"'
    • Updated search_chat_prompts() to use OQL filter 'template_structure = "chat"'
    • Removed client-side template_structure filtering (backend now handles it)
    • Backend filters prompts at database level for better performance

    • Implement twin methods for chat prompt similar to string prompt ones

    • Fix lint errors

    • Merge search methods for caht and string promtps into one, change default value for template_structure to null in retrieve prompt version endpoint

    • Fix lint errors

    • Fix backend error with ambiguous method

    • Update unit tests

    • Fix lint errors for FE

    • Add new BE tests

    • Remove template_structure from prompt version table and some methods

    • Fix lint errors

    • Revision 2: Fix template_structure handling in Python SDK and regenerate REST API clients

    • Move template_structure from PromptVersion object to CreatePromptVersion parameter
    • Regenerate Python and TypeScript REST API clients with updated OpenAPI spec
    • Fix Python SDK client to pass template_structure as separate parameter
    • All 40 e2e tests now passing (test_prompt.py and test_chat_prompt.py)
    • Revision 3: Fix frontend prompt handling
    • Fix Python code snippet: use get_chat_prompt() instead of get_prompt() for chat prompts
    • Add template_structure parameter to prompt version mutation API
    • Pass template_structure when updating chat prompt versions from UI

    This fixes the issue where updating chat prompts from the UI would fail
    because template_structure parameter was missing in the API request.

    • Revision 4: Fix Playground chat prompt saving
    • Add templateStructure parameter to AddNewPromptVersionDialog when creating new version
    • This fixes the issue where saving chat prompts from Playground would fail
    • Now passes template_structure='chat' to the backend API correctly
    • Revision 5: Fix Playground prompt saving issues
    • Remove hardcoded filterByTemplateStructure in LLMPromptMessageActions
    • Fix chat prompt data persistence after loading in Playground
    • Remove reset of selectedChatPromptId after loading chat prompt

    This ensures that:

    1. When you load a chat prompt and save it, the dialog correctly shows
      the loaded prompt and offers to create a new version
    2. The prompt name is pre-filled with the loaded prompt name
    3. Chat prompts are correctly identified by template_structure

    • Improve import/export in playground

    • Revision 6: Persist loaded chat prompt ID in Playground store

    • Add loadedChatPromptId field to PlaygroundPromptType
    • Store selected chat prompt ID in persisted Zustand store instead of component state
    • This ensures the 'Load chat prompt' dropdown displays the selected prompt name
      even after re-renders or page refreshes

    Fixes issue where dropdown showed 'Load chat prompt' placeholder instead of
    the currently loaded prompt name.

    • Revision 7: Set loadedChatPromptId when loading chat prompt via 'Try in Playground'
    • Add loadedChatPromptId to newPrompt when templateStructure is 'chat'
    • This ensures the dropdown displays the loaded prompt name when using
      'Try in Playground' button from the prompt library

    Fixes issue where clicking 'Try in Playground' loaded the prompt content
    but didn't update the dropdown to show the prompt name.

    • Fix backend tests

    • Revision 8: Fix chat prompt tests and add video modality support

    • Disable validate_placeholders by default in ChatPromptTemplate (False instead of True)
    • Add render_video_url_part function for video content support
    • Register video_url renderer with video modality and placeholder
    • All 14 tests in test_message_renderer.py now passing

    This completes the merge conflict resolution from main branch that added
    video modality support to the evaluation models.

    • Add missing video rendering logic, add supported_modalities parameter to format method

    • Fix lint errors

    • Fix dspy test

    • Update prompt tempalte tests to follow new default validation policy

    • Change default validation arg value

    • Post merge cleanup

    • Fix unit test for template

    • Update text to Sentence case

    • Refactor all occurencies of string template_structure to text template_structure

    • Fix lint errors

    • Improve PromptEdit dialog

    • Refactor chatprompt raw view

    • Add gap 0.5 to edit prompt version dialog

    • Update prompt messages view

    • Update prompt views

    • Fix load prompt button in playground

    • Fix lint errors

    • Update the code according to rules

    • Add missing file and rename migrations file

    • Update table columnslogic for prompt library

    • Make ChatPrompt support ALL modalities by default unless specified directly

    • Update autogenerated code

    • Rename string -> text occurencies in prompt docstrings in the SDK

    • Fix import

    • Fix: Remove unused parseContentWithImages import

    • Bump prefix number in template_structure migration

    • Fix: Update ChatPromptMessageReadonly and useLoadPlayground to use new media components and handle MessageContent type

    • Replace PromptMessageImageTags with PromptMessageMediaTags
    • Replace parseContentWithImages with parseLLMMessageContent
    • Add video support to ChatPromptMessageReadonly
    • Use getTextFromMessageContent to handle MessageContent type in useLoadPlayground

    • Fix: Prettier formatting for imports and destructuring

    • Fix review comments

    • Display attachments

    • Fix lint errors

    • Update loading messages logic

    • Update the type hint in opik context

    • Fix lint errors

    • Rename messages to template

    • Update migration script to follow the guidelines

    • Remove redundant factory for template_structure

    • Update BE tests to not use custom podam factory and use explicit template structure setting instead. + start using builders where possible

    • Remove template_structure from the prompt version

    • Regenerate openapi client code

    • If template structure is unknown - use TEXT

    • Add JsonCreator decorator to fromString method of TemplateStructure

    • Revert unnecessary @JsonView formatting changes in PromptVersion

    • Remove templateStructure field (not needed in PromptVersion)
    • Revert @JsonView scope and formatting changes back to original
    • Keep only essential changes for clean PR

    Addresses review feedback about unnecessary @JsonView changes

    • Start using lombok.Getter instead of raw getter

    • Remove constructor boilerplate

    • Follow DRY principle for default template structure value

    • Remove template structure from prompt version response payload, remove validation too

    • Revert template validation

    • retrievePromptVersion now handles template_structure comparison on the DB level

    • Update client code in the SDK

    • Address SDK comments

    • Update migration script name

    • Make backend unaware of template_structure validation logic when new version is created or the version is retrieved

    • Document the logic about template_structure parameter during the prompt version creation

    • Implement client-side prompts validation

    • Update test to make sure prompts remain untouched when errors are raised because of mismatch

    • Fix lint errors

    • Add missing migration script

    • Replace all raw string occurencies of text and chat with enum

    • Replace useEffect with callback for processing raw json chat prompt content

    • Address AddEditPromptDialog comment

    • Implement reusable parseChatTemplateToLLMMessages function

    • useMemo for chatPromptTemplate in PlaygroundPrompt

    • Address race condition in playgroundprompt.tsx

    • Resolve one more race condition in playground prompt

    • Add indentations for promptTemplates

    • Use memo in for attachments in ChatPromtpMessageReadonly.tsx

    • Move unchanged values to const

    • Move value guard from handleImportChatPrompt to the caller code

    • Rename migration script

    • Extract ChatPromptAttachments to a separate component

    • Remove unnecessary memo

    • Re-generate rest api fern code

    • Fix messages borders

    • Fix validation logic and add a red dot to the save button to indicate that there are unsaved changes

    • Fix some linting errors

    • Improve logic to detect unsaved changes for chat prompt in playground

    • Fix lint errors

    • Renamed migration scripts

    • Re-generate openapi spec

    • Fix the issue with inconsistent chat prompt saving behavior

    • Fix linter

    • regenerate openapi spec

    • Remove text prompt invalidation on saving logic

    • Fix cache validation in playground

    • Fix lint errors

    • Fix prompt query invalidation

    • Update autogenerated code

    Co-authored-by: Andres Cruz andresc@comet.com

    下载附件