-
[OPIK-2512] [FE/BE] Multimodal Image Use in LLMJ's and Online Evals (#3420)
frostbyte_neo 发布于
2025-10-16 07:29:20 +00:00 -
feat: added image support to llmj
-
chore: chat message with images
-
chore: playground supports images
-
chore: datatables supports base64 image preview
-
Update llm.ts
-
chore: images working in OAI
-
chore: finalize BE
-
chore: finalise FE
-
feat: image support to llmj evals in SDK
-
feat: java model capability flags
-
fix: FE
-
Update span_cost_upload_daily.yml
-
Update evaluator.py
-
chore: int
-
Update LLMPromptMessageActions.tsx
-
Update apps/opik-backend/src/main/java/com/comet/opik/utils/OpenAiMessageJsonDeserializer.java
Co-authored-by: Copilot 175728472+Copilot@users.noreply.github.com
- Update apps/opik-frontend/src/components/pages-shared/llm/LLMPromptMessages/LLMPromptMessage.tsx
Co-authored-by: Copilot 175728472+Copilot@users.noreply.github.com
-
docs: update
-
Update TryInPlaygroundButton.tsx
-
chore: lint/push
-
chore: visuals
-
chore: spacing
-
chore: ux
-
Update llm.ts
-
chore: patched
-
chore: python sdk changes
-
chore: tests for python
-
chore: frontend improvements and online eval
-
chore: java online eval and tests
-
test: coverage
-
chore: lint
-
test: online evals using langchain4j interface for image inference
-
fix: UX
-
fix: UX
-
fix: edit score flow improvements
-
Update OnlineScoringEngine.java
-
fix: serial input for online eval
-
chore: ux
-
fix: type checks to pass
-
revert: remove Python SDK image support changes
Remove incomplete Python SDK changes for local evaluation image support.
These changes will be addressed in a separate PR with proper implementation.Issues with removed code:
- supports_image_input() was a non-functional stub (returned True for any model)
- Unconditional flattening in create_scoring_inputs() broke vision model support
- Changes not needed for online evaluations (backend handles serialization)
- Requires different design for local SDK evaluations
Changes reverted:
- sdks/python/src/opik/evaluation/evaluator.py
- sdks/python/src/opik/evaluation/metrics/arguments_helpers.py
- sdks/python/src/opik/message_processing/__init__.py
- sdks/python/src/opik/message_processing/image_support.py (deleted)
- sdks/python/tests/message_processing/test_image_support.py (deleted)
This PR now focuses solely on online evaluations (frontend + backend).
🤖 Generated with Claude Code
Co-Authored-By: Claude noreply@anthropic.com
-
chore: lint
-
revert: restore Textarea in AddEditPromptDialog
Reverts CodeMirror changes in prompt library edit dialog that caused
whitespace and rendering issues. Returns to original Textarea implementation.🤖 Generated with Claude Code
Co-Authored-By: Claude noreply@anthropic.com
-
Update OnlineScoringEngine.java
-
Update ChatCompletionService.java
-
Delete ImageModelSupport.java
-
Update MessageContentNormalizer.java
-
Update ModelCapabilities.java
-
Update MessageContentNormalizer.java
-
chore: default lang
-
Update ModelCapabilities.java
-
Update LlmProviderAnthropicMapper.java
-
Update usePromptDatasetItemCombination.ts
-
Update MessageContentNormalizerTest.java
-
Update LlmProviderAnthropicMapper.java
-
Update ModelCapability.java
-
Update ModelCapabilities.java
-
Update MessageContentNormalizer.java
-
Update ChatCompletionService.java
-
Update CostService.java
-
Update OnlineScoringEngine.java
-
Update evaluate_your_llm.mdx
-
Update llm.ts
-
Update ModelCapabilities.java
-
Update OnlineScoringEngine.java
-
Update apps/opik-frontend/src/components/pages-shared/experiments/CompareExperimentsConfigCell/CompareExperimentsConfigCell.tsx
Co-authored-by: Copilot 175728472+Copilot@users.noreply.github.com
- Update apps/opik-frontend/src/components/pages/PlaygroundPage/PlaygroundOutputs/PlaygroundOutputTable/PlaygroundOutputCell.tsx
Co-authored-by: Copilot 175728472+Copilot@users.noreply.github.com
- Update apps/opik-frontend/src/components/pages/PlaygroundPage/PlaygroundOutputs/PlaygroundOutputTable/PlaygroundVariableCell.tsx
Co-authored-by: Copilot 175728472+Copilot@users.noreply.github.com
- Update apps/opik-frontend/src/components/pages/PlaygroundPage/PlaygroundOutputs/PlaygroundOutputTable/PlaygroundOutputTable.tsx
Co-authored-by: Copilot 175728472+Copilot@users.noreply.github.com
-
fix: reverse breaking change
-
refactor: seralizer to libs
-
chore: patch
-
fix: truncation
-
Update usePromptDatasetItemCombination.ts
-
Update PlaygroundVariableCell.tsx
-
Revert unrelated changes and add new FE implementation
-
- revert wrongly merged changes
-
fix prompt tab to show images
-
- fix explainers text
-
- resolve UI review comments
-
- fix useless triggers
-
- fix validation for filters
-
- add validation to allow saves images for models that support it
-
- add validation to allow run playground
-
Implement custom provider model check and integrate into image input support
-
Add test scripts
-
Mustache parsing quick workaround
-
feat: support Mustache unescaped syntax for images
- Add support for {{{variable}}} and {{&variable}} syntax in Mustache templates
- Update frontend parser to extract unescaped variables (token type '&')
- Remove global HTML escaping workaround that was a security risk
- Update CodeMirror plugin to highlight all Mustache syntax types
- Add comprehensive tests for unescaped syntax support
- Maintain security: only explicitly unescaped variables bypass HTML encoding
This allows proper handling of image URLs with special characters
while maintaining security for regular text variables.- docs: simplify Mustache syntax documentation
- Remove {{&variable}} syntax from UI documentation to reduce complexity
- Keep only {{{variable}}} syntax for unescaped variables in user-facing text
- Maintain technical support for both syntaxes in code
- Simplify tooltips and explainers for better user experience
-
Undo mustache patch
-
Revert "docs: simplify Mustache syntax documentation"
This reverts commit
18341c10a1.- Revert "feat: support Mustache unescaped syntax for images"
This reverts commit
a474212139.-
- cleanup code
-
revert documentation changes
-
Enhance ComparePromptVersionDialog to display image changes
- Integrate image parsing and comparison for prompt versions.
- Update the diff view height based on the presence of image changes.
- Add visual representation of images for both base and diff versions when changes are detected.
-
1
-
- add screenshots to documentation
-
Optimised images with calibre/image-actions
-
Potential fix for code scanning alert no. 233: Clear-text logging of sensitive information
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
-
Restrict online evaluation to OpenAI models when using images
-
Attempt to revert documentation changes
-
Refactor PlaygroundVariableCell to use parseImageValue for image handling
- Removed the useMemo hook for processing input data and replaced it with a direct call to parseImageValue.
- Updated ImagesListWrapper to display a single image based on the parsed value.
- Added parseImageValue function to lib/images.ts for improved image URL parsing and handling.
- Refactor PlaygroundVariableCell to streamline image handling
- Removed unused imports related to image content processing.
- Retained only the parseImageValue function for improved clarity and efficiency.
Co-authored-by: Copilot 175728472+Copilot@users.noreply.github.com
Co-authored-by: Claude noreply@anthropic.com
Co-authored-by: andrii.dudar fanat.web@gmail.com
Co-authored-by: Andres Cruz andresc@comet.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
Co-authored-by: andrii.dudar andriid@comet.com下载附件
-