github-featured / comet-ml--opik

发布

0 发布 8,340 Git 标签

RSS 订阅源
  • 1.9.101-4010 98434bb337
    比较

    [NA][SDK] Opik Optimizer SDK v3 Release (#4727)

    frostbyte_neo 发布于 2026-01-27 01:18:38 +00:00

    • Create __init__.py

    • Create wikipedia.py

    • Update core.py

    • Update wikipedia.py

    • Update wikipedia.py

    • tests: wikipedia

    • chore: examples fixed

    • Update test_wikipedia.py

    • Update test_wikipedia.py

    • chore: lint

    • Update pyproject.toml

    • Update pyproject.toml

    • Update wikipedia.py

    • chore: multi-hop agent

    • Create hotpot_multihop_benchmark.py

    • Create build_bm25_wikipedia.py

    • Update litellm_gepa_tiny_test_example.py

    • chore: lint

    • Update wikipedia.py

    • Update sdks/opik_optimizer/src/opik_optimizer/utils/tools/wikipedia.py

    Co-authored-by: Copilot 175728472+Copilot@users.noreply.github.com

    • Update sdks/opik_optimizer/scripts/datasets/build_bm25_wikipedia.py

    Co-authored-by: Copilot 175728472+Copilot@users.noreply.github.com

    • Update sdks/opik_optimizer/benchmarks/agents/hotpot_multihop_agent.py

    Co-authored-by: Copilot 175728472+Copilot@users.noreply.github.com

    • Update sdks/opik_optimizer/scripts/llm_frameworks/adk/adk_agent.py

    Co-authored-by: Copilot 175728472+Copilot@users.noreply.github.com

    • Update wikipedia.py

    • Update build_bm25_wikipedia.py

    • Update hotpot_multihop_benchmark.py

    • Update hotpot_multihop_agent.py

    • Update test_dataset_sources.py

    • Update wikipedia.py

    • Update wikipedia.py

    • Update result_ops.py

    • Update prompts.py

    • Update candidate_ops.py

    • Update meta_prompt_optimizer.py

    • Update meta_prompt_optimizer.py

    • Create test_meta_prompt_optimizer_agents.py

    • Update hotpot_multihop_agent.py

    • Update hotpot_multihop_benchmark.py

    • Update result_ops.py

    • Update meta_prompt_optimizer.py

    • Create bundle_agent.py

    • chore k flag

    • Update hotpot_multihop_benchmark.py

    • fix: metaprompter

    • chore: lint

    • chore: mypy

    • fix: scores

    • Update meta_prompt_optimizer.py

    • Update meta_prompt_optimizer.py

    • Update sdks/opik_optimizer/src/opik_optimizer/utils/tools/wikipedia.py

    Co-authored-by: Copilot 175728472+Copilot@users.noreply.github.com

    • Update sdks/opik_optimizer/src/opik_optimizer/utils/tools/wikipedia.py

    Co-authored-by: Copilot 175728472+Copilot@users.noreply.github.com

    • Update wikipedia.py

    • fix: refactored multiagent

    • fix: refactor finalized

    • Update sequenced_agent.py

    • Update sdks/opik_optimizer/src/opik_optimizer/utils/tools/wikipedia.py

    Co-authored-by: Copilot 175728472+Copilot@users.noreply.github.com

    • Update sdks/opik_optimizer/src/opik_optimizer/utils/llm_logger.py

    Co-authored-by: Copilot 175728472+Copilot@users.noreply.github.com

    • Update sdks/opik_optimizer/benchmarks/agents/sequenced_agent.py

    Co-authored-by: Copilot 175728472+Copilot@users.noreply.github.com

    • Update sdks/opik_optimizer/src/opik_optimizer/algorithms/meta_prompt_optimizer/meta_prompt_optimizer.py

    Co-authored-by: Copilot 175728472+Copilot@users.noreply.github.com

    • fix: tests

    • Update test_wikipedia.py

    • Update meta_prompt_optimizer.py

    • fix: move files

    • Create __init__.py

    • Update result_ops.py

    • chore: mv unit test

    • Update hotpot_multihop_benchmark.py

    • Update prompts.py

    • Update meta_prompt_optimizer.py

    • Update hotpot_multihop_agent.py

    • Add support for agents in metaprompt, hierarchical and fewshot optimizers

    • Fix benchmark tests

    • WIP evolutionary

    • WIP 3.0.0

    • Fix multimodal tests

    • Fix vulture checks

    • Fix pre-commit checks

    • Fix pre-commit checks

    • Add conftest.py for single and multi prompt e2e tests

    • Fix evaluation_ops for multi-prompt

    • Update to preview version for publishing

    • Fix types when using ty

    • [NA][SDK] Optimizer 3.0rc2: Stablize 3.0rc1 fixing broken tests and mypy (#4555)

    • chore: lint/mypy fixes

    • fix: canidates overwriting

    • chore: lint/mypy fixes

    • chore: lint/mypy fixes

    • chore: lint/mypy fixes

    • chore: lint/mypy fixes

    • fix: tests for mypy

    • Update test_dataset_sources.py

    • Update test_multimodal_prompt.py

    • Update test_benchmark_smoke.py

    • Update pyproject.toml

    • Update pyproject.toml

    • Update pyproject.toml

    • fix: multi agent prompt support for task runner outputs

    • Update pytest.ini

    • Update Makefile

    • fix: test failures

    • chore: fix test failure

    • Create conftest.py

    • [NA][SDK] Optimizer rc3 improvements to add ARC-AGI 2 and solver (#4554)

    • feat: added arc-agi2 dataset

    • fix: reasoning_model in base optimizer

    • fix: reasoning_model in HRPO

    • Create arc_agi2_baseline_prompt_optimizer.py

    • Create arc_agi2_hrpo_solver.py

    • Update arc_agi2.py

    • Update arc_agi2_baseline_prompt_optimizer.py

    • Update arc_agi2_hrpo_solver.py

    • Update arc_agi2.py

    • Update pyproject.toml

    • fix: datasets and mapping

    • feat: running now

    • Update arc_agi2.py

    • Update arc_agi2_hrpo_solver.py

    • Update arc_agi2_hrpo_solver.py

    • Update litellm_agent.py

    • Update hierarchical_reflective_optimizer.py

    • Update dataset_utils.py

    • Update multi_metric_objective.py

    • Update arc_agi2.py

    • Update arc_agi2_hrpo_solver.py

    • Update arc_agi2.py

    • chore: in-line changes - to be adapted later

    • feat: filter_by mode for datasets util

    • fix: MultiMetricObjective standardized

    • refactor: encode_image_to_base64_uri

    • chore: image util and filters docs and tests

    • Update arc_agi2_hrpo_solver.py

    • chore: lint

    • fix: dataset splits

    • Update define_datasets.mdx

    • refactor: new arc-agi script

    • Create __init__.py

    • chore: arc-agi mypy

    • chore: updated end state and metric foreground

    • chore: arc_agi image mode

    • Update litellm_agent.py

    • Update pyproject.toml

    • Update evolutionary_optimizer.py

    • chore: patch mcp no longer in place

    • fix: ADK missmatch

    • fix: early stop and checks

    • fix: cost and tool calling usage

    • fix: erroneous kwargs

    • fix: tests

    • tests: additional

    • Update gepa_optimizer.py

    • Update define_datasets.mdx

    • fix: various issues and smells

    • Update prompts.py

    • tests: fix

    • chore: dead code

    • chore: dead code

    • chore: dead code

    • Update generate_fern_docs.py

    • Update arc_agi2_baseline_prompt_optimizer.py

    • chore: arc-agi

    • Update .pre-commit-config.yaml

    • Update pyproject.toml

    • chore: lint

    • feat: prompt library

    • Create AGENTS.md

    • feat: prompt library updates and fixtures

    • chore: gepa PromptOverrides

    • chore: finalize prompt library

    • chore: arc-agi

    • Update sdk_configuration.mdx

    • fix: litellm async and project details

    • fix: supress multivariate optuna warning

    • chore: arc-agi

    • Update logging_config.py

    • chore: cleanup tests

    • docs: update advanced docs

    • feat: native support for n model param

    • refactor: dead tests

    • fix: few ahout and EVO bug

    • chore: n candidates

    • fix: tqdm issue

    • chore: n candidates

    • docs: fern docs optimizer update

    • Update tasks_optimizer.py

    • Update mutation_ops.py

    • fix: test failures

    • Update extending_optimizers.mdx

    • Update _llm_calls.py

    • docs update

    • Update pyproject.toml

    • Update pyproject.toml

    • chore: examples

    • Update reporting_utils.py

    • Update reporting_utils.py

    • Update .pre-commit-config.yaml

    • Update chat_prompt.py

    • chore: fixes

    • docs: changelog

    • fix: empty reasoning panel

    • Update generate_changelog.py

    • tests: fix post refactor

    • refactor: post refactor fixes

    • Update README.md

    • Update test_multi_metric_objective.py

    • docs: refesh docs

    • Update reference.mdx

    • feat: normalize optimizationresult.details

    • Update optimize_prompts.mdx

    • Delete .node-version

    • chore: baz review updates

    • [NA] [SDK] Add comprehensive optimizer unit tests before refactor (#4741)

    • Form testing approach and refactor the tests

    • Fix linter issues: markdown formatting and ruff format

    • [NA][SDK] Optimizer Additional Aggresive Refactor and Dead Code Cleanup (rc) (#4795)

    • Form testing approach and refactor the tests

    • Fix linter issues: markdown formatting and ruff format

    • Refactor: Extract common optimizer setup logic into BaseOptimizer helpers

    Phase 1 & 2 of the optimizer technical debt refactor:

    • Add _normalize_prompt_input() to convert single/dict prompts consistently
    • Add _create_optimization_run() with unified error handling
    • Add _select_evaluation_dataset() for training/validation dataset selection
    • Update all 6 optimizers to use the new helpers:
      • EvolutionaryOptimizer
      • MetaPromptOptimizer
      • FewShotBayesianOptimizer
      • HierarchicalReflectiveOptimizer
      • GepaOptimizer
      • ParameterOptimizer
    • Add unit tests for new BaseOptimizer methods

    This reduces code duplication and makes the optimizer-specific logic
    more visible by extracting common setup boilerplate.

    • Revision: Fix lint errors and refine progress tracking display

    • another refactor

    • fix precommit issues

    • Baz review fixes

    • a few more review fixes

    • test comment fix

    • small review fixes

    • lint

    • fix wrong test logic

    • fix: passing n_threads

    • chore: normalize _reporter

    • fix: project name defaults

    • refactor: normalize n_threads

    • refactor: remove reuse and warn

    • refactor: gepa nomenclature

    • fix: default project_name

    • chore: extend finish reason

    • refactor: better test and validation for scores with _coerce_score

    • chore: lint

    • Update base_optimizer.py

    • test: counters on HRPO optimizer

    • test: score normlization

    • fix: reporter refactor and validation dataset for evo

    • Update test_evolutionary_optimizer.py

    • Update evolutionary_optimizer.py

    • refactor: _reporter pattern

    • fix: pydantic v2 errors

    • chore: terminal UX improvement

    • refactor: centralized "should_stop" logic into _should_stop_context

    • Update test_parameter_optimizer.py

    • chore: update reporting nomenclature

    • refactor: centralized history

    • chore: docs updates

    • refactor: llm counters

    • refactor: dead code

    • refactor: new wiring

    • refactor: parameter optimizer

    • refactor: optimzable agent

    • refactor: utils

    • refactor: wiring up candidate hooks

    • refactor: optimizer types

    • fix: trial id mutation

    • refactor: hrpo evals

    • refactor: optimizer helpers

    • refactor: dead code

    • refactor: display utils

    • refactor: display util

    • refactor: display

    • fix: trial scores

    • Update optimization_result.py

    • refactor: rounds and display

    • refactor: time

    • refactor: baz and timestamp

    • refactor: few shot ops

    • fix: baz

    • refactor: hook nomencalture

    • refactor: reporting

    • refactor: reporting

    • fix: round data

    • refactor: hooks

    • refactor: reporting/display

    • refactor: reporting

    • refactor: display

    • chore: make test all

    • Update display_run.py

    • Delete AGGRESSIVE_REFACTORING_PLAN.md

    • Delete optimization_history_state.md

    • chore: mv file

    • fix: baz

    • refactor: display

    • test: additional regression tests

    • test: additional tests for regressions, baz and clean

    • fix: bugs and errors

    • Update conftest.py

    • refactor: logger

    • refactor: various

    • tests: refactor

    • refactor: tests

    • Update logging.py

    • refactor: core removed and cleaned

    • fix: logger

    • refactor: tests

    • chore: debug logger on all hooks

    • fix: finalized loggers and few shot

    • fix: various ux

    • chore: rules

    • Update AGENTS.md

    • Update evolutionary_optimizer.py

    • refactor: state types

    • fix: tests passing

    • refactor: move core libraries to new loc

    • refactor: cont

    • refactor: core

    • fix: mypy

    • refactor: core

    • refactor: cont on core

    • refactor: initial reorg of core

    • refactor: core continued

    • refactor: hook naming

    • fix: hooks

    • refactor: additional

    • refactor: candidate to trial

    • Update base_optimizer.py

    • refactor: context

    • refactor: cont

    • refactor: metaprompter

    • refactor: metaprompter

    • refactor: metaprompter tested

    • refactor: meta prompter

    • refactor: reporting output

    • Update metaprompt_hotpot_example.py

    • refactor: benchmarks

    • refactor: force pydantic models

    • Update reporting.py

    • Update hotpot_multihop_benchmark.py

    • fix: various issues

    • test: pydantic litellm

    • fix: tests

    • Update reporting.py

    • refactor: evolutionary

    • fix: various

    • refactor: gepa

    • refactor: optuna

    • refactor: evolutionary

    • refactor: multiple

    • refactor: more optimizers and utils

    • refactor: more

    • refactor: optuna

    • fix: gepa posst refactor

    • refactor: final functions

    • Update .pre-commit-config.yaml

    • Update llm_calls.py

    • feat: sigterm and dataset refactor

    • Update gepa_optimizer.py

    • refactor: various improvements to litellm and defaults

    • feat: senamtic cross over and opik tags for optimizer

    • chore: bump version

    • refactor: fix various

    • fix: optimizer wiring

    • chore: update docs

    • chore: cursor rules

    • fix: baz errors

    • chore: update cursor rules

    • refactor: tests

    • refactor: tests and docs

    • chore: lint

    • Update test_multi_prompt_with_agent.py

    • fix: failure on chatprompt

    • Update generate_fern_docs.py

    • Update test_multi_prompt_with_agent.py

    • fix: multiprompt error

    • Update hierarchical_reflective_optimizer.py

    Co-authored-by: Itamar Golan itamarg@comet.com

    • Delete sdks/opik_optimizer/tests/unit/optimizers/OPTIMIZER_TEST_COVERAGE.md

    • [NA][SDK] Optimizer SDK Refactor Tests following Refactor (v3-rc5) (#4836)

    • Create conftest.py

    • Update __init__.py

    • refactor: algo unit tests

    • refactor: tests conf

    • refactor: base tests

    • fix: baz

    • chore: mypy

    • refactor: base optimizer

    • fix: test

    • chore: lint

    • fix: refactor bug introduced

    • refactor: test fixtures

    • refactor: llm call tests

    • chore: lint

    • chore: mypy test

    • chore: baz

    • Update conftest.py

    • refactor: tests param and evo

    • refactor: tests evo and parameter

    • chore: temp restore to avoid large file change on main PR

    • refactor: tests evo, and util fixtures

    • refactor: base and evo

    • refactor: tests core base, optimization and metaprompter

    • refactor: core, hrpo, evo

    • refactor: more tests

    • refactor: tests cont

    • refactor: tests

    • chore: lint

    • refactor: missing tests

    • tests: fix

    • [NA][SDK] Opik Optimizer centralised shuffle, RNG and use evaluate_on_dict_items (#4851)

    • chore: baz

    • feat: rng util

    • feat: sampling util

    • Update __init__.py

    • chore: api wiring

    • feat: sampling plan evals

    • Update base_optimizer.py

    • Update few_shot_bayesian_optimizer.py

    • Update evaluation_ops.py

    • feat(sampler): support sampler on hrpo

    • feat(sampler): support sampler on metaprompter

    • feat(sampler): base changes

    • feat(sampler): support sampler on evolutionary

    • chore: export

    • feat(sampler): support sampler on parameter

    • chore: various

    • chore: update

    • chore: mypy and issues fixes

    • chore: baz

    • chore: docs move

    • docs: update samples

    • chore: defaults and formatting output

    • docs: sampling

    • chore: docstrings

    • feat: sampling improvements and tests

    • chore: update

    • chore: baz

    • fix: same upstream ratelimitter fix on prod (merge-conflict)

    • fix: evaluation on dict

    • fix: ratelimitter mypy upstream

    • fix: datasets and throttle error handlers

    • chore: update litellm and package deps

    • Update apps/opik-documentation/documentation/fern/docs/agent_optimization/advanced/n_samples.mdx

    Co-authored-by: baz-reviewer[bot] <174234987+baz-reviewer[bot]@users.noreply.github.com>

    • Update sdks/opik_optimizer/Makefile

    Co-authored-by: baz-reviewer[bot] <174234987+baz-reviewer[bot]@users.noreply.github.com>

    • chore: lint

    Co-authored-by: baz-reviewer[bot] <174234987+baz-reviewer[bot]@users.noreply.github.com>

    • Update pyproject.toml

    • Update sdks/opik_optimizer/src/opik_optimizer/algorithms/meta_prompt_optimizer/ops/candidate_single_ops.py

    Co-authored-by: baz-reviewer[bot] <174234987+baz-reviewer[bot]@users.noreply.github.com>

    • Fix FewShotBayesianOptimizer multi-prompt variable replacement and prompt structure

    1. Fix variable replacement: Update prompt template to use single curly braces
      and emphasize using exact key names from dataset examples (e.g., {text}, {label})
      instead of assumed names like {expected_output}

    2. Fix prompt flattening: Preserve dict structure in trial config so multi-prompt
      shows separate prompt:analyze and prompt:respond instead of a flattened list

    Co-Authored-By: Claude Opus 4.5 noreply@anthropic.com

    • [NA][SDK] Optimizer RC Fix LiteLLM, Bugs and Add Prompts in Traces (#4919)

    • fix: throttle bug

    • fix: broken litellm version

    • fix: broken test

    • feat: support optimizer prompts in UI

    • fix: threadsafe agents

    • feat: prompt tracing in UI

    • Update few_shot_bayesian_optimizer.py

    • chore: lint

    • chore: lint

    • feat: role constraints

    • chore: baz overidding metadata

    • chore: lint

    • chore: baz and lint

    • chore: baz and mypy

    • chore: lint

    • fix: test timeout

    • chore: baz

    • Update optimizable_agent.py

    • fix: prompts FE

    • fix: hotpot qa dataset

    • fix: hg direct dataset paths

    • feat: prompt library support in FE from optimizer

    • fix: prompts in FE

    • Update dataset.py

    • Update pyproject.toml

    • chore: final cleanup

    • chore: supress hg warning

    • fix: bug with url display

    • fix: notebook display

    • Update dataset.py

    • Update TraceDataViewer.tsx

    • Update prompt_tracing.py

    • Update TraceDataViewer.tsx

    • Update prompt_tracing.py

    • [OPIK-XXXX] [SDK] Fix parameter optimizer to log trial-specific model parameters

    Previously, optimizer_metadata.model_parameters always showed initial
    parameter values instead of the optimized trial-specific values. This
    fix extracts model_kwargs from tuned_prompts and overrides
    optimizer_metadata.model_parameters in the experiment_config passed to
    evaluate_prompt, ensuring each trial logs its actual optimized parameters.

    • Fix prompt logging format inconsistency in baseline evaluation
    • Update prepare_experiment_config() to always use dict format for prompts
    • Ensures baseline evaluation logs prompts as {"prompt-name": [messages]}
      instead of [messages] for single-prompt optimization
    • Makes baseline format consistent with follow-on trial logging
    • Fixes issue where hierarchical, evolutionary, and other optimizers
      logged baseline and follow-on prompts in different formats
    • [NA] [SDK] Fix MetaPromptOptimizer multimodal content support
    • Fix sanitize_generated_prompts to handle multimodal content (list) by extracting text before calling .lower()
    • Update _build_prompts_from_items to always use messages instead of user/system for multimodal compatibility
    • Remove max_tokens from e2e test configs to prevent JSON truncation errors
    • Add unit tests for multimodal content sanitization

    • fix: dataset to pass intergration tests

    • [DOCS] Add comprehensive OptimizableAgent documentation for single and multi-prompt optimization

    • Update docs

    • [SDK] Fix mypy type annotations in litellm_agent.py

    • [NA] [DOCS] Remove duplicated n parameter explanation from multiple_completions.mdx

    • Update .agents/rules/sdks/opik_optimizer/architecture.mdc

    Co-authored-by: baz-reviewer[bot] <174234987+baz-reviewer[bot]@users.noreply.github.com>

    • [SDK] Fix few-shot optimizer to preserve placeholder messages by deferring role constraints

    Move role constraint application from _create_few_shot_prompt_template to
    _reconstruct_prompts_with_examples to ensure placeholder-bearing messages
    are preserved during template creation and role filtering happens at
    injection time when examples are actually added to prompts.

    • [NA] [SDK] Address PR review comments: fix naming, handle ContentPart lists, add docstring
    • Rename fewshot_template to few_shot_template for naming consistency
    • Handle list[ContentPart] when extracting user content from messages
    • Add comprehensive docstring to build_optuna_objective function
    • Remove duplication in algorithm-structure.mdc by linking to canonical source
    • Optimizer v3.0.0

    Co-authored-by: Copilot 175728472+Copilot@users.noreply.github.com
    Co-authored-by: Jacques Verre jverre@gmail.com
    Co-authored-by: itamargolan itamargolan85@gmail.com
    Co-authored-by: Itamar Golan itamarg@comet.com
    Co-authored-by: baz-reviewer[bot] <174234987+baz-reviewer[bot]@users.noreply.github.com>
    Co-authored-by: Claude Opus 4.5 noreply@anthropic.com

    下载附件