发布

RSS 订阅源

2.0.75-5757 ffb5e0cfe3

比较
[OPIK-6924] [BE][FE] feat: simplify Optimization Studio deployment, fix optimization runs, and add separate algorithm model (#7133)

frostbyte_neo 发布于 2026-06-22 10:01:44 +00:00
- [OPIK-6924] [DOCS] docs: add Optimization Studio onboarding & design doc
Document how Optimization Studio works end-to-end from both a logical
(request lifecycle) and infrastructural/architectural (services, subprocess
isolation, deployment config) perspective.

Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
- fix(optimization-studio): use selected model for prompt and guide provider setup
The studio runner created ChatPrompt without a model, so baseline and
per-trial task evaluations fell back to the SDK default (openai/gpt-5-nano)
and ignored the model chosen in the UI — a regression from #4538 that became
a hard failure once #6652 stopped injecting an OpenAI key. Pass the configured
model and parameters onto the prompt.

On the frontend, the new-optimization form seeded a hardcoded gpt-4o-mini
default that renders empty on workspaces without that provider. Default to the
first available model instead, clear the model when no providers are configured
so submission is blocked, and turn the empty state into a "Configure a provider"
action that opens the same dialog the playground uses.

Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
- chore(optimization-studio): remove optimization studio feature toggle
The Optimization Studio feature has graduated, so drop the deployment-level
TOGGLE_OPTIMIZATION_STUDIO_ENABLED flag entirely (matching the OPIK-6434 toggle
cleanups). Removes the ServiceTogglesConfig field, the config.yml/config-test.yml
entries, the docker-compose env var, the FE FeatureToggleKeys member + default,
and all gating usages. The studio is now governed solely by the
OPTIMIZATION_STUDIO_USE / canUseOptimizationStudio RBAC permission.

Generated REST clients and OpenAPI specs still reference the field and need a
fern regeneration in a follow-up.

Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
- fix(optimization-studio): strip unsupported model params before submitting
The new-optimization form sent a default temperature for every model, so models
that deprecate sampling params (e.g. Claude Opus 4.8) failed at the gateway with
"temperature is deprecated for this model". Apply the playground's two-layer
approach: reconcile the config on model change via updateProviderConfig, and
sanitize it via sanitizeConfigForRequest before building studio_config. Also add
the missing Opus 4.8 capability row (supportsSamplingParams: false), which fixes
the same gap in the playground.

Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
- fix(optimization-studio): optimize the prompt's actual roles to avoid GEPA crash
optimize_prompt defaults to optimizing only system messages. A prompt with no
system message (e.g. a single user message, which the studio form produces by
default) left the optimizer with zero editable components; GEPA then hit
(pid + 1) % len(predictors) with an empty predictor list and died with
"integer modulo by zero". Derive optimize_prompts from the roles actually
present in the prompt so every optimizer (GEPA, hierarchical, evolutionary,
few-shot) gets a non-empty candidate and runs to completion for any prompt shape.

Verified: user-only prompt + Claude Opus 4.8 now completes for both GEPA and
hierarchical (previously crashed mid-reflection).

Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
- test(optimization-studio): add gateway-based e2e regression test + workflow
Adds e2e infra for opik-python-backend (mirroring the Python SDK's e2e setup):
an opik_client fixture, a seeded sentiment dataset, and a studio_gateway
fixture that routes the optimizer's LLM calls through the backend /v1/private
gateway using the workspace-stored provider key — no provider key is passed to
LiteLLM, matching the Studio's production path and Opik's key-storage principle.

The test drives the studio the way the job runner does (OptimizationConfig →
factories → run_optimization) on a user-only prompt, and asserts via the
produced traces that the configured model actually ran (not the gpt-5-nano
default) — guarding the model-passing and GEPA empty-candidate regressions for
both GEPA and hierarchical optimizers. It deliberately skips the Java REST
enqueue and the RQ queue. Bounded via OPTIMIZER_MAX_TRIALS so runs stay short.

A dedicated workflow runs it (full stack via opik.sh) whenever
src/opik_backend/studio/** changes; the Anthropic key is provided as a secret
and stored in the workspace, never handed to the optimizer.

Verified locally against a running stack: GEPA and hierarchical both pass
through the gateway with no provider env key.

Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
- fix(optimization-studio): address review — effective model params + test isolation
- Pass the optimizer's effective model_parameters (with the factory-injected
  max_tokens) into ChatPrompt instead of the raw config.model_params, so
  baseline/per-trial task calls don't truncate at the SDK default.
- studio_gateway e2e fixture now uses monkeypatch for the gateway env vars and
  optimizer_runner.OPENAI_API_BASE so they're restored after the test and the
  suite stays order-independent.
Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
- test(optimization-studio): store Anthropic secret in the BE, run via gateway
Make the e2e flow explicit and secret-first: take the ANTHROPIC_API_KEY value
(GitHub secret), store it in the backend workspace via the provider-key API, and
trigger the optimization so the gateway resolves it server-side. Standardize on
Anthropic (claude-haiku) and drop the OpenAI path. Falls back to an already-
configured Anthropic provider for local runs without the secret.

Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
- feat(optimization-studio): separate algorithm model + e2e via real entrypoint
Algorithm model:
- The optimizer/algorithm (GEPA reflection LM, hierarchical reasoning model) can
  now run on a different model than the prompt. It's carried in
  studio_config.optimizer.parameters.model; OptimizationConfig pulls it out so the
  remaining params stay valid optimizer kwargs, and the runner builds the optimizer
  on it (defaulting to the prompt model). FE: an "Algorithm model" selector in the
  Algorithm settings, defaulting to the prompt model. Shared max_tokens default
  factored into ensure_default_model_params so prompt + optimizer both get it.
E2E test:
- Drive the real entrypoint process_optimizer_job (which sets up the gateway and
  runs optimizer_runner.py as a subprocess) instead of re-implementing the wiring,
  so the gateway routing / model prefix / ChatPrompt(model=...) are exercised. The
  test stores the Anthropic key in the workspace and never passes it to the
  optimizer. One test for GEPA, one for hierarchical. Asserts health signals
  (baseline established, optimized prompt produced, no regression, real span
  volume) and verifies via traces that the configured model ran (not gpt-5-nano).
- Flakiness protection: trace polling for eventual consistency, tiny dataset +
  OPTIMIZER_MAX_TRIALS, and --reruns in the workflow for transient LLM failures.
- Separate-algorithm-model wiring is covered by a deterministic unit test (the
  reflection-model calls aren't logged as spans, so they can't be trace-verified).
Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
- test(optimization-studio): verify prompt + algorithm use their configured models
Extract build_optimizer_and_prompt from the runner's main() so the model wiring
is testable, and assert (offline, deterministically) that the prompt is built
with the configured prompt model + params and the optimizer is built with its
own configured model + params (defaulting to the prompt model when unset).

Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
- fix(optimization-studio): address review — clear algo params on reset, e2e skip guard
- "Use prompt model" now also drops model_parameters (not just model) so no
  stale optimizer-model params are serialized into studio_config (high-severity
  review finding).
- anthropic_workspace_key e2e fixture skips early when OPIK_URL_OVERRIDE is unset,
  instead of calling httpx with a None base URL.
Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
- fix(optimization-studio): preserve optimizer params; surface subprocess errors
- build_optimizer_and_prompt now keeps config.optimizer_model_params even when no
  separate optimizer model is set (defaults the model to the prompt model), so
  optimizer.parameters.model_parameters from saved configs / API clients isn't
  silently dropped (review finding). Unit test added.
- The optimization runs in an isolated subprocess; on failure it now emits the
  full traceback in its result JSON and process_optimizer_job propagates it, so
  the error details are visible in CI even though they happen in another process.
- e2e workflow streams logs live (--log-cli-level=INFO) and logs per-test
  durations (--durations=0).
Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
- chore(python-backend): split tests into tests/unit + tests/e2e
Reorganize the opik-python-backend test suite for a clear separation:
- tests/unit — fast tests with no real backend (Flask client, mocks, local
  subprocess/docker executors). Run by the "Python Backend Tests" workflow.
- tests/e2e — tests that need a running Opik backend (the studio optimization
  e2e + the code-metric integration test). Run by the renamed "Python BE E2E
  Tests" workflow (was "Optimization Studio E2E").
Workflows updated to reflect the split: the unit workflow runs pytest unit;
the e2e workflow runs pytest tests/e2e and triggers on any
apps/opik-python-backend/** change. Fixed the moved integration test's
file-relative path and registered the integration marker.

Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
- ci(python-backend): point e2e at the stack's real Redis
The studio job processor streams subprocess logs to Redis via a log collector.
Set REDIS_URL to the stack's password-protected, port-mapped Redis so the e2e
uses a real Redis instead of falling back to an unauthenticated localhost:6379
(which the password-protected stack Redis rejects).

Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
- test(python-backend): add strict types to e2e tests, drop redundant _e2e suffix
The studio test lives in tests/e2e/, so the file-name suffix was redundant:
test_studio_optimization_e2e.py -> test_studio_optimization.py.

Annotate helper/fixture signatures (opik.Opik, opik.Dataset, dict[str, Any],
Iterator[...]) so the e2e suite is fully typed.

Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
- test(python-backend): align code-metric e2e with the studio pipeline patterns
The code-metric test predates the gateway work: it parsed a frontend demo
template via a node subprocess, ran the optimizer factories directly, passed
OPENAI_API_KEY straight to litellm, and asserted a metric name (intent_accuracy)
that the opik-chatbot template no longer produces (it switched to G_EVAL) — so
it was both off-pattern and stale.

Rebuild it as a focused CODE-metric regression that follows the patterns
established in this PR: an inline studio_config with a user BaseMetric, driven
through the real process_optimizer_job entrypoint, with the Anthropic key stored
in the workspace and resolved server-side via the gateway (no provider key
handed to the optimizer). Extract the shared run_studio_job /
assert_optimization_healthy helpers into studio_helpers.py so both e2e tests
reuse them. Fully typed.

Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
- test(python-backend): centralize model names, consolidate studio e2e, clean up entities
- Add tests/llm_constants.py (mirrors the Python SDK convention): every real
  model id used by the unit + e2e suites lives in one place. A tiny
  tests/conftest.py puts it on the path for both pytest tests/e2e and
  cd tests && pytest unit.
- Both e2e modules were studio-optimization tests differing only by metric, so
  merge the code-metric test into test_studio_optimization.py as a variant and
  drop test_code_metric_optimization.py + studio_helpers.py.
- Replace the run helper with a run_studio_optimization fixture that owns the
  optimization record's lifecycle and deletes it on teardown. project_name and
  the (renamed) seeded_sentiment_classification_dataset fixtures already clean
  up the project and dataset, so each test is independent and self-cleaning.
Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
- test(python-backend): assert optimized-prompt shape + model use on both variants
- Tighten the optimized-prompt check from "truthy" to a non-empty list of
  role/content messages that still carries the dataset variable, so a mangled
  or variable-less prompt is caught even when the score looks healthy.
- Factor the model/trace verification into a shared helper and run it on the
  code-metric variant too, so a model-passing regression fails both tests.
Verified against the real pipeline (GEPA, hierarchical, code metric) and the
new assertions only trip on real regressions (validated via fault injection),
not on run-to-run noise.

Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
- ci(python-backend): drop unused e2e env vars
OPENAI_API_KEY is no longer read by any e2e test (the code-metric test now runs
Anthropic through the gateway), and OPIK_WORKSPACE just restated the "default"
that every reader already falls back to. Also correct the REDIS_URL comment:
log flushing is best-effort, so a missing Redis doesn't fail the run — it only
kills the (unasserted) log-streaming path.

Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
- refactor(optimization-studio): type the optimizer job result dict
Add TypedDicts for the studio job result — OptimizationRunResult (success),
OptimizationErrorResult (failure), OptimizationCancelledResult — and the
OptimizationJobResult union process_optimizer_job returns. Annotate the dicts
where they're built (optimizer_runner) and the handler's return/cancel paths,
so the result shape is explicit instead of Dict[str, Any].

Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
- refactor(optimization-studio): drop config mutation in build_optimizer_and_prompt
Address PR review:
- build_optimizer_and_prompt no longer mutates config (it computed gateway-routed
  model/params onto config so the metric could reuse them). Resolve the gateway
  task model locally and have the caller read it from prompt.model when building
  the metric — clearer data flow, no hidden side effects.
- Drop the dead assert "error" not in result in the e2e health check (the error
  path raises inside process_optimizer_job and never returns here); keep the
  reachable cancelled guard.
Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
- fix(optimization-studio): enable the RQ worker in the Helm chart
The Optimization Studio worker runs in the python-backend only when
RQ_WORKER_ENABLED=true. docker-compose sets it (so local works), but the Helm
chart never did — so on k8s the Java backend enqueues studio jobs to
opik:optimizer-cloud and nothing consumes them, leaving runs stuck in
"initialized". Set RQ_WORKER_ENABLED=true on the python-backend deployment.

Also correct the rq_worker_manager docstrings: the code default is "false",
not "true".

Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
- docs(helm): regenerate chart README for RQ_WORKER_ENABLED
helm-docs output for the python-backend RQ_WORKER_ENABLED value added in the
previous commit; keeps the "Generate helm docs" check green.

Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
- refactor(optimization-studio): resolve default model without effects
Address review: replace the seed-a-guess-then-reconcile model default with an
effect-free reactive resolution.
- convertOptimizationStudioToFormData no longer hardcodes gpt-4o-mini; it takes
  the workspace's available models and resolves the default (configured model
  if available, else first available, else "").
- OptimizationsNewPage lifts useProviderKeys + useLLMProviderModelsData +
  useModelOptions to where useForm lives, computes availableModels, and feeds
  the resolved defaults via the reactive values prop with
  resetOptions.keepDirtyValues (preserves a model the user already picked).
- Drops both effects: the manual form.reset and the model-sync useEffect in
  useOptimizationsNewFormHandlers.
- Guards on a providersReady flag so "configured providers, models still
  loading" isn't mistaken for "no models" (no mid-load "Model is required"
  flash under mode: onChange).
Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
- fix(optimization-studio): default RQ_WORKER_ENABLED to true
Running the Optimization Studio job worker is the python-backend's primary
role, so default it on instead of requiring every deployment to set the flag.
This makes the studio work out-of-the-box on k8s (the Helm chart didn't set it,
so studio jobs enqueued to opik:optimizer-cloud were never consumed and runs
sat in "initialized") without a per-chart override.
- env_utils.is_rq_worker_enabled() now defaults to "true"; executor-only
  instances without Redis can opt out with RQ_WORKER_ENABLED=false.
- Drop the now-redundant RQ_WORKER_ENABLED from the Helm chart (regen README);
  restore the rq_worker_manager docstrings to "default: true" (now accurate).
Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com

Co-authored-by: Claude Opus 4.8 (1M context) noreply@anthropic.com
下载附件
- 源代码 (ZIP)
- 源代码 (TAR.GZ)

0 发布 8,340 Git 标签

[OPIK-6924] [BE][FE] feat: simplify Optimization Studio deployment, fix optimization runs, and add separate algorithm model (#7133)