发布

RSS 订阅源

2.0.44-5415 100cc2c610

比较
[OPIK-6408] feat: add visual comparison tests POC (#6793)

frostbyte_neo 发布于 2026-05-21 08:39:47 +00:00
- initial local POC for visual comparison tests
- stabilize existing tests
- adding allure
- improve clean up
- add mask for demo data
- [OPIK-6505] [BE] Split stats query and gate legacy feedback_scores UNION (#6713)
- [OPIK-6505] [BE] Split traces/spans stats query and gate legacy feedback_scores UNION
Splits the project-stats endpoints into two parallel queries (traces+spans
aggregation and feedback-score aggregation) and runs them via Mono.zip. The
per-trace/per-span JOIN against the rich groupArray-tuple feedback CTE was
the dominant cost at scale.

Adds a workspaces.has_legacy_scores flag (defaults TRUE) detected during
workspace version determination; the stats SQL gates the legacy
feedback_scores table UNION on it so workspaces with no legacy data skip the
empty-table scan.

Applies the same split + gate to SpanDAO.getStats. On the OPIK-6505 customer
dataset this brings end-to-end stats wall clock down from ~78s to ~4s
unfiltered and ~0.3s with a trace-side filter, with byte-identical API
output (144-key feedback map matched to float64 precision).
- [OPIK-6505] [BE] Align StatsMapperTest with split-A / split-B contract
The split moved feedback-score row mapping out of mapProjectStats (split-A
no longer carries those columns) into a new mapProjectScoresStats(Row) used
by SELECT_FEEDBACK_SCORES_STATS / SELECT_SPAN_FEEDBACK_SCORES_STATS. Update
the test to assert the new contract:
- mapProjectScoresStats_withSpanFeedbackScores_returnsSpanFeedbackScoresStats
  asserts the new method emits the feedback stat items.
- mapProjectStats_doesNotEmitFeedbackScoreStats asserts the trace+spans
  mapper now intentionally drops feedback-score columns.
- [OPIK-6505] [BE] Address review: NPE guard, consolidate hasLegacyScores, add regression test
- StatsMerger.merge: avoid NPE for projects with no feedback rows by
  falling back to ProjectStats.empty() at the call site instead of guarding
  inside the single-project overload.
- Move the reactive has_legacy_scores lookup onto WorkspacesService
  (Mono hasLegacyScores) so TraceDAO and SpanDAO share one
  implementation instead of duplicating a Schedulers.boundedElastic wrapper.
- Add ProjectsResourceTest coverage that flips the workspace flag and
  re-hits /stats to verify the endpoint stays consistent.
- [OPIK-6505] [BE] Fix CI regressions: thread stats feedback + merge order
- StatsMapper.mapProjectStats: re-emit feedback_scores when the row carries
  that column. Thread stats inline feedback in the same row and call this
  mapper directly with no merge step; the prior linter refactor dropped them
  silently. Split-A trace/span rows omit the column, so the metadata-guard
  is a no-op there.
- StatsMerger.merge: splice the feedback ProjectStats into the base list at
  the canonical position (just before guardrails / error-count entries)
  instead of appending. The pre-split mapper emitted feedback there, and
  StatsUtils-built expected stats assert that order.
- StatsMapperTest: replace the "drops feedback" assertion with two cases —
  emits when the column is present, omits when it is not — matching the
  real contract for split-A vs thread paths.
- [OPIK-6505] [BE] Defend stats Mono.zip against empty traces/spans branch
SELECT_TRACES_SPANS_STATS and SELECT_SPANS_STATS both end with GROUP BY
project_id, so when no rows match the filter the projection emits zero
rows and singleOrEmpty() completes empty. Mono.zip then drops the
feedback branch as well, so the response silently loses feedback-only
stats. Match the safety net already on the feedback side by attaching
.switchIfEmpty(Mono.just(new ProjectStats(List.of()))) to both
traces/spans branches.
- [OPIK-6505] [BE] Address review: gate legacy probe, dedupe stats orchestration
WorkspaceVersionService.persistAndEmitBlocking now reuses the existing
findById(workspaceId) lookup (already loaded for lastKnownVersion) to gate
the ClickHouse probe + MySQL upsert behind the stored has_legacy_scores
flag. Once that flag is false the legacy feedback_scores table only ever
shrinks (no new writes land there), so the per-cache-miss probe is wasted
work. When the flag is true we still probe but only upsert when the result
differs, eliminating the redundant rewrites flagged in review.

Extracted the split-A / split-B Mono.zip orchestration into
StatsMerger.zipAndMerge so SpanDAO.getStats and TraceDAO.getStats invoke
the same logic. The helper centralises the empty-default safety net that
the prior commit introduced ad-hoc on each side — future tweaks to the
zip semantics now live in one place.
- [OPIK-6505] [BE] Address review: short-circuit zipAndMerge, trim comments
zipAndMerge: when the aggregates Mono emits empty, return empty stats
instead of forcing a feedback-only response. This matches the
traces-driven scope the map overload enforces (no resurrecting
projects from feedback alone). Use the existing ProjectStats.empty()
factory instead of allocating new ProjectStats(List.of()).

Trim verbose comments on StatsMerger, mapProjectStats, and the
WorkspaceVersionService gated probe down to one-line whys.
- [OPIK-6505] [BE] Add regression test with real legacy_scores data
Seeds a project via buildProjectStats then writes one feedback score
directly into the legacy feedback_scores ClickHouse table via the
DAO's author=null path (the public API always routes to
authored_feedback_scores). Asserts the /stats response — full object
recursive comparison — includes the legacy score when has_legacy_scores
is true and drops it after flipping the workspace flag to false.

Closes the gap @ldaugusto flagged on the PR: previously the UNION
branch was only verified by manual production benchmarks.
- [OPIK-6505] [BE] Cover all three legacy-UNION endpoints with real-data tests
Each test writes one feedback score directly into the legacy
feedback_scores ClickHouse table via the author=null DAO path (the
public API always routes to authored_feedback_scores), then triggers
the natural workspace-version determination flow via
workspaceResourceClient.getWorkspaceVersion so the probe sees the
legacy rows and persists has_legacy_scores=true the same way it does
in production. Stats assertions use whole-object recursive comparison.
- ProjectsResourceTest: multi-project /projects/stats path.
- MultiValueFeedbackScoresE2ETest: single-project /traces/stats and
  /spans/stats paths (new SQL templates on the span side).
Drops the previous flag=false branch on the multi-project test — that
state is impossible when legacy data exists, the existing
getProjects__whenHasLegacyScoresFlipped__thenStatsStayConsistent test
already covers the no-legacy-data case.
- [NA] [SDK] [DOCS] Update automatically OpenAPI spec and Fern code (#6734)
Co-authored-by: Andres Cruz andresc@comet.com
- [NA] [BE] Update model prices file (#6736)
Co-authored-by: Andres Cruz andresc@comet.com
- [NA] [BE][FE] chore: sync provider model definitions (#6737)
Co-authored-by: Andres Cruz andresc@comet.com
- build(deps): bump dev.langchain4j:langchain4j-bom in /apps/opik-backend (#6738)
Bumps dev.langchain4j:langchain4j-bom from 1.14.0 to 1.15.0.
- Release notes
- Commits
updated-dependencies:
- dependency-name: dev.langchain4j:langchain4j-bom
  dependency-version: 1.15.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  ...
Signed-off-by: dependabot[bot] support@github.com
Co-authored-by: Andres Cruz andresc@comet.com
- build(deps-dev): bump com.diffplug.spotless:spotless-maven-plugin (#6739)
Bumps com.diffplug.spotless:spotless-maven-plugin from 3.4.0 to 3.5.1.
updated-dependencies:
- dependency-name: com.diffplug.spotless:spotless-maven-plugin
  dependency-version: 3.5.1
  dependency-type: direct:development
  update-type: version-update:semver-minor
  ...
Signed-off-by: dependabot[bot] support@github.com
Co-authored-by: Andres Cruz andresc@comet.com
- build(deps): bump org.redisson:redisson in /apps/opik-backend (#6740)
Bumps org.redisson:redisson from 4.3.1 to 4.4.0.
updated-dependencies:
- dependency-name: org.redisson:redisson
  dependency-version: 4.4.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  ...
Signed-off-by: dependabot[bot] support@github.com
Co-authored-by: Andres Cruz andresc@comet.com
- [INFRA] feat: respect pre-set port env vars when initializing worktree ports (#6558)
init_worktree_ports() unconditionally overwrote NGINX_PORT (and every sibling
port) with BASE + PORT_OFFSET, so callers couldn't pin a single port via env
without also disabling the worktree offset for the rest. Switch each assignment
to the ${VAR:-default} idiom so pre-set values win, fix the docker-compose
default for OPIK_REVERSE_PROXY_URL to follow NGINX_PORT, and document the knob
in the docker-compose README.

Co-authored-by: Claude Opus 4.7 (1M context) noreply@anthropic.com
Co-authored-by: Andres Cruz andresc@comet.com
- [OPIK-6544] [FE] feat: add edit option to dataset item row menu (#6741)
Wires setActiveRowId into the row-actions cell via customMeta and adds
an Edit menu item above Delete (with a separator). Works for both
regular dataset items and test suite items since both share the
DatasetItemsTab and DatasetItemRowActionsCell wiring.

Co-authored-by: Claude Opus 4.7 (1M context) noreply@anthropic.com
- [NA] [QA] fix: wait for traces table to fully render before scanning Moderation header (#6743)
The online-scoring moderation flow test polled table thead th with a fixed
3s wait per reload. On production the traces table renders static columns
first and appends dynamic feedback-score columns after a second API call,
so the scan ran before the Moderation header was mounted and every retry
re-raced the same too-short timer. All six provider/model variants were
failing identically only on the prod sanity launches while local, staging,
self-hosted, and post-merge stayed green.

Switched each retry to use Playwright auto-retrying assertions:
toHaveCount on tbody tr to confirm the rows are present, then
toBeVisible on the Moderation header before reading the column index.
Per-attempt budget is bounded so the 25-attempt loop still fits in the
120s test timeout in the happy path.
- docs(changelog): weekly changelog for 2026-05-19 (#6742)
- docs(changelog): weekly changelog for 2026-05-19
Covers releases 2.0.32–2.0.37. Highlights:
- Python/TS SDK client-side prompt caching with trace metadata injection
- New opik migrate dataset CLI command
- opik connect CLI improvements (Rich errors, auto-configure, instant disconnect)
- Environments: auto-assigned colors, inline validation errors, SDK field preservation
- Playground: suppress Gemma 4 reasoning traces, updated Gemini/Vertex AI defaults
- Google ADK integration re-patching fix
- Bug fixes: silent dataset item loss, self-hosted onboarding loop, CSV upload visibility
- Performance: dataset streaming, workspace selector load time
https://claude.ai/code/session_01T9yZkbs2iJSNzHYEFGr4ms
- docs(changelog): remove opik migrate dataset section (to be documented separately)
https://claude.ai/code/session_01T9yZkbs2iJSNzHYEFGr4ms
- docs(changelog): remove introductory boilerplate line
https://claude.ai/code/session_01T9yZkbs2iJSNzHYEFGr4ms
- docs(changelog): move environment items to bug fixes section
Environment feature was already announced in 2026-05-12; these are
follow-up fixes and improvements, not a new feature introduction.

https://claude.ai/code/session_01T9yZkbs2iJSNzHYEFGr4ms
- docs(changelog): consolidate Playground and ADK sections into bug fixes
Keep prompt caching and opik connect as the two featured sections.
Playground fixes, ADK re-patching fix, and all other items are now
under Bug Fixes & Improvements.

https://claude.ai/code/session_01T9yZkbs2iJSNzHYEFGr4ms

Co-authored-by: Claude noreply@anthropic.com
- [OPIK-6417] [SDK] feat: cascade trace + span comments in opik migrate dataset (#6714)
- [OPIK-6417] [SDK] feat: cascade trace + span comments in opik migrate dataset
Slice 4 of opik migrate dataset. The Slice 3 cascade already
copies traces + spans + feedback scores + assertion results, but
comments live as READ_ONLY on the trace/span Public view -- they
can't ride along create_traces / create_spans and were
silently dropped at the destination.

Re-emit them via the dedicated single-comment write endpoints
(POST /v1/private/traces/{id}/comments,
POST /v1/private/spans/{id}/comments) after the destination
trace/span flushes. Each POST is wrapped with the existing
ensure_rest_api_call_respecting_rate_limit helper; order is
preserved by iterating the source comments list in place so the
destination read order matches.

Counters surface on a new cascade_experiments_summary audit
record alongside traces_migrated / spans_migrated (additive --
the existing cascade_experiments action record shape is
unchanged).

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- test(e2e): use verifiers for comment round-trip assertions
Extend verifiers.verify_trace / verify_span with an optional
comments kwarg that asserts equality on the source-ordered list
of comment texts. Refactor the cascade-comments e2e to delegate
to the shared verifiers instead of hand-rolling REST polling and
content/order checks.

Addresses PR review feedback from baz-reviewer[bot] on PR #6714:
comment text/order is part of the TracePublic.comments /
SpanPublic.comments contract, so it belongs on the shared
verifier surface (consistent with how the sibling
test_migrate_dataset__evaluate_shape__round_trips test already
verifies traces via verifiers.verify_trace).

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com

Co-authored-by: Claude Opus 4.7 (1M context) noreply@anthropic.com
- Update versions to 2.0.38 and bump base version to 2.0.39
- [OPIK-6482] [BE] fix: add (workspace_id, scope, id) index on dashboards to avoid filesort OOM (#6733)
- [OPIK-6482] [BE] fix: add (workspace_id, scope, id) index on dashboards to satisfy ORDER BY from index and avoid filesort OOM
The list-dashboards query
SELECT * FROM dashboards WHERE workspace_id = ? AND scope = ? ORDER BY id DESC LIMIT ? OFFSET ?
falls back to filesort because no index covers both the equality filter and the ORDER BY key. With SELECT *
including the JSON config column, the filesort step copies the full row into the sort buffer, and Aurora
MySQL 8.0 raises ER_OUT_OF_SORTMEMORY (errno 1038) even at LIMIT=1 due to JSON addon-field sizing in the sort
buffer. Observed in Netflix opik state DB: 168 failed calls between 2026-05-02 and 2026-05-12, surfacing as
500s for the affected workspace.

Adding (workspace_id, scope, id) lets the optimizer satisfy the equality + ORDER BY id DESC from index order
via a backward index scan, eliminating the filesort step entirely. The JSON column is still selected but is
never copied into a sort buffer.

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- [OPIK-6482] [BE] address review: bump migration to 000070 and add trailing blank line
- Bump 000069 -> 000070 (000069 was taken by add_has_legacy_scores_to_workspaces after this branch was opened)
- Add trailing blank line per Liquibase migration convention
Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com

Co-authored-by: Claude Opus 4.7 (1M context) noreply@anthropic.com
- [OPIK-6511] [FE] fix: hide Assistant sidebar wrapper when ollieEnabled toggle is off (#6725)
- [OPIK-3662] [INFRA] feat: add actionlint workflow validation (#6600)
- [OPIK-3662] [INFRA] feat: add actionlint workflow validation
Adds actionlint as a CI gate for changed GitHub Actions workflow files,
plus a matching pre-commit hook with graceful degradation when the
binary is not installed locally. Both run with SHELLCHECK_OPTS=--severity=warning
so the pre-existing SC2086 backlog (tracked in OPIK-6323) does not gate
PRs while real warning+ findings still do.

Real defects fixed alongside the enabling change:
- 9 outdated action versions (publish_cursor_extension.yml@v2,
  setup-python@v3/v4 in 4 files, setup-node@v3 in 3 files, cache@v3,
  release-drafter@v5)
- Script-injection risk in trigger_test_env_on_label.yaml: pull_request.head.ref
  is now passed via env: instead of interpolated into actions/github-script
  body
- Deprecated set-output -> $GITHUB_OUTPUT (documentation_cookbook_tests.yml)
- SC2046 unquoted command substitution in same file
- workflow_call default value on a required input (build_and_publish_sdk.yaml)
- Undeclared workflow_dispatch input ALLURE_JOB_RUN_ID (test_docs_links.yml)
Deferred to OPIK-6323: 168 SC2086 quoting cleanup, 9 remaining cosmetic
events default removals, 2 remaining expression issues, and tightening
the gate to scan all files at default shellcheck severity.

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- fix(ci): use unicode octopus in workflow name
:octocat: is a GitHub markdown shortcode and only renders in places
that go through GFM (PR descriptions, comments). It does not render in
the Actions UI workflow name, where the raw string :octocat: Lint Workflows was showing instead. Switching to the unicode octopus 🐙
which the Actions UI renders correctly.

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- fix(ci): pin actionlint install via gh release + attestation verify
Addresses Baz comment on PR #6600: the previous bash <(curl ...) install
pulled the download script from a mutable URL with no integrity check.

Replaces it with the upstream-recommended secure install path from
https://github.com/rhysd/actionlint/blob/main/docs/install.md:
1. Resolve the latest release tag via gh release view
2. Download the linux_amd64 tarball as an immutable release asset
3. Verify its GitHub artifact attestation (Sigstore-signed by rhysd's
  release CI) before executing — verifies provenance, not just bytes
4. Echo the version in the workflow log for traceability
This is strictly stronger than the SHA256-pinning Baz suggested:
attestation verification confirms the artifact was built by the
official rhysd/actionlint release workflow, not just that the bytes
match a hardcoded hash. gh is preinstalled on GitHub-hosted runners,
so no extra setup needed.

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- feat(ci): also lint workflows that use changed composite actions
Composite actions in .github/actions/*/action.yml can't be linted by
actionlint directly (it treats them as workflows and complains about
missing on:/jobs:). They're validated transitively when a workflow
that references them is linted.

Extends the gate (CI + pre-commit) to detect when a composite action
is changed without a workflow edit, find the workflows that reference
it via uses: ./.github/actions/<name>, and add those to the lint
set. Avoids the full-tree fallback that would have tripped on the
pre-existing baseline findings deferred to OPIK-6323.

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- refactor(ci): drop unnecessary const layer in trigger_test_env_on_label
Addresses liyaka's review comment on PR #6600: the intermediate const headRef = process.env.HEAD_REF and const baseVersion = ... lines
added ceremony for values used once. Inlines ${process.env.HEAD_REF}
and ${process.env.BASE_VERSION} directly in the template literal.

The env: block is preserved — that's the actual security primitive.
Passing ${{ github.event.pull_request.head.ref }} via env: (rather
than inlining it into the script body) prevents script injection,
because env: values are read at runtime via process.env rather than
substituted into the script source by ${{ }} expansion. Added a
comment block above the env: keys explaining this so the indirection
is no longer mysterious.

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com

Co-authored-by: Claude Opus 4.7 (1M context) noreply@anthropic.com
- Update versions to 2.0.39 and bump base version to 2.0.40
- [OPIK-6580] [BE] [CI] fix: clear actionlint events and expression findings (#6747)
Removes 10 dead default: values on workflow_call inputs that are
also marked required: true (actionlint [events]) across 7 workflows,
and resolves the two remaining [expression] findings:
- quickstart_guide_snippets_test.yml: declare ALLURE_JOB_RUN_ID as a
  workflow_dispatch input so github.event.inputs.ALLURE_JOB_RUN_ID
  is defined.
- release-wrapper-release-n-deploy.yml: drop the debug echo "url: ${{ inputs }}"
  that triggered whole-object eval (next line already echoes
  toJSON(inputs)).
Parent: OPIK-6323. First of 5 subtasks; no SC2086 quoting work here.

Co-authored-by: Claude Opus 4.7 (1M context) noreply@anthropic.com
- [OPIK-6516] [SDK] feat(runner): add opik connect/endpoint stop to cleanly terminate local runners (#6730)
- [NA] [SDK] refactor: group opik connect/endpoint into cli/local_runner with Rich error UX
- Move connect.py, endpoint.py, _run.py, pairing.py, error_view.py under
  cli/local_runner/ so the pairing flow is one cohesive subpackage.
- Banner now shows Workspace alongside Opik URL and Project, with values
  rendered bold so they read before the pairing link.
- Replace the inline error string with a Rich-rendered labelled block
  (Reason / Workspace / URL / Config / Fix / Docs / Run) via a new
  RichClickError + build_config_error_block factory. Plain text stays on
  .message for Sentry/tests; Rich output is used in .show().
- Detect "no config file at ~/.opik.config" and surface a clear
  "run opik configure" call-to-action on every failure path.
- Drop default values on internal kwargs that production callers always
  supply (workspace, base_url, config_file_exists, create_if_missing, etc.)
  so signatures reflect actual usage; add test helpers to keep call sites
  readable.
Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- [NA] [SDK][FE] feat: auto-configure on connect/endpoint + pairing context UX
- opik connect/endpoint auto-launch opik configure when no config file
  exists. Skipped on --non-interactive, --headless, no TTY, or when an
  API key is already supplied via --api-key / OPIK_API_KEY.
- Extract preflight helpers (should_create_project, maybe_auto_configure)
  into cli/local_runner/preflight.py so tests can target a public surface.
- Pair URL now carries &url=<ui-url> with the /api suffix stripped, so
  the pairing page can show the user-facing instance address.
- Pairing error screens render a labelled context card under the subtitle:
  "Workspace: X / Pairing with Opik at: " — URL is a clickable link
  that opens the Opik UI in a new tab.
- scripts/dev-runner.{sh,ps1} export TOGGLE_FORCE_WORKSPACE_VERSION=version_2
  by default so a fresh local backend doesn't trip the "Workspace upgrade
  required" pairing screen. Override via TOGGLE_FORCE_WORKSPACE_VERSION=disabled.
Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- [NA] [FE] feat(pairing): reword settings card and reorder Opik URL/Workspace
- Add caption "The CLI tried to pair using these Opik settings:" above
  the card so the framing is unambiguous — these are the values the CLI
  used, not the user's current session.
- Rename "Pairing with Opik at" → "Opik URL" so the labels match the CLI
  banner (Opik URL / Workspace) verbatim.
- Reorder rows: URL first, then Workspace — same order as the banner.
Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- [NA] [SDK][FE] feat(pairing): include project name in pair link and context card
- CLI build_pairing_link now accepts project_name and appends a URL-encoded
  &project=<name> query param. run_pairing forwards it through.
- PairingPage reads project from the query; PairingStatusScreen renders
  a new Project row under Workspace in the error context card.
- Project names with spaces / / round-trip safely via percent-encoding;
  added a regression test covering the encoding shape.
Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- fix lint
- fix(pairing): scheme allow-list on Opik URL link + URL-encode workspace
Addresses PR #6691 review feedback:
- PairingStatusScreen: validate expectedBaseUrl with URL() and only
  render it as a clickable <a> when the scheme is http/https; otherwise
  render as plain text. Prevents a crafted pair URL like
  ?url=javascript:alert(1) from producing a clickable script link.
- PairingPage: thread expected workspace/project/baseUrl through the
  invalid_link branch as well, so the CLI context card surfaces even
  when the link fragment is malformed.
- PairingStatusScreen: loosen showWorkspaceContext to fire when any of
  workspace/project/url is present; gate the Workspace row individually.
- pairing.py: URL-encode the workspace query param the same way as
  url and project, so workspace names containing &/= don't split
  the query and confuse the FE's URLSearchParams parsing. Added a
  regression test covering the encoding.
Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- fix(tui): use solid bullet for banner mark
The combining-enclosing-circle glyph (⠀⃝) failed to render on
terminals that lack the combining mark, falling back to two red
placeholder boxes before "opik". Switch to U+25CF (●), which renders
consistently as a solid colored dot.

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- refactor(preflight): skip redundant project lookup when preflight already 404'd
should_create_project now returns (create_if_missing, known_missing) so
the interactive path — which already observed the 404 before prompting —
can tell resolve_project_id to skip the duplicate lookup and go straight
to _create_project. Headless still leaves known_missing=False so the
resolver can find an already-existing project across re-runs.

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- [OPIK-6497] [SDK][FE] feat(runner): notify backend on close for instant disconnect
- Supervisor calls disconnect_runner on shutdown (Ctrl+C, SIGTERM, 410
  eviction, clean exit) so the FE flips the runner card off before the
  heartbeat TTL expires. Best-effort: server-side cleanup is idempotent
  and the heartbeat-TTL reaper picks up anything we miss.
- activate.py restores default signal handlers and raises KeyboardInterrupt
  on first signal so a second Ctrl+C force-exits when the child wedges on
  its own SIGINT path. Bound supervisor's graceful timeout to 5s.
- Tighten FE poll interval to 1s so the disconnect surfaces in the runner
  card immediately.
- Cover supervisor disconnect on shutdown, 410 eviction, ApiError tolerance,
  and end-to-end SIGINT/SIGTERM delivery via a subprocess driver.
Implements OPIK-6497.

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- revert(activate): drop SIG_DFL + KeyboardInterrupt force-kill path
The two-stage signal handling (raise KeyboardInterrupt on first signal,
restore SIG_DFL so a second signal force-kills) was a safety net for
wedged agents that swallow KeyboardInterrupt. Removing it: the supervisor's
graceful timeout + SIGKILL escalation already covers wedged children, and
keeping the handler purely cooperative avoids changing the SIGINT contract
that frameworks (uvicorn, etc.) may rely on.

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- [OPIK-6516] [SDK] feat(runner): add opik connect/endpoint stop to cleanly terminate local runners
Adds first-class stop subcommands so Ollie (and humans) can tear down a
headless runner without resorting to pkill on the process group, which often
left orphaned children and stale "Endpoint open" state in the Agent Sandbox.
- opik connect stop --project X | --all | --runner ID and the same for
  endpoint. Each Supervisor writes a small JSON lock file under
  ~/.opik/runners/ on startup (pid, runner_id, runner_type, project,
  workspace, started_at) and removes it in the shutdown finally; stale entries
  are purged opportunistically on the next write.
- Stop sends SIGTERM to the supervisor pid, which routes through the
  OPIK-6497 handler that calls disconnect_runner before exit — the FE flips
  off on its next poll instead of waiting out the heartbeat TTL. SIGKILL is
  reserved for unresponsive supervisors and skips the backend notification
  (the reaper picks them up).
- opik connect and opik endpoint become Click groups with a hidden _run
  subcommand and a shared RunnerGroup.resolve_command fallback, so the
  legacy opik endpoint --project X -- python script.py invocation form
  still works alongside the new stop subcommand. --project is now
  required at the Click layer instead of validated by hand.
- Headless endpoint pairing now surfaces a "Paired ✓" panel with a direct
  Agent Playground link (connect keeps the generic project URL since Ollie
  is the user-facing surface there). Both browser and headless flows share
  pairing.post_pairing_url so the destination logic lives in one place.
- Internal helpers (Supervisor, launch_supervisor) drop default values on
  parameters that aren't part of the user SDK API — every caller passes
  workspace/project_name/runner_type explicitly.
- Intra-package imports in cli/local_runner/ switched to module-style
  (from . import pairing; pairing.RunnerType) for clearer call-site
  attribution and easier patch targets in tests.
Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- ci(guardrails): disable guardrails-ai integration tests
guardrails-ai is on PyPI quarantine, so the install step in the lib
integration runner fails before pytest even starts. Comment out the
job and its references in workflow_dispatch and notify-slack needs;
drop the SUITE_RESULTS entry (a YAML # would survive as literal text
into the JSON payload). Re-enable when the package is back on PyPI.

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- style(runner): collapse RunnerGroup fallback call onto one line
ruff format wants the super().resolve_command(...) call on a single
line; restore that to unblock CI lint.

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- fix(runner): address PR review on opik stop CLI
- stop.py: only drop the pid lock file when the runner actually exited.
  On SIGTERM-denied / SIGKILL-failed / still-alive paths we leave the
  file in place so a subsequent opik <type> stop can rediscover the
  live supervisor instead of being blind to it.
- pid_file.remove(): bump the OSError log line to LOGGER.error — a
  failed pid-file unlink is a real signal that the cleanup contract
  broke, not a debug-level event.
- RunnerGroup: narrow the UsageError-catch fallback. Connect (no
  positional after _run) now surfaces click's native "No such command
  'stp'" for subcommand typos instead of routing them into _run and
  printing a confusing "Missing --project". Endpoint opts in via
  accepts_positional_after_run=True so the legacy
  opik endpoint -- python script.py form keeps surfacing the helpful
  "Missing --project" error when --project is omitted.
- connect.py / endpoint.py: split --runner help text across lines and
  reword it as a usage hint ("Use when a project has more than one
  runner attached to it") instead of the opaque "disambiguates". Brings
  source lines back under the 88-char project limit.
- test_pid_file.py and test_stop.py: rename every test method to the
  python-sdk testing convention
  (test_<WHAT>__<CASE>__<EXPECTED_RESULT> / __happyflow).
Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com

Co-authored-by: Claude Opus 4.7 (1M context) noreply@anthropic.com
- [NA] [BE] fix: stop BaseRedisSubscriberTest.shouldRemoveConsumerOnStop flaking (#6754)
- [NA] [BE] fix: stop BaseRedisSubscriberTest.shouldRemoveConsumerOnStop flaking
BaseRedisSubscriber.stop() calls removeConsumer().block(longPollingDuration)
to wait for Redis to remove the consumer from the group, then disposes
consumerScheduler. If the block timeout fires first, the in-flight
removeConsumer call is cancelled by the disposal and the consumer is left
behind. TestStreamConfiguration had longPollingDuration = 100ms, which is
tighter than Redis-container response under load (CI or otherwise), so the
test failed deterministically locally (8/8) and intermittently in CI.
- Bump test longPollingDuration to 2s (prod default is 5s; this keeps tests
  fast for the happy path while matching prod's headroom ratio).
- Handle the now-reachable empty-group case: stream.listConsumers(...) returns
  an empty Mono (block() -> null) when the group has no consumers, so wrap
  with blockOptional().orElse(List.of()).
Stress-verified locally: 10/10 passing after fix (was 0/8 before).
- fix(tests): use 500ms longPollingDuration to keep both LifecycleTests and RetryTests green
The previous 2s bump fixed shouldRemoveConsumerOnStop but extended the
read-loop cycle past the 2s test budget, so autoClaim never fired and
shouldAckAndRemoveAfterMaxRetries / shouldHandleMixedSuccessRetryableAnd-
NonRetryableMessagesInSameBatch broke instead.

longPollingDuration is the XREADGROUP BLOCK timeout AND stop()'s removeConsumer
block timeout, so it has to be:
- long enough that Redis-container response under load doesn't exceed it
  (or stop() cancels the in-flight removal on scheduler disposal), and
- short enough that one chain finishes inside the 2s test wait so autoClaim
  (gated by claimIntervalRatio polls) actually runs.
500ms gives ~5x headroom for Redis response and keeps the cycle short enough:
with claimIntervalRatio=2 (the retry-tests override), autoClaim fires around
t ~= 1.2s, well inside 2s.

Stress-verified locally: 10/10 BUILD SUCCESS running both LifecycleTests and
RetryTests nested classes.
- [OPIK-6519] [BE] perf: replace FINAL with LIMIT 1 BY in trace threads closing candidate query (#6748)
- [OPIK-6519] [BE] perf: replace FINAL with LIMIT 1 BY in trace threads closing candidate query
The TraceThreadsClosingJob runs FIND_PENDING_CLOSURE_THREADS_SQL every 15s, and
on large self-hosted deployments (Uber: 132M rows, 36+ parts on trace_threads)
the FROM trace_threads FINAL + GROUP BY exceeds the 30s reactor timeout. Under
high-volume ingestion the reactor cancels but the underlying ClickHouse query
keeps running, R2DBC buffers accumulate, and the JVM crashes with OutOfMemoryError.

Replaces the FINAL+GROUP BY dedup with the same LIMIT 1 BY reverse-order pattern
introduced in OPIK-4828 for the thread-view query. Measured on opik_prod
(16.37M rows, 10 parts), median wall time drops from ~2.9s to ~0.28s (~10x),
data read from 2.68 GiB to 76 MiB (~36x), memory from 130 MiB to 36 MiB (~4x),
with the identical result set verified on the same now() snapshot and on 997
threads with active/inactive transitions in the last 24h (100% agreement on
latest status between FINAL and LIMIT 1 BY).
- chore(threads): add log_comment to pending-closure query
Routes FIND_PENDING_CLOSURE_THREADS_SQL through getSTWithLogComment so the
ClickHouse log_comment placeholder is populated, matching the rest of the
DAO. Addresses baz-reviewer feedback on PR #6748.
- [NA] [BE] Apply Spotless formatting to backend sources (#6750)
- [NA] [BE] Apply Spotless formatting to backend sources
Reformat 53 Java files under apps/opik-backend to match Spotless rules
(Google Java Format). Pure whitespace changes around @JsonView annotation
arrays and record/class braces; no behavioral changes.
- [NA] [INFRA] Add Spotless sweep SHA to .git-blame-ignore-revs
Append the branch-head SHA for the backend Spotless reformat
(6fcfa6f8a8) so git blame skips it.

Note: under squash-merge, the SHA that actually lands on main will
differ from the branch head, so a follow-up commit on main may still
be needed to record the merged SHA.
- add hiding demo project selector
- add mask to Back button
- add additional masks
- added additional step for projects test
- [NA] [CI] [GHA] security: use npm ci instead of npm install in Docker and lint workflows (#6755)
- Use npm ci in opik-frontend Dockerfile (post TanStack incident)
- Use npm ci in frontend_linter workflow (post TanStack incident)
- Use npm ci in typescript_sdk_linter and publish_cursor_extension workflows (post TanStack incident)
Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com

Co-authored-by: Daniel Dimenshtein danield@comet.com
Co-authored-by: Claude Opus 4.7 (1M context) noreply@anthropic.com
- Update versions to 2.0.40 and bump base version to 2.0.41
- [OPIK-6581] [BE] [CI] fix: quote GITHUB_OUTPUT and GITHUB_STEP_SUMMARY redirects (#6749)
- [OPIK-6456] [BE] feat: add prompt masks for non-destructive prompt resolution (#6731)
- [OPIK-6456] [BE] feat: add prompt masks for non-destructive prompt resolution
Add version_type ENUM('prompt_version','mask') discriminator column on
prompt_versions, defaulting existing rows to 'prompt_version'. This is
the schema foundation for mask overlays: masks live in the same table
as regular prompt versions but are filtered out of latest_version and
version_count queries, and selectable via mask_id at resolution time.

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- feat(prompts): add PromptVersionType enum and versionType field
Introduces the PROMPT_VERSION/MASK discriminator at the API layer.
versionType defaults to PROMPT_VERSION so existing clients are
unaffected; mask creation will set it to MASK on the embedded
PromptVersion in CreatePromptVersion.

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- feat(prompts): filter masks from list/count queries and support mask_id resolution in findById
- prompt_versions.version_type='prompt_version' filters added to:
  - latest_version and version_count subqueries in PromptDAO findById/find/findByIds/count
  - PromptDAO.findByCommit and findPromptsByCommits (commits are prompt-version-only)
  - PromptVersionDAO.find/findCount when listing by prompt_id (id-based lookups still
    return any type so masks remain reachable by id)
  - PromptVersionDAO.findByCommit
- PromptDAO.findById gains an optional mask_id that, when present, populates
  requested_version with that version row (service validates it is actually a mask)
- INSERT in PromptVersionDAO.save writes the new version_type column
- PromptVersionColumnMapper reads version_type from JSON_OBJECT payloads with a
  PROMPT_VERSION fallback for older serialized rows
Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- feat(prompts): mask-aware getById and bulk retrieveVersionsByIds in service
- getById(UUID, UUID maskId): delegates to PromptDAO.findById with the
  optional mask_id. When a mask_id is supplied and no row matches for
  the given prompt, throws NotFoundException so callers see a clean 404
  instead of a Prompt with a null requestedVersion.
- retrieveVersionsByIds(List): bulk version lookup powering the
  upcoming POST /v1/private/prompts/retrieve endpoint. Returns versions
  in whatever order PromptVersionDAO produces (id DESC); missing ids
  are dropped from the response.
- Existing getById(UUID) delegates to the new overload with maskId=null,
  so all current callers keep their behavior.
- Variable enrichment factored into enrichWithVariables() and applied
  to both latestVersion and requestedVersion on the resolution path.
Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- feat(prompts): mask_id query param on GET /prompts/{id} and POST /prompts/retrieve
- GET /v1/private/prompts/{id} accepts an optional mask_id query param.
  When set, the response Prompt has requestedVersion populated with the
  matching mask version (404 if no such version belongs to this prompt).
- POST /v1/private/prompts/retrieve accepts { ids: [...] } and returns
  List. Powers the FE flow that resolves multiple mask
  overlays in a single round-trip.
- PromptVersionIdsRequest validates 1..1000 non-null UUIDs.
Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- feat(prompts): suppress PROMPT_COMMITTED alerts and lastUpdatedAt bump on mask saves
Masks are non-destructive overlays, not real commits. They should not
trigger PROMPT_COMMITTED alert subscribers and should not move the
parent prompts last_updated_at, which the UI surfaces as recent prompt
activity. Both side effects are now gated on versionType != MASK;
regular prompt_version saves are unchanged.

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- test(prompts): cover mask creation, isolation, mask_id resolution, bulk retrieve
New PromptMasks nested class in PromptResourceTest:
- mask creation via POST /prompts/versions with version_type=mask
- masks excluded from latestVersion and versionCount on GET /prompts/{id}
- masks excluded from GET /prompts/{id}/versions and from commit-based
  lookups (/by-commit and /retrieve-by-commits)
- GET /prompts/{id}?mask_id=... populates requestedVersion with the mask
  while keeping latestVersion pointed at the latest real version
- 404 when mask_id is unknown or belongs to a different prompt
- POST /prompts/retrieve returns versions by id and rejects empty bodies
- creating a mask does not bump the parent prompts lastUpdatedAt
Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- remove unused import
- feat(local-runners): carry multiple prompt masks per job via prompt_masks
Extends CreateLocalRunnerJobRequest and LocalRunnerJob with promptMasks,
a Map<promptId, maskId> that lets a local-runner job overlay masks on
several prompts in a single run. The existing single maskId field stays
in place but is marked @Deprecated for both DTOs.

EndpointJobServiceImpl persists prompt_masks as JSON on the Redis job
hash and restores it via parsePromptMasks on every job read path
(nextJob, getJob, listJobs).

LocalRunnersResourceTest covers create + claim round-trips for the new
field and that promptMasks is null when not provided.

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- bump migration version
- address comments
- bump migration number
- [NA] [SDK] [DOCS] Update automatically OpenAPI spec and Fern code (#6766)
- [NA] [SDK] fix: preserve environment when merging opik_args (#6735)
- Create cleanup_e2e_docker.yaml
- [OPIK-6543] Fix MEMORY_LIMIT_EXCEEDED in span/trace usage queries (#6760)
- [OPIK-6499] [QA] feat: add SDK clients and use public Opik TS SDK for backend inspection (#6768)
- [OPIK-6499] [QA] feat: add SDK clients and use public Opik TS SDK for backend inspection
Add the opik npm package as a runtime dependency of the e2e suite. The
package will back both the new sdkClient.typescript half (OPIK-6499) and
the cutover of core/backend/client.ts away from openapi-fetch (the
OPIK-6126 roll-in deferred to this ticket).

openapi-fetch stays in devDependencies for now because core/backend/client.ts
still imports it; it is removed in a follow-up commit after that file is
rewritten.

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- feat(sdk-clients): add PythonSdkClient and PythonSdkBridgeError
- feat(sdk-clients): add makeTypescriptSdk factory
- feat(sdk-clients): add SdkClient interface and makeSdkClient factory
- feat(backend-client): swap openapi-fetch for new Opik().api.projects.*
- chore(deps): remove openapi-fetch (replaced by opik SDK)
- test(sdk-clients): seed test for cross-client wiring and error mapping
Co-authored-by: Claude Opus 4.7 (1M context) noreply@anthropic.com
- [NA] [DOCS] Fix broken documentation links (#6769)
- [NA] [DOCS] fix: resolve broken links across docs and redirects
- Fix /tracing/ollie → /ollie (observability, agent_sandbox, self_improving_agents)
- Fix /tracing/supported_models → /tracing/advanced/cost_tracking (integrations, concepts)
- Fix /tracing/opentelemetry/overview → /integrations/opentelemetry (new integrations)
- Fix agent_optimization/* → development/optimization-runs/* paths (latest version restructure)
- Fix /reference/python-sdk/overview → external SDK reference URL (log_traces)
- Fix TypeScript SDK evaluation page cross-links (evaluate/evaluatePrompt slugs)
- Fix /prompt_engineering/getting-started → /development/agent-configuration/* paths
- Fix changelog links (opik-university slug, log_traces path)
- Add missing metric pages (g_eval_conversation_metrics, structure_output_compliance) to docs-v2
- Update docs.yml redirect destinations from agent_optimization/* to development/optimization-runs/*
- Update docs.yml tracing/cost_tracking → tracing/advanced/cost_tracking redirect destination
Co-Authored-By: Claude Sonnet 4.6 noreply@anthropic.com
- [NA] [DOCS] fix: remove {#id} anchors from MDX heading causing acorn parse error
MDX in the latest Fern version treats {#...} in headings as JSX expressions.
Acorn rejects #id as invalid JS, breaking the preview build. Strip the anchors;
Fern auto-generates slug anchors from the heading text anyway.

Co-Authored-By: Claude Sonnet 4.6 noreply@anthropic.com
- [NA] [DOCS] ci: retrigger docs preview build
Co-Authored-By: Claude Sonnet 4.6 noreply@anthropic.com
- [NA] [DOCS] fix: remove invalid frontmatter key causing acorn parse error
pytest_codeblocks_skip with literal backslashes is not a valid JS identifier.
Fern exports frontmatter as a JS object, so acorn fails on the backslash.
This field is a pytest-codeblocks testing directive; Fern ignores it.

Co-Authored-By: Claude Sonnet 4.6 noreply@anthropic.com
- [NA] [DOCS] fix: same acorn fixes applied to v1 docs/ counterparts
- Remove {#id} anchors from docs/evaluation/metrics/g_eval_conversation_metrics.mdx
- Remove pytest_codeblocks_skip frontmatter from docs/evaluation/metrics/structure_output_compliance.mdx
Co-Authored-By: Claude Sonnet 4.6 noreply@anthropic.com
- [NA] [CI] align fern-api version to 4.71.5 across all environments
fern.config.json and the deploy workflow both pin 4.71.5, but the preview
workflow used 0.64.26, the OpenAPI auto-update workflow was unpinned (latest),
and package.json resolved to 0.64.26 via ^0.94.6. Pinning everything to
4.71.5 so local dev, preview CI, and deploy CI all use the same version.

Co-Authored-By: Claude Sonnet 4.6 noreply@anthropic.com
- [NA] [DOCS] Remove Gretel integration and fix broken links in preview
- Delete Gretel integration pages (docs/ and docs-v2/) and remove from nav — Gretel no longer exists
- Fix playground.mdx: /v1/configuration/ai_providers → /v1/workspace-settings/ai_providers (section was renamed)
- Fix langserve.mdx: /v1/tracing/integrations/langchain → /v1/integrations/langchain (integrations moved to own tab)
Co-Authored-By: Claude Sonnet 4.6 noreply@anthropic.com

Co-authored-by: Claude Sonnet 4.6 noreply@anthropic.com
- [OPIK-6532] [SDK] feat: cascade optimizations on opik migrate dataset (Slice 5) (#6746)
- [OPIK-6532] [SDK] feat: cascade optimizations on opik migrate dataset
Stops dropping optimization_id during the migrate cascade. Recreates
every optimization referencing the source dataset under the destination
project with a fresh UUID, populates plan.optimization_id_remap, and
re-points each experiment's optimization_id FK at the new destination
optimization id so the destination's Optimization Studio shows the same
rows and trial groupings as the source.
- New CascadeOptimizations planner action ordered between
  ReplayVersions and CascadeExperiments
- New cascade_optimizations module: paginated find_optimizations
  enumeration, per-row recreate with fidelity fields (name,
  objective_name, status, metadata, studio_config, last_updated_at),
  READ-ONLY aggregates (num_trials, baseline_, best_, etc.) omitted
- Experiment cascade re-points optimization_id via the remap; orphan
  path increments experiments_with_orphan_optimization_id counter
- recreate_experiment forwards optimization_id on the migrate path
- Per-optimization migrate_optimization audit records with
  source_id -> destination_id mapping; uses optimization_status key
  so the audit-level status field isn't shadowed
- Unit tests: planner ordering, optimization fidelity, RO-aggregate
  guard, empty optimization, pagination, 409 propagation, audit shape,
  studio-config Read->Write reconstruction, experiment FK re-point,
  orphan path, no-source-id path
- E2E test: 1 optimization + 2 trial experiments + 1 control experiment
  round-trip with grouping preserved at destination
Implements OPIK-6532: opik migrate dataset (Slice 5): cascade optimizations
- fix(migrate): render CascadeOptimizations in plan table + progress bar
Two UX gaps spotted running the cascade against staging:
- _print_plan had no branch for CascadeOptimizations, so the
  dry-run plan table skipped row 4 entirely (jumped from 3 to 5)
  even though the action executed correctly. Add the missing branch.
- _cascade_optimizations ran silently without a Rich progress bar,
  jarring next to the verbose experiment-cascade bar that fires
  immediately after. Thread a progress_callback through
  cascade_optimizations and drive a Rich Progress block in the
  executor that mirrors _cascade_experiments's shape (one tick
  per optimization). Single-level bar -- no nested per-optimization
  detail bar -- because production datasets carry tens to low-hundreds
  of optimizations at most.
The algorithmic core stays console-agnostic; the bar is driven by the
per-optimization callback so unit tests don't need Rich in the loop.

Verified end-to-end on staging: plan table now lists cascade optimizations -> project <target> as row 4, and a "Cascading
optimizations" Rich bar fires for the optimization phase between
"Replaying versions" and "Cascading experiments".
- fix(migrate): zero-opt progress bar + share ProgressCallback type alias
Two follow-ups from PR review (#6746):
- Fix zero-optimization progress-bar bug: the empty-cascade path fired
  one terminal progress_callback(0, 0, "done"), which created the Rich
  task at total=1 but never advanced completed, leaving the bar at 0%
  for an already-finished migration. Set completed = max(total, 1)
  whenever the "done" tick fires (including first-callback creation)
  so the bar renders at 100% on both the empty and populated paths.
- Hoist ProgressCallback / InnerProgressCallback type aliases into
  a new datasets/_progress.py shared module. Removes the previous
  drift surface where the same alias was redefined in
  experiments.py + optimizations.py and re-spelled inline as
  Callable[[int, int, str], None] in version_replay.py. Single
  source of truth + cross-module convention documented (esp.
  label == "done" finalization). Re-exported from experiments.py
  with # noqa: F401 so any external imports keep working.
- [OPIK-6455] [BE] feat: add environment ownership to prompt versions (#6767)
- Add environment to prompts
- fix
- fix(prompt-envs): require existing env on PATCH and address review nits
Address PR #6767 review comments:
- PATCH /versions/{id} now returns 404 when the environment does not exist
  in the workspace, instead of silently auto-creating (which could no-op at
  the workspace env cap and leave a version pointing at an unregistered
  env). Adds EnvironmentService.existsByName + DAO countByName.
- getPromptById's environment query param now validates @Pattern + @Size,
  matching PromptVersionRetrieve / PromptVersion.
- Migration 000072: add adjacent comment above CREATE UNIQUE INDEX and a
  trailing newline, per migrations.md.
Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- test(prompt-envs): route new tests through PromptResourceClient
Replace direct client.target(...) calls in the new PromptEnvironments
nested block with PromptResourceClient helpers (matching the convention
used elsewhere in PromptResourceTest):
- callGetPrompt overload now accepts environment
- callCreatePromptVersion / callGetPromptVersion / getPromptVersion
- callSetPromptVersionEnvironment
- callRestorePromptVersion
Removed local getVersionById helper; postVersionRaw and
patchVersionEnvironment now delegate to the client.

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- refactor(prompt-envs): address review nits on PATCH env endpoint
- Rename PATCH /versions/{versionId} -> /versions/{versionId}/environments
  to scope the endpoint to environment ownership.
- Move the mask+environment cross-field constraint to a class-level
  @AssertTrue on PromptVersion so it is enforced at request validation;
  remove the redundant service-side check on POST.
- Drop LOWER() from EnvironmentDAO.countByName: the environments table
  uses utf8mb4 (case-insensitive default collation) and has a UNIQUE
  (workspace_id, name) index, so the LOWER() wrap was both unnecessary
  and index-defeating.
- Extract withEnvironmentPromotionLock helper shared by
  createVersionWithEnvironment and setVersionEnvironment.
- Update tests for the new path and the 422 status for the AssertTrue
  violation.
Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- style: format PromptVersion @AssertTrue annotation
- [NA] [SDK] [DOCS] Update automatically OpenAPI spec and Fern code (#6772)
- [OPIK-6500] [QA] feat: add base fixtures for Opik 2.0 E2E suite (#6770)
- [OPIK-6500] [QA] feat: add base fixtures for Opik 2.0 E2E suite
- feat(fixtures): add base fixture with worker-scoped clients and testNamespace
- feat(fixtures): add project fixture with SDK-create REST-delete
- feat(fixtures): add scratch-dir fixture with per-test isolation
- feat(fixtures): add failure-artifacts collector with capture API
- feat(fixtures): add composition barrel as canonical import path
- test(fixtures): add local smoke test for staging verification
- test(fixtures): remove local smoke after staging verification
- fix(artifacts): preserve only on failed/timedOut, not skipped/interrupted
- feat(playwright): wire FastAPI bridge via webServer directive
Playwright now spawns the opik-sdk-driver bridge automatically at the start
of every test run, polls /health until ready, and shuts it down on exit.

Subsequent test tickets (OPIK-6128 onwards) no longer need to manually start
the bridge for staging verification — npx playwright test just works.

reuseExistingServer is true outside CI so a locally-running bridge (from
'uv run uvicorn ...' in another terminal during development) is reused
instead of spawned twice.

env block spreads process.env then layers OPIK_URL_OVERRIDE and
OPIK_WORKSPACE on top. The bridge's Python SDK convention is
OPIK_URL_OVERRIDE; the e2e suite uses OPIK_BASE_URL — the mapping happens
here so test authors set one URL var.

Verified against staging: full seed suite passes with bridge auto-spawned
(3/3 sdk-clients tests, 6.6s, no manual uvicorn start).
- [OPIK-6524][BE] Cost Tracking: Add Missing Models and Providers - Elastic Inference Service OTEL (#6759)
- [OPIK-6524][BE] Add support for Elastic Inference Service model/provider resolution in OpenTelemetry pipeline and cost calculations
- Introduced ElasticInferenceServiceResolver to rewrite EIS provider/model attributes to underlying Opik canonical values for accurate cost lookup and downstream filtering.
- Added model_prices_overrides.json to define pricing and alias mappings for EIS-supported models.
- Updated OpenTelemetryMapper to integrate EIS resolution logic, ensuring original provider/model values are recorded in span metadata for traceability.
- Included comprehensive tests for EIS integration, alias resolution, and dot-to-hyphen normalization in cost calculations.
- Introduced end-to-end tests ensuring correct attribute rewriting, provider/model mapping, and non-zero cost verification for supported EIS models.
- [OPIK-6524][BE] Fixed formatting issue in ElasticInferenceServiceResolver
- [OPIK-6524][BE] Update log level and formatting for model price overrides error handling in CostService
- [OPIK-6524][BE] Refactor alias handling in CostService to ensure order-independent overrides
- Introduced two-pass processing logic to separate direct overrides and alias resolution.
- Utilized List<Map.Entry<String, ModelCostData>> for intermediate storage of alias mappings to ensure deterministic behavior during iteration.
- [OPIK-6524][BE] Consolidate EIS provider/model mapping tests into parameterized test
- Replaced individual test cases with a single @ParameterizedTest using CsvSource for provider/model mapping validation.
- Simplified test logic to ensure consistency across multiple provider prefixes.
- [OPIK-6524][BE] Updated code comments to clarify the logic behind direct overrides and alias resolution ordering.
- Update versions to 2.0.41 and bump base version to 2.0.42
- [OPIK-6311] [BE] perf: slim count path + bypass traces hop in target-projects pre-query (#6756)
- [OPIK-6311] [BE] perf: slim count path + bypass traces hop in target-projects pre-query
Two complementary fixes for the experiment-comparison page-load latency:
1. DatasetItemVersionDAO.SELECT_DATASET_ITEM_VERSIONS_WITH_EXPERIMENT_ITEMS_COUNT —
  add a <if(slim_count)> branch that reads count(DISTINCT stable_dataset_item_id)
  directly from experiment_item_aggregates, pre-pruning the dataset_item_versions
  FINAL lookup by (workspace_id, dataset_id). The gate fires when push_top_limit
  would also fire AND there are no filters/search — i.e. the data side already
  takes the fast path, but the count was forced through the heavy CTE chain.
2. ExperimentDAO.SELECT_TARGET_PROJECTS — read project_id directly from the
  experiments table when populated, fall back to experiment_aggregates.project_id
  for experiments without it on the row, and only traverse experiment_items ->
  traces for the experiments missing from BOTH (legacy fallback). Removes the
  workspace-wide experiment_items + traces scan that was the dominant cost.
Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- [OPIK-6311] [BE] fix: scope slim count to requested dataset via experiment_aggregates
Address review feedback (Baz HIGH on DatasetItemVersionDAO.java:461): the
<if(slim_count)> branch was only filtering experiment_item_aggregates by
workspace_id, so EIA rows from other datasets in the same workspace would be
counted. EIA has no dataset_id column, so add an IN subquery against
experiment_aggregates (whose sorting key starts with (workspace_id, dataset_id,
id) → no FINAL needed). The optional experiment_ids predicate now lives inside
that subquery and stays consistent with the legacy branch's scoping.

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- [OPIK-6311] [BE] chore: drop SQL line-comments from SELECT_TARGET_PROJECTS
The CTE names already convey the intent (eia_projects, legacy_scope,
legacy_trace_scope, legacy_projects) and the surrounding Javadoc on the constant
explains the fast/fallback layering. The inline comments were redundant.

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- [OPIK-6311] [BE] perf: extend slim_count to non-search filtered counts
Drops the no-filter precondition on the slim_count gate. Mirroring
top_dataset_items in the data path, the slim count now renders:
- EXPERIMENT_ITEM filters directly on eia
- FEEDBACK_SCORES_AGGREGATED + ..._IS_EMPTY filters on eia.feedback_scores
- DATASET_ITEM filters via a dataset_items_filtered_ids CTE matched against
  eia.dataset_item_id IN arrayJoin([id, row_id]) (same stable-id resolution
  shape the data path uses)
Gate is now hasAggregated && !hasRaw && !hasSearch. Search still falls back
to the legacy heavy path because it requires the trace-side join.

Benchmarked on a production-shaped fixture (workspace with ~2.3M items in
EIA): no filter 3.4 s, EXPERIMENT_ITEM filter 0.4 s, FEEDBACK_SCORES_AGGREGATED
0.5 s, AGGREGATED_IS_EMPTY 1.2 s, combined EIA filters 0.6 s. DATASET_ITEM
filter still pays the DIV pre-resolve scan but skips the heavy upstream CTEs.

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- [OPIK-6311] [BE] fix: skip ZERO_UUID aggregate project_id in SELECT_TARGET_PROJECTS
When populateAggregations runs before any traces exist for an experiment,
experiment_aggregates.project_id is stored as ZERO_UUID
(00000000-0000-0000-0000-000000000000). The legacy SELECT_TARGET_PROJECTS
computed projects from the traces table, so it correctly returned an empty
set in that state. The earlier rewrite read experiments.project_id directly
and returned the experiment's project_id even with no traces, which then
narrowed downstream trace filters to a project_id that had no trace data —
causing FIND_GROUPS / FIND_GROUPS_AGGREGATIONS to return empty
(ExperimentAggregatesIntegrationTest.experimentsWithZeroUuidAggregateProjectIdAreVisibleViaFallback).

The fix:
- Drop the experiments.project_id UNION branch entirely.
- Filter ZERO_UUID out of eia_projects via AND project_id != :zero_uuid.
- Fall back to the experiment_items -> traces traversal for any experiment
  without a valid (non-ZERO_UUID) aggregate.
This matches the legacy "traces are the source of truth" semantics: the
target_project_ids set only includes projects backed by actual trace data.
On observed prod workspaces this drops the result from a buggy 10 (including
ZERO_UUID and three projects with no traces) to the correct 6, latency ~1.3 s
vs the legacy ~23 s.

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- [OPIK-6311] [BE] fix: dedup eia_projects via sorting key instead of FINAL
experiment_aggregates is ReplicatedReplacingMergeTree on
(workspace_id, dataset_id, id) with last_updated_at as the version column.
The earlier eia_projects CTE projected non-key columns (project_id) without
deduplication, so it could read a stale project_id from a prior version row
and incorrectly suppress the experiment_items -> traces fallback via
legacy_scope.

Switch to the same LIMIT 1 BY sorting-key dedup pattern OPIK-6519 uses for
trace_threads: read all candidate rows, sort by sorting key DESC +
last_updated_at DESC, LIMIT 1 BY the sorting key, then filter ZERO_UUID on
the deduped result. Same latency as FINAL on this query (the IN clause
keeps the candidate set small) without the merge cost.

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- [OPIK-6311] [BE] style: bind zero_uuid as UUID, matching other call sites
Address review nit: every other zero_uuid bind in this file passes the
UUID object directly (lines 1940, 1971, 2003, 2209, 2251, 2619); the new
getTargetProjectIdsForExperiments site was the only one passing .toString().
Functionally equivalent, just consistent.

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- [OPIK-6311] [BE] docs: note slim_count CTE divergence on the constant javadoc
Address review nit: flag that the slim branch's dataset_items_filtered_ids
mirrors the push_top_limit one minus the dataset_version_id predicate.

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com

Co-authored-by: Claude Opus 4.7 (1M context) noreply@anthropic.com
- [OPIK-6574] [SDK] feat: add opik migrate prompt command (Slice 6) (#6752)
- [OPIK-6574] [SDK] feat: add opik migrate prompt command (Slice 6)
Introduce opik migrate prompt NAME --to-project=B as a peer to
opik migrate dataset. Moves a single prompt (with its full version
history) from one project to another within the same workspace.

The plan emits RenameSource then CreateDestination then ReplayVersions:
- Rename frees the workspace-unique name; re-passes description + tags
  since the BE rename PUT wipes description (no COALESCE in PromptDAO).
- CreateDestination omits template so the BE does NOT auto-mint a
  v1 (PromptService.create skips version creation when template is
  empty), letting the replay loop carry every source commit verbatim.
- ReplayVersions paginates get_prompt_versions, reverses to
  oldest-first (BE orders pv.id DESC; UUIDv7 ids -> chronological),
  and POSTs each version with the source commit carried verbatim.
  Fresh dest prompt_id means the (workspace_id, prompt_id, commit)
  unique key never collides.
Shared helpers (_create_destination_prompt and
_replay_prompt_versions) are exposed module-level so Slice 7
(OPIK-6575, dataset cascade pulling prompts) can import them
unchanged. MigrationPlan.prompt_version_id_remap carries the
source -> dest version id map for downstream FK rewrites.

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- refactor(migrate): drop --from-project from dataset, redundant under workspace name uniqueness
The dataset (workspace_id, name) unique constraint (see
000001_init_script.sql) is identical to the prompt-library one
(000004), and the datasets INSERT has no ON DUPLICATE IGNORE
fallback, so a dataset name can only ever resolve to a single row
workspace-wide. The Slice 1 docstring claim that "workspace-scoped +
project-scoped collisions are possible" was wrong — there's no soft-
delete carve-out either.

This aligns the dataset and prompt commands on the same resolution
model:
- Drop --from-project from the dataset Click command.
- Drop from_project from build_dataset_plan and resolve_source;
  drop the now-unused project_id filter from _iter_dataset_pages.
- Delete AmbiguityError (unreachable: workspace uniqueness invariant
  is enforced by the BE). If the BE invariant is ever violated, the
  resolver raises ConflictError instead so the failure is still
  explicit.
- Update unit + e2e tests that passed --from-project /
  from_project=None.
Audit-log JSON keys named from_project (e.g. on replay_versions
records) are kept as-is since they're wire artefacts for log
consumers, not flag references.

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- refactor(migrate): extract shared resolver helpers; drop unused prompt_id_remap
Addresses two PR-review findings on #6752:
1. Resolver dedup (baz-reviewer #4). With two concrete implementations
  in tree (datasets + prompts), the right shape is now obvious — the
  moment _base.py's own docstring anticipated. Extracts the four
  entity-agnostic helpers (iter_pages, project_name_for_row,
  ensure_destination_project_exists, name_taken_in_workspace) plus
  the workspace-uniqueness resolve_unique_source_by_name to a new
  cli/migrate/_resolver.py. Each entity's resolver.py keeps only
  the Resolved* dataclass and the entity-specific Fern-surface
  binding (_datasets_list_fn / _prompts_list_fn). Net -198 LOC
  across the touched files.
2. Drop prompt_id_remap from MigrationPlan (baz-reviewer #2). The
  field was declared and documented as "populated by the executor for
  Slice 7", but nothing in the codebase wrote to it (Fern's
  create_prompt returns None, so capturing the dest id would have
  required a follow-up get_prompts lookup that nobody asked for).
  The architecturally consistent fix is the dataset cascade's pattern:
  carry dest_name + dest_project_name through and resolve the
  destination row by name at the cascade's apply time
  (cli/migrate/datasets/experiments.py does exactly this for the
  destination dataset). Workspace uniqueness
  (UNIQUE (workspace_id, name)) makes the lookup unambiguous once
  the rename has fired. Slice 7 will resolve by name when it needs
  the dest prompt id.
323 tests under tests/unit/cli/ pass; make precommit-sdks clean.

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- fix(migrate): restore --from-project + fix case-sensitivity + finish resolver dedup
Restore --from-project on dataset and add to prompt; per the original
Notion design intent, this flag is an optional source-scope hint:
smaller BE result set, sharper "not found in project 'A'" error
message, and explicit targeting of V1/workspace-scoped vs
project-scoped rows during auto-migration. My earlier removal of the
dataset flag (commit 7537e4f64) traded these capabilities for nothing —
workspace name uniqueness makes the source resolution unambiguous
regardless, but the flag still earns its keep on perf and UX axes.

Also fix two correctness/cleanup gaps raised in baz-reviewer's R2 pass
on PR #6752:

[C] Case-sensitive client-side gate vs case-insensitive BE collation
(#3261085067). The BE state DB is utf8mb4_unicode_ci on every
table, so WHERE name LIKE '%mydata%' returns MyData rows. The
previous if row.name != name: continue is Python case-sensitive,
discarding rows the BE legitimately matched — opik migrate prompt MyData against a mydata row raised PromptNotFoundError. Same
pre-existing bug in name_taken_in_workspace (a casing-different
collision would be missed). Centralised the comparison in a new
_name_matches(row_name, lookup_name) helper using str.casefold,
documented the BE invariant it mirrors.

[D] Closure duplication between _datasets_list_fn and
_prompts_list_fn (#3261085070). Extracted make_list_fn(find_fn, *, name, project_id) in _resolver.py; both entity resolvers
now call it with their entity-specific Fern method
(find_datasets vs get_prompts). Net -16 LOC; future
entities and signature changes touch one spot.

A and B (broad-except concerns at _resolver.py:106 and :153) are
skipped — they re-flag the same pattern Slice 1's
datasets/resolver.py shipped with, on the same intentional rationale
(best-effort cosmetic / suggestion lookup; should not abort a
migration), and baz-reviewer acknowledged the equivalent original on
the prior round.

324 tests under tests/unit/cli/ pass; make precommit-sdks clean.

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com

Co-authored-by: Claude Opus 4.7 (1M context) noreply@anthropic.com
- [OPIK-6582] [INFRA] [CI] fix: quote SC2086 vars across 6 high-density workflow files (#6753)
- [OPIK-6582] [INFRA] [CI] fix: quote SC2086 vars in build_and_push_docker.yaml
Quotes 14 GITHUB_OUTPUT/GITHUB_STEP_SUMMARY redirects and the docker
pull/save :$TAG references. The two SC2086 hits on
docker buildx imagetools create $TAG_ARGS $DIGESTS are intentional
argument-list expansion — annotated with # shellcheck disable=SC2086
and a rationale comment instead of quoting (quoting would collapse
each into a single argument and break the docker invocation).

Parent: OPIK-6323. Third of 5 subtasks; SC2086 mid-density cluster
file 1 of 6.

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- fix(typescript_sdk_integration_publish): quote SC2086 redirects in Summary step
All 14 SC2086 findings were >> $GITHUB_STEP_SUMMARY lines in the
Summary step. Quoting is purely defensive — the env var is a runner-
managed file path.
- fix(sync_provider_models): quote SC2086 vars; mark FORCE_REGEN_FLAG arg-list intentional
Quotes 9 GITHUB_ENV/OUTPUT/STEP_SUMMARY redirects plus the
[ \$EXIT_CODE -eq 1 ] integer comparison. One arg-list case
(python ... \$FORCE_REGEN_FLAG ...) is intentionally unquoted —
the variable holds either --force-regen or empty string, acting
as an optional argv slot. Annotated with # shellcheck disable=SC2086
and rationale.
- fix(sdks_generate_openapi_spec_and_fern_code): quote SC2086 GITHUB_* redirects
All 10 SC2086 findings were >> $GITHUB_OUTPUT / >> $GITHUB_ENV
redirects. Quoting is purely defensive — runner-managed file paths.
- fix(opik_wizard_publish): quote SC2086 redirects in Workflow summary step
All 10 SC2086 findings were >> $GITHUB_STEP_SUMMARY in the
Workflow summary block.
- fix(typescript_sdk_publish): quote SC2086 redirects in Summary step
All 7 SC2086 findings were >> $GITHUB_STEP_SUMMARY in the Summary step.

Co-authored-by: Claude Opus 4.7 (1M context) noreply@anthropic.com
- [NA] [BE][FE] chore: sync provider model definitions (#6779)
Co-authored-by: Andres Cruz andresc@comet.com
- [NA] [BE] Update model prices file (#6778)
Co-authored-by: Andres Cruz andresc@comet.com
- [OPIK-5333] Inject trace spans into LLM-as-judge prompts; enrich thread {{context}} with tool calls (#6751)
- [OPIK-5333] [BE] feat: inject spans into LLM-as-judge prompts via {{spans}} variable
A user mapping a variable to the bare string "spans" (the same sentinel
the Python-metric path already uses) now gets the JSON-serialized spans
list substituted into the rendered prompt at evaluation time. Lets a
trace-level LLM-as-judge metric like "count the spans: {{mySpans}}"
actually see the spans — previously the only ways to get spans into the
prompt were the agentic-tools path (over-threshold or toggle-forced) or
switching to a Python metric.
- OnlineScoringEngine.templateReferencesSpans(variables) — public helper
  the trace scorer uses to opt-in to the span fetch on the inline path.
- OnlineScoringEngine.prepareLlmRequest(...) — both overloads now take a
  @NonNull List spans and inject the JSON-serialized list into any
  variable mapped to the "spans" sentinel. Empty list / no sentinel =
  no-op, so the path stays cheap for the common case.
- OnlineScoringLlmAsJudgeScorer.score() — spansNeeded predicate now ORs
  in templateReferencesSpans(variables), so the spanService.getByTraceIds
  fetch fires whenever the template references {{spans}}, not only when
  the agentic-tools path could fire.
- 3 new unit tests in OnlineScoringEngineTest covering sentinel
  detection, span substitution + sort-by-start_time wire order, and
  no-injection when the template doesn't reference the sentinel.
Thread-level LLM-as-judge intentionally NOT updated — threads carry
multiple traces, so {{spans}} would be ambiguous (whose spans?).

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- [OPIK-5333] [FE] feat: auto-fill {{spans}} as a reserved trace-evaluator variable
Pairs with the backend's spans-injection feature: users typing
{{spans}} in a trace-scoped LLM-as-judge prompt no longer have to
manually map the variable to the "spans" sentinel — the FE seeds it
automatically, and the schema validator accepts it without complaint.
- constants/llm.ts: new RESERVED_LLM_JUDGE_TRACE_VARIABLES map
  ({ spans: "spans" }) — extensible if we add more sentinels later.
- AddEditRuleDialog/LLMJudgeRuleDetails.tsx (v1 + v2): when the
  prompt-tag scanner finds a new variable, look it up in the reserved
  map for trace-scope rules and seed the path with the sentinel. User-
  supplied paths still win — we only fill blanks.
- AddEditRuleDialog/schema.ts (v1 + v2): extend the trace-variable
  regex to also accept the bare spans sentinel; updated the error
  message to mention it. Span- and thread-scope schemas unchanged.
Backend already handles the sentinel
(OnlineScoringEngine.injectSpansIntoReplacements substitutes the JSON-
serialized spans list at render time). This commit closes the UX loop:
type {{spans}}, hit Save, done.

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- [OPIK-5333] [FE] feat: auto-fill spans reserved arg for Python trace metrics too
Extends the previous LLM-as-judge fix to the Python metric path: a
score(self, spans, ...) parameter on a trace-scoped Python rule now
auto-maps to the spans sentinel, matching the LLM-as-judge {{spans}}
UX. User no longer has to interact with the variable-mapping dropdown
(which only shows input/output/metadata paths and didn't surface
spans at all).
- Rename RESERVED_LLM_JUDGE_TRACE_VARIABLES →
  RESERVED_TRACE_EVALUATOR_VARIABLES since the same map drives both rule
  types; JSDoc updated to document both code paths.
- PythonCodeRuleDetails.tsx (v1 + v2): when parsePythonMethodParameters
  pulls names off the score method signature, look them up in the
  reserved map for trace-scope rules and seed the path with the
  sentinel. Preserves any existing user-supplied path.
- PythonCodeDetailsTraceFormSchema (v1 + v2): extend the regex to also
  accept the bare spans sentinel; updated error message to mention it.
  Span-scope Python schema unchanged.
UX result: type def score(self, spans, ...), hit Save, done. Same
recipe as LLM-as-judge.

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- [OPIK-5333] [FE] feat: hide reserved trace variables (spans) from the mapping list
Reserved trace-evaluator variables like {{spans}} are auto-filled with a
fixed sentinel path and never need user input. Hide their row from the UI
so users don't see a selector they can't usefully change. The variable
stays in the form value, so the backend still receives the sentinel and
the substitution still happens.

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- [OPIK-5333] [BE] fix: render {{spans}} as [] when trace has no spans
The empty-spans short-circuit in injectSpansIntoReplacements skipped
overwriting the bare "spans" literal that toVariableMapping deposits
into the replacements map, so traces with no children leaked the word
"spans" into the rendered LLM prompt instead of an empty JSON array.

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- [OPIK-5333] [BE] feat: detect {{spans}} in templates without sentinel mapping
The FE auto-fills variables.spans = "spans" whenever the user types
{{spans}} in a prompt, but API-created rules can skip that step and
leave the prompt referencing {{spans}} with no matching entry in the
variables map. In that case the prior gate (variables.containsValue ("spans")) returned false, spans weren't fetched, and the rendered
prompt left {{spans}} unsubstituted.

Extend templateReferencesSpans to OR in a Mustache/Jinja2/Python
parse over the message templates. When a template references {{spans}}
and the variables map doesn't bind spans to anything, mirror the FE
auto-fill server-side: fetch spans and inject the JSON array under the
spans key. Explicit user mappings (e.g. spans → input.foo) take
precedence — only the unbound case opts in implicitly.

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- [OPIK-5333] [FE] perf: hoist hiddenVariableNames to a stable module constant
Each render of LLMJudgeRuleDetails / PythonCodeRuleDetails (v1+v2) was
computing Object.keys(RESERVED_TRACE_EVALUATOR_VARIABLES) inline, allocating
a new ["spans"] array on every render and invalidating
LLMPromptMessagesVariables's variablesList useMemo for nothing.

Pre-compute it once at module scope as a frozen readonly array and reuse the
reference everywhere. The prop type widens to readonly string[] so callers
can pass the frozen constant; the component only iterates it to build a Set.

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- [OPIK-5333] [BE] fix: scan multimodal contentArray for {{spans}} + extract shouldFetchSpans helper
Two related fixes pulled out of the PR review:
1. templateReferencesSpans only walked LlmAsJudgeMessage.content (the
  simple-string field), so {{spans}} inside a multimodal message's
  contentArray[*].text was missed. The renderer happily substitutes
  into structured-content text parts, so detection drifted from
  rendering and a multimodal prompt with {{spans}} would skip the
  fetch and leave the placeholder unsubstituted. Added a
  renderableTextOf(message) helper that streams both shapes; both
  are now scanned.
2. Pulled the spans-fetch routing in score() into a package-private
  shouldFetchSpans(message) helper, mirroring the existing
  shouldUseAgenticTools extraction. Makes the gate unit-testable
  without spinning up the full reactive chain — added a 9-case
  parameterized truth-table test covering the agentic-tools and
  inline-template branches plus the short-circuit interactions.
Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- [OPIK-5333] address PR comments: hide-by-value, DRY auto-fill, rename tests
Three review findings rolled together:
1. (Logical bug 🟠) LLMPromptMessagesVariables was hiding a row whenever
  the variable's name matched a reserved name, regardless of value. A
  user (or API caller) who mapped spans → input.spans couldn't see or
  edit that row — write-only after first parse. Replaced the
  hiddenVariableNames: readonly string[] prop with
  reservedSentinels: Readonly<Record<string, string>>; the row is
  hidden only when the variable's current value equals the sentinel
  for that name. Custom overrides stay visible.
2. (DRY 🟢) The auto-fill block localVariables[v] = variables[v] || reservedDefault || "" was duplicated across v1+v2 ×
  {LLMJudgeRuleDetails, PythonCodeRuleDetails}. Extracted into
  resolveTraceEvaluatorVariableDefault(name, current, scope) in
  lib/llm.ts. Adding a new reserved trace variable now propagates to
  all four editors via a single source of truth.
3. (Style 🟢) Renamed the test methods I added in this PR from the
  testX() form to the scenario-based names per
  .agents/skills/opik-backend/testing.md. Pre-existing methods left
  as-is to keep the rename limited to my own additions.
Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- [OPIK-5333] [BE] feat: gate {{spans}} template path under isAgenticToolsEnabled
Ship the inline {{spans}} template substitution under the same feature
flag as agentic-tools so a single toggle flip turns both pathways on or
off org-wide. When isAgenticToolsEnabled=false and the rule isn't on
the experimentId branch, shouldFetchSpans returns false and
injectSpansIntoReplacements substitutes an empty array into
{{spans}} via the empty list threaded through prepareLlmRequest —
no I/O, no broken rendering.

Truth table updated to cover the new gate: toggle-off + template/sentinel
combinations all expect no fetch; toggle-off + experimentId still fetches
(for the agentic-tools cache seed, with the template substitution
piggy-backing on the same data).

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- [OPIK-5333] [FE] feat: gate {{spans}} auto-fill behind agentic_tools_enabled
Mirror the backend gate (isAgenticToolsEnabled) on the frontend so the
spans-in-prompts feature ships as a single togglable unit.

When the FT is off in the four rule-detail editors (v1+v2 × {LLMJudge,
PythonCode}):
- The spans variable no longer auto-fills to its sentinel value, so the
  user can map it to a custom path like any other variable.
- LLMPromptMessagesVariables receives reservedSentinels={undefined},
  so the spans row stays visible and editable instead of being hidden.
When the FT is on, behavior is unchanged: auto-fill writes spans → "spans", the row is hidden, BE substitutes the spans JSON.

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- [OPIK-5333] fix: normalize feature-toggle state when backend omits a field
Two small fixes for the type contract on FeatureToggles:
- FE: setFeatures(data) was overwriting the entire state object with
  whatever the API returned, leaving keys the backend omits as
  undefined. A newer FE talking to an older BE could land
  AGENTIC_TOOLS_ENABLED = undefined in state, breaking the
  Record<FeatureToggleKeys, boolean> contract. Merge over
  DEFAULT_STATE so omitted keys keep their declared defaults.
- BE: agenticToolsEnabled lacked @NotNull while every other toggle
  in ServiceTogglesConfig has it. Added for consistency — primitive
  booleans can't actually be null at runtime, but the annotation
  documents the contract and matches the surrounding style.
Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- [OPIK-5333] code review polish: docstring + empty-mapping handling
Code-review feedback on the branch surfaced two small things worth tightening:
- OnlineScoringLlmAsJudgeScorer.shouldFetchSpans and
  OnlineScoringEngine.injectSpansIntoReplacements: the prior docstring on
  the toggle implied a clean "kill switch", but in reality the substitution
  runs unconditionally so toggle-off renders Spans: [] rather than the
  literal {{spans}}. Documented why: gating the substitution would
  resurrect the bare-word leak from rules whose variables map still carries
  the sentinel from before the toggle flipped. The current asymmetry is
  intentional, just needed to be spelled out.
- resolveTraceEvaluatorVariableDefault was using if (currentMapping),
  which treated "" as "not set" and re-applied the sentinel auto-fill on
  every prompt re-parse — silently overwriting an API caller's deliberate
  spans: "". Switched to if (currentMapping !== undefined) so explicit
  empty strings stick.
No behavior change in the common paths — existing tests stay green.

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- [OPIK-5333] [BE] feat: enrich thread {{context}} with each turn's spans (tool calls + I/O)
Thread-scope LLM-as-judge rules had no way to see the agent's actual tool-use
behavior — {{context}} only carried each trace's top-level user/assistant
text, and the only path to spans was the agentic-tools tools-branch (which
only fires for big threads, or on the experimentId test-suite-assertion
path). Normal-sized conversations were effectively unobservable from the
judge's POV.

Now: when isAgenticToolsEnabled=true, the thread scorer fetches every span
across every trace in the thread up front and threads them down through
prepareEvaluation. renderThreadMessages substitutes {{context}} with
an enriched shape — each trace's child spans are attached as a spans field
on the assistant entry. Customer writes {{context}} once and gets the full
conversation + reasoning trace.

Wire-shape design (EnrichedThreadChatMessage): keeps the existing
{role, content} shape, adds optional spans field on the assistant entry.
With @JsonInclude(NON_NULL), the spans field is omitted whenever a trace
has no spans (or the toggle is off and the scorer passed an empty list), so
existing rules see today's wire shape unchanged. Backward-compat guaranteed.

Size routing kept honest: estimateThreadContextTokens now serializes the
enriched shape, so an enriched-but-big thread routes correctly to the
agentic-tools path (skeleton + ReadTool drill-down) instead of inline-
rendering an oversized prompt.

Tools-branch path (prepareThreadLlmRequestWithTools) is intentionally
unchanged — it still renders the compact skeleton, and the model uses
ReadTool/JqTool to drill into individual traces' spans on demand. Pulling
all spans inline there would defeat the size optimization the tools branch
exists for.

Tests cover:
- wire-identical {role, content} shape when toggle off (empty spans list)
- enriched shape with spans sorted by start_time when spans provided
- estimateThreadContextTokens reflects span size for the routing gate
Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- [OPIK-5333] [BE] polish: thread spans-fetch comment, FQN cleanup, toggle-gate test
Non-functional polish from an independent code review of the thread-scope
enrichment work:
- Documented the deliberate "fetch spans before path decision" ordering in
  the thread scorer so the size estimate routes honestly. The cost (wasted
  I/O when the tools path wins) is now spelled out in-code, not only in
  the commit history.
- Replaced fully-qualified java.util.stream.Collectors, com.comet.opik .api.Span, com.comet.opik.domain.SpanType references in the test
  file and scorer with the already-imported (or now-imported) short
  forms. Pure cleanup.
- New scorer-level test skipsSpanFetchWhenAgenticToolsDisabled proves
  the toggle gate by asserting verifyNoInteractions(spanService) on a
  full thread-scoring flow with the flag off. Locks in the contract so a
  future "simplification" of the gate would fail fast.
Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- [OPIK-5333] [BE] perf: project Span → SpanForLlm before rendering into prompts
The raw Span record carries ~28 fields — audit metadata (createdBy,
lastUpdatedBy, createdAt), out-of-band annotations (feedbackScores,
comments), cost data (totalEstimatedCost), system IDs (projectId,
projectName), tags, usage, ttft, source, environment — almost none of
which help a judge reason about agent behavior. Pasting them inline
burns tokens on noise and risks confusing the judge ("am I supposed to
defer to these feedback scores?").

Introduce SpanForLlm: a lean projection with just the 11 fields a
judge actually uses — name, type, start/end/duration, in/out, metadata,
model, provider, errorInfo. @JsonInclude(NON_NULL) keeps the per-span
JSON tight (a successful tool span doesn't pad with error_info: null,
a non-LLM span doesn't carry model/provider).

Applied on both render paths:
- Trace-scope {{spans}}: injectSpansIntoReplacements projects before
  serialization (and before the agentic-tools cap, so the cap reflects
  what the judge actually sees).
- Thread-scope {{context}}: EnrichedThreadChatMessage.spans now
  holds List<SpanForLlm>; fromTraceToThreadEnriched projects
  per-trace before nesting.
New test prepareLlmRequestRendersSpansWithLeanProjection builds a Span
populated with every dropped field set to a unique sentinel string and
asserts none of them appear in the rendered prompt. Locks the projection
contract — a future "let me just add this one field" PR breaks loudly.

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- [OPIK-5333] [BE] feat: nest child spans recursively in SpanForLlm projection
A flat sorted list of spans is fine for "what happened in what order"
evals, but loses the call structure that matters for agentic eval —
"did the planner properly decompose into sub-tools? did sub-agent X
call sub-agent Y, or were they siblings?". The judge would have to
mentally resolve parent-id references that we'd dropped anyway in the
lean projection.

Switch SpanForLlm to a recursive shape: each node carries a spans
field with its children (also SpanForLlm), built from parent_span_id
links. Tree depth matches the actual call hierarchy the SDK recorded.

buildSpanTree(spans) reconstructs the tree:
- groups input by parent_span_id
- identifies roots (parent null OR parent not in the input set —
  orphans get promoted so subtree views stay well-formed)
- recursively projects each node, sorting siblings by start_time at
  every level so chronology is preserved within each branch
Applied on both render paths:
- Trace-scope {{spans}} — injectSpansIntoReplacements builds the
  tree directly from the per-trace span list.
- Thread-scope {{context}} — per-trace spans in the input group are
  fed through buildSpanTree before nesting under the assistant entry.
Leaf spans are omitted via @JsonInclude(NON_NULL), so the JSON stays
shaped like normal data (no "spans": [] padding on every leaf).

New test buildSpanTreeReconstructsHierarchyAndPromotesOrphans covers
the structural contract: shuffled input → correct two-root tree
(real root + orphan), sibling ordering, leaf with no children, three
levels of nesting (root → child-A → grandchild).

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- [NA] [BE] test: bump ExperimentProjectMigrationJobTest startup buffer 3s → 10s
The prime-V1 assertion at line 161 runs at ~T+3.1s on busy CI runners
(when this test happens to share an Integration Group with several other
container-heavy tests), past the prior 3s experimentProjectMigration. startupDelay — at which point the migration job has already flipped V1
→ V2, the assertion sees V2, and the test fails deterministically rather
than flakily.

10s gives ~3× headroom for setup contention without measurably extending
the test (the second phase polls until V2 anyway, so the longer
startupDelay just adds time the test would have spent waiting either way).

The test moves between Integration Groups depending on greedy line-count
bin-packing (see .github/scripts/discover-backend-tests.sh) so anyone
whose PR happens to shift this test into a busy group lights the same
flake — this fix is for everyone, not specifically for OPIK-5333.

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- Revert "[NA] [BE] test: bump ExperimentProjectMigrationJobTest startup buffer 3s → 10s"
This reverts commit 68a176033b.
- [OPIK-5333] [BE] feat: enrich thread Python eval conversation with spans
Closes the gap called out in the PR description: thread-scope Python online
evals were stuck on the legacy [{role, content}, ...] conversation shape
regardless of the agentic-tools feature flag, while LLM-as-judge thread
evals already received the enriched shape with each turn's tool calls and
I/O nested under the assistant entry. Same customer use-case ("evaluate the
agent's tool-use across a conversation"), inconsistent backend.

Unify by:
- Promoting SpanForLlm from a nested record inside OnlineScoringEngine
  to a top-level com.comet.opik.api.SpanForLlm. Lives alongside Span
  and Trace so both render paths (LLM-as-judge in OnlineScoringEngine,
  Python in TraceThreadPythonEvaluatorRequest.ChatMessage) can reference
  it without a package cycle.
- Extending TraceThreadPythonEvaluatorRequest.ChatMessage with an
  optional spans: List<SpanForLlm> field. @JsonInclude(NON_NULL) keeps
  the wire shape backward-compatible — when toggle off, the field is
  omitted and the JSON is byte-identical to today.
- Dropping the redundant OnlineScoringEngine.EnrichedThreadChatMessage
  record. fromTraceToThreadEnriched now returns
  List<TraceThreadPythonEvaluatorRequest.ChatMessage> directly so both
  paths share one type.
- Injecting SpanService into OnlineScoringTraceThreadUserDefinedMetric PythonScorer, reactively fetching every span in the thread when the
  toggle is on, and routing through fromTraceToThreadEnriched instead
  of the legacy fromTraceToThread. Mirrors the LLM-as-judge thread
  scorer's pattern exactly.
Python users now get the same nested tool-call tree the LLM judge sees —
their score(self, conversation, ...) method receives a list of dicts
where each assistant entry carries spans with full input/output, model,
provider, error_info, and nested children. The lean SpanForLlm
projection (11 fields) keeps the payload focused on agent behavior, same
as the trace-scope path.

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- [OPIK-5333] [BE] test: lock in toggle-on Python thread enrichment
Adds the end-to-end counterpart to skipsSpanFetchWhenAgenticToolsDisabled —
exercises the full toggle-on path:

toggle on
→ spanService.getByTraceIds(traceIds) fires once with the right id set
→ fromTraceToThreadEnriched grouops the spans by trace and nests them
→ captured ChatMessage list sent to PythonEvaluatorService has user/assistant
entries with spans populated on the assistant turn only

Catches future regressions where someone "simplifies" by routing back to
the legacy fromTraceToThread or quietly drops the SpanService fetch.

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- [OPIK-5333] [BE] feat: unify trace-scope Python spans kwarg on the lean SpanForLlm projection
Closes the last inconsistency in the spans-render matrix: trace-scope
Python metrics were the only path still sending the full ~28-field Span
record to the runner, while every other path (trace LLM-as-judge
{{spans}}, thread LLM-as-judge {{context}}, thread Python conversation)
already projected through SpanForLlm and buildSpanTree.

OnlineScoringEngine.toReplacements(variables, trace, spans) now puts
buildSpanTree(spans) under the spans key instead of the raw flat
list. One shared helper, one wire shape across all four paths.

What the customer's Python score(self, spans, ...) method sees changes:

Before: list of dicts with ~28 fields each (id, project_id, trace_id,
parent_span_id, name, type, in/out, metadata, model, provider,
tags, usage, ttft, error_info, source, environment, created_at,
created_by, last_updated_at, last_updated_by, feedback_scores,
comments, total_estimated_cost, ...)

After: list of root-span dicts with 11 fields each (name, type,
start_time, end_time, duration, input, output, metadata, model,
provider, error_info) plus a recursive spans field carrying
child spans — same projection the LLM judge sees.

This IS a backward-compatible-breaking change for existing trace-scope
Python metrics that read dropped fields (id, parent_span_id, usage,
feedback_scores, etc.). Worth flagging to anyone with such metrics —
they'll need to update them or pull the dropped data elsewhere. Most
"agent-behavior" metrics only read name/type/input/output anyway, so
the practical break should be narrow.

Updated existing fetchesSpansAndPassesThemAsListWhenSpansArgumentPresent
test to assert the new SpanForLlm shape end-to-end, with a known span
name so the projection contract is explicit in the test.

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- [OPIK-5333] [BE] perf: move thread Python scorer's blocking prep onto boundedElastic
The Python thread scorer's prepareScoring calls projectService.get(...) —
synchronous JDBC. It was wrapped in Mono.fromCallable but without
.subscribeOn, so the blocking call ran on whichever thread emitted the
upstream Mono. Before this PR that was the consumer loop thread; after
the toggle-on span-pre-fetch landed, it can also be the spanService DB
thread. Both are bad — consumer loop can stall message draining, DB
thread can starve the JDBC pool.

Pin the callable to Schedulers.boundedElastic() — the standard reactor
scheduler for wrapping blocking calls in reactive glue.

The trace and thread LLM-as-judge scorers are unaffected: trace scorer's
prepareEvaluation is pure CPU and stays on parallel(); thread LLM-as-judge
prepareEvaluation is also pure CPU (its projectService.get() lives in the
post-LLM .map block, which is a separate concern not introduced here).

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com

Co-authored-by: Sasha sasha@Sashas-MacBook-Pro.local
Co-authored-by: Claude Opus 4.7 (1M context) noreply@anthropic.com
Co-authored-by: Aliaksandr Pyrkh aliaksandr@comet.com
- [OPIK-6579] [BE] feat: extend experiment project migration to handle deleted-project and no-inference tail (#6785)
- fix(visual-tests): address baz review — extract helper, rename methods, replace networkidle
- Extract _run_evaluate() helper in experiments.py to deduplicate dataset
  lookup + evaluate pipeline shared by create_experiment and
  create_experiment_for_project
- Rename createTestSuiteDataset/createTestSuiteExperiment to
  createTestSuiteDatasetForProject/createTestSuiteExperimentForProject for
  naming consistency with createDatasetForProject/createExperimentForProject
- Replace waitForLoadState('networkidle') with waitForLoadState('load') in
  all visual page objects to avoid hangs on background polling
Co-Authored-By: Claude Sonnet 4.6 noreply@anthropic.com
- fix(visual-tests): address baz round 2 — add validation, scope dataset lookup
- Add validate_required_fields() to create_dataset_for_project and
  create_test_suite_dataset in datasets.py (import from .utils)
- Pass project_name to get_dataset() in _run_evaluate so dataset lookup
  is scoped to the correct project (SDK supports project_name param)
Co-Authored-By: Claude Sonnet 4.6 noreply@anthropic.com
- fix(visual-tests): pass project_name to get_dataset in create_test_suite_experiment
Missed in previous commit — create_test_suite_experiment had its own
client.get_dataset() call that wasn't scoped by project_name.

Co-Authored-By: Claude Sonnet 4.6 noreply@anthropic.com

Signed-off-by: dependabot[bot] support@github.com
Co-authored-by: Thiago dos Santos Hora thiagoh@comet.com
Co-authored-by: CometActions 126667691+CometActions@users.noreply.github.com
Co-authored-by: Andres Cruz andresc@comet.com
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Nikhil Tilwalli 8410254+ntilwalli@users.noreply.github.com
Co-authored-by: Claude Opus 4.7 (1M context) noreply@anthropic.com
Co-authored-by: Daniel Dimenshtein danield@comet.com
Co-authored-by: Andrei Cautisanu 30831438+AndreiCautisanu@users.noreply.github.com
Co-authored-by: Jacques Verré jverre@gmail.com
Co-authored-by: github-actions github-actions@comet.com
Co-authored-by: avinahradau a.l.vinogradov1986@gmail.com
Co-authored-by: Aliaksandr Kuzmik 98702584+alexkuzmik@users.noreply.github.com
Co-authored-by: EdvardLaub 147416353+EdvardLaub@users.noreply.github.com
Co-authored-by: BorisTkachenko 35521895+BorisTkachenko@users.noreply.github.com
Co-authored-by: Pragnyan Ramtha pragnyanramtha@gmail.com
Co-authored-by: Liya Katz liyak@comet.com
Co-authored-by: Iaroslav Omelianenko yaric_mail@yahoo.com
Co-authored-by: sasha aadereiko@gmail.com
Co-authored-by: Sasha sasha@Sashas-MacBook-Pro.local
Co-authored-by: Aliaksandr Pyrkh aliaksandr@comet.com
下载附件
- 源代码 (ZIP)
- 源代码 (TAR.GZ)

0 发布 8,340 Git 标签

[OPIK-6408] feat: add visual comparison tests POC (#6793)