-
[OPIK-6951] [QA] test: Ollie deeper E2E coverage (sidebar, context, /analyze, /instrument, /improve) (#7108)
frostbyte_neo 发布于
2026-06-16 11:05:32 +00:00 - test(ollie): add sidebar surface, context-awareness, and /analyze coverage
Extend the OlliePage POM and add ollie-agentic.spec.ts with three deeper
flows beyond the initial smoke coverage:- sidebar surface (@t1-smoke): the Ollie sidebar iframe mounts on a project
page and persists across navigation - context awareness (@t2-cuj): the "Traces: N" badge reflects the seeded
trace count for the active project - /analyze (@t3-nightly): running /analyze on a project with traces renders
a structured, non-empty response (directional)
All cloud-gated via envConfig.features.ollie; assertions are structural.
Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
- test(ollie): add Local Runner /instrument + /improve coverage + golden agents
Drive the Ollie
opik connectLocal Runner flows end-to-end and verify
both /instrument and /improve on a connected agent (both green on staging):- agents/: golden fixtures (OPIK-6125) — an uninstrumented agent for
/instrument and a known-failing agent + eval suite for /improve. - core/local-runner/connect.ts: spawn
opik connect, scrape the pairing
URL from the daemon's OSC-8 hyperlink output, plus suite eval/seed helpers. - OlliePage: pairConnectRunner / startInstrument / startImprove /
approveUntilSettled / applyProposal for the agentic approval loop (the
/improve accept step is a chat reply "apply", not a button). - ollie-connect.spec.ts (@t3-nightly): /instrument asserts the agent source
gained opik imports + @track decorators; /improve seeds failing traces,
runs the improve loop, replies "apply", and asserts the eval-suite pass
rate rises (2/6 -> 6/6). Assertions are structural/directional.
Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
Co-authored-by: Claude Opus 4.8 (1M context) noreply@anthropic.com
下载附件