test_llm_judge_validates_compiled_workflow fails: the judge returns
pass=false, reporting structural items from the kitchen-sink agent spec
as missing from the compiled workflowDef.
Listed as a plain reason string rather than run:false — the test makes a
single judge call and fails fast, so there is no CI time to reclaim, and
leaving it running means a fix surfaces as XPASS.
E2E_MIN_PASSED drops 135 -> 134 in the same commit, as the known-failures
README requires: an added entry moves a test out of the PASSED column, and
without the matching decrement the lane fails on the passed-count floor
for an unrelated-looking reason.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The entries had grown into full root-cause write-ups — 2.2KB for one
reason string. That analysis belongs in the tracking issue and the fixing
PR, where it can be discussed and closed out; duplicated here it just goes
stale silently, which is exactly what happened to the suite14 entry.
Each value is now a short statement of what fails plus a tracking link.
_README says to keep it that way. File is 2227 bytes, down from 4781.
Kept the operational parts, which are not history and prevent real
mistakes: the E2E_MIN_PASSED coupling, and the warning that a run:false
xfail can never XPASS so it must be un-listed by hand.
Also repoints suite14 at conductor-oss/python-sdk#459, which supersedes
the now-closed #455.
Verified: plugin still reports both entries matching exactly one test,
no MATCHED NOTHING.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
rc4 is the release that lands the changes several deferred items were
waiting on, so they all come due together:
* env names. rc4's conftest migrated to CONDUCTOR_SERVER_URL /
CONDUCTOR_AGENT_LLM_MODEL, so the rename TODO is resolved and the vars
are live rather than inert. Noted in-file that they must go back if the
pin is ever moved down to rc2 or earlier, which read only AGENTSPAN_*.
* agentspan CLI, removed. rc4 dropped it entirely — no CredentialsCLI, no
CLI_PATH, and test_suite16_cli_skills.py is gone — so provisioning it
buys nothing. Recorded that it WAS load-bearing under rc2, where the
credential suites reached their read-only skip by invoking it and
removing it turned clean skips into FileNotFoundError failures, so it is
not restored on stale reasoning.
* the three Suite16 cli-skills known failures, removed. Their file no
longer exists, and a key matching nothing now raises a
KnownFailuresWarning, so leaving them would add noise every run. History
kept in _HOWTO.
* floor 137 -> 135. This is a coverage LOSS, not a fix: rc4 deleted
test_suite16_cli_skills.py, and while three of its tests were xfail-ed
here, two were genuinely passing (load_serve_and_run_by_name,
run_ephemeral_executes_script_worker). Real coverage of the
ephemeral/by-name CLI skill paths went away with the file, invisibly in
pass/fail terms. Spelled out at the constant so the -2 is not read as
routine.
Verified locally against a server built from this branch: 135 passed,
7 skipped, 3 xfailed, 0 failed; floor gate green; plugin reports both
remaining entries matching exactly one test with no MATCHED NOTHING.
suite14's entry stays — its fixes are in conductor-oss/python-sdk#455,
still unmerged, so rc4 does not carry them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The entry blamed #1356 for dropping the swarm transfer task defs, giving
updateTaskDef NotFound so "agents can't hand off". Reproducing it locally
showed that is wrong on every link: the SDK registers all six
*_transfer_to_* defs successfully, the "No such task by name" server lines
are its benign lookup-before-create, and the handoff itself succeeds —
agent 0's sub-workflow reaches COMPLETED including transfer_msg.
What actually hangs is a tool worker. _register_workers reads
`agent.stateful` on the immediate agent only, but the test hangs swarm_tool
off the swarm MEMBERS, which are not themselves stateful — so the worker
registers with domain=None while the stateful swarm makes the server
domain-route the task. pollCount=0, startTime=0, the FORK's JOIN never
satisfies, workflow times out.
A second, independent bug sits behind it: the test's _find_tasks_by_type
matches only taskDefName, but handoff_check is an INLINE task whose
taskDefName is literally "INLINE", so the assertion fails even though 20
handoff_check tasks exist and are COMPLETED.
Both fixes belong to python-sdk. Verified against the pinned 2.0.0-rc2:
stock FAILED (RUNNING, 304s); SDK fix alone reached COMPLETED but failed
the handoff_check assert; both fixes PASSED in 118s.
Also corrects the timing: one attempt is ~304s, not ~908s — that figure was
three attempts under conftest's unconditional flaky(reruns=2).
Refs #1363
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three review findings on the e2e lane.
1. Stale docs. The plugin docstring said the known-failures list is "empty
when the suite is green" — it has 5 entries and the suite is green, so
the parenthetical read as a contradiction. Reworded, and the JSON
_README now also documents the new match-count warnings and the fact
that adding an entry requires lowering E2E_MIN_PASSED in the same
commit.
2. Silent misconfiguration. A missing/typo'd E2E_KNOWN_FAILURES path loaded
nothing and returned early before any reporting, so every forgiven
failure reported as real with no explanation — the one case the new
diagnostics still could not see. Now warns, and distinguishes that from
the legitimate "no list configured" case, which stays silent.
3. Readiness probe. `curl -sf http://localhost:3001/` treats mcp-testkit's
404-on-root as failure, so it never succeeded: the loop burned all 15
iterations and printed "started" whether or not anything was listening.
Check for any HTTP status instead, warn and tail the log on timeout.
Verified: bad path warns / unset path silent / good path reports 5x1 with
no warnings; probe detects a live mcp-testkit (HTTP 404) and warns when
nothing listens, without aborting under bash -e.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
test_suite21_scheduling.py skips all 11 of its tests unless
GET {SCHEDULER_CONDUCTOR_URL}/scheduler/schedules answers 200. Its default
is port 8089 and the lane's server is on 8080, so the probe never matched
and the suite skipped silently on every run — no failure, no signal.
conductor-oss serves the scheduler itself (conductor.scheduler.enabled=true
by default, SchedulerResource at /api/scheduler); the endpoint was verified
to return 200 on the lane's own server. Pointing the var there runs the
suite: 10 pass, 1 fails.
That one failure is real and now recorded rather than hidden:
get_schedule() on a deleted schedule is expected to return None, but the
server 404s and OrkesSchedulerClient propagates ApiException(404).
Floor raised 127 -> 137 to lock the recovered coverage in. Verified locally:
137 passed, 11 skipped, 6 xfailed, floor check green.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Removing the CLI was wrong. The credential suites (xdist group
`credentials`, suites 2/3/4/5/26) reach their conductor-oss skip THROUGH
the binary: CredentialsCLI.set() shells out, the server rejects the write
because its secret store is env-backed and read-only, and that stderr is
what triggers pytest.skip. conftest's subprocess.run has no try/except, so
with no binary they raise FileNotFoundError and fail.
Verified locally: a run with the CLI absent turned clean skips into hard
failures in suites 2, 4 and 5 before it was stopped. The step's original
rationale was accurate; the prior commit removed it on a wrong reading
that generalized from Suite16's fixture (which does skip) to the
credential fixture (which does not).
Keeps the corrected skills context — server-side skills remain
unsupported, so Suite16 stays xfail-ed rather than skipped.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
conductor-oss does not support the server-side skills API: SkillController
is gated on both conductor.integrations.ai.enabled and
agentspan.skills.enabled, and the latter has no default and is set in one
place repo-wide (AgentSpanDeploymentContractEndToEndTest). The only other
CLI consumers are the credential suites, which skip on this flavor because
the secret store is env-backed and read-only. So the download bought
nothing on every run.
Absent the binary the suites skip cleanly — Suite16's fixture calls
pytest.skip and the credentials fixture only constructs an object — so the
step's stated rationale (FileNotFoundError without it) did not hold either.
Also rewrites the known-failures context: these are an unsupported feature,
not a bug pending a decision, and the server-side skills registry is
distinct from the "Conductor Skills" coding-agent integration in docs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The stateful-swarm-handoff xfail (#1363) is a deterministic ~908s hang that
would burn ~15 min of CI every run. Add per-entry run control to the plugin:
a JSON value may now be a reason string (run=True, XPASSes when fixed) or an
object {"reason":..., "run":false} to xfail WITHOUT executing.
Set the swarm entry to run:false so it's skipped (marked xfailed [NOTRUN], no
hang); the 3 cli-skills entries stay run=True. Un-list the swarm entry
manually when #1363 is fixed (a non-run xfail can't auto-XPASS).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The python-sdk-e2e gate caught a real, deterministic regression on main:
test_stateful_swarm_handoff_completes hangs (workflow stuck RUNNING ~908s,
2/2 runs). Root cause: #1356 ("Enhances A2A/AgentSpan execution") stopped
registering the swarm transfer/handoff task defs (now compiler-owned INLINE),
but the pinned SDK bundle (2.0.0-rc2, the latest release) still PUTs them ->
updateTaskDef NotFound -> agents can't hand off -> workflow never completes.
Passed pre-#1356 (2026-07-17). Server-side fix owned by #1356; no newer SDK
to bump to. Tracked in conductor-oss/conductor#1363.
xfail it so the gate goes green; it XPASSes (remove the entry) once #1363 is
fixed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sharpen the 3 Suite16 cli-skills reasons with the traced root cause: #1288
(rc.9) changed the SkillController gate from agentspan.embedded [1 prop] to
agentspan.embedded + agentspan.skills.enabled [2 props], flipping the skills
API off-by-default on main (served on rc.8, which python-sdk pins). Add a
shared _CONTEXT_cli_skills note (ignored by the plugin) with the full trace
and the #1353 decision. No behavior change — same 3 node-ids xfail-ed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Earlier reason ("skills-register endpoint missing on main") was wrong: the
SkillController exists but is gated on agentspan.embedded (intended
orkes-only for OSS) AND agentspan.skills.enabled, so POST /api/skills/register
returns 404 on conductor-oss. Per design direction, agentspan.embedded should
be false for conductor-oss (the design may change), so the skills API isn't a
settled OSS surface — do not force-enable it on boot.
Keep the 3 Suite16 cli-skills tests xfail-ed with the accurate reason + link
to the tracking issue conductor-oss/conductor#1353. They XPASS (remove the
entries) once the OSS skills story is settled.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The python-sdk-e2e gate correctly surfaced a real server gap: the CLI's
`skill register` calls POST /api/skills/register, which the server built
from main returns 404 for ("No static resource api/skills/register"). The
endpoint exists in the released 3.32.0-rc.8 that python-sdk's own CI pins,
so it's green there but not against a from-source main build.
Add the 3 affected Suite16 cli-skills tests to the conductor-oss
known-failures list so the lane gates green while the skills-register API
gap is investigated (an XPASS will flag it once the endpoint lands). The
other 151 tests — LLM, tools, MCP, guardrails, credential lifecycle — pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds a `python-sdk-e2e` job to CI that boots the conductor server built from
the current commit in SQLite mode (the default persistence — no external DB)
and runs the released python SDK agent e2e suite against it, so a server
change can't silently break the SDK before a release.
- The suite + bundle come from conductor-oss/python-sdk
(conductor-ai-e2e-python-<version>, pinned); fetched at runtime, sha256-verified.
- The server auto-configures the openai provider from OPENAI_API_KEY
(conductor.ai.openai.api-key), so no manual integration setup is needed.
- Known failures are xfail-ed via an external pytest plugin
(.github/agent-e2e/known_failures_plugin.py loaded with -p) + a per-repo
list (known-failures-python.json). The suite is green today so the list is
empty; the mechanism stays so the lane can gate while any future gap is
fixed (a fixed bug XPASSes; a stale entry is a harmless no-op).
- Gating. Runs on push/dispatch and on PRs touching agent-relevant paths
(ai/, conductor-agentspan/, server/, core/, sqlite-persistence/, the
workflow) via a detect-changes `agent` filter, to avoid spending LLM
budget on unrelated PRs.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Ling-Sen Peng