test_llm_judge_validates_compiled_workflow fails: the judge returns
pass=false, reporting structural items from the kitchen-sink agent spec
as missing from the compiled workflowDef.
Listed as a plain reason string rather than run:false — the test makes a
single judge call and fails fast, so there is no CI time to reclaim, and
leaving it running means a fix surfaces as XPASS.
E2E_MIN_PASSED drops 135 -> 134 in the same commit, as the known-failures
README requires: an added entry moves a test out of the PASSED column, and
without the matching decrement the lane fails on the passed-count floor
for an unrelated-looking reason.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
mkdocs.yml declares the mermaid custom fence as
`!!python/name:main.mermaid_fence`, which PyYAML resolves with a plain
`import main` while parsing the config. The deploy action runs the mkdocs
console script inside a Docker container, where sys.path[0] is the script's
bin directory rather than the checkout, so the import fails and the deploy
aborts before building.
Point PYTHONPATH at the container's workspace mount so main.py resolves.
Run 32411004435 failed with:
cannot find module 'main' (No module named 'main')
in "/github/workspace/mkdocs.yml", line 283, column 19
* - Add end to end integration tests
- Update playwright so it can get OPENAI_API_KEY from .env.local
* Bump coverage percentage to 90% to test CI before merging in additional tests
* Update workflow to use coverage report script
* Add tests and remove kitchen
* - Delete orphaned components
- Cleanup local coverage runs
* Make test less flaky
* Run integration tests in docker for consistent snapshots
* Show incompletion reasons
---------
Co-authored-by: Dale Brady <49766562+bradyyie@users.noreply.github.com>
* Create workflow to run enterprise integration tests against OSS branch changes on creation of pull request
* Add to documentation
* Use status context to get status of playwright tests
* Update OSS token
* Trigger CI pipeline
* Add frontend code coverage
* Revert LLM task forms to correct version and add tests
* Fix CI heap out of memory
* Add OPENAI_API_KEY to env for test
* - Update slack icon for coverage report
- Make tests more stable
* Set slack icon back to hidden based on flag
* - Fix coverage report
- Try and improve CI speed
* Add better placeholder and helper text for Chat Complete task
* Make instructions take priority over system messages if instructions are provided
* Improve LLM chat complete UX so it's less confusing - Enterprise you either select a prompt OR provide instructions
* Rename field to Prompt Template instead of AI Prompt and increase spacing from label
* test: await async decide in SubWorkflowRestartSpec setup instead of racing it
CI failure (run 30872035136): IndexOutOfBoundsException at setup line 125 -
a raw tasks.get(1) read the mid-level workflow's task list before the async
decide had scheduled the SUB_WORKFLOW task. The decide normally runs inline
with the task-completion update, but falls to the background sweeper when
the workflow lock is contended, so under CI load the read can outrun it.
Same load-sensitive race class as the WorkflowRetryTests/WorkflowRerunTests
hardening (555d7d069); this spec carried one more instance of the pattern.
Both root and mid-level stages now wait (PollingConditions, 30s ceiling)
for the SUB_WORKFLOW task to exist and for its subWorkflowId to be
populated before dereferencing. The waits also tolerate the system-task
coordinator having already started the task (only manually started when
still SCHEDULED) - the reason the old code's find{SCHEDULED} could be
legitimately null.
Positive eventually-waits only: zero added time on passing runs.
Validated 10/10 consecutive green local runs of the spec.
* test: re-enable 4 healed WorkflowRerunTests, refresh stale @Disabled reasons on the rest
The retry/rerun/restart fixes (388fabfdc lineage) silently repaired several
behaviors these tests were disabled for; nobody re-enabled them. Verified
against a live server on current main:
Re-enabled (passing, incl. 3x consecutive local runs):
- fork-join rerun with DO_WHILE loop task (x3 variants)
- SWITCH re-execution after rerun (sync status variant)
Still failing - @Disabled reasons updated to the ACCURATE current failure
modes (the old reasons described symptoms that no longer occur, which is
how these stayed forgotten):
- fork-join rerun: sibling branch genuinely not rescheduled (stays at 2
tasks through a 30s await)
- SUB_WORKFLOW-inside-FORK rerun (x2, Ticket #7097): sibling branch child
never spawns (subWorkflowId stays null through a 30s await)
- DO_WHILE rerun: task never re-decided from SCHEDULED to IN_PROGRESS
- SWITCH rerun: workflow completes without rescheduling the selected branch
- fork-join with wait/webhook/switch: never reaches the expected fork shape
- SWITCH-inside-DO_WHILE: flaky across consecutive runs
Also hardened the racy one-shot reads in these tests (subWorkflowId and
post-rerun snapshots now await with diagnostics) so the remaining failures
report the engine gap directly instead of NPEs/IndexOutOfBounds - ready
for whoever picks up the engine work.
* test: eliminate @Disabled from WorkflowRerunTests; raise load-flaky await ceilings across e2e suites
WorkflowRerunTests now has ZERO disabled tests:
- The two 'rerun of a RUNNING workflow' tests are rewritten as contract
tests: conductor-oss deliberately rejects rerun on a non-terminal
workflow, so the tests now assert the rejection and that the workflow is
untouched - real coverage of the OSS contract instead of dead tests.
- The seven engine-gap tests (fork-join sibling rescheduling, #7097
SUB_WORKFLOW-in-FORK children, DO_WHILE/SWITCH rerun re-decide) are
ENABLED and tagged engine-gap: excluded from the blocking run via
build.gradle so known gaps do not redden the pipeline, runnable with
-PincludeEngineGaps, each annotated with the precise verified failure.
When the engine work lands, deleting the tag line activates the test.
Await-ceiling sweep over the suites failing nightly on starved CI runners
(all pass locally; every wait is a positive eventually-wait, so raising
ceilings is free on passing runs):
- WorkflowRerunTests: all sub-30s atMost() raised to 30s (119 sites)
- WorkflowRetryTests: WF_AWAIT_SECS 60 -> 120 (funnels all 67 awaits)
- DynamicForkTests: 6 ceilings raised to 30s
- DoWhileEdgeCasesTests: 30s -> 60s
Verified against a live current-main server: WorkflowRerunTests 30/30
(engine-gap excluded), WorkflowRetryTests 16/16, DynamicForkTests 7/7,
DoWhileEdgeCasesTests 3/3.
* test: await task appearance in the do_while rerun-contract test setup
The converted contract test kept the original setup's one-shot orElseThrow()
lookups (WAIT tasks per iteration, iteration-2 SWITCH); under CI load the
loop progression lags the read (NoSuchElementException in dispatch run 1,
redis-es8). Same await treatment as the rest of the suite.
* test: raise status-await ceilings on multi-hop sub-workflow progressions
Census runs on CI show nested rerun/retry progressions intermittently
exhausting 15-30s (and once 120s) status awaits while passing locally:
each nesting hop that loses the inline-decide lock race falls back to the
sweeper backstop, and those waits compound across hops on starved runners.
- WorkflowRerunTests awaitWorkflowStatus default 15s -> 60s (+ 10s/20s
call sites -> 60s), nested-rerun RUNNING await 30s -> 90s
- WorkflowRetryTests WF_AWAIT_SECS 120 -> 180 (FORK_JOIN_DYNAMIC spawns
three children; 2/2 census failures at 120s)
All positive eventually-waits: free on passing runs. If the census still
shows exhaustion at these ceilings, the follow-up is engine-side (decide
re-drive under lock contention), not further test patience.
* ci: cancel superseded PR runs on new pushes (concurrency group)
Two runs of the same PR on different shas were burning runners in parallel.
Same pattern as orkes-conductor's workflows; groups are keyed by event type
so scheduled nightlies and manual dispatches never cross-cancel - only a
stale PR run is cancelled when its PR receives a new push.
* test: fix spotless violation; add WFDUMP diagnostic on awaitWorkflowStatus timeout
spotlessApply on WorkflowRerunTests (broke the build job in the dispatch
census). Port the task-tree dump diagnostic to WorkflowRetryTests: the
FORK_JOIN_DYNAMIC retry-completion test is the census's one deterministic
CI failure (parent stuck RUNNING for 181s on 5/5 flavors while passing
locally) — on the next census runs the WFDUMP marker will show exactly
which task/JOIN/child is non-terminal.
* fix(core): expedite SCHEDULED sibling JOINs too, not only IN_PROGRESS
A JOIN recreated by retry/rerun stays SCHEDULED until every branch is done
(Join#execute only flips status on completion). When such a JOIN's queue
message goes dark under load (popped but its execution dropped), the
expedite added for IN_PROGRESS JOINs skipped it, so the parent workflow
hung RUNNING indefinitely after the last branch completed.
Evidence: WFDUMP from the CI e2e census (FORK_JOIN_DYNAMIC retry test,
2 flavors, run 30884774592) shows all fork branches and their fresh
children COMPLETED while dyn_join_ref sits SCHEDULED for 181+ seconds.
The JOIN backoff itself caps at the system task callback time, so only a
lost/reserved queue message explains a stall that long; the expedite's
push-if-missing is the rescue and must not filter SCHEDULED out.
Unit test: completed sub-workflow branch re-pushes a SCHEDULED sibling
JOIN whose message is gone, postpones an IN_PROGRESS one to 0, and leaves
terminal JOINs untouched.
* test: tag FORK_JOIN_DYNAMIC retry stall engine-gap; restart policy for cassandra server
The FORK_JOIN_DYNAMIC retry test hangs on a real engine gap (SCHEDULED
JOIN whose queue message is lost is never re-evaluated) — deterministic
under CI load, so exclude it from the blocking e2e run via the existing
engine-gap tag until the core expedite fix is validated. Runs locally and
with -PincludeEngineGaps as before; no @Disabled.
The cassandra e2e job dies at boot when conductor-server hits a transient
'session is closed' from a just-healthy Cassandra and never retries;
restart: on-failure:3 lets the boot race resolve within the run script's
existing 300s health wait.
* revert: restore WorkflowRerunTests to main; drop engine-gap machinery and cassandra yml change
Back out the rerun-test re-enabling experiment wholesale: WorkflowRerunTests
returns to main's version (original @Disabled set), the engine-gap tag
exclusion leaves e2e/build.gradle, and the cassandra compose restart policy
is withdrawn. The branch now only hardens tests that already run (await
ceilings, WFDUMP diagnostic, SubWorkflowRestartSpec setup) and carries the
SCHEDULED-JOIN expedite core fix. No running test is disabled.
* fix(core): evaluate JOIN on start() so a retried/rerun JOIN can complete
retry/rerun recreate a FAILED JOIN with status SCHEDULED
(taskToBeRescheduled, rerunWF), but nothing in the engine can evaluate a
SCHEDULED JOIN: AsyncSystemTaskExecutor calls execute() only for
IN_PROGRESS tasks and start() for SCHEDULED ones, Join inherited the
no-op base start(), and decide() does not evaluate async JOINs. The
rescheduled JOIN is popped, no-oped, and postponed forever while the
parent hangs RUNNING after every branch completes. This is why
JoinTaskMapper creates JOINs directly IN_PROGRESS.
Override start() to run the first evaluation.
Reproduced via public API only (plain FORK_JOIN, two SIMPLE branches:
fail the JOIN, retry, complete both branches): without this fix the
parent sticks RUNNING with the JOIN SCHEDULED at pollCount=16; with it
the workflow completes in 5s. Root cause of the chronic nightly e2e
failures in WorkflowRetryTests (FORK_JOIN_DYNAMIC retry),
DynamicForkTests (retried fork), and WorkflowRerunTests (rerun in FORK
branch) — all green against a fixed server.
* test(e2e): raise JOIN-latency ceilings, 90s client read timeout; restore cassandra restart policy
DynamicForkTests: a plain fork branch failure only fails the workflow when
the JOIN's backed-off async evaluation observes it (nothing expedites a
JOIN on task failure), so the 30s/60s ceilings flake under CI load — raise
to 90s/150s. DoWhile stress tests were dying on the SDK client's 30s read
timeout fetching huge workflows, not on assertions — raise to 90s. Restore
restart: on-failure:3 for the cassandra server (boot-time 'session is
closed' from a just-healthy Cassandra killed the job with no retry).
* test(e2e): re-apply await hardening to WorkflowRerunTests (awaits only)
Replace one-shot task lookups with awaits and raise short ceilings in the
enabled WorkflowRerunTests — the census showed the reverted file failing
with the exact pre-hardening signatures (child inner task completed
against a stale task id after nested rerun -> parent FAILED with reason
'null' at ~12s).
Scope guarantee, verified against origin/main: all 13 @Disabled tests
keep main's exact text (nothing re-enabled, no contract rewrites, no
tags); every added line is await/polling machinery. Control run proves
the 3 locally-failing do_while rerun tests fail identically with main's
file version on the same server (pre-existing, static-name state
pollution locally; tracked via census on fresh CI servers).
* fix(cassandra): stop 500ing workflow completion; skip unavailable-capability e2e suites
CassandraExecutionDAO.removeFromPendingWorkflow threw
UnsupportedOperationException from a method its own javadoc calls a dummy
— cassandra has no pending-workflows structure — turning every
completeWorkflow/terminateWorkflow that hits the already-terminal branch
into an HTTP 500. The first census run where the cassandra server
actually booted showed 80/200 e2e failures, the bulk of them updateTask/
terminateWorkflow calls dying on this exception. Make it the no-op it
documents.
The rest of the cassandra failures are true capability gaps: the flavor
runs with conductor.integrations.ai.enabled=false (no skill DAOs) and no
/api/files resource. Introduce E2E_DISABLED_CAPABILITIES (forwarded by
e2e/build.gradle, set to ai,filestorage by run_tests-cassandra-es7.sh)
and skip AgentTaskTests/FileStorageE2ETest via @DisabledIfSystemProperty
instead of failing them against endpoints that do not exist.
* fix(core): JOIN must not fail while a branch failure's retry decision is pending
The async JOIN evaluation races the decider: after a fork branch attempt
fails, decide() either schedules a retry (old attempt gets retried=true),
marks it executed=true when it declines to retry, or fails the workflow
when mandatory retries are exhausted. A JOIN evaluated inside that window
saw a non-successful latest attempt and failed the workflow although a
retry was still owed.
This is the chronic CI failure of the DynamicForkTests retried-fork
tests: with retryDelaySeconds=1 the workflow went FAILED with only 2 of 3
attempts present, deterministically under CI load where the window is
wide (the tests' reversed assertEquals arguments made the reports read
backwards: 'expected FAILED but was RUNNING' was the workflow being
FAILED when it should still be RUNNING).
Treat a terminal, unsuccessful, retriable attempt with retried=false and
executed=false as retry-decision-pending: the JOIN keeps waiting (also
excluded from the all-terminal completion check so it cannot complete
past it). FAILED_WITH_TERMINAL_ERROR/CANCELED are not retriable and fail
the JOIN immediately as before. Existing TestJoin fixtures that meant
'decider declined retry' now set executed=true; new tests cover the
pending window, the retried-attempt re-evaluation, and the non-retriable
fast path.
* test(diagnostic): enrich WFDUMP with failure reasons and retried/executed flags
The remaining CI-only race (parent workflow re-FAILS immediately after
retry/rerun, fresh tasks CANCELED) does not reproduce locally (15/15
green); the previous dump lacked the workflow's reasonForIncompletion and
the per-task retried/executed flags needed to attribute it. Extend the
WorkflowRetryTests dump and add the same dump to WorkflowRerunTests'
awaitWorkflowStatus so the next census runs capture the full evidence.
* fix: drop getFailedTaskId from WFDUMP (not on the client Workflow model)
* Revert "fix(core): JOIN must not fail while a branch failure's retry decision is pending"
This reverts commit 3be90dc2ab5c6aa8d3020f79a59a03a3434f1976.
* test(e2e): await event-handler visibility after registration
EventClientTests read the handler list immediately after registering; on
slower backends (cassandra in the census: 'expected 1 but was 0' at ~4s)
the handler is not yet visible. Await up to 30s instead of a one-shot
read.
* revert(core): drop all engine changes from this PR — tests/CI/docker only
Per review direction, PR #1465 carries only test-side hardening and CI/
flavor infrastructure. The core changes (Join.start evaluation for
rescheduled JOINs, expedite of SCHEDULED sibling JOINs, cassandra
removeFromPendingWorkflow no-op) are removed; the engine issues they
addressed remain documented in the census WFDUMP evidence and commit
history for follow-up.
* fix(core): retry container/join tasks in place, aligning with OrkesWorkflowExecutor
Port OrkesWorkflowExecutor#taskToBeRescheduled's in-place branch: DO_WHILE,
FORK_JOIN, JOIN and EXCLUSIVE_JOIN are retried as the same task (retried=false,
retryCount+1, IN_PROGRESS) instead of a fresh SCHEDULED copy.
JOIN/EXCLUSIVE_JOIN are in the in-place branch here although Orkes' block
lists only DO_WHILE/FORK_JOIN: OrkesJoin is sync so a retried join takes the
sync-system-task copy branch (IN_PROGRESS) there, while conductor-oss's Join
is async — its SCHEDULED copy lands in a queue where the executor only calls
the no-op start(), so the join is popped, never evaluated, and postponed
forever, and the workflow hangs RUNNING after all branches complete. A JOIN
must never be SCHEDULED (the mappers create joins IN_PROGRESS for exactly
this reason).
This is the root cause of the chronic nightly FORK_JOIN_DYNAMIC retry stall
(census WFDUMP: old JOIN FAILED retried=true, new JOIN SCHEDULED
retried=false executed=false, parent RUNNING for 180s+ with every branch
COMPLETED). Validated: deterministic API repro (fail JOIN -> retry ->
complete branches) hangs forever without this and completes in 3s with it;
DynamicForkTests 7/7 and the FORK_JOIN_DYNAMIC retry e2e green locally;
in-place task passes dedupAndAddTasks untouched (already in the task list
with the bumped retryCount) and createTasks upserts by task id.
* ci: run the e2e matrix in parallel
max-parallel: 1 made a full 6-flavor matrix take ~90 minutes (6 x ~14min
sequentially); each matrix job runs on its own runner VM, so parallel
execution completes the same matrix in ~15 minutes with no contention.
* fix(cassandra): removeFromPendingWorkflow is a no-op; SignalTaskTest uses UUID ids
CassandraExecutionDAO.removeFromPendingWorkflow threw
UnsupportedOperationException from a method its own javadoc calls a dummy
(cassandra keeps no pending-workflows structure), turning
completeWorkflow/terminateWorkflow calls that hit the already-terminal
branch into HTTP 500s — dozens of e2e failures on the cassandra flavor.
Make it the documented no-op.
SignalTaskTest's not-found tests used a non-UUID workflow id: cassandra
parses ids as UUIDs and returns 400 on the parse before reaching the
not-found path every backend 404s on. Use a random UUID so all backends
exercise the same not-found path.
* ci: build the server image once and share it across the e2e matrix
Every e2e flavor built the identical server image from source (~6 min per
job, six times per run) — the flavors differ only in CONFIG_PROP and their
compose sidecars, not the image. A build-server-image job now builds it
once, uploads it as an artifact, and the matrix jobs docker-load it;
SKIP_SERVER_BUILD=1 makes the run scripts skip their per-flavor rebuild
(compose up does not rebuild when the image is already present). Saves
~30 runner-minutes per full matrix run; local usage of the scripts is
unchanged.
* fix(core): JOIN must not fail while a branch failure's retry decision is pending
The async JOIN evaluation races the decider: after a fork branch attempt
fails, decide() either schedules a retry (old attempt gets retried=true),
marks it executed=true when it declines to retry, or fails the workflow
when mandatory retries are exhausted. A JOIN evaluated inside that window
saw a non-successful latest attempt and failed the workflow although a
retry was still owed.
This is the chronic CI failure of the DynamicForkTests retried-fork
tests: with retryDelaySeconds=1 the workflow went FAILED with only 2 of 3
attempts present, deterministically under CI load where the window is
wide (the tests' reversed assertEquals arguments made the reports read
backwards: 'expected FAILED but was RUNNING' was the workflow being
FAILED when it should still be RUNNING).
Treat a terminal, unsuccessful, retriable attempt with retried=false and
executed=false as retry-decision-pending: the JOIN keeps waiting (also
excluded from the all-terminal completion check so it cannot complete
past it). FAILED_WITH_TERMINAL_ERROR/CANCELED are not retriable and fail
the JOIN immediately as before. Existing TestJoin fixtures that meant
'decider declined retry' now set executed=true; new tests cover the
pending window, the retried-attempt re-evaluation, and the non-retriable
fast path.
* fix(core): repair siblings before reviving the parent; decide inline (Race B, Orkes parity)
updateAndPushParents persisted the parent as RUNNING before repairing its
stale sibling tasks, then left the first evaluation to an async decider-
queue push. From the moment of that persist, any concurrent decide could
evaluate a RUNNING parent whose CANCELED SUB_WORKFLOW sibling still
pointed at a not-yet-resumed TERMINATED child — the sync path mapped the
stale child status onto the task (TERMINATED, reason 'null') and the
freshly retried parent was terminated again, orphaning the resumed child
(census WFDUMP: parent TERMINATED citing a task whose child is RUNNING
with a fresh SCHEDULED task).
Mirror OrkesWorkflowExecutor's order exactly: apply the parent status
reset in memory, repair every sibling task first, persist the RUNNING
parent last, then decide inline — concurrent decides bounce off the
still-terminal stored parent during the repair window, and the revived
parent's first evaluation runs on fully repaired state.
* ci: disable redis-es7 and cassandra-es7 e2e flavors
redis-es8 becomes the always-on flavor (runs on every PR/push); optional
profiles are postgres, mysql, redis-os3. ES7 coverage is superseded by
the es8 flavor and cassandra support is partial; both run scripts remain
in e2e/ for local use and can be re-added to the matrix later.
* ci: revert shared server image — INDEXING_BACKEND is baked at build time
The server image is NOT identical across e2e flavors: docker/server/
Dockerfile takes INDEXING_BACKEND as a build arg (default elasticsearch;
es8 passes elasticsearch8, os3 passes opensearch3), so the shared default
image left the es8 server without an IndexDAO bean (APPLICATION FAILED TO
START in the verification run). With the matrix reduced to four flavors
spanning three distinct backends, sharing would save a single duplicate
build — not worth per-backend artifact plumbing. Flavors build their own
image again; the SKIP_SERVER_BUILD guard in the run scripts stays
(dormant, default off).
* fix(core): fence late child events from rerun-superseded parent task generations
A rerun from a fork task replaces the parent's fork generation; the old
SUB_WORKFLOW task rows survive in the task store but leave the parent's
task list. A late terminal event from the old generation's child still
propagated through that stale task record and failed the parent's fresh
generation (census WFDUMP: parent FAILED citing a task id absent from its
own task list, child failure reason 'null'). Retry already fences
superseded attempts via isRetried(); rerun-superseded tasks are now
fenced by parent task-list membership in updateParentWorkflowTask, with
the drop logged. Unit test covers the dropped propagation.
* core: restrict core changes to WorkflowExecutorOps; disable the two async-JOIN race tests
Join.java and TestJoin return to main per review scope (core changes only
in WorkflowExecutorOps). Without the JOIN-side guard the async JOIN can
again evaluate between a fork branch attempt's FAILED persist and the
decider scheduling its retry, so the two DynamicForkTests that exercise
retried forks are @Disabled with the race documented; the follow-up is a
test-side rework to explicit task polling + PUT /workflow/decide
sequencing.
* test(e2e): disable rerun-from-FJD test pending rerun/decide snapshot fencing
A rerun issued while the original child-failure propagation is in flight
loses to that decide's pre-rerun snapshot: the parent is re-FAILED citing
a task id absent from its own task list (two census WFDUMPs, postgres).
The generation fence in updateParentWorkflowTask stops late child events;
this door — an in-flight decide committing a verdict computed against the
superseded generation — needs rerun/decide lock-versioning in the engine.
Disabled with the evidence documented until that fix exists.
* test(e2e): disable deeply-nested retry test — same in-flight-decide race family
The multi-level retry walk-up revives the mid-level parent, and an
in-flight decide on a pre-revival snapshot re-terminates it citing the
sibling's superseded TERMINATED state (census WFDUMP; both children's
reasons cite each other's termination). Same engine door as the disabled
rerun-from-FJD test: revival vs decide needs lock-versioning. Disabled
with the evidence until that engine fix exists.
* fix(core): hold the parent's execution lock across the walk-up revival; await SetVariable batch
The repair->persist sequence in updateAndPushParents ran without the
parent's execution lock, so a concurrent decide holding a pre-revival
snapshot could interleave its stale verdict with the revival (census
WFDUMP: revived mid-level parent re-TERMINATED citing a sibling's
superseded state, both children's reasons citing each other). Acquire the
parent's lock across load -> sibling repair -> persist; the inline decide
runs after release, when the repaired state is fully persisted, so any
decide ordering is then safe.
SetVariableTests replaced its fixed 5s sleep with an await on the whole
batch reaching COMPLETED (180s) — under CI load the sleep converted
scheduling latency into assertion failures.
* core: drop the walk-up lock — OrkesWorkflowExecutor takes none; ordering is the contract
Verified against OrkesWorkflowExecutor#updateAndPushParents: it holds no
execution lock; its protection is exactly the repair-first/persist-last/
decide-inline ordering already ported. Remove the lock wrapper so the
method matches Orkes verbatim in structure.
* test(e2e): disable two more rerun-family tests — same deterministic-child-id race
Same family as the two already-disabled rerun tests: the in-place
SUB_WORKFLOW reset regenerates the deterministic child id and the
idempotent start races its own status sync against the old FAILED child
under the same identity, re-failing the parent with the superseded
child's reason (census WFDUMPs across four runs, one family member per
run). Disabled with the evidence pending the startWorkflowIdempotent/
sync engine fix.
Four comment blocks had accumulated the story of how the lane got here —
the rc2-to-rc4 migration, the agentspan CLI that used to be load-bearing,
the floor's 127/137/135 progression with CI run IDs, a cross-reference to
the PR that first pinned mcp. That belongs in git history and the PR, not
in a workflow someone reads to understand the current configuration.
Each block keeps the part that prevents a wrong edit and loses the part
that only records what happened:
* env names — keep "verify against the pinned ref before renaming, a name
the suite does not read falls through to defaults silently"; drop the
AGENTSPAN_* migration account.
* SCHEDULER_CONDUCTOR_URL — keep the probe and the port-8089 default that
makes it necessary; drop "10 of the 11 then pass".
* E2E_MIN_PASSED — keep why the floor exists and the raise/lower rule, and
add that moving the pin can change the test set so the number must be
re-measured; drop the progression log.
* mcp pin — keep the unbounded-dependency mechanism; drop the #1408
cross-reference.
Comments in the job: 69 lines -> 52. No behaviour change.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Mirrors orkes-io/orkes-conductor#3888. Instead of downloading the released
conductor-ai-e2e-python-<version> bundle, check out conductor-oss/python-sdk
at the pinned ref and run its e2e/ in place through run-suite.sh — the same
entrypoint that repo's own agent-e2e.yml uses.
Why: the bundle restated the suite's dependency set on our side, so an
upstream dep change or version bump could leave us testing something stale,
and the bundle had to exist as a published release artifact before we could
point at it. Running in place removes both. The SDK is now built from the
checked-out source rather than resolved from PyPI, so the pin can be any tag,
branch or SHA — set the repo variable to test unreleased SDK work without a
commit here. If upstream breaks its own entrypoint, its CI goes red before
ours does.
run-suite.sh keeps require_path guards on e2e/ and setup.py: that layout is
the only coupling left after dropping the generated manifest, so it fails with
a named cause instead of a confusing pip or pytest error.
Unchanged: the version env var (the _BUNDLE_ in its name is now vestigial but
matches the configured repository variable), known-failures passed through as
trailing pytest args, the passed-count floor, JUnit publishing and artifact
upload.
Also pins mcp-testkit, which is broken today independently of this change:
it declares `mcp[cli]>=1.0.0` unbounded, and mcp 2.0.0 dropped
mcp.server.fastmcp which the testkit imports at start-up, so the unpinned
install yields a testkit that exits immediately and every MCP suite skips.
Verified locally. #1408 already made this fix for the test-harness install;
this job predates it on this branch and never got it.
Verified against a server built from this branch: 135 passed, 7 skipped,
3 xfailed, 0 failed — identical to the bundle-based run, so the floor stays
at 135.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- redis-es7 is now in the matrix for every trigger, so pushes and pull requests
get one full end-to-end run instead of none. Previously e2e only ran on
workflow_dispatch and schedule.
- The matrix runs one backend at a time (max-parallel: 1) in the listed order,
with redis-es7 first so it is also the first to report.
- workflow_dispatch takes an e2e_profiles input to pick which additional
backends run: "all" (default), "none", or a comma separated subset of
redis-es8, postgres, mysql, redis-os3, cassandra-es7. An unrecognised name
fails the matrix job with the list of valid values rather than silently
running a smaller matrix.
The entries had grown into full root-cause write-ups — 2.2KB for one
reason string. That analysis belongs in the tracking issue and the fixing
PR, where it can be discussed and closed out; duplicated here it just goes
stale silently, which is exactly what happened to the suite14 entry.
Each value is now a short statement of what fails plus a tracking link.
_README says to keep it that way. File is 2227 bytes, down from 4781.
Kept the operational parts, which are not history and prevent real
mistakes: the E2E_MIN_PASSED coupling, and the warning that a run:false
xfail can never XPASS so it must be un-listed by hand.
Also repoints suite14 at conductor-oss/python-sdk#459, which supersedes
the now-closed #455.
Verified: plugin still reports both entries matching exactly one test,
no MATCHED NOTHING.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Spell out "PYTHON" to match the lane's name and the other CONDUCTOR_*
variables. Covers all three roles: the job env key, the vars.* lookup for
the commit-free override, and the shell consumer in the fetch step.
Note the vars.* name changes with it, so a repo/org variable set under the
old name stops being honoured and the lane falls back to the in-file pin.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
rc4 is the release that lands the changes several deferred items were
waiting on, so they all come due together:
* env names. rc4's conftest migrated to CONDUCTOR_SERVER_URL /
CONDUCTOR_AGENT_LLM_MODEL, so the rename TODO is resolved and the vars
are live rather than inert. Noted in-file that they must go back if the
pin is ever moved down to rc2 or earlier, which read only AGENTSPAN_*.
* agentspan CLI, removed. rc4 dropped it entirely — no CredentialsCLI, no
CLI_PATH, and test_suite16_cli_skills.py is gone — so provisioning it
buys nothing. Recorded that it WAS load-bearing under rc2, where the
credential suites reached their read-only skip by invoking it and
removing it turned clean skips into FileNotFoundError failures, so it is
not restored on stale reasoning.
* the three Suite16 cli-skills known failures, removed. Their file no
longer exists, and a key matching nothing now raises a
KnownFailuresWarning, so leaving them would add noise every run. History
kept in _HOWTO.
* floor 137 -> 135. This is a coverage LOSS, not a fix: rc4 deleted
test_suite16_cli_skills.py, and while three of its tests were xfail-ed
here, two were genuinely passing (load_serve_and_run_by_name,
run_ephemeral_executes_script_worker). Real coverage of the
ephemeral/by-name CLI skill paths went away with the file, invisibly in
pass/fail terms. Spelled out at the constant so the -2 is not read as
routine.
Verified locally against a server built from this branch: 135 passed,
7 skipped, 3 xfailed, 0 failed; floor gate green; plugin reports both
remaining entries matching exactly one test with no MATCHED NOTHING.
suite14's entry stays — its fixes are in conductor-oss/python-sdk#455,
still unmerged, so rc4 does not carry them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The entry blamed #1356 for dropping the swarm transfer task defs, giving
updateTaskDef NotFound so "agents can't hand off". Reproducing it locally
showed that is wrong on every link: the SDK registers all six
*_transfer_to_* defs successfully, the "No such task by name" server lines
are its benign lookup-before-create, and the handoff itself succeeds —
agent 0's sub-workflow reaches COMPLETED including transfer_msg.
What actually hangs is a tool worker. _register_workers reads
`agent.stateful` on the immediate agent only, but the test hangs swarm_tool
off the swarm MEMBERS, which are not themselves stateful — so the worker
registers with domain=None while the stateful swarm makes the server
domain-route the task. pollCount=0, startTime=0, the FORK's JOIN never
satisfies, workflow times out.
A second, independent bug sits behind it: the test's _find_tasks_by_type
matches only taskDefName, but handoff_check is an INLINE task whose
taskDefName is literally "INLINE", so the assertion fails even though 20
handoff_check tasks exist and are COMPLETED.
Both fixes belong to python-sdk. Verified against the pinned 2.0.0-rc2:
stock FAILED (RUNNING, 304s); SDK fix alone reached COMPLETED but failed
the handoff_check assert; both fixes PASSED in 118s.
Also corrects the timing: one attempt is ~304s, not ~908s — that figure was
three attempts under conftest's unconditional flaky(reruns=2).
Refs #1363
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The 108-pin constraints file worked, but it had to be regenerated on every
CONDUCTOR_PY_E2E_BUNDLE_VERSION bump or pip would fail on the conflict —
too much standing upkeep for the protection it bought on a CI lane.
Drop the file and PIP_CONSTRAINT. Replace with a comment recording that
transitive deps float, that an upstream release can redden the lane with
no change here, and — the useful part — that it will look like a server
regression, so check the pip install output before hunting a server cause.
setup-java v5 from the previous commit is unaffected.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The bundle ships prebuilt and its run.sh does a plain
`pip install -r requirements.txt` at run time. That requirements.txt pins
only conductor-python — pytest, langgraph, mcp-testkit and the provider
SDKs all float, so an upstream release can redden this gating lane with no
change to this repo, presenting as a server regression. Already real: a
langgraph/pydantic interaction with typing.TypedDict breaks suite11 on
Python < 3.12.
Add a full transitive pin set and point PIP_CONSTRAINT at it. pip honours
that env var for every install in the job, including the one inside the
bundle, so the bundle gets pinned without being modified. Constraints only
bound versions, they install nothing, and a conflict with a future bundle
surfaces as a loud resolver error — regenerate rather than delete, per the
header.
Generated with `uv pip compile --python-platform x86_64-unknown-linux-gnu
--python-version 3.12` to target the runner rather than a dev machine; the
CI log was not usable as a source because the earlier --quiet install
swallowed most of the closure (73 of 108 packages).
Verified: PIP_CONSTRAINT demonstrably rebinds resolution (forced pytest
8.3.4 / langgraph 0.6.7 against newer available), and resolving the real
requirements.txt under this file yields 108 packages with all 108 pins
satisfied and no conflict.
Also bumps this job's setup-java v4 -> v5, matching every other job.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three review findings on the e2e lane.
1. Stale docs. The plugin docstring said the known-failures list is "empty
when the suite is green" — it has 5 entries and the suite is green, so
the parenthetical read as a contradiction. Reworded, and the JSON
_README now also documents the new match-count warnings and the fact
that adding an entry requires lowering E2E_MIN_PASSED in the same
commit.
2. Silent misconfiguration. A missing/typo'd E2E_KNOWN_FAILURES path loaded
nothing and returned early before any reporting, so every forgiven
failure reported as real with no explanation — the one case the new
diagnostics still could not see. Now warns, and distinguishes that from
the legitimate "no list configured" case, which stays silent.
3. Readiness probe. `curl -sf http://localhost:3001/` treats mcp-testkit's
404-on-root as failure, so it never succeeded: the loop burned all 15
iterations and printed "started" whether or not anything was listening.
Check for any HTTP status instead, warn and tail the log on timeout.
Verified: bad path warns / unset path silent / good path reports 5x1 with
no warnings; probe detects a live mcp-testkit (HTTP 404) and warns when
nothing listens, without aborting under bash -e.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The plugin's only diagnostic never reached the log. run.sh runs pytest
with -n 3, collection happens in xdist workers, and a worker has no
terminalreporter — so the write was dropped. Confirmed: zero occurrences
of "[known-failures]" in either full CI-shaped run, while a non-xdist run
prints it. With matching unverifiable, a typo'd key and an over-broad key
looked identical to a correct one.
Report per-key match counts over two channels, since neither alone
suffices: warnings (xdist forwards worker warnings to the controller) for
the actionable 0-match and >1-match cases, and terminal lines for the full
table, written inline without xdist or from pytest_testnodedown with it.
Counting is per key rather than per first-match, so an over-broad key is
still visible when another key claimed the item first.
Verified under -n 3: full suite reports all 5 entries at 1x each with no
warnings; an injected bogus key reports 0x and warns. pytest_testnodedown
is marked optionalhook so the plugin still loads without xdist.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
_matches() had a bare `nid.endswith(suf)` arm, so a key like "_completes"
matched every test whose name ended that way and silently xfail-ed
unrelated tests — precisely the hide-a-regression failure the module
docstring claims is impossible.
Anchor on "/" rather than dropping the arm. Node-ids are
"e2e/<file>.py::<Class>::<test>" while the list's keys are
"<file>.py::<Class>::<test>", so neither the exact nor the "::"-anchored
arm matches them; removing the unanchored arm outright would have stopped
all five current entries from matching, un-xfail-ing known failures and
reddening the lane.
Verified: each of the 5 entries still matches exactly one node-id, no
node-id matches two keys, and the bogus keys "_completes" / "completes" /
"returns_none" / "and_delete" / "e" now match nothing. Real collection
still reports "xfail-marked 5 item(s)" over 154 collected.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
test_suite21_scheduling.py skips all 11 of its tests unless
GET {SCHEDULER_CONDUCTOR_URL}/scheduler/schedules answers 200. Its default
is port 8089 and the lane's server is on 8080, so the probe never matched
and the suite skipped silently on every run — no failure, no signal.
conductor-oss serves the scheduler itself (conductor.scheduler.enabled=true
by default, SchedulerResource at /api/scheduler); the endpoint was verified
to return 200 on the lane's own server. Pointing the var there runs the
suite: 10 pass, 1 fails.
That one failure is real and now recorded rather than hidden:
get_schedule() on a deleted schedule is expected to return None, but the
server 404s and OrkesSchedulerClient propagates ApiException(404).
Floor raised 127 -> 137 to lock the recovered coverage in. Verified locally:
137 passed, 11 skipped, 6 xfailed, floor check green.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A pytest skip is not a failure, so CI never reddens on one. A server
change that breaks a test's setup therefore converts pass -> skip and the
lane stays green while covering less — the job built to catch the break
becomes the thing hiding it. The report step is no backstop either:
fail_on_failure and require_tests are both false, so even an empty results
file renders green.
Parse results/junit-e2e.xml after the run and fail if fewer than
E2E_MIN_PASSED tests actually passed, or if the file is absent. Floor is
127, observed identically on two CI runs (30398185123, 30400603109) and
locally; junit folds xfail into skipped, which the arithmetic accounts for.
Also adds -rs so skip reasons appear in the log — without it a test that
quietly stops running leaves no trace in the output.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Public repo, so GitHub withholds secrets from fork-originated runs:
OPENAI_API_KEY arrives empty, the server's openai provider fails to
initialise, and the LLM suites fail for a reason an outside contributor
cannot fix. Skip rather than run a lane that cannot pass.
The fork test is nested inside the pull_request branch on purpose. As a
top-level AND it would also disable push, workflow_dispatch and schedule,
where github.event.pull_request is null so the comparison is false —
which would silently turn off the lane's primary coverage.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Removing the CLI was wrong. The credential suites (xdist group
`credentials`, suites 2/3/4/5/26) reach their conductor-oss skip THROUGH
the binary: CredentialsCLI.set() shells out, the server rejects the write
because its secret store is env-backed and read-only, and that stderr is
what triggers pytest.skip. conftest's subprocess.run has no try/except, so
with no binary they raise FileNotFoundError and fail.
Verified locally: a run with the CLI absent turned clean skips into hard
failures in suites 2, 4 and 5 before it was stopped. The step's original
rationale was accurate; the prior commit removed it on a wrong reading
that generalized from Suite16's fixture (which does skip) to the
credential fixture (which does not).
Keeps the corrected skills context — server-side skills remain
unsupported, so Suite16 stays xfail-ed rather than skipped.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
conductor-oss does not support the server-side skills API: SkillController
is gated on both conductor.integrations.ai.enabled and
agentspan.skills.enabled, and the latter has no default and is set in one
place repo-wide (AgentSpanDeploymentContractEndToEndTest). The only other
CLI consumers are the credential suites, which skip on this flavor because
the secret store is env-backed and read-only. So the download bought
nothing on every run.
Absent the binary the suites skip cleanly — Suite16's fixture calls
pytest.skip and the credentials fixture only constructs an object — so the
step's stated rationale (FileNotFoundError without it) did not hold either.
Also rewrites the known-failures context: these are an unsupported feature,
not a bug pending a decision, and the server-side skills registry is
distinct from the "Conductor Skills" coding-agent integration in docs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The 'agent' filter listed 'conductor-agentspan/**', which matches no
tracked file — ':conductor-agentspan' is the Gradle project name, the
directory is 'agentspan/'. An agentspan-only PR therefore skipped the
python SDK e2e lane. Exactly the kind of change that regressed skills
coverage (#1288 gated SkillController, tracked in #1353).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Records why the rename was rolled back and the order it has to happen
in. The failure mode is silent — the prebuilt bundle's conftest reads
only AGENTSPAN_*, so renaming early drops both vars to conftest defaults
that currently happen to match — hence the explicit "do not rename yet".
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Reverts the rename and the mirroring shim that followed it. The e2e
bundle lives in conductor-oss/python-sdk and arrives here prebuilt, so
its conftest is the thing that has to migrate to CONDUCTOR_*; renaming
on this side only made the vars inert. Restore AGENTSPAN_SERVER_URL /
AGENTSPAN_LLM_MODEL until a migrated bundle is released and pinned.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
CONDUCTOR_SERVER_URL is the canonical name — the SDK's Configuration
reads it in preference to AGENTSPAN_SERVER_URL. The pinned e2e bundle
has not migrated: conftest reads AGENTSPAN_SERVER_URL for its direct
HTTP calls and AGENTSPAN_LLM_MODEL for the model, and reads
CONDUCTOR_AGENT_LLM_MODEL nowhere. Mirror both at the run step off the
canonical job-level values, so the lane targets the CI server and pinned
model instead of silently falling back to the bundle defaults.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Read CONDUCTOR_PY_E2E_BUNDLE_VERSION from `vars` and fall back to the
2.0.0-rc2 pin, so a bundle can be trialled without a commit. An unset
variable is the empty string (falsy), so the fallback applies.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The released python e2e bundle reads CONDUCTOR_SERVER_URL and
CONDUCTOR_AGENT_LLM_MODEL; the AGENTSPAN_* names were ignored, so the
suite fell back to defaults instead of the CI server and pinned model.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The stateful-swarm-handoff xfail (#1363) is a deterministic ~908s hang that
would burn ~15 min of CI every run. Add per-entry run control to the plugin:
a JSON value may now be a reason string (run=True, XPASSes when fixed) or an
object {"reason":..., "run":false} to xfail WITHOUT executing.
Set the swarm entry to run:false so it's skipped (marked xfailed [NOTRUN], no
hang); the 3 cli-skills entries stay run=True. Un-list the swarm entry
manually when #1363 is fixed (a non-run xfail can't auto-XPASS).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The python-sdk-e2e gate caught a real, deterministic regression on main:
test_stateful_swarm_handoff_completes hangs (workflow stuck RUNNING ~908s,
2/2 runs). Root cause: #1356 ("Enhances A2A/AgentSpan execution") stopped
registering the swarm transfer/handoff task defs (now compiler-owned INLINE),
but the pinned SDK bundle (2.0.0-rc2, the latest release) still PUTs them ->
updateTaskDef NotFound -> agents can't hand off -> workflow never completes.
Passed pre-#1356 (2026-07-17). Server-side fix owned by #1356; no newer SDK
to bump to. Tracked in conductor-oss/conductor#1363.
xfail it so the gate goes green; it XPASSes (remove the entry) once #1363 is
fixed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sharpen the 3 Suite16 cli-skills reasons with the traced root cause: #1288
(rc.9) changed the SkillController gate from agentspan.embedded [1 prop] to
agentspan.embedded + agentspan.skills.enabled [2 props], flipping the skills
API off-by-default on main (served on rc.8, which python-sdk pins). Add a
shared _CONTEXT_cli_skills note (ignored by the plugin) with the full trace
and the #1353 decision. No behavior change — same 3 node-ids xfail-ed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Earlier reason ("skills-register endpoint missing on main") was wrong: the
SkillController exists but is gated on agentspan.embedded (intended
orkes-only for OSS) AND agentspan.skills.enabled, so POST /api/skills/register
returns 404 on conductor-oss. Per design direction, agentspan.embedded should
be false for conductor-oss (the design may change), so the skills API isn't a
settled OSS surface — do not force-enable it on boot.
Keep the 3 Suite16 cli-skills tests xfail-ed with the accurate reason + link
to the tracking issue conductor-oss/conductor#1353. They XPASS (remove the
entries) once the OSS skills story is settled.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The python-sdk-e2e gate correctly surfaced a real server gap: the CLI's
`skill register` calls POST /api/skills/register, which the server built
from main returns 404 for ("No static resource api/skills/register"). The
endpoint exists in the released 3.32.0-rc.8 that python-sdk's own CI pins,
so it's green there but not against a from-source main build.
Add the 3 affected Suite16 cli-skills tests to the conductor-oss
known-failures list so the lane gates green while the skills-register API
gap is investigated (an XPASS will flag it once the endpoint lands). The
other 151 tests — LLM, tools, MCP, guardrails, credential lifecycle — pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The re-run failed on infra, not the e2e (which never ran):
- `gh release download --output agentspan` collided with the repo's existing
`agentspan/` module dir ("already exists"). Download the CLI to
`agentspan-cli` instead (+ --clobber) via AGENTSPAN_CLI_PATH.
- The junit report step couldn't create its check ("Resource not accessible
by integration") — the workflow had no `checks: write`. Add a job
permissions block, and mark the report step continue-on-error so gating
stays purely the e2e run step's exit code.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The credential-lifecycle suites (Suite2/4/5 @credentials) shell out to the
`agentspan` binary and errored with FileNotFoundError instead of skipping,
so the first run showed 3 failed / 126 passed. Download the pinned agentspan
CLI release (mirrors conductor-oss/python-sdk agent-e2e.yml) and point
AGENTSPAN_CLI_PATH at it so those suites can run. No product issue — the
server + LLM path were already green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds a `python-sdk-e2e` job to CI that boots the conductor server built from
the current commit in SQLite mode (the default persistence — no external DB)
and runs the released python SDK agent e2e suite against it, so a server
change can't silently break the SDK before a release.
- The suite + bundle come from conductor-oss/python-sdk
(conductor-ai-e2e-python-<version>, pinned); fetched at runtime, sha256-verified.
- The server auto-configures the openai provider from OPENAI_API_KEY
(conductor.ai.openai.api-key), so no manual integration setup is needed.
- Known failures are xfail-ed via an external pytest plugin
(.github/agent-e2e/known_failures_plugin.py loaded with -p) + a per-repo
list (known-failures-python.json). The suite is green today so the list is
empty; the mechanism stays so the lane can gate while any future gap is
fixed (a fixed bug XPASSes; a stale entry is a harmless no-op).
- Gating. Runs on push/dispatch and on PRs touching agent-relevant paths
(ai/, conductor-agentspan/, server/, core/, sqlite-persistence/, the
workflow) via a detect-changes `agent` filter, to avoid spending LLM
budget on unrelated PRs.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
action-junit-report needs checks:write to create the AgentSpan E2E
Test Report check run. Every other job that publishes a JUnit report
(unit-test, test-harness, e2e) already grants this; agentspan-e2e was
missing it, so its check-run creation silently failed with
"Resource not accessible by integration".
Co-authored-by: Kowser <kowser.orkes@gmail.com>
* ci(test-harness): publish a named test-result summary check
The test-harness job passed only report_paths to mikepenz/action-junit-report, so
a run gave no summary of what actually ran — a failure showed up as Gradle's
"There were failing tests" and an exit code, with the counts only reachable by
downloading the report artifact.
Configures the action to surface a "Test-Harness tests" check reading
"N tests run, N passed, N skipped, N failed":
- check_name so the result is its own check rather than folded into the job
- detailed_summary for the per-suite breakdown
- check_title_template for readable annotation titles
- check_retries and flaky_summary to distinguish retried/flaky tests
- include_passed=false to keep the annotation volume down
fail_on_failure is intentionally not set: the Gradle step already fails the job,
so adding it would only mark a second step red.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* ci(test-harness): publish a named test-result summary check
The test-harness job's junit-report step parsed results correctly but could not
publish them:
JUnit Test Report - 338 tests run, 306 passed, 31 skipped, 1 failed.
Failed to create checks using the provided token.
(HttpError: Resource not accessible by integration)
ci.yml declares no permissions, so the job ran with the default read-only token
and the check run was rejected. fail_on_failure defaults to false, so the step
still went green and the failure was invisible — leaving Gradle's exit code as
the only signal of which test failed.
Grants the job checks: write, and names the check plus enables the per-suite
summary so the result is visible without opening the log.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* ci: publish named test-result checks for unit-test, test-harness and e2e
The junit-report steps parsed results correctly but could not publish them:
Unit Test Report - 2802 tests run, 2713 passed, 89 skipped, 0 failed.
Failed to create checks using the provided token.
(HttpError: Resource not accessible by integration)
ci.yml declared no permissions, so these jobs ran with the default read-only
token and every check run was rejected. fail_on_failure defaults to false, so the
steps still went green and the failure was invisible — leaving Gradle's exit code
as the only signal of which test failed.
Grants checks: write to the three jobs that run tests, and names each check plus
enables the per-suite summary. e2e is a matrix, so its check name carries
matrix.name to keep one check per database combination instead of the legs
overwriting each other.
The build job is left alone: it runs with -x test and has no results to publish.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
PRs touching ai/.../providers/** now run the live media suite (the exact
changes these tests exist to catch), while everything else in the repo
still never triggers a paid run. Concurrency group cancels superseded
runs on rapid pushes so spend cannot stack.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- OpenAITest / AzureOpenAITest: testChatCompletionWithImageMedia on
gpt-4o-mini — both ride OpenAIResponsesChatModel's input_image path
(pre-existing support; this locks it live at each endpoint).
- BedrockTest: new BearerIntegrationTests nest gated on
AWS_BEARER_TOKEN_BEDROCK (the dedicated org secret, matching the
server's conductor.ai.bedrock.bearerToken binding) — kept separate
from the access-key nest because the CI AWS keys are provisioned for
artifact publishing and may lack Bedrock permissions. Media flows via
Spring AI Converse image blocks; model claude-3-haiku (vision-capable,
same id as the existing bedrock tests).
- Workflow wires the new secrets (OPENAI, AZURE_OPENAI key+endpoint,
AWS_BEARER_TOKEN_BEDROCK, AWS_REGION).
Verified locally: Anthropic and OpenAI pass against the live APIs;
keyless providers skip. Azure/Bedrock get their first live run on this
PR's workflow run.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Ling-Sen Peng