Compare commits

...

2 Commits

Author SHA1 Message Date
Pat Sukprasert 943fe2c4ae test: remove orphaned d6 fixture agents (referrers deleted)
The two D6 e2e test files removed earlier on this branch were the only
referrers of these fixture agent dirs. With those tests gone, nothing
references them except their own YAML filename, so delete them to finish
the dead-code cleanup:

- tests/_fixtures/agents/d6-direct-cancel-test/
- tests/_fixtures/agents/d6-sdk-async-dispatch-test/

The fan-out fixture (tests/_fixtures/agents/d6-fan-out-test) is untouched.

Co-authored-by: Isaac
2026-06-18 08:56:39 +08:00
Pat Sukprasert bd5444971d test: delete 2 obsolete /v1/responses-route D6 e2e tests (re-homed by #555)
These two suppressed D6 e2e tests drove the removed `/v1/responses` route
(client.responses.stream(...)), so they were red because that route no longer
exists — NOT because of the `_build_terminal_event` bug their suppression text
cited (that bug does not reproduce; #20 hardening 00d9db6 already fixed it).

Their behavioral intent was re-homed onto main by #555 in
tests/integration/test_d6_async_cancel_round_trip.py:
- test_sdk_cancels_local_body_on_llm_cancel_task
  -> test_direct_cancel_parks_then_interrupts_cleanly
- test_sdk_async_client_tool_completes_round_trip
  -> test_client_tool_round_trip

Removed:
- tests/e2e/test_d6_direct_cancel_e2e.py (whole file; single test)
- tests/e2e/test_d6_sdk_async_dispatch_e2e.py (whole file; single test)
- their two terminal-d6 entries in tests/known_failures.yaml

Deliberately KEPT test_d6_parallel_fan_out_e2e (also on the dead route): its
parallel-fan-out behavior is not yet re-homed by #555.

Co-authored-by: Isaac
2026-06-18 08:47:01 +08:00
5 changed files with 0 additions and 467 deletions
@@ -1,31 +0,0 @@
name: d6-direct-cancel-test
description: |
Test-only agent that dispatches one async client tool and
then immediately cancels it.
executor:
model: gpt-5.4
prompt: |
# D6 direct-cancel E2E agent
Each turn, follow this exact sequence:
1. Dispatch the `slow_compute` tool via **`sys_call_async`**
with `tool: "slow_compute"` and
`args: "{\"seconds\": 30}"`.
2. You will receive a `function_call_output` with a handle
JSON: `{"task_id": "...", "tool_name": "slow_compute", ...}`.
Extract the `task_id` value.
3. In your **next** tool_call, call `sys_cancel_task` with that
`task_id`. Do not wait for any system message; call
sys_cancel_task immediately after you see the handle FCO.
4. You will receive a `function_call_output` like
`{"cancelled": true, "prior_status": "in_progress", ...}`
and eventually a `[System: task <id> (client_tool)
cancelled]` user message.
5. Reply with the literal string `CANCELLED_OK` so the test
can confirm you completed the sequence.
Do NOT skip sys_cancel_task. Do NOT wait for the slow_compute
to finish before cancelling.
@@ -1,33 +0,0 @@
name: d6-sdk-async-dispatch-test
description: |
Test-only agent for the Phase 5 D6 SDK-lifecycle E2E.
executor:
model: gpt-5.4
prompt: |
# D6 SDK-async-dispatch E2E agent
You are an agent for an end-to-end test that exercises the
Python SDK's async-client-tool lifecycle. Each turn:
1. Dispatch the `compute` tool via **`sys_call_async`** —
pass `tool: "compute"` and
`args: "{\"value\": \"<your-marker>\"}"`. This dispatches
the work as a background task. You will receive a
`function_call_output` with a handle JSON like
`{"task_id": "...", "tool_name": "compute", "status":
"in_progress", ...}`. **That is not the final result.**
2. After dispatching, do not output text yet. Wait for the
system message `[System: task ... completed]\n<BODY>` to
arrive in the conversation.
3. Once the system message arrives, reply to the user with
the literal string `ANSWER:<BODY>` (everything after
`ANSWER:` is the system message body verbatim) so the
test can pull the result out.
## Dispatch via `sys_call_async`
The wrapped tool is `compute(value: str)`. The tool returns
the value verbatim — your test marker is whatever string you
sent in `value`.
-207
View File
@@ -1,207 +0,0 @@
"""
E2E for D6 test plan #7: direct cancel propagates to the SDK.
Proves that when the LLM calls ``sys_cancel_task`` on a running
async client_tool, the SDK's D6 lifecycle cancels the local
``asyncio.Task`` running the tool body — the body's
``except asyncio.CancelledError`` branch fires, and the body
never returns normally.
Without the SDK-side SSE handling:
- ``SysCancelTaskTool`` still emits ``response.client_task.cancel``
(committed earlier),
- The SDK's ``stream()`` sees the event and calls
``state.asyncio_task.cancel()``,
- The running body (a blocking ``time.sleep`` inside the
asyncio task) would otherwise run to its full duration.
Without the fix this test would time out / take the full
30 s sleep; with the fix the body's ``except`` fires within
a couple seconds of the sys_cancel_task call.
Excluded from default ``pytest`` runs via
``--ignore=tests/e2e``. Invoke with::
pytest tests/e2e/test_d6_direct_cancel_e2e.py \\
--llm-api-key "$(cat /tmp/mykey)" -v
"""
from __future__ import annotations
import asyncio
import time
from pathlib import Path
import httpx
import pytest
from omnigent_client import OmnigentClient
from omnigent_client._events import (
MessageDone,
ResponseCompleted,
ResponseFailed,
ResponseIncomplete,
)
from omnigent_client.tools import build_tool_handler, tool
from tests.e2e.conftest import upload_agent
_FIXTURES_DIR = Path(__file__).resolve().parents[1] / "_fixtures" / "agents"
_FIXTURE = _FIXTURES_DIR / "d6-direct-cancel-test"
# The @tool body sleeps this long. Test must wall-clock
# complete in well under this — proving cancellation short-
# circuited the sleep.
_BODY_SLEEP_S = 30
# Upper bound on total wall-clock. LLM round-trip (~3s) +
# tool dispatch + sys_cancel_task call + PATCH round-trip + LLM
# final response (~3s) ≈ 10s comfortable ceiling. Anything
# approaching _BODY_SLEEP_S means the body wasn't cancelled
# and the sleep ran to completion.
_MAX_WALL_CLOCK_S = 15.0
@pytest.fixture(scope="session")
def direct_cancel_test_agent(http_client: httpx.Client) -> str:
"""Upload the d6-direct-cancel-test fixture."""
return upload_agent(http_client, _FIXTURE)
# Cancellation fingerprint. The body appends one of three
# outcomes per invocation so the assertions can tell:
#
# - "completed_normally" — body's sleep ran to end (bad).
# - "cancelled_mid_sleep" — asyncio.CancelledError raised
# during the sleep (good — SDK cancelled the body).
# - "other_exception:<repr>" — something else went wrong.
#
# Module-level rather than fixture-scoped so the @tool fn
# can append without argument plumbing.
_body_outcomes: list[str] = []
@tool
async def slow_compute(seconds: int) -> str:
"""Sleep for ``seconds`` seconds, then return a marker.
The body is async so ``asyncio.sleep`` (not ``time.sleep``)
is the wait primitive — this is what makes cancellation
observable: ``asyncio.sleep`` raises CancelledError the
moment its surrounding Task is cancelled. A blocking
``time.sleep`` would finish its full duration no matter
what the enclosing Task tried to do.
Args:
seconds: Sleep duration. The agent's AGENTS.md tells
it to always pass 30, long enough that the test
can observe cancellation without racing.
"""
try:
await asyncio.sleep(seconds)
_body_outcomes.append("completed_normally")
return f"slept-{seconds}"
except asyncio.CancelledError:
_body_outcomes.append("cancelled_mid_sleep")
raise
except BaseException as exc:
_body_outcomes.append(f"other_exception:{exc!r}")
raise
@pytest.mark.asyncio
async def test_sdk_cancels_local_body_on_llm_cancel_task(
live_server: str,
direct_cancel_test_agent: str,
) -> None:
"""
When the LLM calls ``sys_cancel_task`` on an in-flight async
client tool, the SDK must cancel the local ``asyncio.Task``
running the tool body — the body's ``except
CancelledError`` branch fires, and the body never returns
normally.
Failure modes this test catches:
- ``SysCancelTaskTool`` doesn't emit
``response.client_task.cancel`` — the SDK never sees
the cancel signal, the body runs to full duration
(30 s) and total wall-clock trips ``_MAX_WALL_CLOCK_S``.
- SDK's ``stream()`` receives ``ClientTaskCancel`` but
doesn't look up the right local task by ``call_id`` —
body uncancelled, same symptom.
- ``local_task.cancel()`` fires but ``asyncio`` can't find
a cancellation point (e.g. body uses ``time.sleep``) —
this test specifically uses ``asyncio.sleep`` so the
cancellation is observable; fan-out covers the
blocking-body case separately.
"""
_body_outcomes.clear()
handler = build_tool_handler([slow_compute])
start = time.monotonic()
async with OmnigentClient(base_url=live_server) as client:
terminal_status: str | None = None
final_text_chunks: list[str] = []
async for event in client.responses.stream(
model=direct_cancel_test_agent,
input=(
"Run the slow_compute+sys_cancel_task sequence from your "
"instructions. Don't skip sys_cancel_task, and don't wait "
"for slow_compute to finish."
),
tool_handler=handler,
):
if isinstance(event, MessageDone):
for block in event.content:
if isinstance(block, dict) and block.get("type") == "output_text":
text = block.get("text") or ""
if isinstance(text, str):
final_text_chunks.append(text)
elif isinstance(event, ResponseCompleted):
terminal_status = "completed"
elif isinstance(event, ResponseFailed):
terminal_status = "failed"
err = event.response.error
if err is not None:
final_text_chunks.append(f"[FAILED] {err!r}")
elif isinstance(event, ResponseIncomplete):
terminal_status = "incomplete"
elapsed = time.monotonic() - start
assert terminal_status == "completed", (
f"Direct-cancel flow should terminate cleanly; got "
f"terminal_status={terminal_status!r}. "
f"final_text_chunks={final_text_chunks!r}"
)
# Load-bearing: the body did NOT complete normally. If the
# SDK had ignored the cancellation, the asyncio.sleep(30)
# would have run to end and the outcome would be
# "completed_normally".
assert "completed_normally" not in _body_outcomes, (
f"Body completed its full {_BODY_SLEEP_S}s sleep — SDK "
f"did not propagate the LLM's sys_cancel_task to the local "
f"asyncio.Task. Outcomes: {_body_outcomes!r}"
)
# At least one cancellation observed. Could be more than
# one if the LLM retries (shouldn't happen with our
# AGENTS.md but guard against it).
cancelled_count = _body_outcomes.count("cancelled_mid_sleep")
assert cancelled_count >= 1, (
f"Expected at least one slow_compute invocation to raise "
f"asyncio.CancelledError during its sleep; got {_body_outcomes!r}. "
f"If the body raised some other exception, the SDK's "
f"cancel path is firing something unexpected."
)
# End-to-end wall-clock. The body's sleep is 30s; if
# cancellation worked, total should be well under that.
assert elapsed < _MAX_WALL_CLOCK_S, (
f"Total stream duration {elapsed:.1f}s exceeds the "
f"ceiling {_MAX_WALL_CLOCK_S}s. The body's sleep is "
f"{_BODY_SLEEP_S}s; anything approaching that means "
f"cancellation didn't fire and the sleep ran to the end."
)
-184
View File
@@ -1,184 +0,0 @@
"""
E2E for the SDK-side async client-tool lifecycle dispatched
via ``sys_call_async``.
Proves the python-client SDK drives the full async path
end-to-end without any caller bookkeeping:
1. SDK exposes ``@tool``-decorated client tools on the wire.
2. Real LLM dispatches one via
``sys_call_async(tool=..., args=...)``.
3. Server's :meth:`SysCallAsyncTool.dispatch_async` creates a
``kind="client_tool"`` task, registers a pending_tool_call
keyed to a synthesized call_id, starts
``client_tool_workflow`` parked on
``CLIENT_TOOL_RESULT_TOPIC``, and synthesizes a
``function_call(action_required)`` SSE event.
4. SDK's action_required handler fires
``_execute_and_patch`` to run the tool body locally and
PATCH ``tool_results`` back when it completes.
5. Server's PATCH handler completes the pending row and
bridges to ``CLIENT_TOOL_RESULT_TOPIC``; the holder
workflow wakes and sends ``async_work_complete`` to the
parent.
6. Parent's drain renders ``[System: task X (client_tool)
completed]\\n<body>`` as a user message.
7. LLM reads the system message and replies ``ANSWER:<body>``.
8. Test asserts the body text round-tripped through both the
tool and the drain.
Excluded from default ``pytest`` runs via
``--ignore=tests/e2e``. Invoke with::
pytest tests/e2e/test_d6_sdk_async_dispatch_e2e.py \\
--llm-api-key "$(cat /tmp/mykey)" -v
"""
from __future__ import annotations
import asyncio
from pathlib import Path
import httpx
import pytest
from omnigent_client import OmnigentClient
from omnigent_client._events import (
MessageDone,
ResponseCompleted,
ResponseFailed,
ResponseIncomplete,
TextDelta,
)
from omnigent_client.tools import build_tool_handler, tool
from tests.e2e.conftest import upload_agent
_FIXTURES_DIR = Path(__file__).resolve().parents[1] / "_fixtures" / "agents"
_FIXTURE = _FIXTURES_DIR / "d6-sdk-async-dispatch-test"
# Marker the @tool body returns. The agent's AGENTS.md instructs
# the LLM to echo the marker back via ``ANSWER:<body>`` after
# the drain delivers it as a system message — finding the
# marker in the LLM's final assistant text proves the entire
# loop closed: SDK dispatch → server drain → LLM reads system
# message → SDK PATCH → drain delivery.
_MARKER = "D6_SDK_ASYNC_LIFECYCLE_OK_77"
@pytest.fixture(scope="session")
def d6_test_agent(http_client: httpx.Client) -> str:
"""Upload the D6 E2E fixture."""
return upload_agent(http_client, _FIXTURE)
# Tool body — the SDK runs this in an asyncio.Task when the
# server's :meth:`SysCallAsyncTool.dispatch_async` synthesizes
# a ``function_call(action_required)`` SSE event for it.
# Returns ``value`` verbatim so the LLM's ``ANSWER:<body>`` can
# be matched against the input marker.
@tool
async def compute(value: str) -> str:
"""Echo the input string back asynchronously.
Args:
value: Marker to echo. Test asserts this is what the
LLM ultimately replies with.
"""
# Tiny await so the body is visibly async (not just sync
# masquerading) — proves the asyncio.Task path is what
# ran, not an inline-execute fallback.
await asyncio.sleep(0.05)
return value
@pytest.mark.asyncio
async def test_sdk_async_client_tool_completes_round_trip(
live_server: str,
d6_test_agent: str,
) -> None:
"""
Full SDK-driven async-client-tool lifecycle, end-to-end.
Failure modes this test catches:
- LLM doesn't dispatch via ``sys_call_async`` (e.g. calls
``compute`` directly) → call lands in
``pending_client_calls`` and runs synchronously, the
test deadlocks waiting for the system message that the
sync path never produces.
- Server's :func:`_dispatch_client_tool_async` doesn't
register the pending_tool_call → the SDK PATCHes
``tool_results`` and the server returns 404 → the
holder workflow runs out the 1h cap (test would hit
its own timeout first).
- PATCH-handler bridge to ``CLIENT_TOOL_RESULT_TOPIC``
regresses → ``client_tool_workflow`` never wakes →
drain never delivers ``completed`` → LLM has nothing
to ANSWER with.
- Server-side audit-fix-#1 routing regresses → drain
message lands on the wrong agent → LLM never sees it.
"""
handler = build_tool_handler([compute])
async with OmnigentClient(base_url=live_server) as client:
# Drive the stream to completion. Collect events so the
# test can assert on terminal status + the assistant
# message body that contains the marker.
final_text_chunks: list[str] = []
terminal_status: str | None = None
failure_diag: str | None = None
async for event in client.responses.stream(
model=d6_test_agent,
input=f"Compute on the value {_MARKER!r}.",
tool_handler=handler,
):
if isinstance(event, TextDelta):
# Server streams assistant text incrementally as
# output_text deltas; the corresponding
# MessageDone fires with empty content (the
# OpenAI streaming convention). Accumulate
# deltas to capture the LLM's actual output.
final_text_chunks.append(event.delta)
elif isinstance(event, MessageDone):
# Empty under the OpenAI streaming convention,
# but harmless to also catch any non-streamed
# content blocks for forward-compat.
for block in event.content:
if isinstance(block, dict) and block.get("type") == "output_text":
text = block.get("text") or ""
if isinstance(text, str) and text:
final_text_chunks.append(text)
elif isinstance(event, ResponseCompleted):
terminal_status = "completed"
elif isinstance(event, ResponseFailed):
terminal_status = "failed"
err = event.response.error
failure_diag = repr(err)[:600] if err is not None else "no error info"
elif isinstance(event, ResponseIncomplete):
terminal_status = "incomplete"
assert terminal_status == "completed", (
f"D6 lifecycle should complete cleanly; got "
f"terminal_status={terminal_status!r}, "
f"failure_diag={failure_diag!r}, "
f"final_text_chunks={final_text_chunks!r}"
)
# The LLM's reply should contain the marker (per the agent
# AGENTS.md instructions: ANSWER:<body> where <body> is the
# system message body, which is the tool's return value,
# which is the marker). Streamed deltas are concatenated
# without separators — they're token-level fragments of one
# continuous text, not separate messages.
joined = "".join(final_text_chunks)
assert _MARKER in joined, (
f"D6 lifecycle round-trip failed: marker {_MARKER!r} not "
f"found in any assistant message text. "
f"final_text_chunks={final_text_chunks!r}. "
f"\nIf the test hangs / times out instead of failing here, "
f"the SDK probably didn't fire ``_execute_and_patch`` for "
f"the synthesized action_required event — see the "
f"action_required branch in omnigent_client/_responses.py "
f"and ``_dispatch_client_tool_async`` server-side."
)
-12
View File
@@ -89,18 +89,6 @@ skips:
issue: 532
cluster: files-uploads-attachments
mode: skip
- id: tests/e2e/test_d6_direct_cancel_e2e.py::test_sdk_cancels_local_body_on_llm_cancel_task
reason: "Server raises in _build_terminal_event during cancel flow; final response surfaces as 'failed' with 'Failed to retrieve final response'. Real server-side bug in cancel→terminal handling."
issue: 532
cluster: terminal-d6
mode: xfail
- id: tests/e2e/test_d6_sdk_async_dispatch_e2e.py::test_sdk_async_client_tool_completes_round_trip
reason: "Same 'Failed to retrieve final response' server bug as test_sdk_cancels_local_body_on_llm_cancel_task; _build_terminal_event raises during D6 async-dispatch round-trip."
issue: 532
cluster: terminal-d6
mode: xfail
- id: tests/e2e/test_repl_approval_e2e.py::test_repl_refusal_shows_deny_sentinel
reason: "Pexpect timeout waiting for 'approval required' regex. Same REPL-pexpect timeout family as the earlier test_repl_* entries."
issue: 532