Compare commits
2 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 943fe2c4ae | |||
| bd5444971d |
@@ -1,31 +0,0 @@
|
||||
name: d6-direct-cancel-test
|
||||
description: |
|
||||
Test-only agent that dispatches one async client tool and
|
||||
then immediately cancels it.
|
||||
|
||||
executor:
|
||||
model: gpt-5.4
|
||||
|
||||
prompt: |
|
||||
# D6 direct-cancel E2E agent
|
||||
|
||||
Each turn, follow this exact sequence:
|
||||
|
||||
1. Dispatch the `slow_compute` tool via **`sys_call_async`**
|
||||
with `tool: "slow_compute"` and
|
||||
`args: "{\"seconds\": 30}"`.
|
||||
2. You will receive a `function_call_output` with a handle
|
||||
JSON: `{"task_id": "...", "tool_name": "slow_compute", ...}`.
|
||||
Extract the `task_id` value.
|
||||
3. In your **next** tool_call, call `sys_cancel_task` with that
|
||||
`task_id`. Do not wait for any system message; call
|
||||
sys_cancel_task immediately after you see the handle FCO.
|
||||
4. You will receive a `function_call_output` like
|
||||
`{"cancelled": true, "prior_status": "in_progress", ...}`
|
||||
and eventually a `[System: task <id> (client_tool)
|
||||
cancelled]` user message.
|
||||
5. Reply with the literal string `CANCELLED_OK` so the test
|
||||
can confirm you completed the sequence.
|
||||
|
||||
Do NOT skip sys_cancel_task. Do NOT wait for the slow_compute
|
||||
to finish before cancelling.
|
||||
@@ -1,33 +0,0 @@
|
||||
name: d6-sdk-async-dispatch-test
|
||||
description: |
|
||||
Test-only agent for the Phase 5 D6 SDK-lifecycle E2E.
|
||||
|
||||
executor:
|
||||
model: gpt-5.4
|
||||
|
||||
prompt: |
|
||||
# D6 SDK-async-dispatch E2E agent
|
||||
|
||||
You are an agent for an end-to-end test that exercises the
|
||||
Python SDK's async-client-tool lifecycle. Each turn:
|
||||
|
||||
1. Dispatch the `compute` tool via **`sys_call_async`** —
|
||||
pass `tool: "compute"` and
|
||||
`args: "{\"value\": \"<your-marker>\"}"`. This dispatches
|
||||
the work as a background task. You will receive a
|
||||
`function_call_output` with a handle JSON like
|
||||
`{"task_id": "...", "tool_name": "compute", "status":
|
||||
"in_progress", ...}`. **That is not the final result.**
|
||||
2. After dispatching, do not output text yet. Wait for the
|
||||
system message `[System: task ... completed]\n<BODY>` to
|
||||
arrive in the conversation.
|
||||
3. Once the system message arrives, reply to the user with
|
||||
the literal string `ANSWER:<BODY>` (everything after
|
||||
`ANSWER:` is the system message body verbatim) so the
|
||||
test can pull the result out.
|
||||
|
||||
## Dispatch via `sys_call_async`
|
||||
|
||||
The wrapped tool is `compute(value: str)`. The tool returns
|
||||
the value verbatim — your test marker is whatever string you
|
||||
sent in `value`.
|
||||
@@ -1,207 +0,0 @@
|
||||
"""
|
||||
E2E for D6 test plan #7: direct cancel propagates to the SDK.
|
||||
|
||||
Proves that when the LLM calls ``sys_cancel_task`` on a running
|
||||
async client_tool, the SDK's D6 lifecycle cancels the local
|
||||
``asyncio.Task`` running the tool body — the body's
|
||||
``except asyncio.CancelledError`` branch fires, and the body
|
||||
never returns normally.
|
||||
|
||||
Without the SDK-side SSE handling:
|
||||
- ``SysCancelTaskTool`` still emits ``response.client_task.cancel``
|
||||
(committed earlier),
|
||||
- The SDK's ``stream()`` sees the event and calls
|
||||
``state.asyncio_task.cancel()``,
|
||||
- The running body (a blocking ``time.sleep`` inside the
|
||||
asyncio task) would otherwise run to its full duration.
|
||||
|
||||
Without the fix this test would time out / take the full
|
||||
30 s sleep; with the fix the body's ``except`` fires within
|
||||
a couple seconds of the sys_cancel_task call.
|
||||
|
||||
Excluded from default ``pytest`` runs via
|
||||
``--ignore=tests/e2e``. Invoke with::
|
||||
|
||||
pytest tests/e2e/test_d6_direct_cancel_e2e.py \\
|
||||
--llm-api-key "$(cat /tmp/mykey)" -v
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
from omnigent_client import OmnigentClient
|
||||
from omnigent_client._events import (
|
||||
MessageDone,
|
||||
ResponseCompleted,
|
||||
ResponseFailed,
|
||||
ResponseIncomplete,
|
||||
)
|
||||
from omnigent_client.tools import build_tool_handler, tool
|
||||
|
||||
from tests.e2e.conftest import upload_agent
|
||||
|
||||
_FIXTURES_DIR = Path(__file__).resolve().parents[1] / "_fixtures" / "agents"
|
||||
_FIXTURE = _FIXTURES_DIR / "d6-direct-cancel-test"
|
||||
|
||||
# The @tool body sleeps this long. Test must wall-clock
|
||||
# complete in well under this — proving cancellation short-
|
||||
# circuited the sleep.
|
||||
_BODY_SLEEP_S = 30
|
||||
|
||||
# Upper bound on total wall-clock. LLM round-trip (~3s) +
|
||||
# tool dispatch + sys_cancel_task call + PATCH round-trip + LLM
|
||||
# final response (~3s) ≈ 10s comfortable ceiling. Anything
|
||||
# approaching _BODY_SLEEP_S means the body wasn't cancelled
|
||||
# and the sleep ran to completion.
|
||||
_MAX_WALL_CLOCK_S = 15.0
|
||||
|
||||
|
||||
@pytest.fixture(scope="session")
|
||||
def direct_cancel_test_agent(http_client: httpx.Client) -> str:
|
||||
"""Upload the d6-direct-cancel-test fixture."""
|
||||
return upload_agent(http_client, _FIXTURE)
|
||||
|
||||
|
||||
# Cancellation fingerprint. The body appends one of three
|
||||
# outcomes per invocation so the assertions can tell:
|
||||
#
|
||||
# - "completed_normally" — body's sleep ran to end (bad).
|
||||
# - "cancelled_mid_sleep" — asyncio.CancelledError raised
|
||||
# during the sleep (good — SDK cancelled the body).
|
||||
# - "other_exception:<repr>" — something else went wrong.
|
||||
#
|
||||
# Module-level rather than fixture-scoped so the @tool fn
|
||||
# can append without argument plumbing.
|
||||
_body_outcomes: list[str] = []
|
||||
|
||||
|
||||
@tool
|
||||
async def slow_compute(seconds: int) -> str:
|
||||
"""Sleep for ``seconds`` seconds, then return a marker.
|
||||
|
||||
The body is async so ``asyncio.sleep`` (not ``time.sleep``)
|
||||
is the wait primitive — this is what makes cancellation
|
||||
observable: ``asyncio.sleep`` raises CancelledError the
|
||||
moment its surrounding Task is cancelled. A blocking
|
||||
``time.sleep`` would finish its full duration no matter
|
||||
what the enclosing Task tried to do.
|
||||
|
||||
Args:
|
||||
seconds: Sleep duration. The agent's AGENTS.md tells
|
||||
it to always pass 30, long enough that the test
|
||||
can observe cancellation without racing.
|
||||
"""
|
||||
try:
|
||||
await asyncio.sleep(seconds)
|
||||
_body_outcomes.append("completed_normally")
|
||||
return f"slept-{seconds}"
|
||||
except asyncio.CancelledError:
|
||||
_body_outcomes.append("cancelled_mid_sleep")
|
||||
raise
|
||||
except BaseException as exc:
|
||||
_body_outcomes.append(f"other_exception:{exc!r}")
|
||||
raise
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_sdk_cancels_local_body_on_llm_cancel_task(
|
||||
live_server: str,
|
||||
direct_cancel_test_agent: str,
|
||||
) -> None:
|
||||
"""
|
||||
When the LLM calls ``sys_cancel_task`` on an in-flight async
|
||||
client tool, the SDK must cancel the local ``asyncio.Task``
|
||||
running the tool body — the body's ``except
|
||||
CancelledError`` branch fires, and the body never returns
|
||||
normally.
|
||||
|
||||
Failure modes this test catches:
|
||||
|
||||
- ``SysCancelTaskTool`` doesn't emit
|
||||
``response.client_task.cancel`` — the SDK never sees
|
||||
the cancel signal, the body runs to full duration
|
||||
(30 s) and total wall-clock trips ``_MAX_WALL_CLOCK_S``.
|
||||
- SDK's ``stream()`` receives ``ClientTaskCancel`` but
|
||||
doesn't look up the right local task by ``call_id`` —
|
||||
body uncancelled, same symptom.
|
||||
- ``local_task.cancel()`` fires but ``asyncio`` can't find
|
||||
a cancellation point (e.g. body uses ``time.sleep``) —
|
||||
this test specifically uses ``asyncio.sleep`` so the
|
||||
cancellation is observable; fan-out covers the
|
||||
blocking-body case separately.
|
||||
"""
|
||||
_body_outcomes.clear()
|
||||
handler = build_tool_handler([slow_compute])
|
||||
|
||||
start = time.monotonic()
|
||||
async with OmnigentClient(base_url=live_server) as client:
|
||||
terminal_status: str | None = None
|
||||
final_text_chunks: list[str] = []
|
||||
|
||||
async for event in client.responses.stream(
|
||||
model=direct_cancel_test_agent,
|
||||
input=(
|
||||
"Run the slow_compute+sys_cancel_task sequence from your "
|
||||
"instructions. Don't skip sys_cancel_task, and don't wait "
|
||||
"for slow_compute to finish."
|
||||
),
|
||||
tool_handler=handler,
|
||||
):
|
||||
if isinstance(event, MessageDone):
|
||||
for block in event.content:
|
||||
if isinstance(block, dict) and block.get("type") == "output_text":
|
||||
text = block.get("text") or ""
|
||||
if isinstance(text, str):
|
||||
final_text_chunks.append(text)
|
||||
elif isinstance(event, ResponseCompleted):
|
||||
terminal_status = "completed"
|
||||
elif isinstance(event, ResponseFailed):
|
||||
terminal_status = "failed"
|
||||
err = event.response.error
|
||||
if err is not None:
|
||||
final_text_chunks.append(f"[FAILED] {err!r}")
|
||||
elif isinstance(event, ResponseIncomplete):
|
||||
terminal_status = "incomplete"
|
||||
|
||||
elapsed = time.monotonic() - start
|
||||
|
||||
assert terminal_status == "completed", (
|
||||
f"Direct-cancel flow should terminate cleanly; got "
|
||||
f"terminal_status={terminal_status!r}. "
|
||||
f"final_text_chunks={final_text_chunks!r}"
|
||||
)
|
||||
|
||||
# Load-bearing: the body did NOT complete normally. If the
|
||||
# SDK had ignored the cancellation, the asyncio.sleep(30)
|
||||
# would have run to end and the outcome would be
|
||||
# "completed_normally".
|
||||
assert "completed_normally" not in _body_outcomes, (
|
||||
f"Body completed its full {_BODY_SLEEP_S}s sleep — SDK "
|
||||
f"did not propagate the LLM's sys_cancel_task to the local "
|
||||
f"asyncio.Task. Outcomes: {_body_outcomes!r}"
|
||||
)
|
||||
|
||||
# At least one cancellation observed. Could be more than
|
||||
# one if the LLM retries (shouldn't happen with our
|
||||
# AGENTS.md but guard against it).
|
||||
cancelled_count = _body_outcomes.count("cancelled_mid_sleep")
|
||||
assert cancelled_count >= 1, (
|
||||
f"Expected at least one slow_compute invocation to raise "
|
||||
f"asyncio.CancelledError during its sleep; got {_body_outcomes!r}. "
|
||||
f"If the body raised some other exception, the SDK's "
|
||||
f"cancel path is firing something unexpected."
|
||||
)
|
||||
|
||||
# End-to-end wall-clock. The body's sleep is 30s; if
|
||||
# cancellation worked, total should be well under that.
|
||||
assert elapsed < _MAX_WALL_CLOCK_S, (
|
||||
f"Total stream duration {elapsed:.1f}s exceeds the "
|
||||
f"ceiling {_MAX_WALL_CLOCK_S}s. The body's sleep is "
|
||||
f"{_BODY_SLEEP_S}s; anything approaching that means "
|
||||
f"cancellation didn't fire and the sleep ran to the end."
|
||||
)
|
||||
@@ -1,184 +0,0 @@
|
||||
"""
|
||||
E2E for the SDK-side async client-tool lifecycle dispatched
|
||||
via ``sys_call_async``.
|
||||
|
||||
Proves the python-client SDK drives the full async path
|
||||
end-to-end without any caller bookkeeping:
|
||||
|
||||
1. SDK exposes ``@tool``-decorated client tools on the wire.
|
||||
2. Real LLM dispatches one via
|
||||
``sys_call_async(tool=..., args=...)``.
|
||||
3. Server's :meth:`SysCallAsyncTool.dispatch_async` creates a
|
||||
``kind="client_tool"`` task, registers a pending_tool_call
|
||||
keyed to a synthesized call_id, starts
|
||||
``client_tool_workflow`` parked on
|
||||
``CLIENT_TOOL_RESULT_TOPIC``, and synthesizes a
|
||||
``function_call(action_required)`` SSE event.
|
||||
4. SDK's action_required handler fires
|
||||
``_execute_and_patch`` to run the tool body locally and
|
||||
PATCH ``tool_results`` back when it completes.
|
||||
5. Server's PATCH handler completes the pending row and
|
||||
bridges to ``CLIENT_TOOL_RESULT_TOPIC``; the holder
|
||||
workflow wakes and sends ``async_work_complete`` to the
|
||||
parent.
|
||||
6. Parent's drain renders ``[System: task X (client_tool)
|
||||
completed]\\n<body>`` as a user message.
|
||||
7. LLM reads the system message and replies ``ANSWER:<body>``.
|
||||
8. Test asserts the body text round-tripped through both the
|
||||
tool and the drain.
|
||||
|
||||
Excluded from default ``pytest`` runs via
|
||||
``--ignore=tests/e2e``. Invoke with::
|
||||
|
||||
pytest tests/e2e/test_d6_sdk_async_dispatch_e2e.py \\
|
||||
--llm-api-key "$(cat /tmp/mykey)" -v
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
from pathlib import Path
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
from omnigent_client import OmnigentClient
|
||||
from omnigent_client._events import (
|
||||
MessageDone,
|
||||
ResponseCompleted,
|
||||
ResponseFailed,
|
||||
ResponseIncomplete,
|
||||
TextDelta,
|
||||
)
|
||||
from omnigent_client.tools import build_tool_handler, tool
|
||||
|
||||
from tests.e2e.conftest import upload_agent
|
||||
|
||||
_FIXTURES_DIR = Path(__file__).resolve().parents[1] / "_fixtures" / "agents"
|
||||
_FIXTURE = _FIXTURES_DIR / "d6-sdk-async-dispatch-test"
|
||||
|
||||
# Marker the @tool body returns. The agent's AGENTS.md instructs
|
||||
# the LLM to echo the marker back via ``ANSWER:<body>`` after
|
||||
# the drain delivers it as a system message — finding the
|
||||
# marker in the LLM's final assistant text proves the entire
|
||||
# loop closed: SDK dispatch → server drain → LLM reads system
|
||||
# message → SDK PATCH → drain delivery.
|
||||
_MARKER = "D6_SDK_ASYNC_LIFECYCLE_OK_77"
|
||||
|
||||
|
||||
@pytest.fixture(scope="session")
|
||||
def d6_test_agent(http_client: httpx.Client) -> str:
|
||||
"""Upload the D6 E2E fixture."""
|
||||
return upload_agent(http_client, _FIXTURE)
|
||||
|
||||
|
||||
# Tool body — the SDK runs this in an asyncio.Task when the
|
||||
# server's :meth:`SysCallAsyncTool.dispatch_async` synthesizes
|
||||
# a ``function_call(action_required)`` SSE event for it.
|
||||
# Returns ``value`` verbatim so the LLM's ``ANSWER:<body>`` can
|
||||
# be matched against the input marker.
|
||||
@tool
|
||||
async def compute(value: str) -> str:
|
||||
"""Echo the input string back asynchronously.
|
||||
|
||||
Args:
|
||||
value: Marker to echo. Test asserts this is what the
|
||||
LLM ultimately replies with.
|
||||
"""
|
||||
# Tiny await so the body is visibly async (not just sync
|
||||
# masquerading) — proves the asyncio.Task path is what
|
||||
# ran, not an inline-execute fallback.
|
||||
await asyncio.sleep(0.05)
|
||||
return value
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_sdk_async_client_tool_completes_round_trip(
|
||||
live_server: str,
|
||||
d6_test_agent: str,
|
||||
) -> None:
|
||||
"""
|
||||
Full SDK-driven async-client-tool lifecycle, end-to-end.
|
||||
|
||||
Failure modes this test catches:
|
||||
|
||||
- LLM doesn't dispatch via ``sys_call_async`` (e.g. calls
|
||||
``compute`` directly) → call lands in
|
||||
``pending_client_calls`` and runs synchronously, the
|
||||
test deadlocks waiting for the system message that the
|
||||
sync path never produces.
|
||||
- Server's :func:`_dispatch_client_tool_async` doesn't
|
||||
register the pending_tool_call → the SDK PATCHes
|
||||
``tool_results`` and the server returns 404 → the
|
||||
holder workflow runs out the 1h cap (test would hit
|
||||
its own timeout first).
|
||||
- PATCH-handler bridge to ``CLIENT_TOOL_RESULT_TOPIC``
|
||||
regresses → ``client_tool_workflow`` never wakes →
|
||||
drain never delivers ``completed`` → LLM has nothing
|
||||
to ANSWER with.
|
||||
- Server-side audit-fix-#1 routing regresses → drain
|
||||
message lands on the wrong agent → LLM never sees it.
|
||||
"""
|
||||
handler = build_tool_handler([compute])
|
||||
|
||||
async with OmnigentClient(base_url=live_server) as client:
|
||||
# Drive the stream to completion. Collect events so the
|
||||
# test can assert on terminal status + the assistant
|
||||
# message body that contains the marker.
|
||||
final_text_chunks: list[str] = []
|
||||
terminal_status: str | None = None
|
||||
failure_diag: str | None = None
|
||||
|
||||
async for event in client.responses.stream(
|
||||
model=d6_test_agent,
|
||||
input=f"Compute on the value {_MARKER!r}.",
|
||||
tool_handler=handler,
|
||||
):
|
||||
if isinstance(event, TextDelta):
|
||||
# Server streams assistant text incrementally as
|
||||
# output_text deltas; the corresponding
|
||||
# MessageDone fires with empty content (the
|
||||
# OpenAI streaming convention). Accumulate
|
||||
# deltas to capture the LLM's actual output.
|
||||
final_text_chunks.append(event.delta)
|
||||
elif isinstance(event, MessageDone):
|
||||
# Empty under the OpenAI streaming convention,
|
||||
# but harmless to also catch any non-streamed
|
||||
# content blocks for forward-compat.
|
||||
for block in event.content:
|
||||
if isinstance(block, dict) and block.get("type") == "output_text":
|
||||
text = block.get("text") or ""
|
||||
if isinstance(text, str) and text:
|
||||
final_text_chunks.append(text)
|
||||
elif isinstance(event, ResponseCompleted):
|
||||
terminal_status = "completed"
|
||||
elif isinstance(event, ResponseFailed):
|
||||
terminal_status = "failed"
|
||||
err = event.response.error
|
||||
failure_diag = repr(err)[:600] if err is not None else "no error info"
|
||||
elif isinstance(event, ResponseIncomplete):
|
||||
terminal_status = "incomplete"
|
||||
|
||||
assert terminal_status == "completed", (
|
||||
f"D6 lifecycle should complete cleanly; got "
|
||||
f"terminal_status={terminal_status!r}, "
|
||||
f"failure_diag={failure_diag!r}, "
|
||||
f"final_text_chunks={final_text_chunks!r}"
|
||||
)
|
||||
|
||||
# The LLM's reply should contain the marker (per the agent
|
||||
# AGENTS.md instructions: ANSWER:<body> where <body> is the
|
||||
# system message body, which is the tool's return value,
|
||||
# which is the marker). Streamed deltas are concatenated
|
||||
# without separators — they're token-level fragments of one
|
||||
# continuous text, not separate messages.
|
||||
joined = "".join(final_text_chunks)
|
||||
assert _MARKER in joined, (
|
||||
f"D6 lifecycle round-trip failed: marker {_MARKER!r} not "
|
||||
f"found in any assistant message text. "
|
||||
f"final_text_chunks={final_text_chunks!r}. "
|
||||
f"\nIf the test hangs / times out instead of failing here, "
|
||||
f"the SDK probably didn't fire ``_execute_and_patch`` for "
|
||||
f"the synthesized action_required event — see the "
|
||||
f"action_required branch in omnigent_client/_responses.py "
|
||||
f"and ``_dispatch_client_tool_async`` server-side."
|
||||
)
|
||||
@@ -89,18 +89,6 @@ skips:
|
||||
issue: 532
|
||||
cluster: files-uploads-attachments
|
||||
mode: skip
|
||||
- id: tests/e2e/test_d6_direct_cancel_e2e.py::test_sdk_cancels_local_body_on_llm_cancel_task
|
||||
reason: "Server raises in _build_terminal_event during cancel flow; final response surfaces as 'failed' with 'Failed to retrieve final response'. Real server-side bug in cancel→terminal handling."
|
||||
issue: 532
|
||||
cluster: terminal-d6
|
||||
mode: xfail
|
||||
|
||||
- id: tests/e2e/test_d6_sdk_async_dispatch_e2e.py::test_sdk_async_client_tool_completes_round_trip
|
||||
reason: "Same 'Failed to retrieve final response' server bug as test_sdk_cancels_local_body_on_llm_cancel_task; _build_terminal_event raises during D6 async-dispatch round-trip."
|
||||
issue: 532
|
||||
cluster: terminal-d6
|
||||
mode: xfail
|
||||
|
||||
- id: tests/e2e/test_repl_approval_e2e.py::test_repl_refusal_shows_deny_sentinel
|
||||
reason: "Pexpect timeout waiting for 'approval required' regex. Same REPL-pexpect timeout family as the earlier test_repl_* entries."
|
||||
issue: 532
|
||||
|
||||
Reference in New Issue
Block a user