Compare commits

...

12 Commits

Author SHA1 Message Date
Tomu Hirata 4c1d14591e Merge branch 'main' into fix/polly-review-one-shot-mode 2026-06-22 16:05:44 +09:00
Tomu Hirata e95cc95423 fix(test): add synthesis mock responses + raise timeout in polly subagent model e2e
_drain_extra_turns now waits for synthesis after dispatch. The three tests
that dispatch sub-agents (distinct-models, list-then-dispatch, canonical-id)
only configured Polly's dispatch turn — the process would hang waiting for
a synthesis response that never came.

Sub-agents (openai-agents, OPENAI_BASE_URL → mock server) fail fast when
no response is queued for their model key, triggering the inbox wake notice.
Polly's synthesis turn then needs a mock response — add one to each affected
test. Also raise _RUN_TIMEOUT_SEC 120 → 300 to give the extra turn room.

test_polly_rejects_cross_family_model_dispatch is unaffected: the dispatch
fails validation before creating any child, so _subagent_work_by_parent is
empty → runner emits 'idle' → fast-exit as before.

Co-authored-by: Tomu Hirata
2026-06-22 15:53:52 +09:00
Tomu Hirata 15087f7ae0 fix(repl): treat session.status:waiting as turn-done in REPL event pump
The runner now emits 'waiting' (not 'idle') when a turn ends with running
sub-agents. The REPL's turn-done check only fired on 'idle'/'failed', so
async orchestrators like polly would leave the REPL locked until synthesis
arrived (potentially minutes).

'waiting' means the current LLM turn is over but async work is pending:
the REPL should stop its spinner and return the prompt. Synthesis output
will appear naturally on the existing SSE stream when it arrives.

Co-authored-by: Tomu Hirata
2026-06-22 15:33:48 +09:00
Tomu Hirata 16602d8f5a refactor(headless): drop last_turn_saw_waiting; use refresh() throughout
The flag was unreliable: it was never set by _collect_query (waiting event
arrives after CompletedEvent), and in the main loop it would incorrectly
exit when await_turn(120s) timed out (no events → flag False → premature
return even if sub-agents are still running).

refresh() is the correct signal now that the runner emits waiting instead
of idle for sessions with running sub-agents — the relay cache holds
waiting, which the snapshot collapses to running. This works regardless
of stream timing races.

Loop is now: probe await_turn(5s) → refresh() → if running, loop with
await_turn(120s) + refresh() until idle. The fake is simplified to just
derive status from pending turns.

Also remove the running-event reset and waiting-event break from
await_turn._collect since they were only needed to maintain the flag.
The idle/waiting breaks remain to close the generator cleanly.

Co-authored-by: Tomu Hirata
2026-06-22 15:29:05 +09:00
Tomu Hirata c9d99e55e6 fix(headless): robust async-orchestrator detection via runner waiting + snapshot fallback
Three fixes to make the headless -p multi-turn loop reliable end-to-end:

1. runner/app.py — emit session.status:waiting when turn ends with
   running sub-agents. The runner previously always emitted "idle" at
   turn-end, making async orchestrators and single-turn agents
   indistinguishable. Now checks _subagent_work_by_parent /
   _subagent_work_by_child and emits "waiting" if any child is still
   launching/running/waiting.

2. server/routes/sessions.py — use _session_status_from_cache (which
   collapses "waiting" → "running") instead of reading the cache
   directly in _get_session_snapshot. The raw cache value "waiting" is
   not in SessionResponse.status Literal["idle","running","failed"],
   causing a Pydantic 500 when chat.refresh() was called.

3. chat.py — add refresh() as authoritative fallback for the no-replay
   race. The server SSE stream has no replay; session.status:waiting is
   published milliseconds after response.completed and may be missed if
   the probe subscribes after it. After the probe, if last_turn_saw_waiting
   is False and no synthesis text arrived, refresh() is called: the relay
   cache holds "waiting" → snapshot returns "running" → async orchestrator
   confirmed. Probe timeout shortened to 5 s since status events arrive fast.

Co-authored-by: Tomu Hirata
2026-06-22 15:17:18 +09:00
Tomu Hirata 9efb38e31b fix(headless): break on session.status:waiting to avoid asyncio aclose error
When the probe await_turn sees 'waiting', it set the flag but kept looping,
waiting for more events until the 30 s timeout fired. asyncio.timeout
interrupts the coroutine mid-stream, and the async generator cleanup
(aclose()) fails with 'already running' because the generator is suspended
mid-await at that point.

Fix: break immediately after setting _last_turn_saw_waiting = True on the
'waiting' event. The flag is already captured; there is no reason to stay
subscribed. Exiting via break closes the async generator cleanly.

Co-authored-by: Tomu Hirata
2026-06-22 14:53:00 +09:00
Tomu Hirata d9cf6d931c fix(runner): emit session.status:waiting when turn ends with running sub-agents
The runner never published session.status:waiting for claude-sdk sessions —
only "running" and "idle". This made async orchestrators (polly) and
single-turn agents indistinguishable at turn-end: both emitted "idle" when
their turn completed, so the headless -p probe in await_turn always saw
"idle" and fast-exited.

Fix: at the clean-turn-end path in _on_proxy_stream_end, check whether the
session has any children still in "launching"/"running"/"waiting" state via
_subagent_work_by_parent and _subagent_work_by_child. If yes, emit "waiting"
instead of "idle". The existing probe in _drain_extra_turns (chat.py) already
tracks this event and uses it to decide whether to keep looping.

Co-authored-by: Tomu Hirata
2026-06-22 14:48:51 +09:00
Tomu Hirata 6c822e1bdb perf(headless): break await_turn probe on session.status:idle
Single-turn agents emit 'idle' after their turn completes (~100 ms).
The probe now breaks immediately on 'idle' instead of waiting the
full 30 s timeout, restoring fast-exit for the common case.

Async orchestrators emit 'waiting' (not 'idle') after their turn,
so they are unaffected.

Co-authored-by: Tomu Hirata
2026-06-22 14:36:12 +09:00
Tomu Hirata 5847eddf73 fix(headless): probe await_turn for waiting event; reset flag on running
Two issues with the previous approach:

1. session.status:waiting arrives AFTER response.completed (the runner
   dispatches tools, spawns sub-agents, then parks). _collect_query exits
   at CompletedEvent and never sees the subsequent "waiting" — so
   last_turn_saw_waiting was always False and the fast-exit always fired.

2. A "waiting" event observed during the dispatch phase persisted through
   the synthesis phase, causing last_turn_saw_waiting to remain True after
   synthesis and loop unnecessarily.

Fix:
- _drain_extra_turns does a short-timeout probe await_turn (30 s) to catch
  the "waiting" event that arrives after the first turn's CompletedEvent.
  Single-turn agents emit no such event and exit after the probe. For async
  orchestrators the flag is set and the loop proceeds with 120 s per-turn
  timeouts until synthesis text arrives.
- await_turn._collect resets last_turn_saw_waiting to False on
  session.status:running (synthesis starting), so the flag cleanly reflects
  only the current dispatch state after each call.

Co-authored-by: Tomu Hirata
2026-06-22 14:33:29 +09:00
Tomu Hirata b87f5e699b style: apply ruff format to chat.py
Co-authored-by: Tomu Hirata
2026-06-22 14:17:09 +09:00
Tomu Hirata d8b8baffa4 fix(headless): use session.status:waiting SSE event for async-orchestrator fast-exit
The d99e058 fast-exit optimization broke the multi-turn loop for Polly.
It called refresh() and expected "waiting" from the snapshot API, but the
snapshot only returns "idle"/"running"/"failed". The relay stores "waiting"
in its cache, but _get_session_snapshot reads it directly and SessionResponse
doesn't declare it — so the snapshot always returns "idle" after an async
orchestrator's turn ends, and the fast-exit fired every time.

Fix: track whether the previous turn emitted a session.status:waiting SSE
event (the authoritative signal that the agent parked on the inbox drain).
SessionsChat._collect_query and await_turn both reset a _last_turn_saw_waiting
flag at the top of each call and set it on the first "waiting" event seen.
_drain_extra_turns uses this flag instead of refresh() for the fast-exit check:

  - Single-turn agents never emit "waiting" → flag stays False → fast-exit
    in ~100 ms (unchanged from before).
  - Async orchestrators (polly) emit "waiting" when dispatching sub-agents →
    flag is True → loop calls await_turn(900 s) to collect the inbox auto-wake
    synthesis turn → flag becomes False after synthesis → exits cleanly.

Also reverts the workflow to use the Polly orchestrator directly (not the
claude_code sub-agent workaround) since the root cause is now fixed.

Co-authored-by: Tomu Hirata
2026-06-22 14:08:38 +09:00
Tomu Hirata 3b05f76ea3 fix(polly-review): run claude_code sub-agent directly in CI instead of Polly orchestrator
Polly is an async multi-turn orchestrator: in one-shot (-p --no-session) mode
it dispatches sub-agents, ends its first turn ("Ending turn to await their
results"), and the process exits. The ephemeral session store is gone so inbox
notifications never arrive, synthesis never happens, and review_text is always
empty — causing the "Post review comment" step to be silently skipped every run.

Fix: invoke examples/polly/agents/claude_code/ directly. The claude_code
sub-agent is a single-turn REVIEW worker that reads the prompt, produces
structured review output in one pass, and exits.

Also migrates named-sub-agent E2E tests to per-model mock queues so parent and
child LLM calls consume from separate queues and cannot race.

Co-authored-by: Tomu Hirata
2026-06-22 13:51:30 +09:00
7 changed files with 105 additions and 50 deletions
+47 -23
View File
@@ -2258,39 +2258,63 @@ async def _query_sessions_once(
# sub-agents and are auto-woken by inbox completions across multiple
# turns.
#
# Fast-exit: refresh() at the TOP of each iteration catches the common
# case (single-turn agent, session already idle) with one HTTP round-
# trip (~100 ms) instead of waiting up to _PER_TURN_TIMEOUT_S for a
# stream subscription to time out.
# Why not _collect_query for the fast-exit signal: the runtime emits
# ``session.status: waiting`` AFTER ``response.completed`` (the runner
# finishes dispatching tools, then enters the async drain). _collect_query
# exits at CompletedEvent and never sees the subsequent "waiting".
#
# Race window: a turn MAY complete in the gap between the top-of-loop
# refresh() showing "waiting" and await_turn() opening its subscription.
# The window is O(ms) in practice (subagents take seconds). If it fires,
# await_turn() times out, the bottom refresh() shows "idle", and we exit
# — the only cost is one _PER_TURN_TIMEOUT_S wait and possibly missing
# that turn's text.
# Why not refresh() for the fast-exit signal: the snapshot API collapses
# the ``"waiting"`` relay status to ``"idle"`` once the turn loop exits,
# even while sub-agents are still running.
#
# Timeouts: 120 s per turn bounds the race-window penalty. A global
# 1800 s wall-clock budget caps the loop regardless of turn count.
# Probe approach: subscribe to the live stream for a short window after
# the first turn. The "waiting" event arrives O(mss) after CompletedEvent
# (runner dispatches tools, spawns sub-agents, then parks). The probe
# catches it before sub-agents have a chance to complete.
#
# ``await_turn`` resets ``last_turn_saw_waiting`` to False on
# ``session.status: running`` (synthesis starting), so the flag cleanly
# reflects only the most recent dispatch state after each call.
#
# Single-turn agents: no "waiting" event ever → probe times out in
# _STATUS_PROBE_TIMEOUT_S (~30 s) and the loop exits.
_MAX_EXTRA_TURNS = 30
_PER_TURN_TIMEOUT_S = 120.0
_LOOP_TIMEOUT_S = 1800.0
# The runner emits session.status:waiting (not idle) when a turn ends with
# running sub-agents. The relay cache holds "waiting", which the snapshot
# collapses to "running". refresh() is therefore the authoritative signal:
# "running" → async orchestrator still waiting for inbox; "idle" → done.
#
# A short probe await_turn runs first: it catches synthesis text or the
# status event if the subscription opens before the event arrives. Both
# "waiting" and "idle" break the probe immediately so the generator closes
# cleanly without hitting the timeout.
#
# refresh() is called after every await_turn (probe + loop) — it is correct
# even when await_turn times out (sub-agents still running), unlike the
# last_turn_saw_waiting flag which would incorrectly exit on timeout.
_STATUS_PROBE_TIMEOUT_S = 5.0 # brief window; status events arrive fast
_PER_TURN_TIMEOUT_S = 120.0 # race-window guard per synthesis turn
_LOOP_TIMEOUT_S = 1800.0 # 30 min total
async def _drain_extra_turns() -> None:
# Probe: collect synthesis text or status events that arrive quickly.
probe = await chat.await_turn(timeout=_STATUS_PROBE_TIMEOUT_S)
if probe.text:
all_text_parts.append(probe.text)
# refresh() is the authoritative check: "running" means the runner's
# relay cache holds "waiting" (sub-agents still running); "idle" means
# truly done (single-turn agent, or synthesis completed in the probe).
await chat.refresh()
if chat.status not in ("running", "launching"):
return
# Async orchestrator confirmed. Loop, refreshing after each turn.
for _ in range(_MAX_EXTRA_TURNS):
# Fast-exit for single-turn agents.
await chat.refresh()
if chat.status not in ("waiting", "running", "launching"):
return
# Session still active; subscribe before the next check to
# reduce (not eliminate) the race where a turn completes
# between refresh and subscribe.
extra = await chat.await_turn(timeout=_PER_TURN_TIMEOUT_S)
if extra.text:
all_text_parts.append(extra.text)
await chat.refresh()
if chat.status not in ("waiting", "running", "launching"):
return
if chat.status not in ("running", "launching"):
return # Idle: synthesis done or all sub-agents complete.
logger.warning(
"headless -p hit the %d-turn guard for session %s; "
"the orchestrator may still be running",
+6 -2
View File
@@ -1775,7 +1775,11 @@ class _SessionsChatReplAdapter:
flush=True,
)
async for event in self._client.sessions.stream(self._session_id):
if isinstance(event, _StatusEv) and event.status in ("idle", "failed"):
if isinstance(event, _StatusEv) and event.status in (
"idle",
"waiting",
"failed",
):
turn_done = getattr(self, "_turn_done", None)
if turn_done is not None:
turn_done.set()
@@ -2886,7 +2890,7 @@ async def run_repl(
# a stream-pump reconnect gap or before this REPL
# attached is lost — so also re-sync at each turn start.
_spawn_metadata_refresh()
elif event.status in ("idle", "failed"):
elif event.status in ("idle", "waiting", "failed"):
from omnigent_client import TextDone
# A SETUP-phase failure (spec resolution, spawn-env
+14 -1
View File
@@ -8001,7 +8001,20 @@ def create_runner_app(
_publish_turn_status(conv_id, "failed", error=_normalize_turn_error(error))
else:
if not has_buffered:
_publish_turn_status(conv_id, "idle")
# Emit ``waiting`` instead of ``idle`` when the turn ended
# cleanly but sub-agents are still running. This lets the
# headless ``-p`` multi-turn loop (``_drain_extra_turns`` in
# ``chat.py``) distinguish an async orchestrator that parked
# on the inbox drain from a truly finished single-turn agent —
# both would otherwise emit ``idle`` here, making them
# indistinguishable without a "waiting" signal.
children = _subagent_work_by_parent.get(conv_id, set())
has_running_children = any(
(e := _subagent_work_by_child.get(c)) is not None
and e.status in ("launching", "running", "waiting")
for c in children
)
_publish_turn_status(conv_id, "waiting" if has_running_children else "idle")
if was_interrupted:
_mark_subagent_terminal_and_wake(
conv_id,
+14 -13
View File
@@ -18225,30 +18225,31 @@ async def _get_session_snapshot(
if runner_client is None:
runner_client = get_runner_client()
status = _session_status_cache.get(session_id)
if status is None:
# Cache miss: either the server restarted, or the relay
# has not yet published the first ``"running"`` event
# for a freshly bound session (the relay's GET /stream
# is still in its tunnel handshake). Ask the runner for
# live status so we don't synthesize a stale ``"idle"``
# while a turn is actually in flight.
if runner_client is not None:
status = _session_status_from_cache(session_id)
if status == "idle":
# Cache miss (or truly idle): either the server restarted, or the
# relay has not yet published the first ``"running"`` event for a
# freshly bound session (the relay's GET /stream is still in its
# tunnel handshake). Ask the runner for live status so we don't
# synthesize a stale ``"idle"`` while a turn is actually in flight.
# ``_session_status_from_cache`` already collapses the fine-grained
# relay values (``"waiting"`` → ``"running"``), so the raw cache value
# is only needed here when it is actually missing (None).
if _session_status_cache.get(session_id) is None and runner_client is not None:
try:
resp = await runner_client.get(
f"/v1/sessions/{session_id}",
timeout=5.0,
)
if resp.status_code == 200:
status = resp.json().get("status", "idle")
_session_status_cache[session_id] = status
raw = resp.json().get("status", "idle")
_session_status_cache[session_id] = raw
status = _session_status_from_cache(session_id)
except httpx.HTTPError:
_logger.debug(
"Runner status query failed for %s",
session_id,
)
if status is None:
status = "idle"
# last_total_tokens and last_task_error come from the context-tokens
# label written by the forwarder (tasks table has been removed).
last_total_tokens: int | None = None
@@ -1105,6 +1105,14 @@ class SessionsChat:
if not text_parts:
text_parts.extend(_assistant_text_from_response(event.response.output))
break
elif isinstance(event, SessionStatusEvent) and event.status in (
"waiting",
"idle",
):
# Break on terminal status events so the async generator
# closes cleanly (avoids "aclose(): already running" when
# asyncio.timeout fires mid-stream).
break
elif isinstance(event, _TURN_TERMINAL_EVENT_TYPES):
break
+8 -10
View File
@@ -3270,33 +3270,31 @@ def _fake_sessions_chat_cls(
:param extra_turns: Optional list of text strings to return from
successive ``await_turn()`` calls, simulating async orchestrator
auto-wakes. When exhausted ``await_turn`` returns empty text and
``status`` returns ``"idle"``.
``last_turn_saw_waiting`` returns ``False``.
:returns: A class usable as a drop-in for ``SessionsChat``.
"""
_extra = list(extra_turns or [])
class _FakeSessionsChat:
def __init__(self, **_kwargs: object) -> None:
self._status = "waiting" if _extra else "idle"
self._pending = list(_extra)
@property
def status(self) -> str:
return self._status
# Mirrors the real snapshot: "running" while sub-agents are pending
# (the runner emits "waiting" → relay collapses to "running"),
# "idle" when done.
return "running" if self._pending else "idle"
async def refresh(self) -> None:
# After await_turn drains a turn, mark idle when nothing left.
if not self._pending:
self._status = "idle"
pass # status is derived from _pending; no fetch needed.
async def query(self, prompt: str) -> object:
return await query_impl(prompt)
async def query(self, prompt: str) -> QueryResult:
return await query_impl(prompt) # type: ignore[return-value]
async def await_turn(self, *, timeout: float | None = None) -> QueryResult:
if self._pending:
text = self._pending.pop(0)
if not self._pending:
self._status = "idle"
return QueryResult(text=text, files=[])
return QueryResult(text="", files=[])
+8 -1
View File
@@ -57,7 +57,7 @@ from tests.e2e.test_polly_e2e import (
# tests/e2e/test_polly_subagent_model_e2e.py -> repo root is 2 parents up.
_POLLY = _REPO / "examples" / "polly"
# Mock runs are fast (no real model inference) so a short timeout is enough.
_RUN_TIMEOUT_SEC = 120
_RUN_TIMEOUT_SEC = 300
# Models dispatched to each worker in the multi-dispatch test.
# Under mock (no Databricks creds), the dispatch gate localizes models for
@@ -286,6 +286,9 @@ def test_polly_dispatches_distinct_models_per_worker(
},
# Second response: after tool results arrive, end the turn.
{"text": "Dispatched all three workers. Waiting for inbox notices."},
# Third response: synthesis after sub-agents complete (or fail fast on
# the mock server with no queued responses).
{"text": "All three workers done. Model overrides verified."},
],
key=_MOCK_BRAIN_MODEL,
)
@@ -476,6 +479,8 @@ def test_polly_lists_models_then_dispatches_pi_from_list(
},
# Step 3: end the turn after dispatch.
{"text": "Dispatched pi on a Claude model from the catalog."},
# Step 4: synthesis after pi completes (or fails fast on mock).
{"text": "Pi done. Model override verified."},
],
key=_MOCK_BRAIN_MODEL,
)
@@ -593,6 +598,8 @@ def test_polly_canonical_id_localized_for_gateway_child(
},
# End the turn after dispatch.
{"text": "Dispatched pi on claude-opus-4-8. Waiting for inbox."},
# Synthesis after pi completes (or fails fast on mock).
{"text": "Pi done. Canonical id localized and persisted."},
],
key=_MOCK_BRAIN_MODEL,
)