Compare commits

...

2 Commits

Author SHA1 Message Date
harry-yao_data d33863faeb docs(benchmarks): stop native_hook_spawn claiming to be the per-chunk path
The journey's comment, docstring, README row and CLI description all say
it spawns "the per-chunk MessageDisplay hook exactly as Claude Code
does". That stopped being true when MessageDisplay moved to a /bin/sh
appender and evaluate-policy moved to a curl against the runner's relay.
Both are pinned by tests — test_message_display_shell_command_round_trips
asserts "python" is absent from the installed command — so the number the
journey reports (~40ms here) is not on any per-chunk or per-tool-call
path.

Left as it was, the number reads as ~40ms of blocked TUI per streamed
chunk, which would make it the largest single cost in the system and the
obvious thing to go fix. It isn't, and I went and measured a replacement
for an optimization the repo already has.

Say what it measures instead: the lifetime of a hook that is Python,
which is what the per-turn hooks (SessionStart / Stop / UserPromptSubmit
/ PreCompact / Task*), the PostToolUse TodoWrite+TaskUpdate matchers, and
the policy hook's pre-relay fallback still pay — and which is the
standing argument for keeping the hot paths off the interpreter. Naming
the tests that pin it points the next reader at the evidence rather than
at a stale comment.

No behaviour change; comments, docstring, description and README only.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: harry-yao_data <harry.yao@databricks.com>
2026-08-22 08:50:20 +00:00
Aravind Segu 5d7aa85132 fix(cli): forward non-uuid session ids on remote resume (#5218)
`omnigent resume <id>` canonicalized every id through the local sqlite store's uuid rule (uuid_to_bytes), even when --server points at a remote server that owns its own id space. A deployment that keys sessions on non-uuid ids (e.g. numeric node ids) had every id rejected client-side with "Invalid session id." before any request was sent.

Only the local path binds the id to the Uuid16 column, so keep the strict uuid guard there; on the remote path forward the id untouched and let the server resolve it, matching how the runner and SDK already pass the id straight through.

Signed-off-by: aravind-segu <aravind.segu@databricks.com>
Signed-off-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
2026-08-21 15:54:45 -07:00
4 changed files with 86 additions and 22 deletions
+14 -3
View File
@@ -72,11 +72,22 @@ they still work with no runner or LLM.
| Journey | Operation timed |
| --- | --- |
| `native_hook_spawn` | Spawn the per-chunk `MessageDisplay` hook exactly as Claude Code does — isolated interpreter, module entrypoint, JSON payload on stdin |
| `native_hook_spawn` | Spawn one **Python** command hook — isolated interpreter, module entrypoint, JSON payload on stdin — and time its whole lifetime |
Claude Code **blocks its TUI** on command hooks, so one hook subprocess's
lifetime is user-visible streaming latency, and the same interpreter+import
cost fronts every statusline refresh and per-tool-call policy hook. The
lifetime is user-visible latency. Read this number as *"what a hook costs if it
is Python"*.
It is **not** the per-chunk streaming cost, and treating it as one leads
straight to wasted work. The hooks that fire per chunk (`MessageDisplay`) and
per tool call (`evaluate-policy`) were deliberately moved off the interpreter —
a `/bin/sh` appender and a `curl` to the runner's relay — and
`test_message_display_shell_command_round_trips` pins that by asserting
`"python"` is absent from the installed command. What still pays this number is
the per-turn set (`SessionStart` / `Stop` / `UserPromptSubmit` / `PreCompact` /
`Task*`), the `PostToolUse` `TodoWrite`+`TaskUpdate` matchers, and the policy
hook's Python fallback before the relay is up. So the journey's real job is to
keep the argument for staying off the interpreter measurable. The
journey needs no server or runner; registering it here rides hook spawn cost
on the same nightly/release regression comparison as everything else
(`omnigent/__init__` re-exports lazily so this stays ~interpreter-sized). The
+22 -8
View File
@@ -881,12 +881,21 @@ async def _measure_cli_startup(env: BenchEnvironment, _ctx: JourneyContext) -> N
# ── native hook spawn (no server involved) ───────────────────
# Claude Code blocks its TUI on command hooks, so one hook subprocess's whole
# lifetime is user-visible latency: the MessageDisplay hook runs once per
# streamed text chunk, and the same interpreter+import cost fronts every
# statusline refresh and per-tool-call policy hook. Spawn the per-chunk hook
# exactly as Claude Code does — isolated interpreter, module entrypoint, JSON
# payload on stdin — and time the full process lifetime. The import-graph side
# of this guarantee is pinned by tests/test_claude_native_message_display_hook.
# lifetime is user-visible latency. This journey times that lifetime for a
# Python hook — isolated interpreter, module entrypoint, JSON payload on stdin —
# which is the cost of ANY hook the bridge installs as a `python -m` command.
#
# It is NOT the per-chunk streaming path. The hooks that fire per chunk
# (MessageDisplay) and per tool call (evaluate-policy) were deliberately moved
# off the interpreter — a /bin/sh appender and a curl to the runner's relay
# respectively — and tests pin that (`test_message_display_shell_command_round_trips`
# asserts "python" is absent from the installed command). What still pays this
# is the per-turn set (SessionStart / Stop / UserPromptSubmit / PreCompact /
# Task*), the PostToolUse TodoWrite+TaskUpdate matchers, and the policy hook's
# Python fallback when the relay is not yet up. So read this number as
# "what a hook costs if it is Python", and as the standing argument for keeping
# the hot paths off it — not as a per-chunk cost. The import-graph side is
# pinned by tests/test_claude_native_message_display_hook.py.
_HOOK_SPAWN_PAYLOAD = json.dumps(
{
"hook_event_name": "MessageDisplay",
@@ -905,7 +914,7 @@ async def _setup_hook_spawn(env: BenchEnvironment) -> JourneyContext:
async def _measure_hook_spawn(env: BenchEnvironment, ctx: JourneyContext) -> None:
"""Spawn the MessageDisplay hook once, as Claude Code does, and wait."""
"""Spawn one Python hook subprocess and wait, as Claude Code would."""
del env
proc = await asyncio.create_subprocess_exec(
sys.executable,
@@ -1082,7 +1091,12 @@ ALL_JOURNEYS: dict[str, Journey] = {
measure=_measure_hook_spawn,
setup=_setup_hook_spawn,
teardown=_teardown_hook_spawn,
description="Spawn the per-chunk MessageDisplay hook exactly as Claude Code does.",
description=(
"Spawn one Python command hook (isolated interpreter, module "
"entrypoint) and time its whole lifetime — what any `python -m` "
"hook costs Claude's blocked TUI. Not the per-chunk path: that "
"one is a /bin/sh appender."
),
),
Journey(
name="cli_startup",
+14 -11
View File
@@ -221,17 +221,20 @@ def _dispatch_by_runtime(
"""
from omnigent.db.db_models import InvalidUuidError, uuid_to_bytes
# Resolve the id the argument contains, then canonicalize to bare hex
# before any lookup: a paste drags punctuation along (trailing period,
# wrapping quotes or backticks), and none of it can ever be part of a
# valid id — so strip it and resume rather than erroring. A malformed
# id would otherwise surface as a raw StatementError traceback from
# the local store's Uuid16 bind, and downstream consumers key
# sessions on the bare spelling.
try:
target = uuid_to_bytes(target.strip(_PASTE_PUNCTUATION)).hex()
except InvalidUuidError as exc:
raise click.ClickException("Invalid session id.") from exc
# Paste punctuation (trailing period, wrapping quotes/backticks) is never
# part of an id, so strip it. Only the local path binds the id to the sqlite
# store's Uuid16 column, so it must be a real uuid — reject a malformed one
# loudly rather than surfacing a raw StatementError. The remote server owns
# its id space (a managed deployment keys sessions on non-uuid ids) and
# validates the id itself, so forward it untouched, like the runner and SDK.
stripped = target.strip(_PASTE_PUNCTUATION)
if server is None:
try:
target = uuid_to_bytes(stripped).hex()
except InvalidUuidError as exc:
raise click.ClickException("Invalid session id.") from exc
else:
target = stripped
if server is not None:
wrapper = _read_wrapper_label_remote(server=server, conv_id=target)
+36
View File
@@ -480,6 +480,42 @@ def test_dispatch_by_runtime_legacy_prefixed_id_canonicalized_to_bare_hex(
assert captured["session_id"] == "415c9954e2fe4b9276083a4d2c66f689"
def test_dispatch_by_runtime_remote_forwards_non_uuid_id(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""
A remote server owns its id space, so a non-uuid id (e.g. a managed
deployment's numeric node id) must reach the lookup and wrapper
unchanged. Forcing the local uuid rule on the remote path would
reject a valid id before the server — the one thing the server is
there to resolve — ever sees it.
"""
seen: dict[str, str] = {}
def _label(*, server: str, conv_id: str) -> str:
"""Record the id the remote lookup receives."""
seen["conv_id"] = conv_id
return "claude-code-native-ui"
monkeypatch.setattr(resume_dispatch, "_read_wrapper_label_remote", _label)
captured: dict[str, Any] = {}
def _capture(**kwargs: Any) -> None:
"""Record the kwargs ``run_claude_native`` was called with."""
captured.update(kwargs)
monkeypatch.setattr("omnigent.claude_native.run_claude_native", _capture)
resume_dispatch._dispatch_by_runtime(
target="2048200000527758",
server="https://example.com",
)
# The raw non-uuid id flows through untouched — not rejected, not reshaped.
assert seen["conv_id"] == "2048200000527758"
assert captured["session_id"] == "2048200000527758"
def test_dispatch_by_runtime_non_wrapper_local_raises_with_hint(
monkeypatch: pytest.MonkeyPatch,
) -> None: