Compare commits

...

1 Commits

Author SHA1 Message Date
harry-yao_data d33863faeb docs(benchmarks): stop native_hook_spawn claiming to be the per-chunk path
The journey's comment, docstring, README row and CLI description all say
it spawns "the per-chunk MessageDisplay hook exactly as Claude Code
does". That stopped being true when MessageDisplay moved to a /bin/sh
appender and evaluate-policy moved to a curl against the runner's relay.
Both are pinned by tests — test_message_display_shell_command_round_trips
asserts "python" is absent from the installed command — so the number the
journey reports (~40ms here) is not on any per-chunk or per-tool-call
path.

Left as it was, the number reads as ~40ms of blocked TUI per streamed
chunk, which would make it the largest single cost in the system and the
obvious thing to go fix. It isn't, and I went and measured a replacement
for an optimization the repo already has.

Say what it measures instead: the lifetime of a hook that is Python,
which is what the per-turn hooks (SessionStart / Stop / UserPromptSubmit
/ PreCompact / Task*), the PostToolUse TodoWrite+TaskUpdate matchers, and
the policy hook's pre-relay fallback still pay — and which is the
standing argument for keeping the hot paths off the interpreter. Naming
the tests that pin it points the next reader at the evidence rather than
at a stale comment.

No behaviour change; comments, docstring, description and README only.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: harry-yao_data <harry.yao@databricks.com>
2026-08-22 08:50:20 +00:00
2 changed files with 36 additions and 11 deletions
+14 -3
View File
@@ -72,11 +72,22 @@ they still work with no runner or LLM.
| Journey | Operation timed |
| --- | --- |
| `native_hook_spawn` | Spawn the per-chunk `MessageDisplay` hook exactly as Claude Code does — isolated interpreter, module entrypoint, JSON payload on stdin |
| `native_hook_spawn` | Spawn one **Python** command hook — isolated interpreter, module entrypoint, JSON payload on stdin — and time its whole lifetime |
Claude Code **blocks its TUI** on command hooks, so one hook subprocess's
lifetime is user-visible streaming latency, and the same interpreter+import
cost fronts every statusline refresh and per-tool-call policy hook. The
lifetime is user-visible latency. Read this number as *"what a hook costs if it
is Python"*.
It is **not** the per-chunk streaming cost, and treating it as one leads
straight to wasted work. The hooks that fire per chunk (`MessageDisplay`) and
per tool call (`evaluate-policy`) were deliberately moved off the interpreter —
a `/bin/sh` appender and a `curl` to the runner's relay — and
`test_message_display_shell_command_round_trips` pins that by asserting
`"python"` is absent from the installed command. What still pays this number is
the per-turn set (`SessionStart` / `Stop` / `UserPromptSubmit` / `PreCompact` /
`Task*`), the `PostToolUse` `TodoWrite`+`TaskUpdate` matchers, and the policy
hook's Python fallback before the relay is up. So the journey's real job is to
keep the argument for staying off the interpreter measurable. The
journey needs no server or runner; registering it here rides hook spawn cost
on the same nightly/release regression comparison as everything else
(`omnigent/__init__` re-exports lazily so this stays ~interpreter-sized). The
+22 -8
View File
@@ -881,12 +881,21 @@ async def _measure_cli_startup(env: BenchEnvironment, _ctx: JourneyContext) -> N
# ── native hook spawn (no server involved) ───────────────────
# Claude Code blocks its TUI on command hooks, so one hook subprocess's whole
# lifetime is user-visible latency: the MessageDisplay hook runs once per
# streamed text chunk, and the same interpreter+import cost fronts every
# statusline refresh and per-tool-call policy hook. Spawn the per-chunk hook
# exactly as Claude Code does — isolated interpreter, module entrypoint, JSON
# payload on stdin — and time the full process lifetime. The import-graph side
# of this guarantee is pinned by tests/test_claude_native_message_display_hook.
# lifetime is user-visible latency. This journey times that lifetime for a
# Python hook — isolated interpreter, module entrypoint, JSON payload on stdin —
# which is the cost of ANY hook the bridge installs as a `python -m` command.
#
# It is NOT the per-chunk streaming path. The hooks that fire per chunk
# (MessageDisplay) and per tool call (evaluate-policy) were deliberately moved
# off the interpreter — a /bin/sh appender and a curl to the runner's relay
# respectively — and tests pin that (`test_message_display_shell_command_round_trips`
# asserts "python" is absent from the installed command). What still pays this
# is the per-turn set (SessionStart / Stop / UserPromptSubmit / PreCompact /
# Task*), the PostToolUse TodoWrite+TaskUpdate matchers, and the policy hook's
# Python fallback when the relay is not yet up. So read this number as
# "what a hook costs if it is Python", and as the standing argument for keeping
# the hot paths off it — not as a per-chunk cost. The import-graph side is
# pinned by tests/test_claude_native_message_display_hook.py.
_HOOK_SPAWN_PAYLOAD = json.dumps(
{
"hook_event_name": "MessageDisplay",
@@ -905,7 +914,7 @@ async def _setup_hook_spawn(env: BenchEnvironment) -> JourneyContext:
async def _measure_hook_spawn(env: BenchEnvironment, ctx: JourneyContext) -> None:
"""Spawn the MessageDisplay hook once, as Claude Code does, and wait."""
"""Spawn one Python hook subprocess and wait, as Claude Code would."""
del env
proc = await asyncio.create_subprocess_exec(
sys.executable,
@@ -1082,7 +1091,12 @@ ALL_JOURNEYS: dict[str, Journey] = {
measure=_measure_hook_spawn,
setup=_setup_hook_spawn,
teardown=_teardown_hook_spawn,
description="Spawn the per-chunk MessageDisplay hook exactly as Claude Code does.",
description=(
"Spawn one Python command hook (isolated interpreter, module "
"entrypoint) and time its whole lifetime — what any `python -m` "
"hook costs Claude's blocked TUI. Not the per-chunk path: that "
"one is a /bin/sh appender."
),
),
Journey(
name="cli_startup",