fix: tool_search_tool_regex deferred and falsely resolved on direct-Anthropic path (#2971)
## Description Direct Anthropic users could receive `400 Tool reference 'tool_search_tool_regex' not found in available tools` when Claude Code sent a typeless `tool_search_tool_regex` entry. Headroom treated it as an ordinary deferrable tool, injected a typed search tool with the same name, and later mistook that typed search mechanism for a valid target of the stale `tool_reference`. This change prevents the duplicate injection and repairs already-poisoned transcripts without stripping valid references to ordinary deferred tools. It addresses the first-party Anthropic regression reported in [PR #2539's follow-up](https://github.com/headroomlabs-ai/headroom/pull/2539#issuecomment-5280259642) and complements the history repair from #2805. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - recognize typeless, case-insensitive `tool_search_tool_*` names as an existing client tool-search surface and skip Headroom's duplicate injection - exclude typed Anthropic search mechanisms from the set of valid `tool_reference` targets - preserve valid regular deferred-tool references and the normal deferral path for similar non-reserved names - add a first-party Anthropic handler regression that proves the outbound tools remain unchanged and stale search bookkeeping is removed - rebase onto #2996, which prevents the native detector from hanging the full CI test shard ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text .venv/Scripts/python.exe -m pytest \ tests/test_issue_746_tool_search.py \ tests/test_anthropic_stage_timings.py \ tests/test_cache_control_ttl_order.py \ tests/test_cache_ttl_preserved.py \ tests/test_proxy/test_tool_search_repair_after_turn_hooks.py \ tests/test_transforms/test_detect_fallback_1123.py \ tests/test_transforms_content_detection.py \ tests/test_transforms_content_router.py \ -q --disable-warnings --maxfail=1 170 passed, 1 warning in 10.27s .venv/Scripts/ruff.exe check . All checks passed! .venv/Scripts/ruff.exe format --check . 1411 files already formatted pre-commit run mypy --all-files Success: no issues found in 519 source files git diff --check (no output) ``` ## Real Behavior Proof - Environment: Windows, Python 3.13.3, first-party Anthropic handler test with `HEADROOM_TOOL_SEARCH` at its default enabled setting - Exact command / steps: run the same helper-level payload against the pre-fix base and this branch, then run `test_anthropic_direct_path_repairs_typeless_tool_search_regression` through `handle_anthropic_messages()` with 20 ordinary tools, one typeless `tool_search_tool_regex`, and a stale self-reference - Observed result: before the fix, Headroom injected a second typed search tool, deferred the typeless client tool, and removed 0 stale blocks; on this branch, it skips duplicate injection, preserves the client tools array, and removes the paired `server_tool_use` and `tool_search_tool_result` blocks before forwarding - Not tested: a live request against a paid Anthropic account; the production handler's outbound body is captured before the network boundary instead ## Runtime Rollout Safety - Rollout-managed feature(s): Anthropic server-side tool-search deferral (`HEADROOM_TOOL_SEARCH`) - Minimum rollout channel: standard CI; narrow corrective change to an existing default-on path - Stable/default behavior changed: yes; reserved typeless client search tools now suppress duplicate injection, and typed search mechanisms no longer satisfy deferred-tool references - Kill switch / disable path: set `HEADROOM_TOOL_SEARCH=0` to disable new injection; history repair remains unconditional so existing poisoned sessions can recover - Unsafe override required: no - Qualification impact: no new rollout surface or configuration; focused handler, helper, cache-control, and hook-order regressions cover the affected path - Rollback path: revert this PR; operators can set `HEADROOM_TOOL_SEARCH=0` while rolling back ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A — proxy request transformation only. ## Additional Notes - Documentation is not changed because this fixes internal request classification and transcript repair without adding a user-facing option or workflow. - Anthropic documents `tool_search_tool_regex` / `tool_search_tool_bm25` as server search mechanisms; deferred definitions, rather than the search mechanism itself, are the valid `tool_reference` targets. - Rebased onto #2996, which fixes the unrelated native-detector hang that timed out shard 4 on the prior merge commit. --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: JerrettDavis <2610199+JerrettDavis@users.noreply.github.com> Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
This commit is contained in:
Copilot
@@ -2858,8 +2858,9 @@ def inject_tool_search_deferral(
|
||||
if not isinstance(tools, list) or len(tools) < _TOOL_SEARCH_MIN_TOOLS:
|
||||
return tools
|
||||
for tool in tools:
|
||||
if isinstance(tool, dict) and str(tool.get("type", "")).startswith(
|
||||
_TOOL_SEARCH_TOOL_TYPE_PREFIX
|
||||
if isinstance(tool, dict) and (
|
||||
str(tool.get("type", "")).startswith(_TOOL_SEARCH_TOOL_TYPE_PREFIX)
|
||||
or str(tool.get("name") or "").lower().startswith(_TOOL_SEARCH_TOOL_TYPE_PREFIX)
|
||||
):
|
||||
return tools # client already uses tool search — leave it alone
|
||||
|
||||
@@ -2974,7 +2975,19 @@ def strip_unsupported_tool_search_blocks(messages: Any, tools: Any) -> tuple[Any
|
||||
return messages, 0
|
||||
|
||||
tool_list = tools if isinstance(tools, list) else []
|
||||
available = {str(t["name"]) for t in tool_list if isinstance(t, dict) and t.get("name")}
|
||||
# Typed search tools (type starts with "tool_search_tool_") are the search
|
||||
# mechanism itself — they are never the target of a tool_reference lookup.
|
||||
# Excluding them from `available` ensures that a stale history entry that
|
||||
# references "tool_search_tool_regex" (from a turn where inject deferred a
|
||||
# typeless client tool with that name) is correctly dropped rather than
|
||||
# falsely kept because the injected typed search tool shares the same name.
|
||||
available = {
|
||||
str(t["name"])
|
||||
for t in tool_list
|
||||
if isinstance(t, dict)
|
||||
and t.get("name")
|
||||
and not str(t.get("type") or "").startswith(_TOOL_SEARCH_TOOL_TYPE_PREFIX)
|
||||
}
|
||||
has_search_tool = any(
|
||||
isinstance(t, dict) and str(t.get("type", "")).startswith(_TOOL_SEARCH_TOOL_TYPE_PREFIX)
|
||||
for t in tool_list
|
||||
|
||||
@@ -337,6 +337,88 @@ def test_anthropic_third_party_upstream_strips_tool_search_tools():
|
||||
)
|
||||
|
||||
|
||||
def test_anthropic_direct_path_repairs_typeless_tool_search_regression():
|
||||
"""Do not double-inject a typeless search tool; heal its stale history."""
|
||||
tools = [
|
||||
{
|
||||
"name": f"mcp_tool_{index}",
|
||||
"description": f"tool {index}",
|
||||
"input_schema": {"type": "object", "properties": {}},
|
||||
}
|
||||
for index in range(20)
|
||||
]
|
||||
tools.append(
|
||||
{
|
||||
"name": "tool_search_tool_regex",
|
||||
"description": "client-provided tool search",
|
||||
"input_schema": {"type": "object", "properties": {}},
|
||||
}
|
||||
)
|
||||
messages = [
|
||||
{"role": "user", "content": "search for a tool"},
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": [
|
||||
{
|
||||
"type": "server_tool_use",
|
||||
"id": "srvtoolu_regex",
|
||||
"name": "tool_search_tool_regex",
|
||||
"input": {"pattern": "regex"},
|
||||
},
|
||||
{
|
||||
"type": "tool_search_tool_result",
|
||||
"tool_use_id": "srvtoolu_regex",
|
||||
"content": {
|
||||
"type": "tool_search_tool_search_result",
|
||||
"tool_references": [
|
||||
{
|
||||
"type": "tool_reference",
|
||||
"tool_name": "tool_search_tool_regex",
|
||||
}
|
||||
],
|
||||
},
|
||||
},
|
||||
],
|
||||
},
|
||||
{"role": "user", "content": "continue"},
|
||||
]
|
||||
request = _build_request(
|
||||
{
|
||||
"model": "claude-sonnet-4-6",
|
||||
"max_tokens": 100,
|
||||
"messages": messages,
|
||||
"tools": tools,
|
||||
},
|
||||
{"authorization": "Bearer sk-ant-api-test"},
|
||||
)
|
||||
handler = _DummyAnthropicHandler()
|
||||
|
||||
import headroom.tokenizers as _tk
|
||||
|
||||
orig_get = _tk.get_tokenizer
|
||||
_tk.get_tokenizer = lambda model: _DummyTokenizer()
|
||||
try:
|
||||
response = anyio.run(handler.handle_anthropic_messages, request)
|
||||
finally:
|
||||
_tk.get_tokenizer = orig_get
|
||||
|
||||
assert response.status_code == 200
|
||||
_, _, _, forwarded_body = handler.captured
|
||||
# The client-owned typeless entry suppresses Headroom's typed search-tool
|
||||
# injection, and the tools array remains byte-for-byte equivalent.
|
||||
assert forwarded_body["tools"] == tools
|
||||
assert not any(tool.get("type") for tool in forwarded_body["tools"])
|
||||
# The stale server-side round trip is removed before Anthropic validates it.
|
||||
block_types = [
|
||||
block.get("type")
|
||||
for message in forwarded_body["messages"]
|
||||
for block in message.get("content", [])
|
||||
if isinstance(message.get("content"), list) and isinstance(block, dict)
|
||||
]
|
||||
assert "server_tool_use" not in block_types
|
||||
assert "tool_search_tool_result" not in block_types
|
||||
|
||||
|
||||
def test_anthropic_http_invalid_body_still_emits_stage_timings(stage_log_capture):
|
||||
async def receive():
|
||||
# Invalid JSON — produces ``ValueError`` from ``_read_request_json``.
|
||||
|
||||
@@ -471,3 +471,141 @@ def test_repair_strips_search_history_when_only_the_tool_is_missing() -> None:
|
||||
_poisoned_transcript(), [{"name": "AskUserQuestion", "input_schema": {}}]
|
||||
)
|
||||
assert removed == 2
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Regression tests for the direct-Anthropic regression reported in PR #2539
|
||||
# comment #5280259642: "Tool reference 'tool_search_tool_regex' not found in
|
||||
# available tools".
|
||||
#
|
||||
# Root cause: when a client sends ``tool_search_tool_regex`` as a *typeless*
|
||||
# tool, ``inject_tool_search_deferral`` would (a) not early-exit because the
|
||||
# guard only checked ``type``, and (b) defer the tool. Anthropic then found
|
||||
# the deferred copy via the server-side search and stored the tool's name in a
|
||||
# ``tool_reference`` entry. On subsequent requests where the typed injected
|
||||
# search tool was present, ``strip_unsupported_tool_search_blocks`` incorrectly
|
||||
# treated the injected search tool's *name* as proof the reference was
|
||||
# resolvable — but the typed server tool is not a valid deferred-tool target, so
|
||||
# Anthropic rejected the request with 400.
|
||||
#
|
||||
# The two-part fix:
|
||||
# 1. ``inject_tool_search_deferral`` early-exit also fires on a name-prefix
|
||||
# match, preventing double-injection when the client carries a typeless
|
||||
# ``tool_search_tool_*`` entry.
|
||||
# 2. ``strip_unsupported_tool_search_blocks`` excludes typed search tools
|
||||
# from the ``available`` set — they are the search mechanism, not targets.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _transcript_with_search_tool_regex_reference() -> list[dict]:
|
||||
"""Transcript where the search found 'tool_search_tool_regex' itself.
|
||||
|
||||
This happens when inject_tool_search_deferral defers a typeless client tool
|
||||
named 'tool_search_tool_regex': Anthropic finds it and stores it as a
|
||||
tool_reference. On subsequent requests the repair must drop the block
|
||||
rather than falsely keep it because the typed injected search-tool shares
|
||||
the same name.
|
||||
"""
|
||||
return [
|
||||
{"role": "user", "content": [{"type": "text", "text": "search for a tool"}]},
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": [
|
||||
{
|
||||
"type": "server_tool_use",
|
||||
"id": "srvtoolu_REGEX",
|
||||
"name": _TOOL_SEARCH_DEFAULT_NAME,
|
||||
"input": {"pattern": "regex"},
|
||||
},
|
||||
{
|
||||
"type": "tool_search_tool_result",
|
||||
"tool_use_id": "srvtoolu_REGEX",
|
||||
"content": {
|
||||
"type": "tool_search_tool_search_result",
|
||||
"tool_references": [
|
||||
{
|
||||
"type": "tool_reference",
|
||||
# The search found the deferred 'tool_search_tool_regex'
|
||||
# typeless tool — this is the broken reference.
|
||||
"tool_name": _TOOL_SEARCH_DEFAULT_NAME,
|
||||
}
|
||||
],
|
||||
},
|
||||
},
|
||||
],
|
||||
},
|
||||
]
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"name",
|
||||
[_TOOL_SEARCH_DEFAULT_NAME, "TOOL_SEARCH_TOOL_BM25"],
|
||||
)
|
||||
def test_inject_deferral_exits_early_on_typeless_tool_search_name(name: str) -> None:
|
||||
# A client that sends tool_search_tool_regex without a ``type`` field should
|
||||
# be treated as already using tool search (name-prefix guard), so Headroom
|
||||
# must not inject a second search tool on top of it.
|
||||
typeless_search = {"name": name, "input_schema": {}}
|
||||
tools = _tools(20) + [typeless_search]
|
||||
result = inject_tool_search_deferral(tools)
|
||||
assert result is tools # no injection
|
||||
|
||||
|
||||
def test_inject_deferral_does_not_false_match_similar_typeless_tool_name() -> None:
|
||||
# Keep ordinary tools whose names merely resemble the reserved prefix on the
|
||||
# normal deferral path; the trailing underscore is part of the match.
|
||||
tools = _tools(20) + [{"name": "tool_search_toolbox", "input_schema": {}}]
|
||||
result = inject_tool_search_deferral(tools)
|
||||
assert result is not tools
|
||||
by_name = {tool.get("name"): tool for tool in result}
|
||||
assert by_name["tool_search_toolbox"]["defer_loading"] is True
|
||||
|
||||
|
||||
def test_repair_drops_search_tool_self_reference_when_inject_ran() -> None:
|
||||
# Regression for PR #2539 comment #5280259642.
|
||||
#
|
||||
# Scenario: inject ran on a previous turn (has_search_tool=True because the
|
||||
# typed search tool is present), but the transcript's tool_reference names
|
||||
# 'tool_search_tool_regex' — the search tool itself. The typed injected
|
||||
# search tool must NOT count as a valid reference target; the block must be
|
||||
# dropped so Anthropic never sees an unresolvable tool_reference.
|
||||
transcript = _transcript_with_search_tool_regex_reference()
|
||||
# tools array after inject: typed search tool + regular deferred tools
|
||||
tools = [
|
||||
_SEARCH_TOOL, # typed search tool — must NOT be in 'available'
|
||||
{"name": "Bash", "input_schema": {}},
|
||||
{"name": "mcp_tool_x", "input_schema": {}, "defer_loading": True},
|
||||
]
|
||||
messages, removed = strip_unsupported_tool_search_blocks(transcript, tools)
|
||||
assert removed == 2 # server_tool_use + tool_search_tool_result both dropped
|
||||
# The assistant turn is entirely stripped (only search blocks were present).
|
||||
assert len(messages) == 1
|
||||
assert messages[0]["role"] == "user"
|
||||
|
||||
|
||||
def test_repair_noop_when_referenced_tool_is_regular_deferred_tool() -> None:
|
||||
# Baseline: when the transcript references a normal deferred tool (not the
|
||||
# search tool itself) and that tool is in the current tools array, the block
|
||||
# must be kept — no false-positive stripping from the typed-search exclusion.
|
||||
transcript = _poisoned_transcript() # references "AskUserQuestion"
|
||||
tools = [
|
||||
_SEARCH_TOOL,
|
||||
{"name": "AskUserQuestion", "input_schema": {}, "defer_loading": True},
|
||||
]
|
||||
messages, removed = strip_unsupported_tool_search_blocks(transcript, tools)
|
||||
assert removed == 0
|
||||
assert messages is transcript
|
||||
|
||||
|
||||
def test_repair_drops_when_referenced_tool_absent_despite_search_tool_present() -> None:
|
||||
# The referenced tool is NOT in the current tools array even though the
|
||||
# typed search tool is present (e.g. a compact request with a different tool
|
||||
# subset). The block must be dropped.
|
||||
transcript = _poisoned_transcript() # references "AskUserQuestion"
|
||||
tools = [
|
||||
_SEARCH_TOOL,
|
||||
{"name": "Bash", "input_schema": {}},
|
||||
# AskUserQuestion intentionally absent
|
||||
]
|
||||
messages, removed = strip_unsupported_tool_search_blocks(transcript, tools)
|
||||
assert removed == 2
|
||||
|
||||
Reference in New Issue
Block a user