* Python: add ATR validation FunctionMiddleware sample (execution-boundary validation, #5366)
Adds python/samples/02-agents/middleware/atr_validation_middleware.py: a
FunctionMiddleware that validates tool arguments at the execution boundary and
raises MiddlewareTermination before call_next() when they match an attack
pattern, so the tool never runs. This is the deterministic, single-enforcement-
point pattern named in #5366 and answers its open follow-up about a recommended
validation-at-execution-boundary sample.
The check is a small self-contained deny-list mirroring Agent Threat Rules (ATR)
intent (prompt injection, exfiltration, credential access in tool args); a
docstring notes how to swap in the full open ruleset via pyatr. No external
dependency, so the sample stays import-clean.
Updates the middleware README Files table.
Signed-off-by: Adam Lin <adam@agentthreatrule.org>
* Python: Samples: run the real ATR engine in atr_validation_middleware
Address review on #6528:
- Load and run the real ATR ruleset via pyatr (ATREngine + AgentEvent
tool_call event) instead of re-implementing a regex deny-list; the
built-in deny-list is now only a fallback when pyatr is not installed.
- Add re.DOTALL (and a whole-text scan) to the fallback patterns so
multiline injection payloads are not missed.
- Move load_dotenv() into main() so importing the module has no side
effects.
- Route the middleware block/allow messages through a module logger
instead of print().
- Include the matched ATR rule id in the log and in the
MiddlewareTermination message for auditability.
- Update the middleware README entry to match.
* fix(samples): make ATR validation middleware pass ty/pyrefly typing CI
Resolve the three type-checker errors flagged on the samples typing jobs
(ty + pyrefly, reportMissingImports/reportAttributeAccessIssue via pyright):
- pyatr is an optional, unstubbed runtime dependency that is not installed
in the typing CI env; mark its imports with `# type: ignore` so the
unresolved-import error is suppressed while keeping the graceful
ImportError -> deny-list fallback intact.
- Replace the function-attribute engine cache
(`_detect_with_atr._engine`), which ty/pyrefly reject, with a clean
`functools.lru_cache`-backed `_load_atr_engine()` loader.
- Type the argument-scanning helpers to accept the real
`FunctionInvocationContext.arguments` type (`BaseModel | Mapping[str, Any]`)
and normalise a pydantic model via `model_dump()` before scanning, fixing
the invalid-argument-type error.
ty / pyrefly / pyright (samples config) / ruff check + format all clean on
the file; runtime block/allow behaviour verified for both dict and BaseModel
arguments.
* Python: Samples: simplify ATR middleware to plain pyatr import
Address review feedback (@eavanvalkenburg): now that the sample runs the
real pyatr engine, drop the optional-import scaffolding.
- Add a dependency header declaring pyatr (pip install pyatr).
- Switch to a plain top-level `import pyatr` and remove the
try/except ImportError fallback path.
- Remove the regex deny-list (_FALLBACK_PATTERNS, _detect_with_fallback);
keep 2-3 representative pattern shapes inline as a reference comment so
readers still see the kind of rules ATR encodes. Detection is now a
single straight-line engine call.
- Keep the prior typing fixes: `# type: ignore` on the pyatr import
(unstubbed, absent in the typing CI env), the functools.lru_cache
engine loader, and the BaseModel | Mapping[str, Any] signatures.
* fix: use PEP 723 inline script metadata for sample dependencies
---------
Signed-off-by: Adam Lin <adam@agentthreatrule.org>
Co-authored-by: eeee2345 <eeee2345@users.noreply.github.com>
Co-authored-by: Eduard van Valkenburg <eavanvalkenburg@users.noreply.github.com>
* Python: Add AgentLoopMiddleware for re-running agents in a loop
Add `AgentLoopMiddleware`, an `AgentMiddleware` that re-runs the wrapped
agent in a loop. A single configurable class covers three common patterns,
each with a convenience classmethod factory:
- Ralph loop (`.ralph(...)`): no exit criteria, with feedback tracking
(`record_feedback`/`progress`), progress injection (`inject_progress`),
optional fresh context per iteration (`fresh_context`), and an early-stop
completion signal (`is_complete`).
- Predicate (`.with_predicate(...)`): loop while a `should_continue` callable
returns True (e.g. paired with `todos_remaining`/`background_tasks_running`).
- Judge (`.with_judge(...)`): a second chat client decides whether the original
request was answered, using a `JudgeVerdict` structured-output response.
The loop also auto-resolves pending function-approval / user-input requests via
an `on_approval_request` callable (bounded by `max_approval_rounds`), and the
next iteration's input is controlled by `next_message`. Supports both streaming
and non-streaming runs.
Exports `AgentLoopMiddleware`, `JudgeVerdict`, `todos_remaining`, and
`background_tasks_running`. Adds tests, a sample, and docs.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Python: Refine AgentLoopMiddleware API and sample
- with_judge: add criteria list with {{criteria}} templating into judge
instructions plus an agent-side instruction; add fresh_context, additional
judge feedback relay; default judge max_iterations.
- should_continue is now required and positional; supports (bool, str|None)
feedback tuples surfaced to next_message/record_feedback via feedback kwarg.
- Judge forwards full multi-modal request and response messages.
- Default max_iterations=10 (explicit None = unbounded); removed is_complete and
Ralph terminology; ShouldContinueResult is a real TypeAlias.
- Sample: stream all loops, print iteration counts via injected user-block
boundaries (robust to function calling), <role>: content formatting, per-method
expected output, and a looping todo sample.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Python: Fix CI checks for AgentLoopMiddleware
- Resolve pyright errors in _loop.py: drop the always-true final_result None
check (the while loop always assigns it) and cast finish_reason to the
AgentResponse constructor's expected type.
- Apply pyupgrade --py310-plus: import TypeAlias from typing.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Python: Resolve mypy/pyright disagreement on finish_reason
pyright infers AgentResponse.finish_reason as including str and rejects the
direct assignment, while mypy considers a cast redundant. Drop the cast and
suppress only pyright with a targeted reportArgumentType ignore, satisfying
both type checkers.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Python: Add todo+judge AgentLoopMiddleware sample
Add a second AgentLoopMiddleware sample that composes two criteria in one
should_continue predicate: a TodoProvider check (evaluated first) and a
report-style judge chat client (evaluated once todos are complete) that grades
the assembled report against shared requirements. Register it in the middleware
samples README.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Python: Compose todo+judge loops as two middleware
Rework the todo+judge sample to compose two AgentLoopMiddleware on the agent
itself (middleware=[judge_loop, todo_loop]) instead of a single hand-written
predicate. The inner todos_remaining loop drafts the report todo-by-todo and the
outer with_judge loop re-runs it until an editor chat client judges the report
publication-ready, reusing the built-in helpers.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Reset session for fresh_context loops via snapshot/restore
AgentLoopMiddleware.fresh_context previously only reset context.messages,
so with an attached session each iteration still reloaded the local
transcript or re-threaded the service-side conversation id and the model
saw the accumulated history. Snapshot the session once before the loop
(via to_dict) and restore it (from_dict + field copy) between iterations,
so every pass starts from the pre-loop baseline. The final iteration's
pass is persisted (no restore after the terminating iteration), so a
subsequent agent.run continues from there.
Removed the obsolete warning, updated docstrings and core AGENTS.md, and
added tests: a snapshot/restore round-trip, a session-reset
streaming x fresh_context x inject_progress x store matrix across multiple
runs and loop iterations, and response_format parsing across the loop.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Updated samples and docstrings
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* [BREAKING] Refactor middleware layering and raw clients
Reorder chat client layers so function invocation wraps chat middleware, and chat middleware stays outside telemetry while still running for each inner model call. Add middleware pipeline caching, refresh docs and samples, and split Anthropic into raw and public clients to match the standard layering model.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Tighten typing ignores in ancillary modules
Add targeted typing ignores in workflow visualization and lab modules so pyright stays clean alongside the middleware refactor work.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix categorize_middleware to unpack tuple/Sequence and use relative MRO assertions
- Broaden isinstance check in categorize_middleware from list to Sequence
so tuples and other Sequence types are properly unpacked instead of
being appended as a single item.
- Replace fragile hardcoded MRO index assertions in anthropic test with
relative ordering via mro.index().
- Add regression tests for categorize_middleware with tuple, list, and
None inputs.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix middleware string decomposition, add middleware param to FunctionInvocationLayer, and add tests (#4710)
- Guard categorize_middleware Sequence check against str/bytes to prevent
character-by-character decomposition of accidentally passed strings
- Add explicit middleware parameter to FunctionInvocationLayer.get_response
and merge it into client_kwargs before categorization, fixing the
inconsistency where only OpenAIChatClient supported this parameter
- Add assertions that RawAnthropicClient does not inherit convenience layers
- Add chat middleware cache test with non-empty base middleware
- Add tests for single unwrapped middleware item and string input
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Apply pre-commit auto-fixes
* Apply pre-commit auto-fixes
* Address review feedback for #4710: review comment fixes
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <copilot@github.com>
Eduard van Valkenburg