feature/python-serial-message-delivery
512 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Tao Chen
|
c5e1acf561 | Fix formatting | ||
|
Tao Chen
|
c508c42220 | Improve comments | ||
|
Tao Chen
|
b8d34dc482 |
Use lazy-None init for the event queue, matching the executor lock
Initialize _event_queue to None and create it on first use in _get_event_queue, mirroring the per-executor lock. Avoids constructing a queue in __init__/reset_for_new_run that is immediately discarded once the running loop is known. |
||
|
Tao Chen
|
1e12403759 |
Re-create runner context event queue lazily under the running loop
Like the per-executor lock, the runner context's asyncio.Queue bound to the first event loop it was awaited under, so reusing a workflow across loops (e.g. successive asyncio.run calls) raised 'bound to a different event loop'. Re-create the queue lazily via _get_event_queue() when the running loop changes. Adds an integration test reusing a workflow across event loops. |
||
|
Tao Chen
|
5ed5d7ab7c |
Create per-executor lock lazily under the running loop
asyncio.Lock created in Executor.__init__ would bind to the first event loop it was awaited under, so reusing an executor/workflow across loops (e.g. successive asyncio.run calls) raised 'bound to a different event loop'. Create the lock lazily via _get_execution_lock(), re-creating it when the running loop changes. Adds a loop-scoped lock test. |
||
|
Tao Chen
|
252735a604 |
Process messages to an executor serially within a superstep
Add a per-executor asyncio.Lock in Executor.execute so each executor processes its messages one at a time within a superstep, while preserving concurrency across distinct executors. Includes a regression test. |
||
|
Tao Chen
|
3c3feb8705 |
Python: Refactor runner/workflow responsibilities and fix checkpoint ancestry bug (#6695)
* Refactor runner/workflow responsibilities, add concurrency guards, and fix checkpoint ancestry bug Move runner-state ownership out of Workflow into Runner for clearer responsibilities. Add a weakref-based concurrent-run guard in Workflow and fix the stream-drop race in run_until_convergence. Fix the checkpoint ancestry bug by tracking the previous checkpoint id as runner instance state so parent pointers persist across resumed runs. Move Runner to a deprecated lazy __getattr__ export (backward-compatible with DeprecationWarning) and export CheckpointID. * Scope runtime checkpoint storage to its owning run Close the stream-drop race where a dropped run's deferred async-generator finalizer could leave a runtime checkpoint storage override set (inherited by a new run) or clear a successor run's storage. run() now defensively clears any stale override before starting, and _run_core only clears the override if this run still owns it (mirroring the _active_run ownership guard). Adds regression tests for both the inheritance and clobber cases. * Collapse runtime-storage ownership into the active-run weakref _runtime_storage_owner always held the same weakref as _active_run, so the two ownership conditions were equivalent. Derive ownership from a single owns_run = (_active_run is my_active_run) captured before the active-run clear, and remove the redundant field. No behavior change. * Nest runtime-storage clear under the owns_run guard Both the active-run release and the runtime-storage clear are gated on owns_run, so fold the storage clear inside the if owns_run block. No behavior change. * Reset resume flag in a finally so it can't leak across runs _resumed_from_checkpoint was only cleared on the success path of run_until_convergence, so a failure during a resumed run (e.g. executor failure) left it True. The next fresh run then skipped the superstep-0 checkpoint and parented later checkpoints to the stale resume point. Move the reset into a finally. Add a regression test that fails a resumed run via an executor error and asserts the next fresh run creates the superstep-0 checkpoint. * Fix tests and formatting * Fix formatting * Address comments * Update type ignore statements |
||
|
Giles Odigwe
|
9f1ee23a4b |
Python: [BREAKING] Refactor FileSkillsSource for depth-based discovery and predicate filters (#6488)
* Python: [Breaking] Refactor FileSkillsSource for depth-based discovery and predicate filters Refactors FileSkillsSource to make script and resource discovery more flexible. ## Changes - **Drops** resource_directories / script_directories options (preconfigured directory whitelists). - **Adds** search_depth option (>= 1, default 2): controls how deep the recursive scan goes within each skill directory. - **Adds** script_filter / resource_filter predicate options that receive a FileSkillFilterContext (skill_name + relative_file_path), allowing whitelist/blacklist filtering by file path. - **Adds** FileSkillFilterContext class exported from agent_framework. ## Notes - The Skills API is marked @experimental -- the option removals are intentional breaking changes within the experimental surface. - Security checks (path containment, symlink detection) are preserved and continue to use the skill root directory as the trusted boundary. - Ports the same refactoring from .NET PR #6109 while following Python conventions (instance methods, Callable type hints, __slots__). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR feedback: clarify depth constants and skip nested skill directories - Add clarifying comments distinguishing MAX_SEARCH_DEPTH (SKILL.md discovery) from DEFAULT_SEARCH_DEPTH (per-skill resource/script scanning). - Stop recursing into subdirectories that contain their own SKILL.md, preventing child skill files from being attached to the parent skill. - Add test verifying nested skill boundary is respected. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove __slots__ from FileSkillFilterContext and add type-ignore comments - Remove __slots__ from FileSkillFilterContext per reviewer feedback — the optimization is negligible and inconsistent with sibling classes. - Add type: ignore[attr-defined] / ty: ignore[unresolved-attribute] comments to test lines accessing private _resources/_scripts attributes, matching the convention established on main. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Simplify filter predicates: remove FileSkillFilterContext, use Callable[[str, str], bool] Address reviewer feedback: - Remove FileSkillFilterContext class — a dedicated class for two strings is overkill in Python. Filters now receive (skill_name, relative_file_path) directly as positional args. - Update docstrings to describe behavior instead of referencing private instance attributes. - Remove FileSkillFilterContext from exports and __all__. - Update all test lambdas and remove TestFileSkillFilterContext class. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Use DEFAULT_SEARCH_DEPTH as default argument directly Instead of accepting int | None and resolving None to the default internally, use DEFAULT_SEARCH_DEPTH as the parameter default value on both FileSkillsSource.__init__() and SkillsProvider.from_paths(). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Giles Odigwe
|
91f639a694 |
Python: Explicitly emit available_resources and available_scripts in skill content (#6694)
Skill content now always emits <available_resources> and <available_scripts> blocks, using self-closing elements when empty, so models receive an authoritative list per category and do not hallucinate resource/script names. FileSkill now also emits its resources block. Closes #6348 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Eduard van Valkenburg
|
acb28a63b5 |
Python: Fix MCP metadata and tool name handling (#6656)
* Fix MCP metadata and tool name handling Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address MCP review feedback Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Eduard van Valkenburg
|
f2d02e58b3 |
Python: Add hosting core and Responses channel (#6580)
* Add Python hosting core and Responses channel Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address hosting core review feedback Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Adopt source pyright typing setup for hosting packages Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Cover ResponsesChannel custom path routing Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Align hosting tests with package layout Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix hosting workflow fixture imports in aggregate tests Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Apply useful Responses channel hardening Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix hosting package typing checks Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix hosting pyright under Python 3.11 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Avoid static diskcache dependency in hosting Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix aggregate typing and Docker test resilience Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Simplify local Responses workflow sample Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Clarify generic hosting is not Foundry hosting Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Revert "Clarify generic hosting is not Foundry hosting" This reverts commit 73b584d919053bed43a258d75dc2b76406e9c181. * Clarify isolation key source flexibility Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Clarify isolation header reuse boundary Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Support multimodal Responses channel outputs Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Preserve multimodal streaming Responses output Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Stream Responses output items from updates Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Improve Responses streaming output handling * Tighten Responses channel default option handling - Restore full option parsing in parse_responses_request: known fields are remapped (max_output_tokens→max_tokens, parallel_tool_calls→ allow_multiple_tool_calls), transport/session keys excluded, None values dropped, everything else forwarded as-is so run_hook can inspect the full set. - Add a default _strip_options_hook on ResponsesChannel that removes all parsed options before reaching the agent. Callers cannot inject generation params (temperature, instructions, tools, …) unless the host explicitly allows it. - A custom run_hook replaces the default entirely and receives the full ChannelRequest.options plus the raw protocol_request. - Update tests to cover remap, default-strip, and custom-hook paths. - Clarify host debug-log docstring to match new option flow. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Tao Chen
|
36420c515e |
Python: Align serialized tool format to OTel GenAI tool def format (#6556)
* Align serialized tool format to OTel GenAI tool def format * Cache serialized tools |
||
|
Tao Chen
|
7051a4920d |
Python: Add MCP as a hard dep in Foundry Hosting (#6634)
* Add MCP as a hard dep in Foundry Hosting * Pin GitHub SDK * Fix formatting * Fix formatting |
||
|
westey
|
a2018b40f9 |
Python: [BREAKING] Require approval for file-access tools with read-only auto-approval (#6599)
* Require approvals for file-access and expose auto approval funcs for it * Scope file-access auto-approval rules to local tools; fix base-Agent sample Address PR #6599 review feedback: - read_only/all_tools auto-approval rules now reject any call carrying a server_label so they stay scoped to FileAccessProvider's local tools and never auto-approve a same-named hosted tool. - Expand the FileAccessProvider docstring to explain the runtime effect of approval_mode="always_require" and point to ToolApprovalMiddleware / create_harness_agent. - Fix the base-Agent file_access_data_processing sample, which would otherwise stop executing file tools under the new always_require defaults, by adding ToolApprovalMiddleware with all_tools_auto_approval_rule. - Add tests covering hosted (server_label) calls and update docs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Clean up comments * Update sample after merge --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Tao Chen
|
7f2e19ca2f |
Python: Ensure spans created inside sync preparations in streaming call are correctly nested (#6552)
* Make sure spans created inside sync ops in streaming path are correctly nested * Add tests * Fix comments * Fix typing |
||
|
westey
|
7b6f582b13 |
Python: Agent Harness blog post accompanying samples part 1 (#6605)
* Add samples for harness blog post part 1 * Add readme for python samples * Update python instructions to match dotnet instructions * Address PR comments * Add link to blog posts * Fix blog post naming. * Add more blog post links |
||
|
westey
|
d108d4b549 |
Python: [BREAKING] Integrate looping into HarnessAgent (#6607)
* Integrate looping into harness * Address PR comments * Address PR comments. * Fix typing error |
||
|
Ahmed Muhsin
|
148f57020a |
Python: host MAF workflows on a standalone Durable Task worker (#6418)
* feat(durabletask): host MAF workflows on a standalone Durable Task worker
Add a host-agnostic workflow execution engine to agent-framework-durabletask so a MAF Workflow can run as a durable orchestration outside Azure Functions:
- WorkflowOrchestrationContext protocol + DurableTaskWorkflowContext adapter, the superstep orchestrator, serialization helpers, capturing runner context, and the shared non-agent activity body (including the yield-output classifier so intermediate executors are not surfaced as final outputs).
- DurableAIAgentWorker.configure_workflow auto-registers agent executors as entities, non-agent executors as activities, and the workflow orchestrator.
- plan_workflow_registration centralizes the 'what to register' decision so it can be shared across hosts.
- run_agent_coroutine runs all agent coroutines on one persistent event loop, fixing a cross-loop hang when shared chat clients/credentials bind their asyncio primitives to a dead loop.
- DurableWorkflowClient (start/await workflow + HITL discover/respond); DurableAIAgentClient stays agent-only.
* refactor(azurefunctions): delegate workflow execution to agent-framework-durabletask
AgentFunctionApp now reuses the shared orchestrator, activity body, and registration planner from agent_framework_durabletask instead of maintaining its own copies; _workflow.py becomes a thin host-specific adapter (AzureFunctionsWorkflowContext).
- Run agent entity coroutines on the shared persistent event loop, fixing the cross-loop hang.
- Relocate state-diff unit tests to the durabletask package; update entity loop tests.
* feat(core): expose durabletask workflow symbols via agent_framework.azure
Lazily re-export WORKFLOW_ORCHESTRATOR_NAME and DurableWorkflowClient from the agent_framework.azure namespace so standalone hosts can import them without depending on internal module paths.
* docs(samples): add standalone durabletask workflow and HITL samples
Add two samples under samples/04-hosting/durabletask demonstrating MAF workflows on a standalone Durable Task worker (no Azure Functions):
- 08_workflow: conditional spam-detection workflow started via DurableWorkflowClient.start_workflow / await_workflow_output.
- 09_workflow_hitl: content-moderation workflow that pauses with ctx.request_info and is resumed via DurableWorkflowClient.get_pending_hitl_requests / send_hitl_response.
Also add the durabletask workflow integration test (test_08_dt_workflow).
* fix: address PR review feedback
- Sanitize HITL external-event responses with strip_pickle_markers in the orchestrator (defense-in-depth for callers that bypass DurableWorkflowClient).
- Raise WorkflowConvergenceException when max_iterations is reached with pending messages, matching the core WorkflowRunner instead of silently returning partial output.
- Route falsy 'sent' messages (use 'is not None' instead of truthiness).
- Normalize None shared_state_snapshot/source_executor_ids in execute_workflow_activity.
- Cast Any returns in AzureFunctionsWorkflowContext to satisfy mypy/pyright.
- Fix sample docstrings to reference DurableWorkflowClient.
* fix: resolve pyright Package Checks errors
- Use typed locals instead of cast in AzureFunctionsWorkflowContext (mypy sees Any, pyright sees concrete types -> avoid reportUnnecessaryCast).
- Annotate shared_state_snapshot and cast partially-typed durabletask SDK returns / HITL custom-status parsing to satisfy reportUnknownVariableType/reportUnknownMemberType.
- Drop the dead deserialize/serialize re-export in _workflow.py and mark the intentional private _extract_message_content re-export.
* fix(durabletask): agent-executor identity and typed workflow input
Register each workflow agent entity under the executor id that the orchestrator dispatches to (instead of the agent name), so AgentExecutor(agent, id=...) works when the id differs from agent.name. The azure-functions host mirrors this.
Reconstruct the start executor declared input type from the workflow initial JSON payload in the shared engine (mirroring in-process delivery) instead of string-coercing it per host. Untrusted input is stripped of pickle markers before reconstruction to prevent deserialization RCE.
* fix(samples): type durable workflow start executors for reconstructed input
The HITL and parallel workflow samples no longer hand-parse a JSON string. Their start executors now declare their real input type (ContentSubmission / DocumentInput), which the durable engine reconstructs from the client payload before delivery.
* test(durabletask): unit coverage for registration, client, worker, and input coercion
Add unit tests for plan_workflow_registration, DurableWorkflowClient, the agent-executor identity registration (entity keyed by executor id), and the typed initial-input coercion including pickle-marker neutralization.
* test(durabletask): HITL and parallel durable workflow integration tests
Add an integration test for the standalone durabletask HITL workflow sample via a new workflow_client fixture. Re-enable the Azure Functions parallel workflow test, consolidated into one end-to-end case so the work-stealing xdist scheduler cannot spawn multiple func hosts for this sample.
* refactor(durabletask): group workflow modules into a _workflows subpackage
Move the eight workflow modules into a private _workflows/ subpackage and drop the redundant _workflow_ prefix (orchestrator.py, registration.py, activity.py, client.py, context.py, dt_context.py, runner_context.py, serialization.py). The public API and __all__ are unchanged; only direct internal-module imports were repointed (package __init__, the worker, the azure-functions shared shim, and the affected unit tests).
* fix(durabletask): harden workflow type resolution and HITL response handling
- resolve_type returns only real classes (avoids issubclass TypeError in reconstruct_to_type)
- re-wait on HITL responses rejected by pickle-marker sanitization instead of dropping the request and losing the run
- American spelling in strip_pickle_markers docstring
- unit tests for resolve_type
* fix(durabletask): treat async edge conditions as not-matched on the synchronous host
The durabletask orchestrator evaluates edge conditions synchronously and does not support async edge conditions. Such an edge is now treated as not matched (the edge is not traversed) rather than assuming a result. Adds unit coverage; full async-condition support will be handled separately.
* fix(durabletask): reconstruct typed workflow outputs at the host boundary
await_workflow_output and the Azure Functions status endpoint now decode the checkpoint-encoded outputs the shared activity produces, via a shared deserialize_workflow_output helper. The client returns the original objects; the AF endpoint emits clean domain JSON instead of checkpoint-marker dicts, keeping the two hosts consistent.
* fix(durabletask): address review findings on workflow hosting
- AF: register workflow agents through add_agent(entity_id=...) so they remain tracked in app.agents / get_agent() (restores documented behavior) while keying by the executor id the orchestrator dispatches to; mirrors DurableAIAgentWorker.add_agent.
- async bridge: treat the shared loop as reusable only while its backing thread is alive, so a dead loop thread is replaced instead of hanging future.result() forever.
- client: add get_runtime_status; the standalone HITL sample now stops polling and reports the real terminal state instead of a generic timeout.
- tests: guard send_hitl_response pickle-marker stripping and add get_runtime_status coverage.
* fix(durabletask): wait indefinitely for HITL responses, matching core
The durable workflow host previously raced HITL responses against a 72h timer and failed the orchestration on elapse. MAF core's request_info has no timeout concept (it waits for the response), and the .NET durable host waits too, so the durable Python host now does the same: it stays paused until a response arrives. Removes the hitl_timeout_hours parameter and DEFAULT_HITL_TIMEOUT_HOURS constant from both hosts. A configurable timeout can be added later once core defines the contract (what happens on elapse).
* feat(durabletask): typed workflow event streaming and async client API
Add a brokerless workflow event stream to the durable host. Each non-agent executor runs inside a durable activity that captures its real WorkflowEvents (with data payloads); the orchestrator replays them into the orchestration custom status after each superstep, and the client streams them back as typed WorkflowEvent objects with reconstructed data. Agent executors contribute synthesized invoked/completed lifecycle events.
Add async client methods run_workflow (start with optional wait) and stream_workflow (typed event iterator), plus is_replaying plumbing through the orchestration context protocol and both host adapters so live status is published only on non-replay execution.
* docs(samples): standalone durabletask workflow streaming sample
Add sample 10_workflow_streaming demonstrating the async DurableWorkflowClient API on a standalone Durable Task worker: run_workflow(wait=False) to start without blocking, then stream_workflow to consume typed WorkflowEvent objects as a WriterAgent -> ReviewerAgent -> publish pipeline runs.
* refactor(durabletask): internal-only checkpoint codec and host-scoped workflow event streaming
Two related hardening changes to the durable workflow hosting layer, plus a
rebase-restored improvement.
Internal-only serialization codec (MSRC follow-up):
- Rename serialize_value/deserialize_value -> _serialize_value/_deserialize_value
in the shared durabletask serialization module and update all call sites, so the
pickle-backed checkpoint codec is unambiguously framework-internal. Untrusted
input is still neutralized with strip_pickle_markers at the HTTP boundary.
- Remove the duplicate agent_framework_azurefunctions._serialization module and
import strip_pickle_markers from the shared durabletask module instead. Move its
unique serialization/strip-marker tests into the durabletask test suite.
Scope workflow event streaming to hosts that can carry it:
- Add WorkflowOrchestrationContext.supports_event_streaming. The standalone
DurableTask host returns True (no custom-status size cap, has a stream_workflow
consumer); the Azure Functions host returns False.
- The orchestrator now accumulates and publishes the WorkflowEvent timeline to the
orchestration custom status only when the host supports streaming. On Azure
Functions the custom status returns to its pre-streaming shape
({state[, pending_requests]}), which fixes orchestrator failures with
"The size of the JSON-serialized payload must not exceed 16 KB" and stops leaking
pickle markers into the HTTP status response. The Azure Functions status endpoint
never consumed the event stream.
Workflow start endpoint:
- Accept text/plain raw request bodies (fall back from get_json to the raw body),
restoring an improvement from main that the rebase conflict resolution dropped.
* fix(azurefunctions): scope workflow status/respond endpoints to the workflow orchestrator
The workflow/status/{instanceId} and workflow/respond/{instanceId}/{requestId}
HTTP endpoints resolved durable instances by ID only. The durable client looks up
IDs across every orchestration in the task hub (agent entities, any
user-registered orchestrations, and other apps sharing the hub), so a caller
holding one instance ID could read another orchestration's status -- including
pending HITL request payloads -- or inject external events into it.
Add AgentFunctionApp._is_workflow_orchestration() and gate both endpoints on it:
an instance whose orchestration name is not WORKFLOW_ORCHESTRATOR_NAME now returns
404 instead of leaking state or accepting events. send_hitl_response now fetches
the orchestration status and validates ownership before raising the external
event. Legitimate workflow instances are unaffected.
Mirrors the .NET fix in PR #6608.
* fix(durabletask): resolve CI typing failures
- serialization: rename _serialize_value/_deserialize_value back to
serialize_value/deserialize_value to follow the package convention for
cross-module internal helpers (matches strip_pickle_markers, resolve_type).
The leading underscore tripped pyright reportPrivateUsage on cross-module
imports under the strict source gate; internal-only status is preserved by
not exporting them from the public API.
- Remove type-ignore comments pyright flags as unnecessary
(reportUnnecessaryTypeIgnoreComment) in _worker.py, orchestrator.py,
serialization.py.
- test_08_dt_workflow: add AgentClientFactoryProtocol and annotate the
agent_client_factory fixture as type[AgentClientFactoryProtocol] (matching
test_01-07) so mypy/ty stop reporting "type has no attribute create".
- samples (08_workflow, 09_workflow_hitl): pass structured output via
FoundryChatOptions[Any](response_format=...) instead of a plain dict so the
samples pyright (basic) config accepts default_options.
---------
Co-authored-by: Gavin Aguiar <80794152+gavin-aguiar@users.noreply.github.com>
|
||
|
Ben Thomas
|
54a30571aa |
Dotnet - Add support for Foundry Adaptive evals (#6267)
* .NET: feat(evals): RubricScore type + EvalScoreResult.Dimensions Adds the core rubric-evaluator surface that mirrors the Python work in PR #6101 (commit e45b934cc). Provider-agnostic types only — no Foundry coupling. Subsequent commits will wire these into FoundryEvals. - RubricScore: per-dimension score record (Id, Score?, Applicable, Weight, Reason). - EvalScoreResult.Dimensions: optional init-only list of RubricScore. Null for non-rubric (built-in) evaluators. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * .NET: feat(evals): GeneratedEvaluatorRef + assertion helpers Adds the provider-agnostic surface for referencing a pre-existing rubric evaluator and gating CI on per-item / per-dimension thresholds. Mirrors Python PR #6101 commits e5830dd7f (ref type) and 4bc60462d (asserts). - GeneratedEvaluatorRef: name + optional version/display-name, plus a Latest(name) factory for versionless refs (discouraged for CI; consumers should warn at run time). - AgentEvaluationResults.AssertScoreAtLeast: walks DetailedItems[].Scores, optionally filtered by evaluator name, recurses into SubResults. - AgentEvaluationResults.AssertDimensionScoreAtLeast: walks each score's Dimensions list, skips non-applicable dimensions by default, supports requireApplicable to flip that, recurses into SubResults. - AgentEvaluationResults.AssertNoFailedItems: walks DetailedItems for fail/error statuses, recurses into SubResults. All helpers throw InvalidOperationException (matches existing AssertAllPassed). Truncates offender lists to the first 5 with a '+N more' suffix to keep CI output readable, mirroring the Python helpers. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * .NET: feat(foundry-evals): accept GeneratedEvaluatorRef in evaluators= Adds FoundryEvaluatorSpec, a readonly-struct union with implicit conversions from both string and GeneratedEvaluatorRef so call sites can mix built-in evaluator names with rubric evaluator references: var evals = new FoundryEvals( projectClient, model, new GeneratedEvaluatorRef("policy-rubric", "3"), FoundryEvals.Relevance, FoundryEvals.Coherence); FoundryEvals constructors (3 overloads), EvaluateTracesAsync, and EvaluateFoundryTargetAsync now take FoundryEvaluatorSpec[]/params instead of string[]/params. Existing call sites using string literals or string[] keep working unchanged via implicit conversion. FoundryEvalConverter.BuildTestingCriteria emits the documented Foundry wire format for rubric refs: { "type": "azure_ai_evaluator", "name": <DisplayName ?? Name>, "evaluator_name": <Name>, "evaluator_version": <Version>, // omitted when null "initialization_parameters": { "deployment_name": <model> }, "data_mapping": { conversation arrays, optional tool_definitions } } WireTestingCriterion gains an optional EvaluatorVersion field. Rubric refs are preserved through FilterToolEvaluators (tool-aware but not tool-required) and ignored by FindMissingGroundTruthEvaluators. A versionless ref emits a Trace.TraceWarning at criterion-build time so CI authors notice the floating version (mirrors the Python warning). Adds 6 new Foundry unit tests (3 BuildTestingCriteria rubric paths, 1 FindMissingGroundTruthEvaluators, 1 FilterToolEvaluators preservation, 1 mixed-order). 369/369 Foundry tests pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * .NET: feat(foundry-evals): parse rubric dimension_scores into RubricScore Adds FoundryEvals.ParseRubricScores, called per result inside ParseDetailedItem. Each EvalScoreResult now populates Dimensions when the evaluator's sample carries a rubric breakdown. Accepts three shapes for forward compatibility with provider SDK iterations: 1. sample.properties.dimension_scores (canonical Foundry runtime shape) 2. sample.properties.rubric_scores (preview/legacy key) 3. top-level sample.dimension_scores / sample.rubric_scores (defensive fallback) Entries missing 'id', 'weight', or 'applicable' are skipped without invalidating well-formed siblings. Non-applicable dimensions may omit 'score' (parsed as null). Adds 6 unit tests covering canonical and legacy keys, top-level fallback, no-match returns null, malformed-entry skipping, and the non-applicable null-score path. 375/375 Foundry tests pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * .NET: feat(samples): Evaluation_FoundryRubric end-to-end sample Adds dotnet/samples/05-end-to-end/Evaluation/Evaluation_FoundryRubric mirroring the Python evaluate_with_rubric_sample.py: - Fetches a pre-existing Foundry agent via AgentAdministrationClient (GetAgentAsync for latest, GetAgentVersionAsync when FOUNDRY_AGENT_VERSION is pinned). - References a rubric evaluator by GeneratedEvaluatorRef(name, version); falls back to GeneratedEvaluatorRef.Latest(name) with the documented floating-version warning. - Mixes the rubric with FoundryEvals.Relevance and FoundryEvals.Coherence in a single FoundryEvals run (implicit string-and-ref conversion). - Prints per-dimension breakdowns from EvalScoreResult.Dimensions for each item. - Demonstrates a CI quality gate with AssertDimensionScoreAtLeast("general_quality", 3.0). Documents the FOUNDRY_PROJECT_ENDPOINT footgun (must be project-scoped URL .../api/projects/<project>, not the bare Azure OpenAI endpoint) and the Eval-Definition-vs-Rubric-Evaluator distinction in the README. Ships a .env.example with the FOUNDRY_* variables. Registers the project in agent-framework-dotnet.slnx and cross-links from the sibling Evaluation_Multimodal / Evaluation_ExpectedOutputs READMEs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(foundry-evals): harden FoundryEvals public surface for review Address PR #6267 review comments on the .NET FoundryEvals integration: - Add source-compat overloads accepting `string[] evaluators` for `FoundryEvals` ctor, `EvaluateTracesAsync`, and `EvaluateFoundryTargetAsync` so existing callers passing string arrays keep compiling unchanged. New overloads forward via a private `ToSpecs` helper that wraps each name through the implicit `string -> FoundryEvaluatorSpec` conversion. - Guard against `default(FoundryEvaluatorSpec)` entries (both `BuiltinName` and `GeneratedRef` null) that would NRE the downstream converter. Adds `FoundryEvaluatorSpec.IsValid` / `EnsureValid` plus an internal `EnsureAllSpecsValid` helper, wired into the main ctor and both static evaluation entry points. - Add 6 unit tests covering the new validation surface. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(sample): set ExitCode=1 when rubric dimension gate trips PR #6267 review comment: the FoundryRubric sample swallowed the AssertDimensionScoreAtLeast failure, so a CI run that included it as a quality gate would still exit 0 even when the rubric regressed. Set `System.Environment.ExitCode = 1` in the catch so CI fails while still letting the rest of the sample's logging complete cleanly. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(foundry-evals): search typed Sample directly for rubric scores PR #6267 review comment: `_extract_rubric_scores` only searched the `properties` dict when the sample exposed one. When the Azure AI Projects typed SDK returns a Sample object that puts `dimension_scores` / `rubric_scores` directly on the instance (no `properties` wrapper), we missed them and surfaced no per-dimension scores. Add an `else: containers.append(sample)` branch so non-dict typed samples are also inspected for the score keys. Covered by two new tests: one with `dimension_scores` directly on a typed Sample without a `properties` wrapper, and one with the legacy `rubric_scores` key in the same shape. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * test(evals): cover assert_score_at_least and assert_no_failed_items PR #6267 review comments: both assertion helpers shipped without unit tests. Add `TestAssertScoreAtLeast` (above threshold, below w/ offenders, evaluator filter, sub_results recursion) and `TestAssertNoFailedItems` (all passing, failed/errored statuses, sub_results recursion) with a shared `_score_results` fixture builder. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docs(samples): remove dead rubric-evaluator doc link from FoundryRubric sample The Azure AI Foundry rubric evaluator concept doc page has not yet been published, so the link in the sample README and Program.cs comment 404s. Drop the references until the upstream doc is live. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Address PR 6267 review nits Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Ben Thomas <25218250+alliscode@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> |
||
|
westey
|
dc445592ed |
Python: [BREAKING] Port FileMemoryProvider and integrate FileMemoryProvider & FileAccess into the harness agent (#6547)
* Port FileMemoryProvider to python and integrate it and FileAccessProvider into the harness * Address PR comments * Address PR comments * Create FileSystemAgentFileStore root lazily on first write Construction no longer calls mkdir, so building a store (and therefore a default create_harness_agent, which wires default file-memory and file-access stores under the CWD) performs no filesystem writes and does not fail in read-only working directories. The root directory is created on the first write_file / create_directory call; all read/list/search operations already tolerate a missing root. Updates docstrings and adds a regression test. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix typing * Fixing typing errors --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Eduard van Valkenburg
|
6e95517659 |
Python: Split type checkers by target (pyright source, 5 checkers on tests/samples) (#6443)
* Python: Split type checkers by target (pyright source, 5 checkers on tests/samples) Rework the typing setup along the lines of the 'too many type checkers' approach: - Pyright (strict) is now the sole source-code type checker; mypy is removed from source and its [tool.mypy] block becomes a relaxed profile used only for tests/samples. - Tests are checked by all five checkers (pyright relaxed, mypy, pyrefly, ty, zuban); samples by pyright, pyrefly, and ty. All run in a relaxed/ basic profile so authors aren't forced into over-annotation. - Add pyrightconfig.tests.json and bump sample pyright configs to basic. - Unify test/sample typing onto the same parallel fan-out used by source pyright via run_command_items in task_runner.py. - Make version-conditional imports symmetric: keep or drop the '# type: ignore' on both branches so results match across interpreter versions (local vs CI). - Update SKILL.md, DEV_SETUP.md, and CODING_STANDARD.md for the five gating checkers and pyright on source+tests+samples. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Python: Fix merge regressions from main (typing + runtime) Merging main into the type-checker split branch surfaced regressions that the new five-checker test suite and unit tests caught: Runtime fixes: - anthropic: restore the dropped `cache_read_input_token_count` mapping in _parse_usage_from_anthropic (lost during merge conflict resolution). - gemini: _get_function_calling_mode test helper returned str(enum) ('FunctionCallingConfigMode.AUTO') instead of the enum value ('AUTO'). - openai: _response_id_from_token test helper was an infinite self-recursion; return token['response_id']. - orchestrations: reset output_events per approval iteration so the terminal output assertion counts only the final run. - core: drop a stale duplicate harness test whose message ('non-negative') contradicted the source ('positive'). - purview: import PolicyLocation/PolicyScope/ProtectionScopeActivities/ ExecutionMode used by the processor tests. Type-checker fixes (tests, relaxed profile): - core: pyright/mypy/pyrefly/ty/zuban green-ups across the harness, MCP, observability and types tests. - anthropic/openai: route provider-namespaced UsageDetails keys through a dict cast (extra_items TypedDict unsupported by mypy/ty). - purview: typed model constructors and cache-mock casts. - ag-ui: annotate WorkflowContext[Any, Any] so yield_output accepts test payloads, guard Optional forwarded_props, and ty-ignore intentional bad args. Source pyright (sole source checker) flagged unnecessary ignores newly introduced by merged code in core _tools.py and declarative _declarative_base.py. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Python: Isolate per-package mypy cache in test-typing fan-out The parallel test-typing fan-out runs many mypy processes concurrently, all defaulting to a single shared ./.mypy_cache. Concurrent writes corrupt the cache and mypy aborts with INTERNAL ERROR (intermittently, depending on worker timing) -- which is why CI's Test Typing job failed on a shifting set of packages while a single-package run was fine. Give each mypy invocation an isolated cache dir keyed by its target paths so incremental caching still works per package without races. Other checkers (zuban/pyrefly/ty/pyright) maintain their own caches and are unaffected. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Python: Make lab pyright-only on source (drop source mypy) Lab was the last package still running mypy on its source code, requiring mypy-only `# type: ignore` comments that pyright (the sole source checker everywhere else) flags as unnecessary. Align lab with the rest of the monorepo: - Remove the lab source mypy poe tasks (mypy-gaia/lightning/tau2) and the now-dead strict [tool.mypy] config block. - Drop the 'Run lab mypy' CI step; lab source is type-checked by pyright only. Lab tests remain covered by the workspace test-typing fan-out (mypy, pyrefly, ty, zuban, pyright over tests using the relaxed root config). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Python: Fix test-typing regressions from latest main merge A fresh merge from main brought in new test code never run under the five-checker test-typing suite. Green up across the affected packages: - core: narrow Optional span.attributes with 'and' guards in span filters and assert+cast the json.loads(...attributes[...]) reads (test_observability); match the existing as_agent ignore on the protocol-typed fixture (test_clients). - openai: align new streaming tests with the established chat_options dict pattern (ChatOptions TypedDict isn't assignable to dict), route Optional .annotations[0] access through a small _first_annotation helper (mirrors the file's assert-not-None convention), and annotate a mapped ResponseStream. - foundry_hosting: annotate error: dict[str, Any] = body.get(...) or {} (zuban needs the annotation). - foundry: narrow ignores for the live AIProjectClient credential arg (pyrefly) and connections.get_default (zuban) SDK type gaps. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * updated pyright version * pyright fix * Python: Fix source typing for pyright 1.1.410 Pyright 1.1.410 tightened several checks. Apply the same source fixes as upstream PR #6275: - anthropic: import AsyncAnthropicBedrock from anthropic.lib.bedrock and AsyncAnthropicVertex from anthropic.lib.vertex (no longer re-exported from the anthropic top-level package -> reportPrivateImportUsage). - core _types.py: cast the transform-hook result to UpdateT (reportAssignmentType). - core _workflows/_events.py: annotate the @contextmanager helper as Generator[None] instead of Iterator[None] (reportDeprecated). - redis: build the combined filter expression with an explicit loop instead of reduce(and_, ...), which pyright could no longer fully type (drops the now unused functools.reduce / operator.and_ imports). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Python: Accept plain-text body in Azure Functions workflow/run endpoint The workflow_orchestrator already accepts plain strings as well as JSON objects via context.get_input(), but the start_workflow_orchestration HTTP handler only accepted JSON and returned 400 for any non-JSON body. This made the functions integration tests that POST text/plain to /api/workflow/run (e.g. test_09_workflow_shared_state) fail consistently with 400 != 202. Fall back to the raw request body (decoded as UTF-8) when the body is not JSON, rejecting only a truly empty body. The JSON path is unchanged. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Evan Mattson
|
b55992bb67 |
Bump Python package versions for 1.9.0 release (#6583)
Selective, CHANGELOG-driven version bumps for the 2026-06-18 release. Released tier: agent-framework-core and the root agent-framework go to 1.9.0 (minor). Core ships new public APIs (agent-loop middleware, tool-approval middleware and harness integration, shell-tool harness integration, AG-UI thread snapshot persistence, context-provider telemetry) plus two behavioral breaking changes on evolving surfaces: MCP sampling now denies server-initiated requests by default, and the FileAccess tools were aligned with the .NET implementation. These are treated as within-1.x changes because every package caps core at <2; a major bump would require rewriting those caps. The foundry and openai packages go to 1.8.2 (patch, bug fixes only). The root agent-framework-core[all] pin was moved to 1.9.0 in lockstep with core. Release-candidate tier: ag-ui to 1.0.0rc5 and declarative to 1.0.0rc2 for their respective changes. orchestrations is promoted to stable 1.0.0; PACKAGE_STATUS and the README install hint were updated accordingly. Prerelease tier (new Pacific date stamp 260618): anthropic (beta), azure-contentunderstanding (alpha) and foundry-hosting (alpha). No beta cohort bump was applied; only packages with changes this cycle were stamped. Dependency floors: following the established convention, the core floor was raised to >=1.9.0 on every non-core package bumped this cycle, preserving the existing <2 upper bound. Also resolves two pre-existing failures in the dependency-bounds validator that are unrelated to the version bumps. Hosted-environment detection now catches a bare ImportError so optional Foundry hosting probing cannot crash user-agent setup. The harness shell-tool integration, which lazily imports the separate agent-framework-tools package to avoid a circular runtime dependency, is now type-checked and tested in isolated environments via a core dev dependency-group, with the shell-tool tests guarded to skip when that package is absent. |
||
|
Eduard van Valkenburg
|
4ff952e100 |
Python: Capture context provider instructions in agent telemetry (#6515)
* Fix agent instructions telemetry Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Simplify agent instructions telemetry guard Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix observability mypy cast Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Eduard van Valkenburg
|
8e10c0399a |
Python: Remove unsupported as_agent function_invocation_configuration (#6520)
* Remove unsupported as_agent config parameter Fixes #6313 Remove the unsupported function_invocation_configuration parameter from BaseChatClient.as_agent(), which currently forwards an invalid kwarg into Agent.__init__(). This keeps the existing TypeError behavior for callers but changes the error source to the public API boundary, which we do not consider a breaking change. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix sample --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
westey
|
0db9305625 |
Python: Integrate tool approval into the harness (#6522)
* Integrate auto tool-approval feature into harness * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Rename disable_tool_approval to disable_tool_auto_approval Addresses PR review feedback that the parameter name was unclear. The flag toggles the auto/standing tool-approval middleware. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Eduard van Valkenburg
|
d7e8d2206d |
Python: Fix Python OTel usage detail attributes (#6493)
* fix python otel usage detail attributes Map cached/read/reasoning usage detail fields to standard OTel GenAI attributes while preserving provider-specific legacy keys. Add focused coverage for direct response spans, aggregated agent spans, and provider usage parsing. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * address usage detail review feedback Omit missing OpenAI Responses usage detail counts while preserving zero-valued counts. Record zero-valued token usage in OTel histograms and add regression coverage. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
westey
|
d7027fc1f9 |
Python: [BREAKING] Align FileAccess tools with .NET — directory discovery and recursive search (#6476)
* Align FileAccess tools with .Net; add directory discovery and recursive search * Fix choices field description: spacing, line length, grammar Addresses PR review: separate concatenated string literals with proper spacing/newlines, wrap lines under the 120-char Ruff limit, and fix "doesn't" -> "don't". Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR comments --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Eduard van Valkenburg
|
1acd242550 |
Python: Add AgentLoopMiddleware for re-running agents in a loop (#6174)
* Python: Add AgentLoopMiddleware for re-running agents in a loop Add `AgentLoopMiddleware`, an `AgentMiddleware` that re-runs the wrapped agent in a loop. A single configurable class covers three common patterns, each with a convenience classmethod factory: - Ralph loop (`.ralph(...)`): no exit criteria, with feedback tracking (`record_feedback`/`progress`), progress injection (`inject_progress`), optional fresh context per iteration (`fresh_context`), and an early-stop completion signal (`is_complete`). - Predicate (`.with_predicate(...)`): loop while a `should_continue` callable returns True (e.g. paired with `todos_remaining`/`background_tasks_running`). - Judge (`.with_judge(...)`): a second chat client decides whether the original request was answered, using a `JudgeVerdict` structured-output response. The loop also auto-resolves pending function-approval / user-input requests via an `on_approval_request` callable (bounded by `max_approval_rounds`), and the next iteration's input is controlled by `next_message`. Supports both streaming and non-streaming runs. Exports `AgentLoopMiddleware`, `JudgeVerdict`, `todos_remaining`, and `background_tasks_running`. Adds tests, a sample, and docs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Python: Refine AgentLoopMiddleware API and sample - with_judge: add criteria list with {{criteria}} templating into judge instructions plus an agent-side instruction; add fresh_context, additional judge feedback relay; default judge max_iterations. - should_continue is now required and positional; supports (bool, str|None) feedback tuples surfaced to next_message/record_feedback via feedback kwarg. - Judge forwards full multi-modal request and response messages. - Default max_iterations=10 (explicit None = unbounded); removed is_complete and Ralph terminology; ShouldContinueResult is a real TypeAlias. - Sample: stream all loops, print iteration counts via injected user-block boundaries (robust to function calling), <role>: content formatting, per-method expected output, and a looping todo sample. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Python: Fix CI checks for AgentLoopMiddleware - Resolve pyright errors in _loop.py: drop the always-true final_result None check (the while loop always assigns it) and cast finish_reason to the AgentResponse constructor's expected type. - Apply pyupgrade --py310-plus: import TypeAlias from typing. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Python: Resolve mypy/pyright disagreement on finish_reason pyright infers AgentResponse.finish_reason as including str and rejects the direct assignment, while mypy considers a cast redundant. Drop the cast and suppress only pyright with a targeted reportArgumentType ignore, satisfying both type checkers. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Python: Add todo+judge AgentLoopMiddleware sample Add a second AgentLoopMiddleware sample that composes two criteria in one should_continue predicate: a TodoProvider check (evaluated first) and a report-style judge chat client (evaluated once todos are complete) that grades the assembled report against shared requirements. Register it in the middleware samples README. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Python: Compose todo+judge loops as two middleware Rework the todo+judge sample to compose two AgentLoopMiddleware on the agent itself (middleware=[judge_loop, todo_loop]) instead of a single hand-written predicate. The inner todos_remaining loop drafts the report todo-by-todo and the outer with_judge loop re-runs it until an editor chat client judges the report publication-ready, reusing the built-in helpers. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Reset session for fresh_context loops via snapshot/restore AgentLoopMiddleware.fresh_context previously only reset context.messages, so with an attached session each iteration still reloaded the local transcript or re-threaded the service-side conversation id and the model saw the accumulated history. Snapshot the session once before the loop (via to_dict) and restore it (from_dict + field copy) between iterations, so every pass starts from the pre-loop baseline. The final iteration's pass is persisted (no restore after the terminating iteration), so a subsequent agent.run continues from there. Removed the obsolete warning, updated docstrings and core AGENTS.md, and added tests: a snapshot/restore round-trip, a session-reset streaming x fresh_context x inject_progress x store matrix across multiple runs and loop iterations, and response_format parsing across the loop. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Updated samples and docstrings --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Evan Mattson
|
76b2b1bf39 |
Python: Add opt-in AG-UI thread snapshot persistence and hydration (#6471)
* feat(ag-ui): add thread snapshot store primitives Key decisions:\n- Introduce an AGUIThreadSnapshot model limited to replayable messages, optional Shared State, and optional interrupt state.\n- Define AGUIThreadSnapshotStore as an async protocol keyed by explicit Snapshot Scope and AG-UI Thread id.\n- Add InMemoryAGUIThreadSnapshotStore as memory-only, latest-only, bounded local/demo/test storage; no file-backed store is introduced.\n- Require snapshot_scope_resolver whenever an endpoint is configured with a snapshot store, including pre-wrapped runners, so thread ids are not authorization boundaries.\n\nFiles changed:\n- packages/ag-ui/agent_framework_ag_ui/_snapshots.py\n- packages/ag-ui/agent_framework_ag_ui/__init__.py\n- packages/ag-ui/agent_framework_ag_ui/_agent.py\n- packages/ag-ui/agent_framework_ag_ui/_workflow.py\n- packages/ag-ui/agent_framework_ag_ui/_endpoint.py\n- packages/core/agent_framework/ag_ui/__init__.py\n- packages/core/agent_framework/ag_ui/__init__.pyi\n- packages/ag-ui/tests/ag_ui/test_snapshots.py\n- packages/ag-ui/tests/ag_ui/test_endpoint.py\n- packages/ag-ui/tests/ag_ui/test_public_exports.py\n- packages/ag-ui/AGENTS.md\n\nVerification:\n- uv run pytest packages/ag-ui/tests/ag_ui/test_snapshots.py packages/ag-ui/tests/ag_ui/test_public_exports.py packages/ag-ui/tests/ag_ui/test_endpoint.py::test_endpoint_requires_snapshot_scope_resolver_when_store_configured packages/ag-ui/tests/ag_ui/test_endpoint.py::test_endpoint_accepts_snapshot_store_with_scope_resolver -q\n- uv run pytest packages/ag-ui/tests/ag_ui/test_endpoint.py::test_endpoint_requires_snapshot_scope_resolver_when_store_configured packages/ag-ui/tests/ag_ui/test_endpoint.py::test_endpoint_requires_snapshot_scope_resolver_when_wrapped_runner_has_store packages/ag-ui/tests/ag_ui/test_endpoint.py::test_endpoint_accepts_snapshot_store_with_scope_resolver -q\n- uv run poe syntax -P ag-ui -C\n- uv run poe pyright -P ag-ui\n- uv run poe syntax -P core -C\n- uv run poe pyright -P core\n- uv run poe typing -P ag-ui\n- uv run poe typing -P core\n- uv run poe test -P ag-ui\n- uv run poe check -P ag-ui\n- git diff --check\n- git diff --cached --check\n\nBlockers / next iteration:\n- No blockers. Next slice can use the store contract to capture and hydrate agent snapshots.\n- uv repeatedly refreshed azure-ai-projects in uv.lock during local runs; reverted the generated lockfile churn because this change does not alter dependencies.\n- The poe-check commit hook was skipped after manual verification because it reformatted unrelated core MCP files outside this task. * feat(ag-ui): hydrate agent threads from snapshots Key decisions: - Resolve Snapshot Scope per endpoint request and pass it to the AG-UI runner only when snapshot storage is active. - Treat empty messages with no resume payload as an agent Hydrate Request when a scoped snapshot store is configured, replaying stored Shared State and message snapshots without invoking the wrapped agent. - Save the latest replayable agent message snapshot and Shared State at normal completion under Snapshot Scope plus AG-UI Thread id; no durable or file-backed store is introduced. Files changed: - packages/ag-ui/agent_framework_ag_ui/_agent_run.py - packages/ag-ui/agent_framework_ag_ui/_endpoint.py - packages/ag-ui/agent_framework_ag_ui/_snapshots.py - packages/ag-ui/tests/ag_ui/test_endpoint.py Verification: - uv run pytest packages/ag-ui/tests/ag_ui/test_endpoint.py::test_agent_endpoint_hydrates_stored_thread_snapshot_without_invoking_agent -q - uv run pytest packages/ag-ui/tests/ag_ui/test_endpoint.py::test_agent_endpoint_hydrates_stored_thread_snapshot_without_invoking_agent packages/ag-ui/tests/ag_ui/test_endpoint.py::test_agent_endpoint_hydrates_snapshots_by_scope_and_thread -q - uv run pytest packages/ag-ui/tests/ag_ui/test_endpoint.py::test_endpoint_empty_messages packages/ag-ui/tests/ag_ui/test_endpoint.py::test_agent_endpoint_hydrates_stored_thread_snapshot_without_invoking_agent packages/ag-ui/tests/ag_ui/test_endpoint.py::test_agent_endpoint_hydrates_snapshots_by_scope_and_thread -q - uv run poe syntax -P ag-ui -C - uv run poe pyright -P ag-ui - uv run poe typing -P ag-ui - uv run poe test -P ag-ui - uv run poe check -P ag-ui - git diff --check - git diff --cached --check Blockers / next iteration: - No blockers. Next slice can reconstruct normal new-user agent turns from stored snapshots. - uv repeatedly refreshed azure-ai-projects in uv.lock during local runs; reverted the generated lockfile churn because this change does not alter dependencies. - The poe-check commit hook was skipped after manual verification because it refreshed unrelated uv.lock dependency resolution. * feat(ag-ui): reconstruct agent turns from snapshots Key decisions: - Load scoped thread snapshots for non-hydrate agent requests only when snapshot storage is active and no resume payload is present. - Rebuild prior AG-UI history from stored snapshot messages, preserving the incoming new user suffix and treating stored snapshot content as authoritative over conflicting prior client history. - Merge stored Shared State with request state overrides before schema defaults and existing state-context injection. Files changed: - packages/ag-ui/agent_framework_ag_ui/_agent_run.py - packages/ag-ui/tests/ag_ui/test_endpoint.py Verification: - uv run pytest packages/ag-ui/tests/ag_ui/test_endpoint.py::test_agent_endpoint_prepends_stored_snapshot_for_new_user_turn -q - uv run pytest packages/ag-ui/tests/ag_ui/test_endpoint.py::test_agent_endpoint_deduplicates_full_history_and_merges_fresh_state -q - uv run pytest packages/ag-ui/tests/ag_ui/test_endpoint.py::test_endpoint_empty_messages packages/ag-ui/tests/ag_ui/test_endpoint.py::test_agent_endpoint_hydrates_stored_thread_snapshot_without_invoking_agent packages/ag-ui/tests/ag_ui/test_endpoint.py::test_agent_endpoint_hydrates_snapshots_by_scope_and_thread packages/ag-ui/tests/ag_ui/test_endpoint.py::test_agent_endpoint_prepends_stored_snapshot_for_new_user_turn packages/ag-ui/tests/ag_ui/test_endpoint.py::test_agent_endpoint_deduplicates_full_history_and_merges_fresh_state -q - uv run pytest packages/ag-ui/tests/ag_ui/test_endpoint.py -q - uv run poe syntax -P ag-ui -C - uv run poe pyright -P ag-ui - uv run poe test -P ag-ui - uv run poe check -P ag-ui - uv run poe typing -P ag-ui - git diff --check - git diff --cached --check Blockers / next iteration: - No blockers. Next slice can enable workflow AG-UI Thread Snapshot persistence and hydration. - uv repeatedly refreshed azure-ai-projects in uv.lock during local runs; reverted the generated lockfile churn because this change does not alter dependencies. - The poe-check commit hook was skipped after manual verification because it refreshes unrelated uv.lock dependency resolution. * feat(ag-ui): hydrate workflow threads from snapshots Key decisions: - Handle workflow Hydrate Requests before resolving or invoking the wrapped workflow when snapshot storage and Snapshot Scope are active. - Capture only replayable workflow protocol data: workflow-emitted state snapshots, workflow-emitted message snapshots, and synthesized messages from text/tool output. - Keep workflow snapshot capture inactive without configured persistence, and skip saving snapshots when the workflow stream emits RUN_ERROR. Files changed: - packages/ag-ui/agent_framework_ag_ui/_workflow.py - packages/ag-ui/tests/ag_ui/test_endpoint.py Verification: - uv run pytest packages/ag-ui/tests/ag_ui/test_endpoint.py::test_workflow_endpoint_hydrates_emitted_snapshots_without_invoking_workflow packages/ag-ui/tests/ag_ui/test_endpoint.py::test_workflow_endpoint_hydrates_synthesized_text_and_tool_snapshot -q - uv run pytest packages/ag-ui/tests/ag_ui/test_endpoint.py -q - uv run pytest packages/ag-ui/tests/ag_ui/golden/test_scenario_workflow.py -q - uv run poe syntax -P ag-ui -C - uv run poe pyright -P ag-ui - uv run poe test -P ag-ui - uv run poe typing -P ag-ui - uv run poe check -P ag-ui - git diff --check - git diff --cached --check Blockers / next iteration: - No blockers. Next slice can preserve interruption state and protect snapshots on errors across agent and workflow endpoints. - uv repeatedly refreshed azure-ai-projects in uv.lock during local runs; reverted the generated lockfile churn because this change does not alter dependencies. - The poe-check commit hook was skipped after manual verification because it refreshes unrelated uv.lock dependency resolution. * feat(ag-ui): preserve interrupted thread snapshots Key decisions: - Capture workflow RUN_FINISHED interrupt metadata in replayable AG-UI Thread Snapshots so Hydrate Requests can restore pending workflow actions without invoking or resuming the workflow. - Keep failed agent and workflow runs from replacing the last good snapshot; RUN_ERROR streams leave the previous snapshot available for hydration. - Verify interruption hydration through endpoint-level AG-UI streams for both agent and workflow wrappers, including Shared State replay and no wrapped runner invocation. Files changed: - packages/ag-ui/agent_framework_ag_ui/_workflow.py - packages/ag-ui/tests/ag_ui/test_endpoint.py Verification: - uv run pytest packages/ag-ui/tests/ag_ui/test_endpoint.py::test_workflow_endpoint_hydrates_interrupted_thread_without_invoking_workflow -q - uv run pytest packages/ag-ui/tests/ag_ui/test_endpoint.py::test_agent_endpoint_hydrates_interrupted_thread_without_invoking_agent packages/ag-ui/tests/ag_ui/test_endpoint.py::test_agent_endpoint_run_error_does_not_overwrite_previous_snapshot packages/ag-ui/tests/ag_ui/test_endpoint.py::test_workflow_endpoint_hydrates_interrupted_thread_without_invoking_workflow packages/ag-ui/tests/ag_ui/test_endpoint.py::test_workflow_endpoint_run_error_does_not_overwrite_previous_snapshot -q - uv run pytest packages/ag-ui/tests/ag_ui/test_endpoint.py -q - uv run pytest packages/ag-ui/tests/ag_ui/golden/test_scenario_workflow.py -q - uv run poe syntax -P ag-ui -C - uv run poe pyright -P ag-ui - uv run poe test -P ag-ui - uv run poe typing -P ag-ui - uv run poe check -P ag-ui - git diff --check - git diff --cached --check Blockers / next iteration: - No blockers. Next slice can document AG-UI Thread Snapshot security and usage. - uv repeatedly refreshed azure-ai-projects in uv.lock during local runs; reverted the generated lockfile churn because this change does not alter dependencies. - The poe-check commit hook was skipped after manual verification because it refreshes unrelated uv.lock dependency resolution. * docs(ag-ui): document thread snapshot security Key decisions: - Document AG-UI Thread Snapshot persistence as opt-in and disabled unless a snapshot_store is configured. - Place Snapshot Scope guidance next to endpoint authentication guidance, making clear that AG-UI Thread ids identify threads but do not authorize snapshot access. - Describe built-in storage as in-memory only, process-local, latest-only, and not durable production storage; durable stores remain app-owned implementations of AGUIThreadSnapshotStore. - Call out snapshot confidentiality impact and that no file-backed AG-UI snapshot store is provided. Files changed: - packages/ag-ui/README.md Verification: - uv run python scripts/check_md_code_blocks.py packages/ag-ui/README.md --no-glob - git diff --check - git diff --cached --check - commit hook without SKIP ran changed-package lint/format and AG-UI README markdown-code-lint successfully before stopping because uv.lock was modified - uv run poe markdown-code-lint (failed due existing unrelated packages/mistral/README.md missing agent_framework_mistral import resolution; changed AG-UI README blocks passed) Blockers / next iteration: - No blockers. Local issue/PRD planning artifacts remain uncommitted. - uv refreshed azure-ai-projects in uv.lock during markdown lint and the commit hook; reverted the generated lockfile churn because this documentation change does not alter dependencies. - The poe-check commit hook was skipped after manual verification because it refreshes unrelated uv.lock dependency resolution. * fix(ag-ui): harden thread snapshot persistence edge cases - Persist the completed confirm_changes turn with interrupt=None so hydration no longer replays a stale pending interrupt after the user responds; resume requests prepend stored history so the persisted thread is not truncated. - Defer endpoint default_state application to the runners when snapshot persistence is active, filling only keys missing from both the stored snapshot state and the request state so defaults never reset persisted Shared State. - Always fold the turn's output into the persisted messages snapshot even when the outbound MESSAGES_SNAPSHOT event is suppressed for predictive tools without confirmation. - Load the stored snapshot on workflow follow-up turns, reconstruct full thread history into the run input, and seed the snapshot builder with merged state so saving a new turn no longer replaces prior history. - Move snapshot message reconstruction helpers to _run_common for reuse by the workflow runner; load stored agent snapshots on resume turns for state merge. - Add endpoint regression tests for all four scenarios. * fix(ag-ui): protect snapshot history on resume and harden suffix trust - Prepend stored thread history when persisting snapshots for resume runs on both the agent and workflow paths, so a resumed interrupt no longer overwrites the stored thread with just the resume turn's output. - Filter the incoming message suffix during thread reconstruction: only user turns and tool results answering backend-issued tool calls (stored tool calls or pending interrupts) may extend authoritative history. Client-forged assistant and tool messages are dropped and logged instead of being persisted and replayed. - Close the workflow snapshot builder's tool-call group when a tool result or text message lands, so synthesized transcripts keep tool results adjacent to their tool_calls message and stay valid as provider replay history. - Export DEFAULT_MAX_THREAD_SNAPSHOTS from agent_framework_ag_ui and expose SnapshotScopeResolver through the core ag_ui facade and stub. - Add regression tests for agent and workflow resume history preservation, forged suffix rejection, builder tool-call grouping, and the export surface. * fix(ag-ui): tolerate snapshot save failures and scope workflow cache - Wrap snapshot_store.save() on both the agent and workflow paths so a transient store failure (timeout, connection refused) is logged instead of propagating. Previously a failing save converted an already-streamed successful run into RUN_ERROR, and on the workflow path emitted RUN_ERROR after RUN_FINISHED, violating the single-terminal-event invariant. The previous snapshot stays available for hydration. - Key the workflow_factory instance cache by (snapshot_scope, thread_id). The Snapshot Scope is the authorization boundary, so the same thread id under different scopes no longer shares an in-memory workflow instance. clear_thread_workflow accepts an optional snapshot_scope and clears all scopes for the thread when omitted. - Add tests: save-failure tolerance for agent and workflow endpoints, scope-isolated workflow cache, async snapshot_scope_resolver support, and in-memory store key validation errors. * fix(ci): ignore all dotnet.microsoft.com links in linkspector The existing ignore pattern only matched https://dotnet.microsoft.com/download, but Microsoft sites insert a locale segment between host and path (e.g. /en-us/download/dotnet/10.0), so localized links slip past the pattern and get checked. dotnet.microsoft.com bot-blocks CI link checkers with intermittent 403s across the whole site, which fails markdown-link-check on unrelated pull requests since linkspector scans the entire repository. Ignore the domain wholesale, matching how platform.openai.com is already handled for the same reason. A 403 from bot blocking is indistinguishable from a removed page, so the checker cannot produce a meaningful signal for this domain either way. * ag-ui: simplify raw_messages assignment and drop OrderedDict - Replace list(cast(...)) with a typed annotation for raw_messages (_agent_run.py:866) per review suggestion - Replace OrderedDict with a plain dict in InMemoryAGUIThreadSnapshotStore (_snapshots.py:136); regular dicts are insertion-order-safe since Python 3.7, so OrderedDict is unnecessary. Update _evict_oldest to use next(iter(...)) for FIFO removal instead of popitem(last=False). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review feedback for #2458: review comment fixes --------- Co-authored-by: Copilot <copilot@github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
westey
|
3d5421edc1 |
Python: Integrate shell tool into harness agent (#6451)
* Integrate shell tool into AgentHarness * Validate shell_executor exposes as_function() with a clear TypeError Addresses PR review feedback: a public factory should fail fast with an actionable error rather than a cryptic AttributeError when an incompatible shell_executor is supplied. Validation happens upfront, regardless of whether the client supports shell tools. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Type shell harness params via TYPE_CHECKING import Addresses PR review feedback: type shell_executor and shell_environment_provider_options instead of Any, using a TYPE_CHECKING import from agent_framework_tools.shell. The import never executes at runtime, so there is no circular dependency, and the lazy runtime import of ShellEnvironmentProvider is retained. Since ShellExecutor is a protocol without as_function(), the validated getattr result is invoked directly. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Eduard van Valkenburg
|
df29af611c |
Python: Add tool approval middleware (#6414)
* Add Python tool approval middleware Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix tool approval restored state handling Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Gate hidden approvals on explicit approval responses Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Handle string inputs in approval replay scan Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Cover argument-scoped approval rules Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Refine tool approval state and budgets Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix tool approval PR CI failures Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Revert DevUI Aspire README link change Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
chetantoshniwal
|
4149f24791 |
Python: [Generated by SRE Agent] Fix MCP allowed_tools empty list handling (#6296)
* Fix MCP allowed_tools empty list handling When allowed_tools is set to an empty list [], the falsy check 'if not self.allowed_tools' incorrectly treats it as unconfigured (same as None), causing all tools to be exposed. Change to an explicit 'is None' check so that an empty list correctly results in no tools being allowed. Co-authored-by: Azure SRE Agent <noreply@microsoft.com> * Clarify allowed_tools docstring: None vs [] semantics Per Eduard's review on PR #6296: explicitly document that None exposes all tools and [] exposes none, across all four MCPTool / MCPStdioTool / MCPStreamableHTTPTool / MCPWebsocketTool docstrings. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * allowed_tools docstring: recommend load_tools=False for full disable Per Eduard's follow-up on PR #6296: `load_tools=False` is the cleaner idiom when you don't want to expose any tools. Reframe `allowed_tools=[]` in the docstring as a runtime guard / inspection-only path and cross-reference `load_tools`. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Azure SRE Agent <noreply@microsoft.com> Co-authored-by: Giles Odigwe <79032838+giles17@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
westey
|
8dde9ef627 |
Python: HarnessAgent: Disable compaction when max tokens not provided (#6410)
* HarnessAgent: Disable compaction when max tokens not provided * Fix regression. * Address PR comments * Require max_output_tokens to be positive Reject max_output_tokens=0 (must be positive), mirroring max_context_window_tokens. Addresses PR review feedback. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Giles Odigwe
|
93cbf6b3f0 |
Python: Parse MCP CallToolResult.structuredContent field to prevent tool results returning None (#6421)
* Parse structuredContent from MCP CallToolResult (#3313) The _parse_tool_result_from_mcp method only iterated over the content field from CallToolResult, ignoring the structuredContent field entirely. MCP servers that return JSON data via structuredContent (e.g., Power BI MCP) appeared to return None. Add handling for structuredContent: when present, serialize it as JSON text and append it to the result list. This preserves the data for the LLM while maintaining backward compatibility with existing behavior. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Python: Parse MCP CallToolResult.structuredContent field to prevent tool results returning None Fixes #3313 * Address review feedback: add default=str to json.dumps and remove .checkpoints/ - Add default=str to json.dumps for structuredContent serialization so non-JSON-serializable values (e.g. bytes) degrade gracefully instead of raising TypeError - Remove all .checkpoints/ runtime artifacts from the repository - Add **/.checkpoints/ to .gitignore to prevent future accidental commits - Add test for non-serializable structuredContent values Fixes #3313 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review feedback for #3313: Python: MCP CallToolResult.structuredContent field is not parsed, causing tool results to return None --------- Co-authored-by: Copilot <copilot@github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Eduard van Valkenburg
|
9a56bc9f16 |
Python: [BREAKING] Add sampling guardrails to MCP tools (#6413)
* Add sampling guardrails to MCP tools Add approval, token, and request-count controls to the MCP sampling callback used when an MCPTool is configured with a chat client. - Add `sampling_approval_callback`, `sampling_max_tokens`, and `sampling_max_requests` parameters to `MCPTool` and its `MCPStdioTool`, `MCPStreamableHTTPTool`, and `MCPWebsocketTool` subclasses, positioned directly after `client`. - Gate each server-initiated `sampling/createMessage` request behind the approval callback, which denies by default when no callback is provided. - Clamp the requested `maxTokens` to `sampling_max_tokens` and enforce a per-session request count via `sampling_max_requests`. - Log incoming sampling requests at WARNING level (counts only). - Export `SamplingApprovalCallback` from the public API. - Add tests, a sample, and documentation updates. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Make sampling denial message context-aware Distinguish the deny-by-default case (no approval callback configured) from an explicit denial by a configured `sampling_approval_callback`, so the returned ErrorData message is accurate for callback-driven denials and exceptions. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Copilot
|
3daed114ee |
Python: bump package versions for 1.8.1 release (#6420)
* Python: bump package versions for 1.8.1 release * Python: bump agent-framework-foundry-hosting for 1.8.1 release * Python: bump ag-ui and azurefunctions for 1.8.1 release * Remove incorrect agent-framework-foundry changelog entry for #6259 * Add [1.8.1] changelog compare link and update [Unreleased] base --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> |
||
|
chetantoshniwal
|
632f67b92e |
Python: [Generated by SRE Agent] docs: clarify checkpoint storage security model and deserialization trust boundaries (#6295)
* docs: clarify checkpoint storage security model and deserialization trust boundaries Add Security Model documentation sections to the checkpoint encoding and Azure Functions serialization modules explaining: - Checkpoint storage is a trusted data source requiring access controls - The RestrictedUnpickler allowlist is defense-in-depth, not a security boundary - Developer responsibilities for securing storage backends - Guidance on using allowed_types and strip_pickle_markers Co-authored-by: Azure SRE Agent <noreply@microsoft.com> * Apply suggestions from code review Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Azure SRE Agent <noreply@microsoft.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> |
||
|
Eduard van Valkenburg
|
cfb033e5d4 |
Python: Filter MCP tool kwargs to declared params via allowlist (#6399)
* Filter MCP tool kwargs to declared params via allowlist Previously MCPTool combined framework runtime kwargs (from FunctionInvocationContext.kwargs) with the LLM-supplied arguments and stripped only a hardcoded denylist of known framework keys before forwarding to the MCP server. Any new framework-injected kwarg leaked to the server unless the denylist was updated. Switch to an allowlist built from each tool's declared parameters (inputSchema.properties). Only declared params are forwarded; everything else is stripped. Add an `additional_tool_argument_names` constructor argument so users can opt extra names back in, globally (Sequence[str]) and/or per remote tool name (Mapping with reserved "*" global key). The existing denylist is kept as a safety net for framework-named params a server declares in its schema; explicitly opted-in extras always win. The reserved _meta handling is unchanged. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address MCP allowlist review comments and fix reload arg loss - Fix pyright reportUnknownArgumentType in _load_tools (cast schema properties). - Register declared param names before the existing-tool skip guard so that tool-list reloads preserve the allowlist for already-loaded tools (previously unchanged tools silently dropped all declared args after a background reload). - Handle bare-string values in an additional_tool_argument_names mapping instead of iterating their characters. - Clarify the framework denylist comment: explicit extras override the denylist. - Make the extras-override-denylist test unambiguous (opt in a denylisted name). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Eduard van Valkenburg
|
7e0767a0a0 |
Python: Fix per-service-call history persistence with server-storing clients (#6310)
* Fix per-service-call history persistence with server-storing clients When an Agent set require_per_service_call_history_persistence=True together with a HistoryProvider, and the chat client stored history server-side by default (e.g. OpenAIChatClient, STORES_BY_DEFAULT=True), the external history provider was silently never persisted. Unify persistence on the per-service-call middleware: when the flag is set and a HistoryProvider exists, the middleware is always installed and owns persistence. service_stores_history now only selects middleware behavior: - service does not store: load providers and drive the function loop with a local sentinel conversation id, or - service stores: skip loading (the service owns history) and persist each service call while the real conversation id flows through. Also rationalize chat-options handling in _prepare_run_context: - _merge_options now skips None overrides and strips remaining None values, so an unset `store` is never forwarded and the service decides its own default. - Resolve `store` and `conversation_id` once from a single combined view (effective_options) instead of probing both default and runtime dicts; the auto-injection and per-service-call resolution now agree on conversation_id. Fixes #5798 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Correct as_agent() docstring: persistence is per service call, not once per run Address PR review: when the client stores history server-side, the per-service-call middleware still persists after each model call; only provider loading is skipped. The previous "persist once per run()" wording contradicted the implementation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR review: docs, missing-conversation-id warning, and tests - Clarify that require_per_service_call_history_persistence is a no-op when no HistoryProvider is present (docstrings in _agents.py and _clients.py). - Warn on every service call when the client stores history server-side but returns no conversation_id, so the (uncommon) loss of cross-turn resumability cannot fail silently. - Add tests: storing client + existing conversation_id does not raise and the id propagates; two runs on the same session keep persisting with a stable service_session_id and no provider loading; storing-without-conversation-id warns per call. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Tao Chen
|
dcc218dbac |
Python: feat(python): Add MCP client OTel spans per GenAI semantic conventions (#6349)
* feat(python): Add MCP client OTel spans per GenAI semantic conventions Implement MCP client spans per the OTel GenAI Semantic Conventions for MCP (https://opentelemetry.io/docs/specs/semconv/gen-ai/mcp/#client). Operations instrumented: - initialize: CLIENT span capturing MCP session setup - tools/list: CLIENT span for tool listing (per-page) - prompts/list: CLIENT span for prompt listing (per-page) - tools/call: CLIENT span (nested under execute_tool when called via FunctionTool) - prompts/get: CLIENT span Span attributes follow the MCP semantic conventions: - Required: mcp.method.name - Conditional: error.type, gen_ai.tool.name, gen_ai.prompt.name - Recommended: gen_ai.operation.name, mcp.protocol.version, mcp.session.id, network.transport, server.address, server.port Transport-specific attributes per subclass: - MCPStdioTool: network.transport=pipe - MCPStreamableHTTPTool: network.transport=tcp, network.protocol.name=http - MCPWebsocketTool: network.transport=tcp, network.protocol.name=websocket All span creation gated behind OBSERVABILITY_SETTINGS.ENABLED. Closes #3624 Closes #4697 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * refactor: simplify MCP spans — remove enrichment logic and protocol version caching - Always create nested CLIENT spans for tools/call instead of enriching the parent execute_tool span - Remove _ACTIVE_TOOL_EXECUTION_SPAN contextvar (no longer needed) - Remove enrich_span_with_mcp_attributes() helper - Remove _otel_error_type preservation in FunctionTool.invoke() - Remove _mcp_protocol_version instance variable; protocol version is only set on the initialize span where it is available Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Refine copilot solution * fix: enable automatic exception recording on MCP spans Remove record_exception=False and set_status_on_exception=False from create_mcp_client_span. Let OTel handle exception recording and status setting automatically. The manual set_mcp_span_error calls for tools/call still correctly set error.type (which OTel's automatic handling doesn't touch), so tool_error is preserved. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Reduce number of lines * Add comment to sample * test: address PR review comments on MCP observability tests - Fix initialize test to call mocked session.initialize() and read protocolVersion from the result instead of hardcoding it - Add tools/call McpError error-path test - Add prompts/get McpError error-path test Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix export error --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Tao Chen
|
9cafd7e58b |
Python: Refactor workflow as agent pending request handling (#6259)
* WIP: Refactor Workflow as agent pending request handling * WIP: debugging empty message bug * Working: Workflow as agent with function approval * Address Copilot comments * Fix mypy * Address comments and fix pipeline * Request info non function approval now becomes function call * Revert uv.lock * Fix mypy * Bump min version of azure-ai-project * Remove RequestInfoFunctionArgs * fix tests * Fix failing tests * Fix sample |
||
|
Peter Ibekwe
|
bf4ad48cf2 |
Python: MCP long-running task support in Python (#6319)
* MCP long-running task support in Python * Fix pyupgrade and AGENTS.md reconnect description - pyupgrade: drop forward-reference string annotations in _mcp.py (Python 3.10+ resolves them natively now that MCPTaskOptions is defined before use). - AGENTS.md: align reconnect description with current behavior. Phase 1 (initial tools/call) does NOT retry on connection loss; raises 'connection lost; task state unknown' instead, so a server that accepted the request but lost the response cannot start the operation twice. Phase 2 (tasks/get / tasks/result) still reconnects once against the same task_id. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix bandit nosec marker for CI pipeline * Address PR feedbacks * Clarifiied comments and addressed more PR feedbacks. --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Giles Odigwe
|
01fc518b29 |
Python: bump package versions for 1.8.0 release (#6351)
- Released cohort (core, openai, foundry, root): 1.7.0 -> 1.8.0 - agent-framework-github-copilot: promote to RC (1.0.0rc1) - agent-framework-orchestrations: rc2 -> rc3 (bug fix) - Beta/alpha packages with changes: a2a, anthropic, azurefunctions, bedrock, foundry-hosting, mistral bumped to new date stamp (260604) - Inter-package dependency bounds updated for changed packages - CHANGELOG.md and PACKAGE_STATUS.md updated Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Eduard van Valkenburg
|
f970a699d8 |
Python: Fix compaction message-id collisions and tool-loop summary persistence (#6299)
* Fix compaction message-id collisions and tool-loop summary persistence Fixes two bugs in the compaction strategies: - #5237: incremental group annotation assigned message ids by position within the re-annotated slice, so moving the re-annotation start back to a previous group start restarted ids at 0 and produced collisions (e.g. a user message reusing an assistant message's id), merging groups and causing tool-result compaction to wrongly exclude messages. group_messages/_ensure_message_ids now take an id_offset and guard against existing-id collisions; annotate_message_groups threads the slice start index through as the offset. - #4991: the function-invocation loop copied the message list each iteration, so summaries inserted by compaction landed in a throwaway copy and were lost across tool-loop iterations (only the persistent excluded flags survived). _prepare_messages_for_model_call now compacts the list in place when messages is a list, so inserted summaries persist. Adds regression tests (incremental id uniqueness, existing-id collision avoidance, idempotency, and tool-loop summary persistence including streaming and conversation-id modes). Also adds a summarization.py sample demonstrating SummarizationStrategy directly with a real client, and reworks advanced.py with tool-call groups and a real summarizer. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Guard incremental message-id assignment against prefix-id collisions Addresses PR review on #5237: _ensure_message_ids only guarded against collisions within the re-annotated slice. A preexisting (e.g. user-supplied) id in the preserved prefix could still be reassigned in the suffix when the id was numerically out of position, merging groups across the re-annotation boundary again. group_messages/_ensure_message_ids now accept reserved_ids, and annotate_message_groups passes the preserved prefix's ids so auto-assigned suffix ids never collide across the full list. Adds a regression test reproducing the out-of-position prefix-id collision. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Yufeng He
|
f29bae8fbc |
Python: run sync tools off the event loop (#5773)
* fix: run sync tools off event loop * chore: silence harness tool marker type check |
||
|
semenshi-m
|
c6951c21f6 |
Python: Add MCP-based skills discovery (McpSkillsSource) (#6169)
* Add MCP-based skills discovery (McpSkill, McpSkillsSource, McpSkillResource) Implement Agent Skills discovery over MCP following the SEP-2640 convention: - McpSkillsSource: reads skill://index.json to discover skills served by an MCP server - McpSkill: lazily fetches SKILL.md content via resources/read on demand - McpSkillResource: wraps MCP resource results (text and binary) - Path traversal protection in get_resource for defense in depth - Samples for Foundry Toolbox and standalone MCP skills server - Comprehensive unit tests (514 lines) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR review comments: rename to MCP* convention, fix error handling and samples - Rename McpSkill/McpSkillResource/McpSkillsSource to MCPSkill/MCPSkillResource/MCPSkillsSource - Add data-URI prefix stripping for blob resource decoding - Let non-McpError exceptions propagate from get_resource() - Fix contradictory test comment - Use interactive input() in mcp_based_skill sample - Remove misleading sample output block Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Restore debug logging for McpError in get_resource() Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Use AzureCliCredential in Foundry toolbox skills sample for consistency Replace DefaultAzureCredential with AzureCliCredential to match the credential convention used in all other samples. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Use MCPStreamableHTTPTool in MCP skills sample Replace raw mcp library imports (ClientSession, streamable_http_client) with the framework's MCPStreamableHTTPTool to keep MCP server connections consistent regardless of whether skills are enabled. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Branch on McpError.error.code so only not-found errors return empty Previously _try_read_index() and get_resource() swallowed every McpError as 'no skills available', making auth failures, server crashes, and connection drops indistinguishable from a server that simply has no skills. Now only two codes are treated as not-found: - -32002 (MCP-spec Resource not found) - -32601 (METHOD_NOT_FOUND — server lacks resources/read) All other McpError codes and non-McpError exceptions propagate with a warning log, surfacing real failures visibly. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add tests for non-McpError and non-not-found error propagation in MCP skills Cover the re-raise branch in MCPSkill.get_resource for plain ConnectionError/TimeoutError, the generic McpError (code 0) propagation on get_resource, and TimeoutError propagation in _try_read_index. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Revert "Use MCPStreamableHTTPTool in MCP skills sample" This reverts commit f31ed0ded914e094f3ac5d811997b2cefc55836b. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Introduce MCP_SKILLS experimental feature for MCP skill classes Add a separate MCP_SKILLS feature ID to ExperimentalFeature enum and use it for MCPSkillResource, MCPSkill, and MCPSkillsSource, since their promotion timeline is partly outside of our control. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Eduard van Valkenburg
|
49a6e433a3 |
Python: progressive tool exposure via FunctionInvocationContext (#6233)
* Python: progressive tool exposure via FunctionInvocationContext Add first-class progressive tool exposure to the Python core function-calling loop. Tools can now add or remove real FunctionTool schemas at runtime via the injected FunctionInvocationContext, taking effect on the next iteration of the loop. - FunctionInvocationContext gains a live `tools` list plus experimental `add_tools()` / `remove_tools()` helpers (feature: PROGRESSIVE_TOOLS). - The function-calling loop establishes a run-local, normalized tools list and threads it into the context at both invocation paths so mutations propagate. - Add a sample (dynamic_tool_exposure.py) and a tools samples README, including a note that CodeAct providers (Monty/Hyperlight) use their own provider-level tool management instead. Supersedes #3877. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Validate non-negative input in dynamic_tool_exposure sample tools Address review feedback: factorial and fibonacci now return an error message for negative n instead of producing incorrect results. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Make add_tools atomic and surface swallowed function errors Address review feedback on progressive tool exposure: - add_tools now validates the full batch against a throwaway copy before committing, so a duplicate-name clash partway through a sequence leaves the live tool list unchanged (all-or-nothing). - _auto_invoke_function now logs a warning (with traceback) when a tool raises, so contract errors such as a duplicate-name ValueError from add_tools are debuggable without enabling include_detailed_errors. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Avoid retaining tracebacks when logging swallowed function errors Logging with exc_info=exc fed the exception traceback to the logging machinery, whose frame references created reference cycles collected lazily by the cyclic GC. On Windows that could drop a hyperlight WasmSandbox on a non-owning thread ("unsendable, dropped on another thread"), crashing the xdist worker. Log a pre-formatted message with the exception repr instead, so no traceback object is retained. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * added missing decorator --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Peter Ibekwe
|
6086a74302 |
Python: Promote agent-framework-declarative package to RC (#6256)
* Promote agent-framework-declarative package to RC * Update missed package status file. |
||
|
Dineshsuriya D
|
a5f355e04a |
Python: Fix OTLP HTTP base-endpoint losing /v1/{signal} auto-append (#5913)
* Python: Fix OTLP HTTP base-endpoint losing /v1/{signal} auto-append
Per the OTel spec, OTEL_EXPORTER_OTLP_ENDPOINT is a *base* URL for HTTP —
the SDK auto-appends /v1/traces, /v1/metrics, /v1/logs when it reads the
env var directly. Signal-specific endpoint env vars are *full* URLs used
verbatim.
_get_exporters_from_env read the base endpoint and forwarded it as the
constructor ``endpoint=`` argument, which the SDK always treats as a full
signal URL. As a result, with OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318
and HTTP protocol, the exporter sent to http://localhost:4318 instead of
http://localhost:4318/v1/traces (and likewise for metrics/logs).
Replicate the spec's auto-append here when falling back to the base
endpoint under HTTP. gRPC behavior is unchanged.
* Python: Fix mypy type errors in OTLP endpoint assignment
Pre-declare traces_endpoint, metrics_endpoint, logs_endpoint as
str | None before the if/else block. Mypy inferred str from the
if-branch f-string assignments and then rejected the str | None
expressions in the else-branch as incompatible.
|
||
|
Ben Thomas
|
e0d0ad16a0 |
Python: feat(evals): Foundry Adaptive Evals integration (rubric-generation) (#6101)
* Python: feat(evals): RubricScore type + EvalScoreResult.dimensions Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Python: feat(foundry-evals): RubricDimension + GeneratedEvaluatorRef + accept in evaluators= Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Python: feat(evals): parse rubric_scores from output items + assertion helpers Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Python: feat(evals): BaseAgent.as_eval_source / Workflow.as_eval_source Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Python: feat(foundry-evals): EvalGenerationSource + generate_rubric helper Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Python: feat(foundry-evals): YAML config loader + sample Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Python: fix(evals): address PR review feedback Addresses 4 Copilot review comments on PR #6101: 1. assert_dimension_score_at_least: drop the (not evaluator or found_any) guard so require_applicable=True correctly raises when the named evaluator produces no entries for the dimension. Adds TestRubricAssertions covering the regression. 2. GeneratedEvaluatorRef docstring: reword to describe actual behaviour (pinning recommended, not required) so it matches the dataclass default and FoundryEvals warning path. 3. _poll_generation_job: switch from asyncio.get_event_loop() to get_running_loop() and bound the per-iteration sleep by remaining time, matching _poll_eval_run. 4. generate_rubric: type category as Literal['quality','safety'] and validate at the entry point with a ValueError; drop the silent 'invalid -> quality' rewrite in _generation_job_to_ref. Adds a regression test. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Python: feat(foundry-evals): hosted-agent-aware rubric generation * Auto-detect hosted Foundry agents in agent_as_eval_source: when the agent's chat_client exposes a string agent_name (the convention used by RawFoundryAgentChatClient for PromptAgents/HostedAgents), emit a type='agent' EvalGenerationSource so the service fetches instructions and tools from the agent registry instead of relying on the local wrapper (which holds neither for hosted agents). * Add hosted_agent_version kwarg and a new agent_version field on EvalGenerationSource so PromptAgent runs can pin to a specific hosted version for reproducible rubric generation. * Add force_prompt_source escape hatch to bypass auto-detection and always emit a rendered prompt dossier - useful when the local wrapper carries overrides the service-side agent doesnt see. * Fix _to_sdk_source for dataset sources: SDK ctor takes name=/version=, not dataset_name=/dataset_version=. The mismatch would raise TypeError against the real azure-ai-projects 2.3.0a* SDK; only unmocked integration paths were affected. Tests cover: auto-detection happy path, versionless hosted agent, explicit hosted_agent_version forwarding, force_prompt_source override, non-string chat_client attrs (MagicMock test doubles) not mis-detected, agent_version forwarded through _to_sdk_source, and the corrected dataset SDK kwarg names. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(foundry-evals): accept canonical dimension_scores key per docs The published Foundry rubric-evaluator output (Microsoft Learn 'Rubric evaluators' reference) places per-dimension breakdowns under properties.dimension_scores, not properties.rubric_scores. The parser now tries dimension_scores first and falls back to rubric_scores for preview-build compatibility, and tolerates non-list payloads (e.g. MagicMock auto-attrs) by trying the next candidate when parsing yields zero entries. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat(foundry-evals): add manual create_rubric_evaluator Adds FoundryEvals.create_rubric_evaluator as the agent-framework surface over project_client.beta.evaluators.create_version. This is the manual counterpart to generate_rubric: callers supply RubricDimension instances (authored locally, ported from another framework, or hand-tuned) and we POST a RubricBasedEvaluatorDefinition. The service auto-attaches the non-editable residual dimension (general_quality for quality, general_policy_compliance for safety). Per the Microsoft Learn 'Rubric evaluators' reference, the auto-generation path (create_generation_job) is primarily a portal/UI feature; external SDK clients with rich local agent context are better served by manual create_version. This keeps generate_rubric for users who want to round-trip through a Foundry-registered agent. Validation up front: weight must be in [1,10], ids unique, descriptions non-empty, pass_threshold in [0,1]. The returned GeneratedEvaluatorRef is identical in shape to one obtained from generate_rubric, so downstream evaluators= lists work unchanged. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * samples(foundry-evals): manual rubric sample + namespace re-exports Adds evaluate_with_manual_rubric_sample.py demonstrating the end-to-end dev scenario for FoundryEvals.create_rubric_evaluator: hand-author a list of RubricDimension, register via create_rubric_evaluator, then use the pinned GeneratedEvaluatorRef alongside built-in evaluators in an agent regression run. Also re-exports RubricDimension, GeneratedEvaluatorRef, build_sources, and load_evals_config from agent_framework.foundry (both the lazy runtime shim and the type stub) so the rubric samples can import everything from a single namespace; the auto-generate sample was previously broken because the shim was missing build_sources / load_evals_config. Updates the foundry-evals README with a chooser entry for the two rubric paths. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat(foundry-evals): remove rubric creation flows; keep consumption only Reframes agent-framework as a pure consumer of Foundry rubric evaluators: scoring against rubrics that already exist (authored in the Foundry portal or via the dedicated SDK / REST surface) instead of creating them from the SDK. Removed creation surface area: - FoundryEvals.generate_rubric (auto-generate path) and create_rubric_evaluator (manual path), plus all _GenerationSdkTypes / _ManualRubricSdkTypes / _to_sdk_dimensions / _coalesce_generation_sources / _to_sdk_source / _poll_generation_job / _generation_job_to_ref / _evaluator_version_to_ref / _get_beta_evaluators / _import_*_sdk_types helpers. - EvalGenerationSource (the input source discriminator), RubricDimension (the input dimension type), agent_as_eval_source / workflow_as_eval_source / _detect_hosted_foundry_agent helpers, and the YAML-config loader (_evals_config.py with RubricGenerationSpec / RubricSourceSpec / parse_evals_config / load_evals_config / build_sources). - BaseAgent.as_eval_source / Workflow.as_eval_source plus the _render_agent_dossier / _render_workflow_dossier helpers in core. These existed only to feed the now-removed generation pipeline. - Samples evaluate_with_generated_rubric_sample.py, evaluate_with_manual_rubric_sample.py, and evaluators.yaml. Replaced with a short README section showing how to reference an existing rubric evaluator via GeneratedEvaluatorRef. Kept (consumption surface): - GeneratedEvaluatorRef, slimmed to (name, version, display_name). Still accepted alongside built-in evaluator strings in FoundryEvals(evaluators=[...]). Versionless refs still warn. - RubricScore on EvalScoreResult.dimensions plus EvalResults.assert_dimension_score_at_least for per-dimension CI gates. - _parse_dimension_entries / _extract_rubric_scores output parsing (both canonical dimension_scores and the legacy rubric_scores key). Tests: 160/160 foundry unit tests and 71/71 core local-eval tests pass; pyright is clean across changed files. The pre-existing tests/core/test_telemetry.py::test_detect_hosted_fallback_import_error failure is unrelated and reproduces on the prior commit. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * samples(foundry-evals): add evaluate_with_rubric_sample Adds a runnable end-to-end sample showing how to consume a pre-existing rubric evaluator created in Foundry: reference it with GeneratedEvaluatorRef(name, version), mix it with built-in evaluators in FoundryEvals, and gate CI with assert_dimension_score_at_least on a specific dimension. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(foundry-evals): satisfy mypy on _fetch_output_items mypy infers OutputItemListResponse.sample as dict[str, object] | None while pyright correctly infers the typed Sample model. Cast to Any so both type checkers accept the attribute access pattern, rename the local to avoid shadowing the inner-loop sample binding, and drop the now-stale pyright suppressions. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docs(foundry-evals): drop unpublished rubric-evaluators learn.microsoft.com link The Adaptive Evals authoring docs are not yet published on Microsoft Learn, so the link 404s. Keep the descriptive text without the broken hyperlink; we can re-add it once the docs ship. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * test(foundry-evals): hoist repeated local imports to module top Per code review feedback (eavanvalkenburg): the test file repeated 'from agent_framework_foundry._foundry_evals import ...' inside 22 test bodies and 'from agent_framework_foundry import GeneratedEvaluatorRef' inside 8 more. Move all of them to the existing top-level imports; the symbols are the same across tests and the local imports were redundant. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Ben Thomas <25218250+alliscode@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |