github-featured/microsoft--agent-framework
2
0
Fork 0
Code Issues Pull Requests Deployments Models Agent Notes Packages Projects Releases Wiki Activity
2,849 Commits 301 Branches 183 Tags
main
Commit Graph

66 Commits

Author SHA1 Message Date
Tao Chen e85b3c8ba8 Python: Fix FHA session ID translation (#7608) 
* Fix FHA session ID traslation

* Fix tests

* Address comments and fix tests

* Fix typing

* Show how to use user created sessions

* Update README
 2026-08-11 17:30:56 +00:00
Evan Mattson 48e547506b Python: Make encrypted reasoning opt-in for Foundry chat (#7536) 
* Python: Make Foundry encrypted reasoning opt-in

* Python: Opt hosted replay test into encrypted reasoning
 2026-08-10 02:36:48 +00:00
MohammadHaroonAbuomar 7302d0bf23 Python: agent-hooks interception contract as a first-class experimental core feature (#7515) 
* feat(python): add agent-hooks middleware as experimental core feature

Implement the AGENT-HOOKS-0.1 interception contract as a first-class
experimental feature in agent_framework core.

- Single public factory agent_hooks_middleware() returning a private
  agent/chat/function middleware trio (one object per middleware
  category); partial or stacked installs fail closed with loud errors.
- All eight interception points: input/output at the agent seam,
  pre/post_model_call at the chat seam, pre/post_tool_call at the
  function seam, agent_startup/agent_shutdown bracketing each run.
- Fail-closed enforcement throughout: transforms write back into the
  native contexts (messages, arguments, results) or raise; content is
  preserved as Content objects; MiddlewareTermination short-circuits
  are guarded at every seam; enforcement-layer failures halt the run;
  interceptor crashes surface as host_error denies.
- Streaming is fully buffered per spec buffered_output semantics: no
  update egresses before the post_model_call/output verdicts; a deny
  at pull time releases zero updates; run state stays active across
  lazy pulls with cleanup on every exit path.
- Session scoping: per-run by default (startup/shutdown bracket each
  run) or host-owned via emitter/builder parameters for one session
  spanning multiple runs.
- agent-hooks-sdk is an opt-in agent-hooks extra (not in all),
  lazy-imported per the _mcp.py pattern; core imports cleanly without
  it and the factory raises a clear ModuleNotFoundError.
- ExperimentalFeature.AGENT_HOOKS + @experimental decorator, lazy root
  export, typing surface, PACKAGE_STATUS.md entry.
- 55 tests built on real Agent/mock-client flows covering deny-before-
  execution, transform write-back, rich-content preservation, complete
  streaming ordering, error cleanup, concurrency isolation, nested
  agents, and importability without the optional SDK.

Signed-off-by: MohammadHaroonAbuomar <40180927+MohammadHaroonAbuomar@users.noreply.github.com>

* style(python): unquote ResponseStream annotation per pyupgrade

The pre-commit pyupgrade hook rewrites the quoted forward reference;
ResponseStream is imported at runtime in this module, so the quotes
were unnecessary.

Signed-off-by: MohammadHaroonAbuomar <40180927+MohammadHaroonAbuomar@users.noreply.github.com>

* refactor(python): address agent-hooks review feedback

Reworks the agent-hooks feature per PR review:

- Verdicts now precede durability: a run-scoped persistence gate
  (_sessions.py) defers per-service-call history persistence and
  after-run provider work until the covering post_model_call/output
  verdict permits; denied content never persists, transforms persist
  post-write-back. Unhooked runs are unchanged (verified against an
  instrumented baseline).
- ResponseStream.buffered_and_gated: a buffered-gate combinator that
  applies the run's pending stream hooks before the gate, then seals
  the stream, so no middleware can rewrite egress after the output
  verdict. Replaces the hand-rolled replay iterator.
- MiddlewareBundle (public, _middleware.py): the factory returns an
  indivisible bundle categorize_middleware splits, making partial
  installs impossible by construction; members are validated at
  construction. Bare (non-sequence) middleware at agent construction
  is now normalized instead of silently dropped, and unrecognized
  middleware logs a warning instead of vanishing.
- Factory split and rename: create_agent_hooks_middleware (per-run
  sessions) and create_agent_hooks_middleware_from_emitter
  (host-owned); the sentinel parameter-diffing is gone.
- Wire conversions live in per-point codec classes owning to_wire and
  write_back. Fixes in that code: tool-call name transforms apply or
  raise; non-object args transforms raise; argument write-back merges
  only changed keys (original values, including bytes, preserved by
  identity); message-list write-back matches by identity, not index.
- function_approval_request objects on the normal return path pass
  through un-emitted, preserving the human approval pause.
- Hosted (service-executed) tool calls surface in the post_model_call
  content projection; the tool-seam limitation is documented.
- Import probe covers the full SDK surface and re-raises as
  missing-extra only for the agent_hooks module; module logger added;
  _json_safe replaced by make_json_safe (which gained bytes support);
  tools_registered uses normalize_tools; dependency-pyright analyzes
  the module again via the test dependency-group.
- Tests: 75 in the feature suite (persistence gating, stream-hook
  sealing, approval passthrough, codec units, bundle validation,
  bare-bundle installs), full core suite green.

Signed-off-by: MohammadHaroonAbuomar <40180927+MohammadHaroonAbuomar@users.noreply.github.com>

* refactor(python): second review round for agent-hooks

Addresses the second review round on the agent-hooks feature:

- Nested-run persistence ownership: RawAgent.run stamps a run identity
  over the run's dynamic extent (including streaming pulls and result
  hooks); the persistence gate binds to its owning run via an
  offer/adopt handshake keyed to the agent instance and accepts only
  its owner's persists — nested runs persist inline regardless of how
  they were started (tool calls, middleware, custom run loops). The
  tool-seam suspension remains for custom-loop sub-agents invoked as
  tools; the one residual case (custom loop nested in a custom loop
  off the tool path) is fail-closed and documented. Fixes a latent
  pre-existing re-deferral: flush() now drains with the gate context
  suspended, so a nested hooked run's permitted after-run persistence
  no longer re-defers into an enclosing gate.
- as_tool stream_callback consumes the released (verdicted) stream;
  observers cannot see denied or pre-transform content. Both
  directions are regression-tested.
- categorize_middleware gained supported_categories: a bundle member
  landing in a category a call site cannot install raises; bare
  middleware warns like _add_middleware. Wired at the chat-client
  sites and the provider seam.
- ResponseStream.buffered_and_gated owns the re-derivation rule via a
  rederive callable (gates cannot choose released updates) and is
  marked experimental.
- Wire codecs compare with bool-aware equality (Python == equates
  1 == True, which made bool/number transforms look untouched and get
  dropped) and _ToolResultCodec.write_back owns the untouched-wire
  rule via the before value.
- middleware parameters accept a bare middleware or bundle everywhere
  the runtime does (constructors, run overloads, as_agent, telemetry
  and harness layers, foundry); the bare-source rule has a single
  owner in categorize_middleware; bare middleware assigned to the
  attribute now executes (documented behavior change).
- MiddlewareBundle is experimental and validates members; approval
  passthrough, typing-check fixes (ty ignores mypy-coded ignore
  comments), logging, and documentation updates per review.

Test count: 85 feature tests plus 12 new this round across sessions,
middleware, agents; full core suite green; typing checked under
mypy, pyrefly, ty, zuban, and pyright.

Signed-off-by: MohammadHaroonAbuomar <40180927+MohammadHaroonAbuomar@users.noreply.github.com>

* docs(python): drop previous-behavior notes from middleware docstrings

Per review: docstrings describe current behavior only. The
bare-middleware behavior change stays recorded in the PR description
and commit history.

Signed-off-by: MohammadHaroonAbuomar <40180927+MohammadHaroonAbuomar@users.noreply.github.com>

* fix(python): gate ownership survives retrying middleware

A retry or fallback middleware issuing a second call_next() gave the
new attempt a fresh run identity that the persistence gate's
first-bind-wins ownership rejected, so the retried attempt's history
persisted inline before the output verdict — a denied response became
durable again. The gate now accumulates every identity adopted
through its own offer ticket: all attempts' persistence stays behind
the one final verdict (deny drops all of it, allow flushes all of
it). Accumulation over rebind-replace is deliberate: rebinding would
flip an earlier attempt's still-running background work from deferred
to inline, which is the fail-open direction. A foreign agent still
cannot bind: tickets are minted only by the covered pipeline's final
handler and adoption is instance-keyed.

Also consolidates the bare-middleware-source rule into a single
_as_middleware_list owner used by every interpretation site (the
harness merge, BaseAgent.__init__, categorize_middleware, both
client-kwargs merges, get_response, SessionContext.extend_middleware),
including the str/bytes exclusion the stray copies missed. The
constructor now stores a copy of the caller's sequence; assign to the
middleware attribute for post-construction changes.

Retry regression tests cover denied and allowed retried runs in both
stream modes and fail with first-bind-wins restored.

Signed-off-by: MohammadHaroonAbuomar <40180927+MohammadHaroonAbuomar@users.noreply.github.com>

* fix(python): streaming seam runs pipeline descent inside the gate

The streaming agent seam ran call_next() outside the persistence
gate (only _consume entered it later), so a retry middleware that
drained a successful attempt with get_final_response() and discarded
it persisted that attempt's exchange before any verdict existed; a
later deny dropped only the retry attempt's deferred work. The
descent is now wrapped in the gate exactly like the non-streaming
seam: attempt identities adopted during descent are accepted owners,
so in-pipeline draining defers, deny drops every attempt, and a
middleware that raises after draining strands the pending persists
unexecuted. The bind_owner docstring now states the actual soundness
invariant covering both bind sites: every bind comes from a run
inside the covered pipeline.

New tests cover drained-and-discarded attempts (deny and allow, both
stream modes) and a sub-agent tool inside a drained attempt; the
streaming deny variant fails with the gate wrap reverted.

Signed-off-by: MohammadHaroonAbuomar <40180927+MohammadHaroonAbuomar@users.noreply.github.com>

* fix(python): flush deferred persistence on streaming no-result termination

With the pipeline descent now running inside the persistence gate, a
middleware that drains a successful attempt and then terminates
without a result left that attempt's deferred persistence stranded:
the streaming no-result termination path raised before any flush, so
history of exchanges that really happened and passed their own
verdicts quietly vanished (streaming only; non-streaming already
flushes before its re-raise). The path now flushes before re-raising
the termination, with a state.halted guard first so an enforcement
failure during the drained attempt still strands pending fail-closed
and surfaces the halt, mirroring the non-streaming ordering exactly.

The regression test covers both seams; the streaming variant fails
without the fix.

Signed-off-by: MohammadHaroonAbuomar <40180927+MohammadHaroonAbuomar@users.noreply.github.com>

---------

Signed-off-by: MohammadHaroonAbuomar <40180927+MohammadHaroonAbuomar@users.noreply.github.com>
 2026-08-07 00:25:09 +00:00
Eduard van Valkenburg b64a2e2f82 Python: add feature-usage User-Agent telemetry (#7420) 
* Python: add first-pass feature usage telemetry

Add the 128-bit feature accumulator, package-local indexes, activation markers, and destination-scoped User-Agent emission for the initial Python implementation slice.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 346bf168-b668-4c4a-a8db-a67282ee5e5f

* Python: track declarative feature usage

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 346bf168-b668-4c4a-a8db-a67282ee5e5f

* Python: complete feature usage telemetry

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 346bf168-b668-4c4a-a8db-a67282ee5e5f

* Python: report core version in User-Agent

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 346bf168-b668-4c4a-a8db-a67282ee5e5f

* Python: configure Lab telemetry import path

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 346bf168-b668-4c4a-a8db-a67282ee5e5f

* Python: preserve telemetry transport behavior

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 346bf168-b668-4c4a-a8db-a67282ee5e5f

* Python: preserve caller-owned Foundry transports

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 346bf168-b668-4c4a-a8db-a67282ee5e5f

* Python: remove stale Anthropic test import

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 346bf168-b668-4c4a-a8db-a67282ee5e5f

---------

Copilot-Session: 346bf168-b668-4c4a-a8db-a67282ee5e5f
 2026-07-30 10:24:34 +00:00
Thota Sai Karthik 4d67eefa5f Python: Fix FoundryAgent inheriting OPENAI_CHAT_MODEL for agent-reference requests (#7283) 
* Python: Fix FoundryAgent inheriting OPENAI_CHAT_MODEL for agent-reference requests (#7272)

* Python: Fix FoundryAgent inheriting OPENAI_CHAT_MODEL for agent-reference requests

* fix(foundry): update test typing annotations to pass mypy, pyrefly, and ty

---------

Co-authored-by: Eduard van Valkenburg <eavanvalkenburg@users.noreply.github.com>
 2026-07-30 10:05:33 +00:00
Evan Mattson a2927c1c09 Python: Fix stateless replay of reasoning-paired tool calls (#7233) 
* Python: Fix reasoning-paired client tool replay

* Python: Handle middleware-terminated reasoning tool loops

* Python: Replay encrypted reasoning function groups

Key decisions:
- Request encrypted reasoning on client-managed Responses calls while preserving caller include values.
- Store encrypted payloads in Content.protected_data and reconstruct one provider reasoning item per reasoning id.
- Replay active and completed function call/result groups; retain continuation-owned history behavior and the existing orphan-safe MCP path.

Files changed:
- python/packages/openai/agent_framework_openai/_chat_client.py
- python/packages/openai/tests/openai/test_openai_chat_client.py

Next iteration:
- Extend encrypted reasoning preservation to streaming and framework serialization boundaries.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Python: Preserve encrypted reasoning through streaming

Key decisions:
- Capture encrypted reasoning from terminal streamed output items in Content.protected_data.
- Preserve summary and private reasoning as distinct framework contents while reconstructing one provider reasoning item per id.
- Prove replay after Message JSON and workflow checkpoint round trips, including encrypted-only and completed function groups.

Files changed:
- python/packages/core/agent_framework/_types.py
- python/packages/openai/agent_framework_openai/_chat_client.py
- python/packages/openai/tests/openai/test_openai_chat_client.py

Next iteration:
- Extend lossless stateless reasoning replay to hosted MCP call/output groups.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Python: Replay hosted MCP reasoning groups

Key decisions:
- Preserve hosted MCP call/output groups in client-managed history instead of deleting them when reasoning cannot be reconstructed.
- Keep call/result coalescing and orphan-result exclusion intact, while retaining continuation-owned duplicate avoidance.
- Cover completed, active, and multi-call reasoning groups plus the public outgoing request boundary.

Files changed:
- python/packages/openai/agent_framework_openai/_chat_client.py
- python/packages/openai/tests/openai/test_openai_chat_client.py

Next iteration:
- Preserve middleware-terminated and parallel function groups atomically.
- Add preflight rejection for non-replayable reasoning groups in the dedicated validation slice.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Python: Preserve terminated parallel reasoning groups

Key decisions:
- Return ordinary function results when middleware terminates a loop, removing the provider-specific durable marker.
- Preserve every parallel call and available sibling result as one encrypted reasoning group in stateless replay.
- Prove successful and policy-blocked batches through the public two-agent Foundry workflow and outgoing HTTP boundary.

Files changed:
- python/packages/core/agent_framework/_tools.py
- python/packages/core/tests/core/test_function_invocation_logic.py
- python/packages/openai/tests/openai/test_openai_chat_client.py
- python/packages/foundry/tests/foundry/test_foundry_agent.py

Next iteration:
- Add preflight rejection for non-replayable and partially compacted reasoning groups.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Python: Reject unsafe stateless reasoning replay

Key decisions:
- Validate client-managed reasoning groups after compaction and report every affected reasoning and call identifier before transport.
- Permit service-owned continuation and fully excluded atomic groups while rejecting partial compaction projections.
- Surface encrypted-reasoning capability failures without lossy retries.

Files changed:
- python/packages/openai/agent_framework_openai/_chat_client.py
- python/packages/openai/tests/openai/test_openai_chat_client.py

Next iteration:
- Run the resource-specific Foundry proof and finish PR #7233; that live proof remains intentionally local and requires the configured developer resource.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Python: Preserve reasoning metadata in Foundry hosting

* Python: Avoid duplicating reasoning text metadata

* Python: Gate encrypted reasoning for Foundry agents

* Python: Type stateless reasoning integration test

* Python: Narrow Foundry mock call arguments

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-07-22 23:51:22 +00:00
Evan Mattson c218067646 Python: Consolidate dependency updates (#7204) 
* Bump uv from 0.11.28 to 0.11.29 in /python

Bumps [uv](https://github.com/astral-sh/uv) from 0.11.28 to 0.11.29.
- [Release notes](https://github.com/astral-sh/uv/releases)
- [Changelog](https://github.com/astral-sh/uv/blob/main/CHANGELOG.md)
- [Commits](https://github.com/astral-sh/uv/compare/0.11.28...0.11.29)

---
updated-dependencies:
- dependency-name: uv
  dependency-version: 0.11.29
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Bump zuban from 0.8.2 to 0.9.0 in /python

Bumps [zuban](https://github.com/zubanls/zubanls-python) from 0.8.2 to 0.9.0.
- [Release notes](https://github.com/zubanls/zubanls-python/releases)
- [Commits](https://github.com/zubanls/zubanls-python/compare/v0.8.2...v0.9.0)

---
updated-dependencies:
- dependency-name: zuban
  dependency-version: 0.9.0
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* PR #7146: Bump ty from 0.0.55 to 0.0.60 in /python

* Bump ruff from 0.15.20 to 0.15.22 in /python

Bumps [ruff](https://github.com/astral-sh/ruff) from 0.15.20 to 0.15.22.
- [Release notes](https://github.com/astral-sh/ruff/releases)
- [Changelog](https://github.com/astral-sh/ruff/blob/main/CHANGELOG.md)
- [Commits](https://github.com/astral-sh/ruff/compare/0.15.20...0.15.22)

---
updated-dependencies:
- dependency-name: ruff
  dependency-version: 0.15.21
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Bump mypy from 2.2.0 to 2.3.0 in /python

Bumps [mypy](https://github.com/python/mypy) from 2.2.0 to 2.3.0.
- [Changelog](https://github.com/python/mypy/blob/master/CHANGELOG.md)
- [Commits](https://github.com/python/mypy/compare/v2.2.0...v2.3.0)

---
updated-dependencies:
- dependency-name: mypy
  dependency-version: 2.3.0
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Bump prek from 0.4.8 to 0.4.10 in /python

Bumps [prek](https://github.com/j178/prek) from 0.4.8 to 0.4.10.
- [Release notes](https://github.com/j178/prek/releases)
- [Changelog](https://github.com/j178/prek/blob/master/CHANGELOG.md)
- [Commits](https://github.com/j178/prek/compare/v0.4.8...v0.4.10)

---
updated-dependencies:
- dependency-name: prek
  dependency-version: 0.4.10
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Bump azure-ai-projects from 2.2.0 to 2.3.0 in /python

Bumps azure-ai-projects from 2.2.0 to 2.3.0.

---
updated-dependencies:
- dependency-name: azure-ai-projects
  dependency-version: 2.3.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Bump types-python-dateutil in /python

Bumps [types-python-dateutil](https://github.com/python/typeshed) from 2.9.0.20260518 to 2.9.0.20260716.
- [Commits](https://github.com/python/typeshed/commits)

---
updated-dependencies:
- dependency-name: types-python-dateutil
  dependency-version: 2.9.0.20260716
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Bump mypy from 2.2.0 to 2.3.0 in /python

Bumps [mypy](https://github.com/python/mypy) from 2.2.0 to 2.3.0.
- [Changelog](https://github.com/python/mypy/blob/master/CHANGELOG.md)
- [Commits](https://github.com/python/mypy/compare/v2.2.0...v2.3.0)

---
updated-dependencies:
- dependency-name: mypy
  dependency-version: 2.3.0
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Bump botocore from 1.43.45 to 1.43.49 in /python

Bumps [botocore](https://github.com/boto/botocore) from 1.43.45 to 1.43.49.
- [Commits](https://github.com/boto/botocore/compare/1.43.45...1.43.49)

---
updated-dependencies:
- dependency-name: botocore
  dependency-version: 1.43.49
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* PRs #7144/#7147: Align lab uv and ruff pins

* Regenerate lockfile for Python Dependabot PRs #7144-#7153

* Python: Support azure-ai-projects 2.3 session operations (#7150)

* Python: Apply Ruff 0.15.22 suppression updates (#7147)

* Python: Update tests for ty 0.0.60 (#7146)

* Python: Update Foundry samples for azure-ai-projects 2.3 (#7150)

* Python: Address dependency rollup review comments

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2026-07-20 18:01:24 +00:00
Nick Brady 54617557e6 Update Foundry branding (#6999) 
Replace user-facing Azure AI Foundry branding with Microsoft Foundry across docs, samples, comments, and display text while preserving technical identifiers.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Evan Mattson <35585003+moonbox3@users.noreply.github.com>
 2026-07-14 06:44:26 +00:00
Vaibhav Patel 23bfa49575 Python: Strip tools from Foundry agent request on the preview path (#6644) 
The Foundry service rejects requests that include tool declarations when an
agent is specified (HTTP 400 invalid_payload, "Not allowed when agent is
specified."). RawFoundryAgentChatClient._prepare_options stripped tools,
tool_choice, and parallel_tool_calls only on the non-preview path, so when
allow_preview=True (where the agent identity is bound on the OpenAI client via
get_openai_client(agent_name=...)) the tool fields were still sent and the call
failed.

This client always targets a pre-provisioned agent, so it must never send tool
declarations. Drop the tool fields unconditionally and log a single warning
when the caller supplied tools, noting they are used only for client-side
function dispatch. The non-agent FoundryChatClient (model-based) is unaffected.

Fixes #5130.

Co-authored-by: Eduard van Valkenburg <eavanvalkenburg@users.noreply.github.com>
 2026-07-09 01:53:39 +00:00
Eduard van Valkenburg fc3111c391 Python: Add FoundryAgent conversation session helper (#6623) 
* Add FoundryAgent conversation session helper

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Simplify Foundry conversation session helper

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Rename Foundry conversation helper

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* use named kw

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-06-22 08:41:15 +00:00
Ben Thomas 54a30571aa Dotnet - Add support for Foundry Adaptive evals (#6267) 
* .NET: feat(evals): RubricScore type + EvalScoreResult.Dimensions

Adds the core rubric-evaluator surface that mirrors the Python work in

PR #6101 (commit e45b934cc). Provider-agnostic types only — no Foundry

coupling. Subsequent commits will wire these into FoundryEvals.

- RubricScore: per-dimension score record (Id, Score?, Applicable, Weight, Reason).

- EvalScoreResult.Dimensions: optional init-only list of RubricScore.

  Null for non-rubric (built-in) evaluators.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* .NET: feat(evals): GeneratedEvaluatorRef + assertion helpers

Adds the provider-agnostic surface for referencing a pre-existing rubric

evaluator and gating CI on per-item / per-dimension thresholds. Mirrors

Python PR #6101 commits e5830dd7f (ref type) and 4bc60462d (asserts).

- GeneratedEvaluatorRef: name + optional version/display-name, plus a

  Latest(name) factory for versionless refs (discouraged for CI; consumers

  should warn at run time).

- AgentEvaluationResults.AssertScoreAtLeast: walks DetailedItems[].Scores,

  optionally filtered by evaluator name, recurses into SubResults.

- AgentEvaluationResults.AssertDimensionScoreAtLeast: walks each score's

  Dimensions list, skips non-applicable dimensions by default, supports

  requireApplicable to flip that, recurses into SubResults.

- AgentEvaluationResults.AssertNoFailedItems: walks DetailedItems for

  fail/error statuses, recurses into SubResults.

All helpers throw InvalidOperationException (matches existing AssertAllPassed).

Truncates offender lists to the first 5 with a '+N more' suffix to keep

CI output readable, mirroring the Python helpers.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* .NET: feat(foundry-evals): accept GeneratedEvaluatorRef in evaluators=

Adds FoundryEvaluatorSpec, a readonly-struct union with implicit conversions
from both string and GeneratedEvaluatorRef so call sites can mix built-in
evaluator names with rubric evaluator references:

    var evals = new FoundryEvals(
        projectClient, model,
        new GeneratedEvaluatorRef("policy-rubric", "3"),
        FoundryEvals.Relevance,
        FoundryEvals.Coherence);

FoundryEvals constructors (3 overloads), EvaluateTracesAsync, and
EvaluateFoundryTargetAsync now take FoundryEvaluatorSpec[]/params instead of
string[]/params. Existing call sites using string literals or string[] keep
working unchanged via implicit conversion.

FoundryEvalConverter.BuildTestingCriteria emits the documented Foundry wire
format for rubric refs:
  {
    "type": "azure_ai_evaluator",
    "name": <DisplayName ?? Name>,
    "evaluator_name": <Name>,
    "evaluator_version": <Version>,   // omitted when null
    "initialization_parameters": { "deployment_name": <model> },
    "data_mapping": { conversation arrays, optional tool_definitions }
  }

WireTestingCriterion gains an optional EvaluatorVersion field. Rubric refs
are preserved through FilterToolEvaluators (tool-aware but not tool-required)
and ignored by FindMissingGroundTruthEvaluators. A versionless ref emits a
Trace.TraceWarning at criterion-build time so CI authors notice the floating
version (mirrors the Python warning).

Adds 6 new Foundry unit tests (3 BuildTestingCriteria rubric paths, 1
FindMissingGroundTruthEvaluators, 1 FilterToolEvaluators preservation, 1
mixed-order). 369/369 Foundry tests pass.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* .NET: feat(foundry-evals): parse rubric dimension_scores into RubricScore

Adds FoundryEvals.ParseRubricScores, called per result inside ParseDetailedItem.
Each EvalScoreResult now populates Dimensions when the evaluator's sample carries
a rubric breakdown.

Accepts three shapes for forward compatibility with provider SDK iterations:

  1. sample.properties.dimension_scores  (canonical Foundry runtime shape)
  2. sample.properties.rubric_scores     (preview/legacy key)
  3. top-level sample.dimension_scores / sample.rubric_scores  (defensive fallback)

Entries missing 'id', 'weight', or 'applicable' are skipped without invalidating
well-formed siblings. Non-applicable dimensions may omit 'score' (parsed as null).

Adds 6 unit tests covering canonical and legacy keys, top-level fallback, no-match
returns null, malformed-entry skipping, and the non-applicable null-score path.
375/375 Foundry tests pass.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* .NET: feat(samples): Evaluation_FoundryRubric end-to-end sample

Adds dotnet/samples/05-end-to-end/Evaluation/Evaluation_FoundryRubric mirroring
the Python evaluate_with_rubric_sample.py:

  - Fetches a pre-existing Foundry agent via AgentAdministrationClient
    (GetAgentAsync for latest, GetAgentVersionAsync when FOUNDRY_AGENT_VERSION
    is pinned).
  - References a rubric evaluator by GeneratedEvaluatorRef(name, version);
    falls back to GeneratedEvaluatorRef.Latest(name) with the documented
    floating-version warning.
  - Mixes the rubric with FoundryEvals.Relevance and FoundryEvals.Coherence
    in a single FoundryEvals run (implicit string-and-ref conversion).
  - Prints per-dimension breakdowns from EvalScoreResult.Dimensions for each
    item.
  - Demonstrates a CI quality gate with AssertDimensionScoreAtLeast("general_quality", 3.0).

Documents the FOUNDRY_PROJECT_ENDPOINT footgun (must be project-scoped URL
.../api/projects/<project>, not the bare Azure OpenAI endpoint) and the
Eval-Definition-vs-Rubric-Evaluator distinction in the README. Ships a
.env.example with the FOUNDRY_* variables.

Registers the project in agent-framework-dotnet.slnx and cross-links from
the sibling Evaluation_Multimodal / Evaluation_ExpectedOutputs READMEs.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(foundry-evals): harden FoundryEvals public surface for review

Address PR #6267 review comments on the .NET FoundryEvals integration:

- Add source-compat overloads accepting `string[] evaluators` for `FoundryEvals` ctor, `EvaluateTracesAsync`, and `EvaluateFoundryTargetAsync` so existing callers passing string arrays keep compiling unchanged. New overloads forward via a private `ToSpecs` helper that wraps each name through the implicit `string -> FoundryEvaluatorSpec` conversion.

- Guard against `default(FoundryEvaluatorSpec)` entries (both `BuiltinName` and `GeneratedRef` null) that would NRE the downstream converter. Adds `FoundryEvaluatorSpec.IsValid` / `EnsureValid` plus an internal `EnsureAllSpecsValid` helper, wired into the main ctor and both static evaluation entry points.

- Add 6 unit tests covering the new validation surface.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(sample): set ExitCode=1 when rubric dimension gate trips

PR #6267 review comment: the FoundryRubric sample swallowed the AssertDimensionScoreAtLeast failure, so a CI run that included it as a quality gate would still exit 0 even when the rubric regressed. Set `System.Environment.ExitCode = 1` in the catch so CI fails while still letting the rest of the sample's logging complete cleanly.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(foundry-evals): search typed Sample directly for rubric scores

PR #6267 review comment: `_extract_rubric_scores` only searched the `properties` dict when the sample exposed one. When the Azure AI Projects typed SDK returns a Sample object that puts `dimension_scores` / `rubric_scores` directly on the instance (no `properties` wrapper), we missed them and surfaced no per-dimension scores.

Add an `else: containers.append(sample)` branch so non-dict typed samples are also inspected for the score keys. Covered by two new tests: one with `dimension_scores` directly on a typed Sample without a `properties` wrapper, and one with the legacy `rubric_scores` key in the same shape.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* test(evals): cover assert_score_at_least and assert_no_failed_items

PR #6267 review comments: both assertion helpers shipped without unit tests. Add `TestAssertScoreAtLeast` (above threshold, below w/ offenders, evaluator filter, sub_results recursion) and `TestAssertNoFailedItems` (all passing, failed/errored statuses, sub_results recursion) with a shared `_score_results` fixture builder.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* docs(samples): remove dead rubric-evaluator doc link from FoundryRubric sample

The Azure AI Foundry rubric evaluator concept doc page has not yet been published, so the link in the sample README and Program.cs comment 404s. Drop the references until the upstream doc is live.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Potential fix for pull request finding

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

* Address PR 6267 review nits

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Ben Thomas <25218250+alliscode@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
 2026-06-18 20:50:28 +00:00
Eduard van Valkenburg 6e95517659 Python: Split type checkers by target (pyright source, 5 checkers on tests/samples) (#6443) 
* Python: Split type checkers by target (pyright source, 5 checkers on tests/samples)

Rework the typing setup along the lines of the 'too many type checkers'
approach:

- Pyright (strict) is now the sole source-code type checker; mypy is
  removed from source and its [tool.mypy] block becomes a relaxed profile
  used only for tests/samples.
- Tests are checked by all five checkers (pyright relaxed, mypy, pyrefly,
  ty, zuban); samples by pyright, pyrefly, and ty. All run in a relaxed/
  basic profile so authors aren't forced into over-annotation.
- Add pyrightconfig.tests.json and bump sample pyright configs to basic.
- Unify test/sample typing onto the same parallel fan-out used by source
  pyright via run_command_items in task_runner.py.
- Make version-conditional imports symmetric: keep or drop the
  '# type: ignore' on both branches so results match across interpreter
  versions (local vs CI).
- Update SKILL.md, DEV_SETUP.md, and CODING_STANDARD.md for the five
  gating checkers and pyright on source+tests+samples.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Python: Fix merge regressions from main (typing + runtime)

Merging main into the type-checker split branch surfaced regressions that
the new five-checker test suite and unit tests caught:

Runtime fixes:
- anthropic: restore the dropped `cache_read_input_token_count` mapping in
  _parse_usage_from_anthropic (lost during merge conflict resolution).
- gemini: _get_function_calling_mode test helper returned str(enum)
  ('FunctionCallingConfigMode.AUTO') instead of the enum value ('AUTO').
- openai: _response_id_from_token test helper was an infinite self-recursion;
  return token['response_id'].
- orchestrations: reset output_events per approval iteration so the terminal
  output assertion counts only the final run.
- core: drop a stale duplicate harness test whose message ('non-negative')
  contradicted the source ('positive').
- purview: import PolicyLocation/PolicyScope/ProtectionScopeActivities/
  ExecutionMode used by the processor tests.

Type-checker fixes (tests, relaxed profile):
- core: pyright/mypy/pyrefly/ty/zuban green-ups across the harness, MCP,
  observability and types tests.
- anthropic/openai: route provider-namespaced UsageDetails keys through a
  dict cast (extra_items TypedDict unsupported by mypy/ty).
- purview: typed model constructors and cache-mock casts.
- ag-ui: annotate WorkflowContext[Any, Any] so yield_output accepts test
  payloads, guard Optional forwarded_props, and ty-ignore intentional bad args.

Source pyright (sole source checker) flagged unnecessary ignores newly
introduced by merged code in core _tools.py and declarative _declarative_base.py.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Python: Isolate per-package mypy cache in test-typing fan-out

The parallel test-typing fan-out runs many mypy processes concurrently,
all defaulting to a single shared ./.mypy_cache. Concurrent writes corrupt
the cache and mypy aborts with INTERNAL ERROR (intermittently, depending on
worker timing) -- which is why CI's Test Typing job failed on a shifting set
of packages while a single-package run was fine.

Give each mypy invocation an isolated cache dir keyed by its target paths so
incremental caching still works per package without races. Other checkers
(zuban/pyrefly/ty/pyright) maintain their own caches and are unaffected.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Python: Make lab pyright-only on source (drop source mypy)

Lab was the last package still running mypy on its source code, requiring
mypy-only `# type: ignore` comments that pyright (the sole source checker
everywhere else) flags as unnecessary. Align lab with the rest of the
monorepo:

- Remove the lab source mypy poe tasks (mypy-gaia/lightning/tau2) and the
  now-dead strict [tool.mypy] config block.
- Drop the 'Run lab mypy' CI step; lab source is type-checked by pyright only.

Lab tests remain covered by the workspace test-typing fan-out (mypy, pyrefly,
ty, zuban, pyright over tests using the relaxed root config).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Python: Fix test-typing regressions from latest main merge

A fresh merge from main brought in new test code never run under the
five-checker test-typing suite. Green up across the affected packages:

- core: narrow Optional span.attributes with 'and' guards in span filters
  and assert+cast the json.loads(...attributes[...]) reads (test_observability);
  match the existing as_agent ignore on the protocol-typed fixture (test_clients).
- openai: align new streaming tests with the established chat_options dict
  pattern (ChatOptions TypedDict isn't assignable to dict), route Optional
  .annotations[0] access through a small _first_annotation helper (mirrors the
  file's assert-not-None convention), and annotate a mapped ResponseStream.
- foundry_hosting: annotate error: dict[str, Any] = body.get(...) or {}
  (zuban needs the annotation).
- foundry: narrow ignores for the live AIProjectClient credential arg (pyrefly)
  and connections.get_default (zuban) SDK type gaps.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* updated pyright version

* pyright fix

* Python: Fix source typing for pyright 1.1.410

Pyright 1.1.410 tightened several checks. Apply the same source fixes as
upstream PR #6275:

- anthropic: import AsyncAnthropicBedrock from anthropic.lib.bedrock and
  AsyncAnthropicVertex from anthropic.lib.vertex (no longer re-exported from
  the anthropic top-level package -> reportPrivateImportUsage).
- core _types.py: cast the transform-hook result to UpdateT (reportAssignmentType).
- core _workflows/_events.py: annotate the @contextmanager helper as
  Generator[None] instead of Iterator[None] (reportDeprecated).
- redis: build the combined filter expression with an explicit loop instead of
  reduce(and_, ...), which pyright could no longer fully type (drops the now
  unused functools.reduce / operator.and_ imports).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Python: Accept plain-text body in Azure Functions workflow/run endpoint

The workflow_orchestrator already accepts plain strings as well as JSON
objects via context.get_input(), but the start_workflow_orchestration HTTP
handler only accepted JSON and returned 400 for any non-JSON body. This made
the functions integration tests that POST text/plain to /api/workflow/run
(e.g. test_09_workflow_shared_state) fail consistently with 400 != 202.

Fall back to the raw request body (decoded as UTF-8) when the body is not
JSON, rejecting only a truly empty body. The JSON path is unchanged.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-06-18 15:06:20 +00:00
Eduard van Valkenburg 571cae426c Python: Fix Azure AI Search citation URLs (#6453) 
* Fix Azure AI Search citation URLs

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Enrich MCP search citation metadata

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix Azure AI Search citation enrichment follow-ups

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address PR #6453 review comments

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* updated filter for paths

* also updated python paths

* reverted dotnet-format change

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: westey <164392973+westey-m@users.noreply.github.com>
 2026-06-16 16:13:30 +01:00
Evan Mattson 6b94315161 Python: Add timeout parameter to FoundryAgent to fix ConnectTimeout on multi-turn conversations (#6263) 
* Python: fix ConnectTimeout on multi-turn FoundryAgent conversations (#6241)

Expose a `timeout` parameter on `RawFoundryAgentChatClient`,
`_FoundryAgentChatClient`, `RawFoundryAgent`, `FoundryAgent`, and
`RawOpenAIChatClient` so callers can override the HTTP timeout used by
the underlying AsyncOpenAI client.

Root cause: `RawFoundryAgentChatClient.__init__` called
`project_client.get_openai_client()` without configuring any timeout,
inheriting the OpenAI SDK default of `httpx.Timeout(connect=5.0)`.
When connections are recycled between turns under load, the 5 s connect
timeout fires and surfaces as `openai.APITimeoutError`.

Fix:
- `load_openai_service_settings` (`_shared.py`): accept `timeout` and
  include it in `client_args` for all three `AsyncOpenAI`/
  `AsyncAzureOpenAI` construction paths.
- `RawOpenAIChatClient.__init__` (`_chat_client.py`): accept `timeout`
  and forward to `load_openai_service_settings`.
- `RawFoundryAgentChatClient.__init__` (`_agent.py`): accept `timeout`
  and set `openai_client.timeout = timeout` on the client returned by
  `get_openai_client()` before passing it to the base class.
- `_FoundryAgentChatClient`, `RawFoundryAgent`, `FoundryAgent`: accept
  and propagate `timeout` through the construction chain.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Add timeout parameter to FoundryAgent and RawOpenAIChatClient

Expose a timeout parameter on RawFoundryAgentChatClient,
_FoundryAgentChatClient, RawFoundryAgent, FoundryAgent, and
RawOpenAIChatClient. When provided, the value is applied to the
underlying AsyncOpenAI client so that connect timeouts under load
or after connection recycling can be tuned by callers.

Previously, get_openai_client() was called without any timeout
override, so the SDK default of httpx.Timeout(connect=5.0) was
inherited and could fire on multi-turn conversations where the
underlying connection is recycled between turns.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Python: Add `timeout` parameter to `FoundryAgent` to fix `ConnectTimeout` on multi-turn conversations

Fixes #6241

* fix(foundry): use with_options to avoid mutating shared OpenAI client timeout (#6241)

Replace direct assignment  with
 in
RawFoundryAgentChatClient.__init__.

The Azure AI Projects SDK caches and returns a shared AsyncOpenAI client
per AIProjectClient. Mutating its .timeout attribute leaked the override
to all other code paths sharing that client (other agents, user code).
with_options() returns a new client instance with the override applied,
leaving the original shared client untouched.

Update tests to assert with_options is called with the correct timeout
and that the original shared client's timeout attribute is not mutated.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* test(foundry): assert with_options return value flows to instance.client (#6241)

The four timeout propagation tests verified that with_options was called
but did not confirm that the returned (timeout-configured) client was
actually stored on the instance. A silent discard of the return value
would have left the tests green while the timeout had no effect.

Each test now captures the constructed instance and asserts:
  assert <instance>.client is openai_client_mock.with_options.return_value

Affected tests:
- test_raw_foundry_agent_chat_client_init_applies_timeout_to_openai_client
- test_raw_foundry_agent_chat_client_init_applies_timeout_with_preview_enabled
- test_foundry_agent_chat_client_init_propagates_timeout
- test_foundry_agent_init_propagates_timeout_to_openai_client

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <copilot@github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-06-04 18:25:18 +00:00
Benke Qu fa8cfb7567 Python: Fix FoundryAgent stripping model from PromptAgent requests (#5526) 
* Fix FoundryAgent stripping model from PromptAgent requests

Move run_options.pop('model', None) inside the _uses_foundry_agent_session()
conditional so that model is only stripped for hosted agent sessions (where
the server manages the model) and preserved for PromptAgent requests that
require it in the Responses API call.

Fixes #5525

* test: add coverage for resp_* continuation preserving model

Adds test_raw_foundry_agent_chat_client_prepare_options_preserves_model_for_resp_continuation
to explicitly verify that HostedAgent v1 / v2-no-session paths (where conversation_id
starts with resp_) preserve model and previous_response_id without triggering the
hosted-session gate.

---------

Co-authored-by: Benke Qu <bequ@microsoft.com>
Co-authored-by: Evan Mattson <35585003+moonbox3@users.noreply.github.com>
 2026-06-02 18:30:04 +00:00
Ben Thomas e0d0ad16a0 Python: feat(evals): Foundry Adaptive Evals integration (rubric-generation) (#6101) 
* Python: feat(evals): RubricScore type + EvalScoreResult.dimensions

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Python: feat(foundry-evals): RubricDimension + GeneratedEvaluatorRef + accept in evaluators=

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Python: feat(evals): parse rubric_scores from output items + assertion helpers

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Python: feat(evals): BaseAgent.as_eval_source / Workflow.as_eval_source

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Python: feat(foundry-evals): EvalGenerationSource + generate_rubric helper

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Python: feat(foundry-evals): YAML config loader + sample

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Python: fix(evals): address PR review feedback

Addresses 4 Copilot review comments on PR #6101:

1. assert_dimension_score_at_least: drop the (not evaluator or found_any) guard so require_applicable=True correctly raises when the named evaluator produces no entries for the dimension. Adds TestRubricAssertions covering the regression.

2. GeneratedEvaluatorRef docstring: reword to describe actual behaviour (pinning recommended, not required) so it matches the dataclass default and FoundryEvals warning path.

3. _poll_generation_job: switch from asyncio.get_event_loop() to get_running_loop() and bound the per-iteration sleep by remaining time, matching _poll_eval_run.

4. generate_rubric: type category as Literal['quality','safety'] and validate at the entry point with a ValueError; drop the silent 'invalid -> quality' rewrite in _generation_job_to_ref. Adds a regression test.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Python: feat(foundry-evals): hosted-agent-aware rubric generation

* Auto-detect hosted Foundry agents in agent_as_eval_source: when the
  agent's chat_client exposes a string agent_name (the convention used
  by RawFoundryAgentChatClient for PromptAgents/HostedAgents), emit a
  type='agent' EvalGenerationSource so the service fetches instructions
  and tools from the agent registry instead of relying on the local
  wrapper (which holds neither for hosted agents).
* Add hosted_agent_version kwarg and a new agent_version field on
  EvalGenerationSource so PromptAgent runs can pin to a specific hosted
  version for reproducible rubric generation.
* Add force_prompt_source escape hatch to bypass auto-detection and
  always emit a rendered prompt dossier - useful when the local wrapper
  carries overrides the service-side agent doesnt see.
* Fix _to_sdk_source for dataset sources: SDK ctor takes name=/version=,
  not dataset_name=/dataset_version=. The mismatch would raise TypeError
  against the real azure-ai-projects 2.3.0a* SDK; only unmocked
  integration paths were affected.

Tests cover: auto-detection happy path, versionless hosted agent,
explicit hosted_agent_version forwarding, force_prompt_source override,
non-string chat_client attrs (MagicMock test doubles) not mis-detected,
agent_version forwarded through _to_sdk_source, and the corrected
dataset SDK kwarg names.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(foundry-evals): accept canonical dimension_scores key per docs

The published Foundry rubric-evaluator output (Microsoft Learn 'Rubric evaluators' reference) places per-dimension breakdowns under properties.dimension_scores, not properties.rubric_scores. The parser now tries dimension_scores first and falls back to rubric_scores for preview-build compatibility, and tolerates non-list payloads (e.g. MagicMock auto-attrs) by trying the next candidate when parsing yields zero entries.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* feat(foundry-evals): add manual create_rubric_evaluator

Adds FoundryEvals.create_rubric_evaluator as the agent-framework surface over project_client.beta.evaluators.create_version. This is the manual counterpart to generate_rubric: callers supply RubricDimension instances (authored locally, ported from another framework, or hand-tuned) and we POST a RubricBasedEvaluatorDefinition. The service auto-attaches the non-editable residual dimension (general_quality for quality, general_policy_compliance for safety).

Per the Microsoft Learn 'Rubric evaluators' reference, the auto-generation path (create_generation_job) is primarily a portal/UI feature; external SDK clients with rich local agent context are better served by manual create_version. This keeps generate_rubric for users who want to round-trip through a Foundry-registered agent.

Validation up front: weight must be in [1,10], ids unique, descriptions non-empty, pass_threshold in [0,1]. The returned GeneratedEvaluatorRef is identical in shape to one obtained from generate_rubric, so downstream evaluators= lists work unchanged.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* samples(foundry-evals): manual rubric sample + namespace re-exports

Adds evaluate_with_manual_rubric_sample.py demonstrating the end-to-end dev scenario for FoundryEvals.create_rubric_evaluator: hand-author a list of RubricDimension, register via create_rubric_evaluator, then use the pinned GeneratedEvaluatorRef alongside built-in evaluators in an agent regression run.

Also re-exports RubricDimension, GeneratedEvaluatorRef, build_sources, and load_evals_config from agent_framework.foundry (both the lazy runtime shim and the type stub) so the rubric samples can import everything from a single namespace; the auto-generate sample was previously broken because the shim was missing build_sources / load_evals_config.

Updates the foundry-evals README with a chooser entry for the two rubric paths.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* feat(foundry-evals): remove rubric creation flows; keep consumption only

Reframes agent-framework as a pure consumer of Foundry rubric evaluators: scoring against rubrics that already exist (authored in the Foundry portal or via the dedicated SDK / REST surface) instead of creating them from the SDK.

Removed creation surface area:

- FoundryEvals.generate_rubric (auto-generate path) and create_rubric_evaluator (manual path), plus all _GenerationSdkTypes / _ManualRubricSdkTypes / _to_sdk_dimensions / _coalesce_generation_sources / _to_sdk_source / _poll_generation_job / _generation_job_to_ref / _evaluator_version_to_ref / _get_beta_evaluators / _import_*_sdk_types helpers.

- EvalGenerationSource (the input source discriminator), RubricDimension (the input dimension type), agent_as_eval_source / workflow_as_eval_source / _detect_hosted_foundry_agent helpers, and the YAML-config loader (_evals_config.py with RubricGenerationSpec / RubricSourceSpec / parse_evals_config / load_evals_config / build_sources).

- BaseAgent.as_eval_source / Workflow.as_eval_source plus the _render_agent_dossier / _render_workflow_dossier helpers in core. These existed only to feed the now-removed generation pipeline.

- Samples evaluate_with_generated_rubric_sample.py, evaluate_with_manual_rubric_sample.py, and evaluators.yaml. Replaced with a short README section showing how to reference an existing rubric evaluator via GeneratedEvaluatorRef.

Kept (consumption surface):

- GeneratedEvaluatorRef, slimmed to (name, version, display_name). Still accepted alongside built-in evaluator strings in FoundryEvals(evaluators=[...]). Versionless refs still warn.

- RubricScore on EvalScoreResult.dimensions plus EvalResults.assert_dimension_score_at_least for per-dimension CI gates.

- _parse_dimension_entries / _extract_rubric_scores output parsing (both canonical dimension_scores and the legacy rubric_scores key).

Tests: 160/160 foundry unit tests and 71/71 core local-eval tests pass; pyright is clean across changed files. The pre-existing tests/core/test_telemetry.py::test_detect_hosted_fallback_import_error failure is unrelated and reproduces on the prior commit.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* samples(foundry-evals): add evaluate_with_rubric_sample

Adds a runnable end-to-end sample showing how to consume a pre-existing rubric evaluator created in Foundry: reference it with GeneratedEvaluatorRef(name, version), mix it with built-in evaluators in FoundryEvals, and gate CI with assert_dimension_score_at_least on a specific dimension.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(foundry-evals): satisfy mypy on _fetch_output_items

mypy infers OutputItemListResponse.sample as dict[str, object] | None while pyright correctly infers the typed Sample model. Cast to Any so both type checkers accept the attribute access pattern, rename the local to avoid shadowing the inner-loop sample binding, and drop the now-stale pyright suppressions.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* docs(foundry-evals): drop unpublished rubric-evaluators learn.microsoft.com link

The Adaptive Evals authoring docs are not yet published on Microsoft Learn, so the link 404s. Keep the descriptive text without the broken hyperlink; we can re-add it once the docs ship.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* test(foundry-evals): hoist repeated local imports to module top

Per code review feedback (eavanvalkenburg): the test file repeated 'from agent_framework_foundry._foundry_evals import ...' inside 22 test bodies and 'from agent_framework_foundry import GeneratedEvaluatorRef' inside 8 more. Move all of them to the existing top-level imports; the symbols are the same across tests and the local imports were redundant.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Ben Thomas <25218250+alliscode@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-06-01 23:01:56 +00:00
Eduard van Valkenburg 8091d052d8 Python: refresh dev dependencies and validate runtime bounds (#6238) 
Updates third-party dev dependencies across the Python workspace and
validates that all runtime dependency bounds still hold at both ends.

Dev dependency bumps (root, lab, declarative, durabletask):
- uv 0.11.6 -> 0.11.17, ruff 0.15.8 -> 0.15.15,
  pytest-asyncio 1.3.0 -> 1.4.0, mcp 1.27.0 -> 1.27.2,
  azure-monitor-opentelemetry 1.8.7 -> 1.8.8,
  poethepoet 0.42.1 -> 0.46.0, prek 0.3.9 -> 0.4.3,
  types-python-dateutil and types-PyYaml stub bumps.
- Transitive Dependabot items swept via lock: idna 3.11 -> 3.17,
  pip 26.0.1 -> 26.1.2.

Deliberately excluded:
- opentelemetry-sdk stays 1.40.0: azure-monitor-opentelemetry (incl.
  1.8.8) hard-pins opentelemetry-sdk==1.40.
- mypy stays 1.20.0 and pyright stays 1.1.408: the 2.1.0 / 1.1.409
  bumps introduce new diagnostics that fail type checking and need
  dedicated PRs.
- rich kept as a range: agentlightning (lab[lightning]) forces
  rich==13.9.4.

Code/formatting changes driven by the ruff upgrade:
- devui lifespan now uses try/finally so shutdown cleanup always runs
  (ruff RUF075).
- Removed unused TYPE_CHECKING imports in core and foundry flagged by
  ruff 0.15.15.
- Reapplied ruff 0.15.15 formatting to the files it changed.

Validation: validate-dependency-bounds-test "*" passes (31/31 lower +
31/31 upper); typing 62/62; lint 31/31; devui tests pass.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-06-01 17:53:56 +00:00
Tao Chen 0578f4c910 Backfill chat span request model if it's unknown and response model is avaliable (#6160) 2026-05-28 20:03:46 +00:00
Yufeng He 55dc3ce734 Python: fix: pass Foundry agent default headers (#6040) 
* fix: pass Foundry agent default headers

* test: loosen Foundry default header assertions
 2026-05-28 10:08:14 +00:00
Eduard van Valkenburg d5c07f2623 Python: feat(foundry): add to_prompt_agent / deploy_as_prompt_agent (experimental) (#5959) 
* feat(foundry): add experimental to_prompt_agent converter

Adds `to_prompt_agent(agent)`, an experimental converter
(`ExperimentalFeature.TO_PROMPT_AGENT`) that turns an Agent Framework
`Agent` into a Foundry `PromptAgentDefinition` ready to publish via
`AIProjectClient.agents.create_version(...)`.

Behaviour:

* `agent.client` must be a `FoundryChatClient` (or subclass); otherwise
  `TypeError` is raised. The model deployment name is lifted from the
  bound client so the same Agent definition used for local runs can be
  published as a hosted prompt agent without restating the model.
* Foundry SDK tool instances (from `FoundryChatClient.get_*_tool()`) are
  passed through unchanged. AF `FunctionTool`s (and `@tool`-decorated
  callables) are emitted as Foundry `FunctionTool` declarations.
* Local AF MCP tools cannot be expressed in a `PromptAgentDefinition`;
  the converter raises `ValueError` and points at
  `FoundryChatClient.get_mcp_tool()` for hosted MCP servers.
* The converter walks both `agent.default_options["tools"]` and
  `agent.mcp_tools` because `normalize_tools()` splits local MCP off
  into its own list.

Re-exported through the `agent_framework.foundry` lazy-loading namespace
(updates both `__init__.py` and the `__init__.pyi` type stub).

Adds a portable-agent sample showing the same `Agent` driven through
both `agent.run(...)` and `to_prompt_agent(agent)`, and a README section
covering the new converter.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* chore(samples): remove snippet tags from portable agent sample

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* chore(samples): inline FoundryChatClient and enable prompt-agent publish

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* chore(samples): drop async credential context manager

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* docs(foundry): trim README to_prompt_agent example to publish-only flow

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* docs(foundry): note FoundryAgent runs @tool callables for deployed prompt agents

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(foundry): address review comments on to_prompt_agent converter

* Construct `PromptAgentDefinition` `Tool` from a dict via `**tool_item`
  unpacking rather than the positional Mapping constructor \u2014 cleaner and
  matches the typical Pydantic / Azure SDK pattern.
* Drop the redundant `isinstance(mcp_tool, MCPTool)` guard in
  `_convert_tools`; the parameter is already typed `Iterable[MCPTool]` so
  the second `raise` was unreachable. The remaining single `raise`
  fires for every entry as intended.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(foundry): match Agent.__init__ model resolution in to_prompt_agent

* Read the model from `agent.default_options.get("model")` first,
  falling back to `agent.client.model`. This mirrors the order
  `Agent.__init__` uses (`_agents.py:740`) when assembling
  default_options, so the model the agent runs with is the same model
  the converter publishes \u2014 e.g. when the caller passes
  `default_options={"model": "..."}` to override the bound client.
* Updated the missing-model error message to point at both the client
  and the default_options paths.
* Added tests:
  * tool-only agent with no `instructions` produces a definition
    where `instructions` is `None` and is omitted from the dict
    payload (`Agent.__init__` strips None values from default_options
    before storing them).
  * `default_options['model']` wins over the bound client's model.
  * Fallback to client.model when default_options has no model.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* feat(foundry): add deploy_as_prompt_agent helper + samples

Adds `deploy_as_prompt_agent(agent)`, a convenience wrapper around
`to_prompt_agent` that reuses the bound FoundryChatClient's project
client to call `project_client.agents.create_version(...)`. Defaults
`agent_name` / `description` from `agent.name` / `agent.description`
so the Agent stays the single source of truth.

* Exposed from `agent_framework_foundry` and the lazy-loading
  `agent_framework.foundry` namespace (including the .pyi stub).
* Marked experimental with the existing
  `ExperimentalFeature.TO_PROMPT_AGENT` tag.
* Tests cover the happy path, name/description defaulting, explicit
  override, no-name error, metadata + description forwarding, extra
  kwargs passthrough, and the experimental metadata.

Samples:
* Renamed the existing sample to `creating_prompt_agents.py`, drops
  'portable' wording, presents `deploy_as_prompt_agent` first as the
  recommended path and `to_prompt_agent` + `AIProjectClient` as the
  two-step alternative, and adds a cleanup step that deletes the
  published agent so re-runs stay idempotent.
* New `using_prompt_agents.py` shows the end-to-end loop: deploy the
  agent, connect to it with `FoundryAgent` passing the same local
  `@tool` callable, run a query against the deployed prompt agent,
  then clean up.

README updated to introduce `deploy_as_prompt_agent` as the
recommended path and link to both runnable samples.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(foundry): restore missing-model ValueError in to_prompt_agent

The check was accidentally dropped while reworking docstrings in the
previous commit. Test `test_to_prompt_agent_rejects_missing_model`
exercises this path and was failing on CI as a result.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* refactor(foundry): rename deploy_as_prompt_agent -> create_prompt_agent

Renames the helper across the foundry package, core lazy-loader stubs,
tests, README and samples. The new name better matches the action
performed (a prompt-agent definition is created in Foundry) and is
consistent with the surrounding ''create_*'' API surface.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* refactor(foundry): drop create_prompt_agent, enrich to_prompt_agent params

Remove the create_prompt_agent helper and consolidate on to_prompt_agent.
Expose every PromptAgentDefinition parameter that has either an Agent
Framework equivalent (sourced from default_options) or no equivalent
(accepted as a keyword argument).

* default_options-sourced (with kwarg overrides):
  temperature, top_p, string tool_choice
* kwarg-only Foundry knobs:
  reasoning, text, structured_inputs, rai_config, ToolChoiceParam tool_choice

Precedence is always: explicit keyword > default_options entry > unset.

Tests cover every path (defaults, default_options, kwargs, kwarg override).
Samples and README rewritten around the enriched to_prompt_agent.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* refactor(foundry): single source of truth for prompt-agent options

Stop duplicating the generation-parameter surface between FoundryChatOptions
and to_prompt_agent. Translate every field with an Agent Framework equivalent
(temperature, top_p, tool_choice, reasoning, response_format/text/verbosity)
from agent.default_options via a new RawFoundryChatClient helper
_prepare_prompt_agent_options. Only Foundry-specific fields with no AF
equivalent — structured_inputs and rai_config — remain as keyword arguments
on to_prompt_agent.

- tool_choice is dropped when there are no tools (mirrors _prepare_options
  semantics and avoids polluting tool-less prompt agents with Agent.__init__'s
  'auto' default).
- response_format Pydantic models route through
  openai.lib._parsing._responses.type_to_text_format_param; dict shapes go
  through the existing _prepare_response_and_text_format helper.
- default_options is not mutated; text dict is defensively copied.

Tests, README, and creating_prompt_agents.py sample updated to reflect the
new single-source model.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* docs(foundry): consolidate prompt-agent sample

Drop creating_prompt_agents.py (the publish-only variant) and rename
using_prompt_agents.py to foundry_prompt_agents.py so the single sample
covers the full convert -> publish -> connect -> run loop. Update the
README link list accordingly.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* docs(foundry): run local Agent + deployed agent in same sample

Add an agent.run() call against the local Agent before publishing, then run
the deployed prompt agent on the same query. Expand the docstring with a
compare-and-contrast covering runtime/latency, configurability, and
persistence/sharing differences between the two execution paths.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* test(foundry): cover conflicting response_format + text.format in to_prompt_agent

Exercises the ValueError path when a Pydantic response_format would overwrite
an explicit text.format mapping with a different shape. Lifts _chat_client.py
coverage from 89% to 90%.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* refactor(foundry): move _prepare_prompt_agent_options into _to_prompt_agent

Lift the translation helper off RawFoundryChatClient and into the
_to_prompt_agent module as a module-private function that takes the client
as its first argument. The chat client no longer needs to carry a method
whose only consumer is the prompt-agent converter, while still serving as
the source of the request-path helper (_prepare_response_and_text_format)
that the converter reuses for dict-shaped response_format values.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* docs(python): codify GA terminology + post-run docs review

Add two pieces of guidance to python/AGENTS.md:

* Terminology - reserve 'GA' for hosted services; use 'released' or 'stable'
  for Agent Framework code/features to match the feature-lifecycle stages.
* Maintaining Documentation - review AGENTS.md and skills at the end of every
  run and update any guidance the conversation made stale; before adding a
  new principle, ask the user to confirm it should be captured.

Also pulls in a docstring fix in foundry_prompt_agents.py that swaps the
stray 'GA' for 'released', applying the new terminology rule.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* address PR review: strict=True default, Tool._deserialize dispatch, sample cleanup safety

- FunctionTool published as strict=True so the server-side schema validation
  matches what the local FoundryAgent(tools=[same_callable]) dispatcher
  enforces. AF FunctionTool has no 'strict' attribute, so the safer default
  is used uniformly instead of silently downgrading to a permissive contract.
- _validate_mapping_tool now dispatches through ProjectsTool._deserialize so
  dict-shaped tools rehydrate to the concrete subclass (FunctionTool,
  WebSearchTool, ...) via the 'type' discriminator instead of returning a
  generic Tool. Added a test that asserts isinstance(WebSearchTool) and a
  new test for the function-typed dict path.
- foundry_prompt_agents.py sample now wraps credential + project client in
  async with and the create_version / run flow in try/finally so a failure
  on connect or run still deletes the published prompt agent rather than
  leaving an orphaned, billable resource in the user's Foundry project.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(ci): correct linkspector ignorePattern typo (./pulls -> ./pull)

GitHub PR URLs use the singular segment /pull/N (compare to /issues/N
for issues). The existing './pulls' ignore pattern never matched
anything as a result, so legitimately stale PR links (e.g. PRs deleted
from forks) surface as linkspector failures on unrelated PRs.

This is the same convention the './issues' rule above already follows.
Fixes the markdown-link-check failure on a dangling link in
dotnet/src/Microsoft.Agents.AI.DurableTask/CHANGELOG.md.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-05-27 13:31:21 +00:00
Yufeng He 46ed66cfd5 Python: include tool definitions for Foundry agent evals (#5974) 2026-05-21 16:23:36 +00:00
Eduard van Valkenburg 47f5c3397f Python: feat(foundry): add experimental hosted tool factories on FoundryChatClient (#5958) 
* feat(foundry): add experimental hosted tool factories on FoundryChatClient

Adds eight new `@experimental` static factory methods on `FoundryChatClient`
covering Foundry-hosted tools that previously had no helper:

- get_azure_ai_search_tool
- get_sharepoint_tool
- get_fabric_tool
- get_memory_search_tool
- get_computer_use_tool
- get_browser_automation_tool
- get_bing_custom_search_tool
- get_a2a_tool

All factories are marked with the new `ExperimentalFeature.FOUNDRY_TOOLS` tag
and resolve the underlying `azure-ai-projects` preview classes lazily through
a `_require_sdk_class` helper so older SDK versions still import cleanly and
fail with a clear `ImportError` only on use.

Tests cover each factory's return type and field wiring, the experimental
metadata, and the missing-SDK-class fallback.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* test(foundry): address review comments on tool-factory tests

* Skip preview-tool tests gracefully (`_skip_if_sdk_class_missing`) when
  the installed `azure-ai-projects` does not expose the required preview
  class, matching the lazy-import guard in production code so the test
  suite stays green on older SDK installs.
* Add `filterwarnings("ignore::FutureWarning")` to each new tool-factory
  test (and the parametrized metadata test) so they remain stable under
  strict warning configurations \u2014 the global dedup in
  `_feature_stage._WARNED_FEATURES` makes `pytest.warns` brittle across
  ordered runs.
* Use `monkeypatch.setattr(..., None, raising=False)` instead of
  `delattr` in the missing-SDK-class test so it works for modules that
  implement PEP 562 `__getattr__`.
* Split the long `get_bing_custom_search_tool` return into two lines for
  readability.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(foundry): harden tool-factory kwargs against silent override

* Reorder the dict-literal kwargs assembly in get_azure_ai_search_tool,
  get_memory_search_tool, and get_bing_custom_search_tool so explicit
  parameters always take precedence over **kwargs (matching the safe
  pattern already used in get_a2a_tool). This prevents a caller
  passing `project_connection_id`, `index_name`, `memory_store_name`,
  `scope`, or `instance_name` through `**kwargs` from silently
  overriding the explicit security-sensitive arguments.
* Update the README experimental note to reflect once-per-feature-id
  dedup semantics of `_feature_stage._WARNED_FEATURES` rather than
  claiming a per-factory "first use" warning.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* feat(foundry): split FOUNDRY_TOOLS / FOUNDRY_PREVIEW_TOOLS, add bing-grounding

- Add ExperimentalFeature.FOUNDRY_PREVIEW_TOOLS to distinguish wrappers around
  preview Foundry SDK tool classes (Sharepoint/Fabric/Memory/ComputerUse/
  BrowserAutomation/BingCustomSearch/A2A) from FOUNDRY_TOOLS, which is for
  GA-SDK wrappers that are simply new in agent-framework-foundry
  (AzureAISearch, BingGrounding).
- Add get_bing_grounding_tool factory and a 'Choosing a web grounding tool'
  comparison block on get_web_search_tool / get_bing_grounding_tool /
  get_bing_custom_search_tool docstrings.
- Drop the _require_sdk_class lazy resolver: every guarded class is available
  at azure-ai-projects>=2.1.0 (the package floor), so import them eagerly.
  Concrete return types replace 'Any'.
- README: split the experimental factories into two tables, one per feature
  flag, with a note explaining the distinction.
- Tests: split into FOUNDRY_TOOLS / FOUNDRY_PREVIEW_TOOLS factory cases;
  drop the obsolete missing-SDK-class ImportError test.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-05-21 08:39:08 +00:00
Tao Chen 72a6157c6a [BREAKING] Python: Enable instrumentation by default (#5865) 
* Enable instrumentation by default

* Update samples

* Optimization when span is not recording

* Address Copilot comments

* Revert uv.lock

* Add warning

* Formatting

* Fix mypy

* Add disable_instrumentation() with sticky user-intent semantics

Add a public disable_instrumentation() entry point so users can explicitly opt
out of Agent Framework telemetry, with a sticky-disable flag that makes the
user's intent "leading" — no framework code path (foundry's
configure_azure_monitor, configure_otel_providers, enable_instrumentation,
enable_sensitive_telemetry, or direct OBSERVABILITY_SETTINGS.enable_*
writes) can re-enable instrumentation until the user explicitly clears the
disable with enable_instrumentation(force=True) /
enable_sensitive_telemetry(force=True).

Also addresses the two remaining unresolved review threads on the PR:
1. test_observability_settings_defaults_instrumentation_true pins the new
   "ENABLE_INSTRUMENTATION defaults to True when env unset" behavior.
2. test_enable_instrumentation_reads_env_sensitive_data restores coverage
   for the post-import load_dotenv() fallback path.

Implementation:
- ObservabilitySettings.enable_instrumentation / enable_sensitive_data become
  properties backed by _enable_*. While _user_disabled is True, the getters
  return False and the setters drop True writes (defense in depth so third-
  party writes can't subvert the disable).
- Public is_user_disabled read-only property lets integrations (e.g. foundry's
  configure_azure_monitor) cheaply check the disable state without poking at
  privates.
- enable_instrumentation() and enable_sensitive_telemetry() short-circuit with
  an info log when disabled; gain a force=True kwarg that clears the disable.
- configure_otel_providers() still creates providers / exporters / views so a
  later force-enable can use them, but logs an info message when called while
  disabled.
- Foundry's FoundryChatClient.configure_azure_monitor and
  FoundryAgent.configure_azure_monitor early-return when the user has
  disabled, so Azure Monitor's global providers aren't installed unnecessarily.

Tests: 11 new tests covering default-on, env re-read at call time, sticky
behavior against each re-enable surface (enable_instrumentation,
enable_sensitive_telemetry, configure_otel_providers, direct attribute
writes), force=True override, re-arming the disable, and the __all__ export.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* docs: document disable_instrumentation() and force=True paths

Add a "Disabling instrumentation" section to the observability sample README
that walks through:

- The distinction between the ENABLE_INSTRUMENTATION env var (initial,
  non-sticky) and disable_instrumentation() (process-wide, sticky).
- Why the sticky semantics matter: framework integrations like
  FoundryChatClient.configure_azure_monitor() can call
  enable_instrumentation() as part of their setup, and the user's opt-out
  needs to win.
- All five surfaces guarded by the sticky disable (property reads, public
  enable functions, configure_otel_providers, direct attribute writes,
  is_user_disabled-aware integrations).
- The force=True escape hatch on both enable_instrumentation() and
  enable_sensitive_telemetry().
- How third-party integrations should consult OBSERVABILITY_SETTINGS.is_user_disabled.
- The limits of the disable (does not tear down existing providers /
  in-flight spans / third-party instrumentation, does not persist across
  processes).

Cross-links the new section from the ENABLE_INSTRUMENTATION row in the env
vars table.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* docs: soften disable_instrumentation() overclaim about telemetry guarantees

Replace 'no telemetry will be emitted no matter what' (which is too strong,
since callers can still pass force=True or mutate private attributes) with
language framing the disable as a user-intent contract that library and
framework code is expected to honor: the framework actively short-circuits
the public enable paths, force=True and private-attribute writes are
acknowledged as out-of-contract escape hatches that integrations should
not use on the user's behalf.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* docs: correct observability Dependencies section

- opentelemetry-sdk is no longer a hard dependency; it is lazily imported by
  create_resource(), create_metric_views(), and configure_otel_providers()
  with a clear ImportError when missing. Day-to-day instrumentation works
  with opentelemetry-api alone provided some other component configures the
  global OpenTelemetry providers (Azure Monitor, an APM agent, application
  bootstrap, etc.).
- opentelemetry-semantic-conventions-ai is no longer used anywhere in the
  source; remove it from the listed dependencies.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* docs: replace stale observability migration guide with current PR's only relevant migration

The old guide documented the move away from setup_observability(otlp_endpoint=...)
which was an earlier-release API change unrelated to this PR and stale enough that
it's more confusing than helpful at this point. Replace it with a short note on the
single migration this PR introduces: callers of
enable_instrumentation(enable_sensitive_data=True) should switch to
enable_sensitive_telemetry(). Cross-link to the Disabling instrumentation section
for the rare 'force on without enabling sensitive data' use case where
enable_instrumentation() still applies.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Eduard van Valkenburg <eavanvalkenburg@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-05-20 11:52:08 +00:00
Evan Mattson e56e6dad4d Python: Remove bespoke Foundry toolbox helpers; standardize on MCP for toolbox consumption (#5671) 
* Remove Foundry toolbox helpers; standardize on MCP for toolbox consumption

- Remove RawFoundryChatClient.get_toolbox() and its fetch_toolbox import
- Remove fetch_toolbox, select_toolbox_tools, get_toolbox_tool_name,
  get_toolbox_tool_type, FoundryHostedToolType, ToolboxToolSelectionInput
  from agent_framework_foundry._tools
- Remove ExperimentalFeature.TOOLBOXES from _feature_stage.py (no consumers)
- Drop toolbox re-exports from agent_framework_foundry/__init__.py and
  agent_framework.foundry namespace
- Update _sanitize_foundry_response_tool docstring to remove toolbox framing;
  sanitization logic itself is unchanged
- Update _agent.py docstring: 'toolbox-fetched MCP' → 'hosted MCP'
- Delete tests/test_toolbox.py (all tests covered removed helpers)
- Update test_foundry_chat_client.py: rename/redoc tests that mentioned
  toolbox but test sanitization that remains
- Delete foundry_chat_client_with_toolbox.py (bespoke toolbox API sample)
- Delete foundry_toolbox_context_provider.py (relied on select_toolbox_tools)
- Rename foundry_chat_client_with_toolbox_mcp.py →
  foundry_chat_client_with_toolbox.py (canonical MCP pattern)
- Rewrite 04_foundry_toolbox/main.py to use MCPStreamableHTTPTool
- Update provider/README, context_providers/README, 04_foundry_toolbox/README

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(samples): update 06_files sample to consume toolbox via MCP (#5670)

Replace removed get_toolbox/select_toolbox_tools APIs with
MCPStreamableHTTPTool, using allowed_tools=["code_interpreter"] to
select only the code interpreter from the toolbox endpoint.

Update .env.example and README to use FOUNDRY_TOOLBOX_ENDPOINT
instead of TOOLBOX_NAME.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(foundry): remove non-existent toolbox helper APIs from README (#5670)

Remove the 'fetch, optionally filter, and pass tools directly' pattern
from the FoundryChatClient toolbox documentation, as select_toolbox_tools
and get_toolbox were removed. Only the MCP endpoint pattern is documented.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(foundry): remove residual toolbox docstring references and reproduction report

Remove REPRODUCTION_REPORT.md (workflow artifact that should not be committed),
and update two remaining docstring references that still said 'toolbox reads'
/'toolbox definition' after the toolbox helpers were removed.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Python: Remove bespoke Foundry toolbox helpers; standardize on MCP for toolbox consumption

Fixes #5670

* fix(#5670): resolve toolbox endpoint from TOOLBOX_NAME fallback; add namespace regression tests

- Add _resolve_toolbox_endpoint() helper in 04_foundry_toolbox/main.py and
  06_files/main.py that prefers FOUNDRY_TOOLBOX_ENDPOINT but falls back to
  deriving the MCP URL from FOUNDRY_PROJECT_ENDPOINT + TOOLBOX_NAME — fixing
  the startup KeyError when agents are deployed via azd provision (which injects
  TOOLBOX_NAME, not FOUNDRY_TOOLBOX_ENDPOINT).
- Update 04_foundry_toolbox/.env.example to use FOUNDRY_TOOLBOX_ENDPOINT
  (consistent with 06_files).
- Add TOOLBOX_NAME env var to 06_files/agent.yaml so deployed agents have it
  available for the fallback derivation.
- Update both READMEs to document the two ways to supply the toolbox endpoint.
- Add test_foundry_namespace_no_longer_exposes_toolbox_helpers() with negative
  assertions for FoundryHostedToolType, get_toolbox_tool_name,
  get_toolbox_tool_type, and select_toolbox_tools — guarding against accidental
  re-introduction of removed symbols.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(samples): fail fast on empty FOUNDRY_TOOLBOX_ENDPOINT; add unit tests

Addresses review feedback for #5670:

- In _resolve_toolbox_endpoint() (04_foundry_toolbox/main.py and
  06_files/main.py) change the walrus-operator check from a truthy
  test to an explicit 'is not None' guard.  An explicitly set empty
  string now raises ValueError immediately with a clear message
  instead of silently falling through to the fallback URL
  construction.

- Add tests/samples/hosting/test_toolbox_endpoint.py covering both
  sample modules:
    (a) FOUNDRY_TOOLBOX_ENDPOINT set → returned as-is
    (b) FOUNDRY_TOOLBOX_ENDPOINT set to empty string → ValueError
    (c) fallback constructs URL from FOUNDRY_PROJECT_ENDPOINT + TOOLBOX_NAME,
        stripping trailing slashes
    (d) neither variable group set → KeyError

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address review feedback: remove extraneous test and docstring content

- Remove test_foundry_namespace_no_longer_exposes_toolbox_helpers (no longer warranted)
- Remove docstring from _agent.py _prepare_tools_for_openai (extraneous)
- Trim _chat_client.py _prepare_tools_for_openai docstring to one-liner (toolbox references no longer relevant)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix: remove remaining extraneous docstring from RawFoundryChatClient._prepare_tools_for_openai

Address review comment on PR #5671: reviewer noted the description
isn't warranted now that toolbox helpers have been removed. Matches
the pattern in RawFoundryAgentChatClient which has no docstring.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <copilot@github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-05-06 23:56:16 +00:00
Evan Mattson f3db60fa65 Python: Support GPT-5 verbosity option and restore Foundry agent_reference (#5619) 
* Python: Support GPT-5 verbosity option and restore Foundry agent_reference

Adds verbosity as a typed Literal["low","medium","high"] field on
OpenAIChatOptions (Responses API) and OpenAIChatCompletionOptions (Chat
Completions API), set in the same way as the existing reasoning options.
For the Responses API, top-level verbosity is translated to the nested
text.verbosity shape the OpenAI service expects. The same field flows
through to FoundryChatClient via the existing FoundryChatOptions alias.

Also fixes #5582: PR #5447 removed the agent_reference injection from
RawFoundryAgentChatClient._prepare_options, so first-turn calls against
a Foundry Prompt Agent went out without model and without agent_reference
and were rejected by the Responses API with "Missing required parameter:
'model'". Restores the injection on the non-preview path
(allow_preview=False) and adds a guard test that asserts the preview
path does not inject agent_reference, since the preview SDK injects it
via project_client.get_openai_client(agent_name=...).

Closes #5516
Closes #5582

* Python: Address Copilot review on PR #5619

- Foundry verbosity sample docstring: replace the misleading "set deployment
  name on model=" instruction with the actual env-var pattern the sample relies
  on (FOUNDRY_PROJECT_ENDPOINT and FOUNDRY_MODEL).
- _build_agent_reference docstring: clarify the helper is used for both
  Prompt Agents and HostedAgents on the non-preview path.
- Add a Responses API test that locks in the documented precedence rule:
  when both top-level verbosity and text["verbosity"] are supplied, the
  top-level value wins.

* Python: Drop redundant Foundry verbosity sample and list OpenAI sample in README

- Remove samples/02-agents/providers/foundry/foundry_chat_client_verbosity.py
  per review feedback. The verbosity functionality is identical across the
  OpenAI and Foundry clients (FoundryChatOptions is an alias of
  OpenAIChatOptions), so a single sample on the OpenAI side is sufficient.
- Add the new client_verbosity.py entry to the OpenAI samples README.
 2026-05-04 21:21:40 +00:00
Giles Odigwe 7b70f80036 Python: Surface oauth_consent_request events from Responses API in Foundry clients (#5070) 
* Fix Foundry clients not surfacing oauth_consent_request events (#5054)

Override _parse_chunk_from_openai in both RawFoundryChatClient and
RawFoundryAgentChatClient to intercept response.output_item.added
events with item.type == 'oauth_consent_request'. The consent link
is validated (HTTPS required) and converted to
Content.from_oauth_consent_request, which the AG-UI layer already
knows how to emit as a CUSTOM event.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address PR review feedback for #5054 OAuth consent parsing

- Extract shared helper (try_parse_oauth_consent_event) to avoid
  duplicated logic between RawFoundryChatClient and
  RawFoundryAgentChatClient
- Use urllib.parse.urlparse() for HTTPS validation instead of
  case-sensitive startswith check
- Sanitize log messages to avoid leaking consent_link tokens;
  log only item id
- Add model=self.model to ChatResponseUpdate to match parent behavior
- Add assertions on role, raw_representation, and model in happy-path
  tests
- Add test for empty-string consent_link
- Add test verifying non-oauth events delegate to super()

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Handle response.oauth_consent_requested top-level event (#5054)

Add support for the top-level response.oauth_consent_requested stream
event in addition to the response.output_item.added variant. The
service may emit either form; handle both so the consent link is
reliably surfaced.

Extract _validate_consent_link helper within _oauth_helpers.py to
reduce nesting.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address review feedback for #5054: Python: [Bug]: `FoundryAgent` (Responses API) Does Not Surface `oauth_consent_request` as a CUSTOM AG-UI Event

* Address review feedback: defensive getattr and dedicated helper tests (#5054)

- Use getattr(event, 'type', None) in try_parse_oauth_consent_event
  for defensive access against malformed events without a type attribute
- Add test_oauth_helpers.py with unit tests for _validate_consent_link
  and try_parse_oauth_consent_event covering edge cases:
  - HTTPS URL with empty netloc (https:///path)
  - Warning log messages for rejected consent links
  - Event objects missing 'type' attribute

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address review feedback for #5054: Python: [Bug]: `FoundryAgent` (Responses API) Does Not Surface `oauth_consent_request` as a CUSTOM AG-UI Event

* Fix mypy: match _parse_chunk_from_openai signature with superclass

Add seen_reasoning_delta_item_ids parameter to _parse_chunk_from_openai
overrides in both RawFoundryChatClient and RawFoundryAgentChatClient to
match the updated superclass signature on main. Update super() calls and
test assertions accordingly.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <copilot@github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Evan Mattson <evan.mattson@microsoft.com>
 2026-04-24 09:59:14 +00:00
Eduard van Valkenburg 62e02da698 Python: update FoundryAgent for hosted agent sessions (#5447) 
* fixes to FoundryAgent to connect to new hosted agents

Co-authored-by: Copilot <copilot@github.com>

* fix mypy

Co-authored-by: Copilot <copilot@github.com>

* Python: remove Foundry service session helpers

Remove the public hosted-agent service session CRUD helpers from FoundryAgent and drop the related feature-stage inventory entry.

Update the hosted-agent sample to create and delete service sessions directly through the preview AIProjectClient APIs, and tighten a few test harnesses surfaced by full workspace validation.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix from merge

* fix hosted env detection

Co-authored-by: Copilot <copilot@github.com>

* reverted sample update

* fix tests and code

Co-authored-by: Copilot <copilot@github.com>

* remove aenter

* skipping some tests

Co-authored-by: Copilot <copilot@github.com>

---------

Co-authored-by: Copilot <copilot@github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-04-24 09:25:03 +00:00
Tao Chen 0989e68d1c Python: Fix user agent prefix (#5455) 
* Fix hosting user agent missing

* Fix other providers

* Add more tests

* comments

* Fix tests
 2026-04-23 23:40:38 +00:00
Evan Mattson b084d0461d Python: (foundry): stop emitting [TOOLBOXES] warning for every FoundryChatClient call (#5440) 
* Python: Foundry: make response tool sanitizer internal, drop TOOLBOXES warning

sanitize_foundry_response_tool runs on every tool passed to the Foundry
Responses API, so its @experimental(TOOLBOXES) decorator was emitting a
[TOOLBOXES] ExperimentalWarning for any FoundryChatClient call, even when
no toolbox was involved. The function isn't in __all__ and has no external
callers. Rename to _sanitize_foundry_response_tool and drop the decorator;
the actual toolbox-facing public helpers remain gated.

* Python: Foundry: silence pyright on intentional cross-module private import
 2026-04-23 22:19:37 +00:00
Evan Mattson fffd0acb3e Python: fix(foundry): reconcile toolbox hosted-tool payloads with Responses API (#5414) 
* fix(foundry): reconcile toolbox hosted-tool payloads with Responses API

* docs(foundry): update create_sample_toolbox docstring to reflect all tools created
 2026-04-22 17:43:26 +00:00
chetantoshniwal aa582d021d Python: feat(evals): add ground_truth support for similarity evaluator (#5234) 
* feat(evals): add ground_truth support for similarity evaluator

- Include expected_output as ground_truth in Foundry JSONL dataset rows
- Add ground_truth to item schema and data mapping for similarity evaluator
- Add expected_output parameter to evaluate_workflow
- Add similarity Pattern 3 to evaluate_agent and evaluate_workflow samples
- Add tests for ground_truth in dataset, schema, and evaluate_workflow

* Apply suggestions from code review

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* fix: wrap long line to satisfy ruff E501

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
 2026-04-21 19:40:53 +00:00
Evan Mattson 04aaf0c1fe Python: Add support for Foundry Toolboxes (#5346) 
* Add support for the Foundry Toolbox in MAF

Introduces a Foundry Toolbox integration: FoundryChatClient gains a
get_toolbox() helper plus select_toolbox_tools(), normalize_tools in
the core package flattens tool-collection wrappers (ToolboxVersionObject
and generic iterables, while leaving Pydantic BaseModel instances
alone), and the new agent_framework.foundry namespace re-exports the
toolbox helpers. Ships with unit tests, a sample, and a design doc.

azure-ai-projects is pinned to the public >=2.0.0,<3.0 range and the
lockfile resolves from public PyPI. The toolbox test module skips when
Toolbox* types are unavailable so CI stays green until the public 2.1.0
SDK lands. OMC tooling directories (.omc/, .omx/) are gitignored.

* Update to latest azure ai projects package

* Improve sample

* Rename ADR to 0025

* Update ADR

* Apply suggestion from @alliscode

Co-authored-by: Ben Thomas <ben.thomas@microsoft.com>

* Improve samples

* Update test

---------

Co-authored-by: Ben Thomas <ben.thomas@microsoft.com>
 2026-04-20 23:56:01 +00:00
Matt Van Horn 18d1ba3624 Python: Strip tools from FoundryAgent request when agent_reference is present (#5101) 
_prepare_options() now removes tools, tool_choice, and parallel_tool_calls
from run_options after injecting agent_reference. The Foundry API rejects
requests containing both fields. FunctionTools are still invoked client-side
by the function invocation layer.

Fixes #5087

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Evan Mattson <35585003+moonbox3@users.noreply.github.com>
 2026-04-08 04:35:01 +00:00
Evan Mattson 36cafe4e5a Python: Raise clear handler registration error for unresolved TypeVar annotations (#4944) 
* Raise clear handler registration error for unresolved TypeVar (#4943)

Detect unresolved TypeVar in message parameter annotations during handler
registration in both _validate_handler_signature (Executor) and
_validate_function_signature (FunctionExecutor). Raises a ValueError with
an actionable message recommending @handler(input=..., output=...) or
@executor(input=..., output=...) instead of letting TypeVar leak through
to a confusing TypeCompatibilityError during workflow edge validation.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address review feedback for #4943: reorder checks and harden function executor

- Move TypeVar check before validate_workflow_context_annotation in
  _executor.py so users see the more actionable error first
- Wrap get_type_hints in try/except in _function_executor.py matching
  the defensive pattern in _executor.py
- Repurpose duplicate test to cover bounded TypeVar rejection
- Add test_function_executor_allows_concrete_types for test symmetry

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Narrow get_type_hints except clause and add missing tests (#4943)

- Narrow `except Exception` to `except (NameError, AttributeError, RecursionError)`
  in both _executor.py and _function_executor.py so unexpected failures in
  get_type_hints are not silently swallowed.
- Add test_handler_unresolvable_annotation_raises to test_function_executor_future.py
  exercising the except branch of get_type_hints in the function executor path.
- Add test_function_executor_rejects_bounded_typevar_in_message_annotation to
  test_function_executor.py for parity with the Executor bounded TypeVar test.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Add error ordering test for TypeVar vs WorkflowContext priority (#4943)

Add test_handler_typevar_error_takes_priority_over_context_error to verify
that when a handler has both a TypeVar message and an unannotated ctx, the
TypeVar error is raised first (the more actionable issue).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Python: Fix image content serialization sending null file_id to Foundry API

Omit file_id from input_image dict when not present instead of including
it as null, which Azure AI Foundry's stricter schema validation rejects.

* Python: Fix Foundry API rejecting rich content in function_call_output

Azure AI Foundry does not support list-format output in function_call_output
items. Add SUPPORTS_RICH_FUNCTION_OUTPUT flag (default True) to
RawOpenAIChatClient, set to False in RawFoundryChatClient so Foundry
falls back to string output for tool results with images/files.

Also omit file_id from input_image dicts when not set, since Foundry
rejects explicit nulls.

* Python: Surface rich tool content as user message when Foundry lacks support

When SUPPORTS_RICH_FUNCTION_OUTPUT is False, image/file items from tool
results are injected as a follow-up user message so the model can still
process the visual content via Foundry's supported user message format.

* Xfail Foundry image integration test for the meantime

---------

Co-authored-by: Copilot <copilot@github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-04-07 16:59:17 +00:00
Eduard van Valkenburg 3446eb8d5d Python: [BREAKING] update to v1.0.0 (#5062) 
* updates to final deprecated pieces and versions

* fix mypy

* fix readme links
 2026-04-02 15:26:30 +00:00
Eduard van Valkenburg 95fd5ec658 Python: [BREAKING] Python: move Azure AI embeddings to Foundry (#5056) 
* renamed AzureAIINferenceEmbeddings and lazy load azure-cosmos and env var rename

* updated coverage

* fix readme
 2026-04-02 11:26:35 +00:00
Copilot 7607acd009 Python: Add experimental decorator to all Evals pieces (#5040) 
* Initial plan

* feat(python): add experimental decorator to all Evals pieces

Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/99d71249-a5d6-4977-a5b5-6ffe0a3be2bc

Co-authored-by: TaoChenOSU <12570346+TaoChenOSU@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: TaoChenOSU <12570346+TaoChenOSU@users.noreply.github.com>
 2026-04-02 09:18:35 +00:00
Eduard van Valkenburg 6acab3d1d6 Python: [BREAKING] Standardize model selection on model (#4999) 
* Refactor Anthropic model option and provider clients

Rename the Anthropic client model option from model_id to model, add provider-specific Anthropic wrappers for Foundry, Bedrock, and Vertex, and expose them through the Anthropic, Foundry, Amazon, and Google namespaces. Update core option handling, docs, samples, and tests accordingly.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix Anthropic skills sample typing

Cast the Anthropic beta client to Any in the skills sample so the pre-commit sample pyright check no longer fails on beta skills and files endpoints that are not exposed by the current SDK stubs.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* undo sample mypy

* Retry CI after transient external failures

Retrigger PR validation after an unrelated Copilot review workflow SAML failure and a transient external tau2 git fetch failure in the Windows Python test setup.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address review feedback on model option merging

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address Anthropic compatibility review feedback

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* moved all to `model`

* fixes for azure ai search

* Python: standardize remaining sample env var names

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Python: fix foundry-local pyright compatibility

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* updated env vars in cicd

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-04-01 19:00:18 +00:00
Eduard van Valkenburg b065a4ce51 Python: [BREAKING] update context provider APIs, middleware, and per-service-call history persistence (#4992) 
* Rename provider base APIs

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Allow provider-added chat and function middleware

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Simulate service-stored history per model call

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix typing regressions in CI

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix response ID suppression review feedback

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Rename per-service-call history persistence APIs

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address context persistence review feedback

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Stabilize markdown sample docs

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Persist service continuation state per call

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-04-01 16:13:11 +00:00
Ben Thomas 35adfdb318 Python: Foundry Evals integration for Python (#4750) 
* Foundry Evals integration for Python

Merged and refactored eval module per Eduard's PR review:

- Merge _eval.py + _local_eval.py into single _evaluation.py
- Convert EvalItem from dataclass to regular class
- Rename to_dict() to to_eval_data()
- Convert _AgentEvalData to TypedDict
- Simplify check system: unified async pattern with isawaitable
- Parallelize checks and evaluators with asyncio.gather
- Add all/any mode to tool_called_check
- Fix bool(passed) truthy bug in _coerce_result
- Remove deprecated function_evaluator/async_function_evaluator aliases
- Remove _MinimalAgent, tighten evaluate_agent signature
- Set self.name in __init__ (LocalEvaluator, FoundryEvals)
- Limit FoundryEvals to AsyncOpenAI only
- Type project_client as AIProjectClient
- Remove NotImplementedError continuous eval code
- Add evaluation samples in 02-agents/ and 03-workflows/
- Update all imports and tests (167 passing)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix: resolve mypy redundant-cast errors while keeping pyright happy

Use cast(list[Any], x) with type: ignore[redundant-cast] comments to
satisfy both mypy (which considers casting Any redundant) and pyright
strict mode (which needs explicit casts to narrow Unknown types).

Also fix evaluator decorator check_name type annotation to be
explicitly str, resolving mypy str|Any|None mismatch.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix: CI failures — pyupgrade, evaluator overloads, sample API, reset attr

- Apply pyupgrade: Sequence from collections.abc, remove forward-ref quotes
- Add @overload signatures to evaluator() for proper @evaluator usage
- Fix evaluate_workflow sample to use WorkflowBuilder(start_executor=) API
- Fix _workflow.py executor.reset() to use getattr pattern for pyright
- Remove unused EvalResults forward-ref string in default_factory lambda

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix: skip gRPC-dependent observability test

The test_configure_otel_providers_with_env_file_and_vs_code_port test
triggers gRPC OTLP exporter creation, but the grpc dependency is
optional and not installed by default. Add skipif decorator matching
the pattern used by all other gRPC exporter tests in the same file.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix: add nosec B101 for bandit assert check

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* style: align eval samples with repo conventions

- Move module docstrings before imports (after copyright header)
- Add -> None return type to all main() and helper functions
- Fix line-too-long in multiturn sample conversation data
- Add Workflow import for typed return in all_patterns_sample

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address PR review feedback: async fixes, sample bugs, deprecation warnings

- Simplify _ensure_async_result to direct await (async-only clients)
- Replace get_event_loop() with get_running_loop()
- Narrow _fetch_output_items exception handling to specific types
- Add warning log when _filter_tool_evaluators falls back to defaults
- Add DeprecationWarning to options alias in Agent.__init__
- Add DeprecationWarning to evaluate_response()
- Rename raw key to _raw_arguments in convert_message fallback
- Fix evaluate_agent_sample.py: replace evals.select() with FoundryEvals()
- Fix evaluate_multiturn_sample.py: use Message/Content/FunctionTool types
- Fix evaluate_workflow_sample.py: replace evals.select() with FoundryEvals()
- Update test mocks to use AsyncMock for awaited API calls

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Add test coverage for review feedback items

- Add num_repetitions=2 positive test verifying 2×items and 4 agent calls
- Add _poll_eval_run tests: timeout, failed, and canceled paths
- Add evaluate_traces tests: validation error, response_ids path, trace_ids path
- Add evaluate_foundry_target happy-path test with target/query verification

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix ruff ISC004 lint error and apply formatter

- Wrap implicit string concatenation in parens in evaluate_multiturn_sample.py
- Apply ruff formatter to 6 other files with minor formatting drift

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Remove core type changes (extracted to fix/workflow-stale-session branch)

Reverts changes to _agents.py, _agent_executor.py, and _workflow.py
back to upstream/main. These fixes are now in a separate PR.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address PR review round 2: bugs, tests, and architecture

Code fixes:
- Fix _normalize_queries inverted condition (single query now replicates
  to match expected_count)
- Fix substring match bug: 'end' in 'backend' matched; use exact set
  lookup for executor ID filtering
- Fix used_available_tools sample: tool_definitions→tools param, use
  FunctionTool attribute access instead of dict .get()
- Add None-check in _resolve_openai_client for misconfigured project
- Add Returns section to evaluate_workflow docstring
- Cache inspect.signature in @evaluator wrapper (avoid per-item reflection)

Architecture:
- Extract _evaluate_via_responses as module-level helper; evaluate_traces
  now calls it directly instead of creating a FoundryEvals instance
- Move Foundry-specific typed-content conversion out of core to_eval_data;
  core now returns plain role/content dicts, FoundryEvals applies
  AgentEvalConverter in _evaluate_via_dataset

Tests:
- evaluate_response() deprecation warning emission and delegation
- num_repetitions > 1 with expected_output and expected_tool_calls
- Mock output_items.list in test_evaluate_calls_evals_api
- Update to_eval_data assertions for plain-dict format
- Unknown param error now raised at @evaluator decoration time

Skipped (separate PR): executor reset loop, xfail removal, options alias

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix CI: revert test_full_conversation, fix pyright errors

- Revert test_full_conversation.py to upstream/main (the session
  preservation test was incorrectly changed to assert clearing)
- Fix pyright reportUnnecessaryComparison on get_openai_client() None
  check by adding ignore comment
- Fix pyright reportPrivateUsage: add public EvalItem.split_messages()
  method and use it in FoundryEvals._evaluate_via_dataset instead of
  accessing private _split_conversation

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address PR review round 3: reliability, test gaps, cleanup

- Add try/except guard for non-numeric score in _coerce_result
- Add poll_interval minimum bound (0.1s) to prevent tight loops
- Add runtime async client check in _resolve_openai_client
- Remove _ensure_async_result wrapper (10 call sites → direct await)
- Better error message when queries provided without agent
- Import-time asserts for evaluator set consistency
- Remove 28 redundant @pytest.mark.asyncio decorators
- Add doc note about _raw_arguments sensitive data
- Tests: tool_called_check mode=any, _normalize_queries branches,
  _extract_result_counts paths, _extract_per_evaluator, bare check
  via evaluate_agent, output_items assertion, modulo wrapping,
  async client check, queries-without-agent error

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix CI: ruff S101 assert, pyright and mypy arg-type errors

- Replace module-level assert with if/raise for evaluator set
  consistency checks (ruff S101 disallows bare assert)
- Add type: ignore[arg-type] and pyright: ignore[reportArgumentType]
  on OpenAI SDK evals API calls that pass dicts where typed params
  are expected (SDK accepts dicts at runtime)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address PR review round 4: bugs, reliability, test fixes

- Fix all_passed ignoring parent result_counts when sub_results present
- Fix _extract_tool_calls: parse string arguments via json.loads before
  falling back to None (real LLM responses use string arguments)
- Sanitize _raw_arguments to '[unparseable]' to avoid leaking sensitive
  tool-call data to external evaluation services
- Add NOTE comment on to_eval_data message serialization dropping
  non-text content (tool calls, results)
- Eliminate double conversation split in _evaluate_via_dataset: build
  JSONL dicts directly from split_messages + AgentEvalConverter
- Raise poll_interval floor from 0.1s to 1.0s to prevent rate-limit
  exhaustion
- Fix MagicMock(name=...) bug in test: sets display name not .name attr
- Fix mock_output_item.sample: use MagicMock object instead of dict so
  _fetch_output_items exercises error/usage/input/output extraction

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address PR review round 5: reliability, docs, test coverage

Code fixes:
- Move import-time RuntimeError checks to unit tests (avoids breaking
  imports for all users on developer set-drift mistake)
- _filter_tool_evaluators now raises ValueError when all evaluators
  require tools but no items have tools (was silently substituting)
- Add poll_interval upper bound (60s) to prevent single-iteration sleep
- Log exc_info=True in _fetch_output_items for debugging API changes
- Fix evaluate() docstring: remove claim about Responses API optimization
- Validate target dict has 'type' key in evaluate_foundry_target
- Document to_eval_data() limitation: non-text content is omitted

Tests:
- TestEvaluatorSetConsistency: verify _AGENT/_TOOL subsets of _BUILTIN
- TestEvaluateTracesAgentId: agent_id-only path with lookback_hours
- TestFilterToolEvaluatorsRaises: ValueError on all-tool no-items
- TestEvaluateFoundryTargetValidation: target without 'type' key
- Assert items==[] on failed/canceled poll results
- Mock output_items.list in response_ids test for full flow
- TestAllPassedSubResults: result_counts=None + sub_results delegation
  and parent failures override sub_results
- TestBuildOverallItemEmpty: empty workflow outputs returns None

Skipped r5-07 (_raw_arguments length hint): marginal debugging value,
could leak content size information.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix error message: evaluate_responses() → evaluate_traces(response_ids=...)

The referenced function doesn't exist; the correct API is
evaluate_traces(response_ids=...) from the azure-ai package.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Remove dead to_eval_data() method, fix docstring claims

- Remove to_eval_data() from EvalItem (dead code after r4-05 JSONL refactor)
- Migrate 15 tests from to_eval_data() to split_messages()
- Update sample to use split_messages() + Message properties
- Remove unimplemented Responses API optimization docstring claim
- Update split_messages() docstring to not reference removed method

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Reduce default eval timeout from 600s to 180s (3 minutes)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Remove dead _evaluate_via_responses method from FoundryEvals

The method was never called — evaluate() uses _evaluate_via_dataset,
and evaluate_traces() calls _evaluate_via_responses_impl directly.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Revert unrelated formatting changes to get-started samples

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix pyright: remove phantom FoundryMemoryProvider import, apply ruff format

- Remove import of non-existent _foundry_memory_provider module
  (incorrectly kept during rebase conflict resolution)
- Apply ruff formatter to test_local_eval.py and get-started samples

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix eval samples: use FoundryChatClient for Agent()

The upstream provider-leading client refactor (#4818) made client=
a required parameter on Agent(). Update the three getting-started
eval samples to use FoundryChatClient with FOUNDRY_PROJECT_ENDPOINT,
matching the standard pattern from 01-get-started samples.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Simplify self-reflection sample using FoundryEvals

Replace ~80 lines of manual OpenAI evals API code (create_eval,
run_eval, manual polling, raw JSONL params) with FoundryEvals:

- evaluate_groundedness() uses FoundryEvals.evaluate() with EvalItem
- Remove create_openai_client(), create_eval(), run_eval() functions
- Remove openai SDK type imports (DataSourceConfigCustom, etc.)
- run_self_reflection_batch creates FoundryEvals instance once,
  reuses it for all iterations across all prompts

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Update eval samples to FoundryChatClient and FOUNDRY_PROJECT_ENDPOINT

- Migrate all foundry_evals samples from AzureOpenAIResponsesClient to FoundryChatClient
- Update env var from AZURE_AI_PROJECT_ENDPOINT to FOUNDRY_PROJECT_ENDPOINT
- Use AzureCliCredential consistently across all samples
- Fix README.md: correct function names (evaluate_dataset -> FoundryEvals.evaluate, evaluate_responses -> evaluate_traces)
- Update self_reflection .env.example and README.md

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix lint errors in eval samples (E501, ASYNC240, formatting)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Remove evaluate_all_patterns_sample.py (redundant with focused samples)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix async credential mismatch: use azure.identity.aio for async AIProjectClient

AIProjectClient from azure.ai.projects.aio requires an async credential.
Switch all foundry_evals samples from azure.identity.AzureCliCredential
to azure.identity.aio.AzureCliCredential. Also pass project_client to
FoundryChatClient instead of duplicating endpoint+credential.

Close credential in self_reflection sample to avoid resource leak.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Revert test_observability.py to upstream/main (not our test)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address moonbox3 review: sphinx docstrings, pagination, isinstance check

- Convert all Example:: / Typical usage:: code blocks to .. code-block:: python
  format matching codebase convention (both _evaluation.py and _foundry_evals.py)
- Add async pagination in _fetch_output_items via async for (handles large result sets)
- Replace hasattr(__aenter__) with isinstance(client, AsyncOpenAI) in _resolve_openai_client
- Move AsyncOpenAI import from TYPE_CHECKING to runtime (needed for isinstance)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix test failures and address remaining moonbox3 review comments

- Fix tests: use MagicMock(spec=AsyncOpenAI) for project_client mocks
  (isinstance check now requires proper type, not duck-typing)
- Fix tests: replace mock_page.__iter__ with _AsyncPage helper for async for
- Fix evaluate_response: auto-extract queries from response messages when
  query is not provided (previously always raised ValueError)
- Add debug logging when skipping internal _-prefixed executor IDs

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address Tao's PR review comments on Foundry Evals

- T1: Add comment explaining builtin.* pass-through in _resolve_evaluator
- T2: Add comment referencing OpenAI evals API for testing_criteria dict
- T3: Document Mustache-style {{item.*}} template placeholders
- T4: Document poll loop 60s sleep upper bound rationale
- T5: Narrow run type to RunRetrieveResponse, use typed field access
  instead of vars()/getattr dance in _extract_result_counts and
  _extract_per_evaluator; use run.error and run.report_url directly
- T6: Clarify openai_client docstring re: Azure Foundry endpoint
- T8: Remove misleading empty expected_tool_calls from sample
- Update tests to match real SDK PerTestingCriteriaResult shape

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Remove unnecessary Any union from run type annotations

RunRetrieveResponse is the correct type — no backward compat needed
for a brand new feature.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Accept FoundryChatClient instead of raw AsyncOpenAI

FoundryEvals now takes client: FoundryChatClient as its primary
parameter instead of openai_client: AsyncOpenAI.  The builtin.*
evaluators require a Foundry endpoint, so the type should reflect that.

- FoundryEvals.__init__: client: FoundryChatClient replaces openai_client
- evaluate_traces / evaluate_foundry_target: same change
- _resolve_openai_client: extracts .client from FoundryChatClient
- project_client fallback retained for standalone functions
- All samples updated to construct FoundryChatClient and pass as client=
- Tests updated (openai_client= → client=)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Remove implicit 60s upper bound on poll interval

If a developer sets a higher poll_interval, respect it. Only clamp
to remaining time and enforce a 1s minimum for rate-limit protection.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Remove 1s floor on poll interval — let the developer control it

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Update python/samples/05-end-to-end/evaluation/foundry_evals/.env.example

Co-authored-by: Eduard van Valkenburg <eavanvalkenburg@users.noreply.github.com>

* Update python/samples/02-agents/evaluation/evaluate_agent.py

Co-authored-by: Eduard van Valkenburg <eavanvalkenburg@users.noreply.github.com>

* Address eavanvalkenburg review (round 2) on Python eval PR

- Rename model_deployment -> model across FoundryEvals and all samples
- Make model param optional, resolves from client.model
- Convert EvalResults from dataclass to regular class
- Remove deprecated evaluate_response() function
- Refactor splitters: BUILT_IN_SPLITTERS dict + standalone functions
- Change per_turn_items from classmethod to staticmethod
- Simplify EvalCheck type alias to use Awaitable[CheckResult]
- Remove errored property from EvalResults
- Remove default value from Evaluator protocol eval_name
- Rename assert_passed -> raise_for_status, add EvalNotPassedError
- Type agent param as SupportsAgentRun | None
- Fix Arguments docstring
- Update __init__.py exports
- Update all tests and samples

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Move FoundryEvals to foundry package, split tool eval sample

- Move _foundry_evals.py from azure-ai to foundry package
- Move test_foundry_evals.py to foundry/tests/
- Update lazy re-exports in agent_framework.foundry namespace
- Update .pyi type stubs
- All samples now import from agent_framework.foundry
- Split tool-call evaluation into evaluate_tool_calls_sample.py
- Fix all_passed to check errored count from result_counts
- Fix raise_for_status to include errored item details

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Auto-create FoundryChatClient from env vars when no client provided

FoundryEvals() now works zero-config when FOUNDRY_PROJECT_ENDPOINT and
FOUNDRY_MODEL environment variables are set. Auto-creates a FoundryChatClient
under the hood, matching the established env var pattern.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix pyright errors: remove dead _normalize_queries, suppress EvalAPIError check

- Remove unused _normalize_queries function and its tests
- Add pyright ignore for EvalAPIError None check (defensive guard)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Support multimodal image content in eval pipeline

Add image (data/uri) content handling to AgentEvalConverter.convert_message()
so that Content.from_data() and Content.from_uri() image payloads are
preserved as input_image parts in the Foundry evaluator format.

- Handle Content type='data' and type='uri' → emit input_image parts
- Add 6 unit tests for image content through convert_message/convert_messages
- Add integration test verifying images flow through EvalItem → JSONL path
- Add evaluate_multimodal.py sample demonstrating local image eval

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address remaining review comments

- Fix project_client docstring to say async-only (not sync/async)
- Add builtin evaluator name validation warning in _resolve_evaluator
- Replace getattr with typed attribute access in _poll_eval_run,
  _extract_result_counts, _extract_per_evaluator, _fetch_output_items
- Remove cast import from _foundry_evals (no longer needed)
- Tighten _coerce_result: honour explicit 'passed' when both 'score'
  and 'passed' are present; remove performative cast
- Fix self_reflection sample: add env file existence check
- Fix traces sample: correct Pattern 2 section label
- Update all Foundry eval samples to FoundryChatClient + FOUNDRY_MODEL
  (remove AIProjectClient + AZURE_AI_MODEL_DEPLOYMENT_NAME pattern)
- Add eval_name and OpenAI client docs to FoundryEvals docstring
- Update test mocks to match typed SDK objects (_MockResultCounts)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix ruff lint errors (E501, SIM108, SIM102)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix pyright errors: type-narrow dict to dict[str, Any], add ignore comments

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Replace ConversationSplitter type alias with Protocol

ConversationSplitter is now a runtime-checkable Protocol with a named
'conversation' parameter, making the expected signature self-documenting.

ConversationSplit enum members gain a __call__ method so they satisfy
the protocol directly -- ConversationSplit.LAST_TURN(conversation) works.

This simplifies _split_conversation from an isinstance dispatch to a
single split(conversation) call.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Standardize on AZURE_AI_MODEL_DEPLOYMENT_NAME and fix Unicode in samples

- Replace FOUNDRY_MODEL with AZURE_AI_MODEL_DEPLOYMENT_NAME in all
  eval samples to match repo convention
- Replace Unicode symbols with ASCII equivalents in all eval sample
  print statements to avoid cp1252 encoding errors on Windows

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Update python/samples/03-workflows/evaluation/evaluate_workflow.py

Co-authored-by: Eduard van Valkenburg <eavanvalkenburg@users.noreply.github.com>

* Apply suggestions from code review

Co-authored-by: Eduard van Valkenburg <eavanvalkenburg@users.noreply.github.com>

* Rename ADR 0020 to 0023 (foundry evals integration)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: alliscode <bentho@microsoft.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Eduard van Valkenburg <eavanvalkenburg@users.noreply.github.com>
 2026-03-31 15:53:06 +00:00
Eduard van Valkenburg b1b528e4a8 Python: [BREAKING] Remove deprecated kwargs compatibility paths (#4858) 
* [BREAKING] Remove deprecated kwargs compatibility paths

Remove the deprecated kwargs compatibility shims across core agents, clients, tools, middleware, and telemetry.

Keep workflow kwargs behavior intact in this branch and follow up separately in #4850.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix PR CI fallout for kwargs removal

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address PR review feedback

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* updates

* Fix Azure AI CI fallout

Remove the stale _get_current_conversation_id override from the Azure AI client after the OpenAI base helper was deleted.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fixed new classes

* Fix Assistants deprecated import gating

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix integration replay regressions

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Switch multi-agent hosting samples to Azure chat completions

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Simplify Azure multi-agent sample config

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-03-27 21:00:12 +00:00
Eduard van Valkenburg cc0cfaaac8 [BREAKING] Python: fix OpenAI Azure routing and provider samples (#4925) 
* Python: fix OpenAI Azure routing and provider samples

Prefer OpenAI when OPENAI_API_KEY is present unless Azure is explicitly requested. Clarify constructor docs, keep deprecated Azure wrappers compatible with stricter settings validation, and refresh the provider samples and tests to use the current client patterns.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix bandit

* Python: align OpenAI embedding Azure routing

Extend the shared OpenAI-vs-Azure routing and credential behavior to the embedding client, add Azure embedding regression coverage, and refresh the embedding samples to use the generic client path.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Python: fix embedding client pyright check

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Python: thin OpenAI embedding wrapper

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Python: document embedding overload routing

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Python: fix callable OpenAI key routing

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Python: fix Azure credential routing tests

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Python: address OpenAI review feedback

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Python: narrow Azure routing markers

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Python: refine OpenAI model fallback order

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Python: narrow Azure deployment docs

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Python: remove embedding routing wording

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Python: run embedding Azure integration tests

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* changed variable name

* Python: expand OpenAI package README

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* clarified readme

* Python: fix Azure OpenAI integration setup

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Python: correct Azure integration env mapping

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* updated code to fix int tests

* test updates

* test fix

* fix test setup

* updates to tests and setup

* remove openai assistants int tests

* improvements in int tests

* fix env var

* fix env vars

* fix azure responses test

* trigger actions

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-03-27 13:33:39 +00:00
Eduard van Valkenburg c012aac5f2 Python: [BREAKING] Reduce core dependencies and simplify optional integrations (#4904) 
* improved dependencies and some fixes

* fix for mypy

* improve mcp
 2026-03-25 18:03:43 +00:00
Eduard van Valkenburg 5e056b672e Python: [BREAKING] Python: Provider-leading client design & OpenAI package extraction (#4818) 
* Python: Provider-leading client design & OpenAI package extraction

Major refactoring of the Python Agent Framework client architecture:

- Extract OpenAI clients into new `agent-framework-openai` package
- Core package no longer depends on openai, azure-identity, azure-ai-projects
- Rename clients for discoverability: OpenAIResponsesClient → OpenAIChatClient,
  OpenAIChatClient → OpenAIChatCompletionClient
- Unify `model_id`/`deployment_name`/`model_deployment_name` → `model` param
- New FoundryChatClient for Azure AI Foundry Responses API
- New FoundryAgent/FoundryAgentClient for connecting to pre-configured Foundry agents
- Remove OpenAIBase/OpenAIConfigMixin from non-deprecated client MRO
- Deprecate AzureOpenAI* clients, AzureAIClient, OpenAIAssistantsClient
- Reorganize samples: azure_openai+azure_ai+azure_ai_agent → azure/
- ADR-0020: Provider-Leading Client Design

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix: missing Agent imports in samples, .model_id → .model in foundry_local sample

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix: CI failures — mypy errors, coverage targets, sample imports

- azure-ai mypy: add type ignores for TypedDict total=, model arg, forward ref
- Coverage: replace core.azure/openai targets with openai package target
- project_provider: add type annotation for opts dict

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix: populate openai .pyi stub, fix broken README links, coverage targets

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fixes

* updated observabilitty

* reset azure init.pyi

* fix errors

* updated adr number

* fix foundry local

* fixed not renamed docstrings and comments, and added deprecated markers to old classes

* fix tests and pyprojects

* fix test vars

* updated function tests

* update durable

* updated test setup for functions

* Fix Foundry auth in workflow samples

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Stabilize Python integration workflows

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Update hosting samples for Foundry

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Trigger full CI rerun

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Trigger CI rerun again

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* trigger rerun

* trigger rerun

* fix for litellm

* undo durabletask changes

* Move Foundry APIs into foundry namespace

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix Foundry pyproject formatting

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Split provider samples by Foundry surface

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Restore hosting sample requirements

Also fix the Foundry Local sample link after the provider sample move.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* updated tests

* udpated foundry integration tests

* removed dist from azurefunctions tests

* Use separate Foundry clients for concurrent agents

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix client setup in azfunc and durable

* disabled two tests

* updated setup for some function and durable tests

* improved azure openai setup with new clients

* ignore deprecated

* fixes

* skip 11

* remove openai assistants int tests

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-03-25 09:56:29 +00:00
Eduard van Valkenburg 9355329dfd Python: [BREAKING] updated structure and samples (#875) 
* updated structure and samples

* updated names and removed cross tests

* updated projects etc

* updated tests

* updated test

* test fixes

* removed devui for now

* updated all-tests task

* removed old style configs

* remove coverage from tests

* updated to unit tests with all-tests

* updated foundry everywhere

* fix azure ai tests

* fix merge tests

* fix mypy
 2025-09-25 07:02:53 +00:00
Eduard van Valkenburg 9eb68ab2f9 Python: additional Foundry Tools (#611) 
* initial work on additional foundry tools

* fixes

* fix tests

* fix import

* updated lock

* added hosted MCP for foundry

* fixes

* fix for test

* updated samples

* fix result parsing
 2025-09-23 11:20:20 +00:00
Eduard van Valkenburg 2576e7a091 Python: Telemetry and observability follow-up (#833) 
* updated telemetry work

* updated telemetry

* slight improvement

* updated tests

* fixes for telemetry

* fixes for mypy

* added settings setup to runner to avoid error

* streamline usage

* updated tests

* updated tests

* further refinement

* fix dumped item for otel

* removed enable_workflow_otel

* final fixes

* final fixes

* updated samples

* removed exporters

* fix tests

* fixed last import'

* fixed devui
 2025-09-23 06:21:56 +00:00
Eduard van Valkenburg 89cb94b5c2 replaced HEADERS with user_agent (#732) 2025-09-16 07:57:37 +00:00
Evan Mattson d00538eb26 Foundry client ToolOutput to FunctionToolOutput (#749) 2025-09-15 09:06:43 +00:00
Eduard van Valkenburg 82ca4065cb Python: Improved telemetry setup (#421) 
* test with stack and simplified names

* quick demo of agent decorator

* moved builder to protocol to enhance functionality

* undid chatclientAgent -> agent rename

* one more

* reverted AIAgent rename

* final reverts

* fixed foundry import

* revert changes

* streamlined otel and fcc decorators

* cleanup of telemetry

* further refinement

* lots of updates

* fixed typing

* fix for mypy

* added input and output atttributes

* fix import

* initial work on baking in otel

* major update to telemetry

* final fixes after rename

* fix

* fix test

* updated tests

* fix for tests

* fixes for tests

* updated based on comments

* removed agent decorator

* fix for Python: ServiceResponseException when using multiple tools
Fixes #649

* addressed comments

* fix tests

* fix tests

* fix tools tests

* fix for conversation_id in assistants client

* fix responses test

* fix tests and mypy

* updated test

* foundry fix

---------

Co-authored-by: Chris <66376200+crickman@users.noreply.github.com>
 2025-09-10 14:52:42 +00:00