Python: Support prompt cache breakpoints for GPT-5.6 models in OpenAI clients (#7163)

* Python: Support prompt cache breakpoints for GPT-5.6 models in OpenAI clients Add request-level prompt_cache_options to OpenAIChatOptions and OpenAIChatCompletionOptions, and forward a per-part prompt_cache_breakpoint from Content.additional_properties onto the content blocks each API supports. Text parts that carry a breakpoint keep typed list content, since the plain-string form cannot hold one; without a breakpoint the existing string forms are unchanged. * Clarify system-message content-shape comment * Address review: SDK prompt cache types, private helper, add sample Replace the custom PromptCacheOptions TypedDict with the openai SDK's own types for each API, which raises the openai floor to 2.45.0 where those types were introduced. Make the breakpoint helper private to the two chat clients. Add a prompt caching sample with a README entry, and unquote the helper's Content annotation so the pyupgrade hook passes. * Guard the prompt cache options import for older openai versions The SDK's PromptCacheOptions types only exist in openai 2.45.0 and later, so each client falls back to a local mirror when the import fails and the dependency floor stays at 2.25.0. A TYPE_CHECKING-only import is not enough because the options classes are introspected with get_type_hints() at runtime. Verified against openai 2.25.0: the package imports, the fallback resolves, and part-level breakpoints still work; sending the option itself requires 2.45.0, which the field docstrings now note. * Make the old-openai fallback for PromptCacheOptions deliberately empty Assigning None instead, as suggested in review, trips pyright's reportInvalidTypeForm on the field annotation (the symbol becomes type | None after the try/except). An empty TypedDict gives the same effect for users on older openai versions: any content they put in prompt_cache_options is flagged by their type checker, since the option cannot be sent on those versions anyway, while get_type_hints() on the options classes keeps working at runtime. * Guard prompt_cache_options at runtime instead of via an empty fallback type The empty-TypedDict fallback flagged valid `prompt_cache_options` usage under pyright on every openai version — including this PR's own `client_prompt_caching.py` sample (`poe check -S`) — because pyright resolves the try/except symbol to the fallback shape regardless of the installed openai, while mypy/ty resolve the failed import to `Any` and never warn. So a type-only "warn on old openai" signal is not achievable cleanly across type checkers. Restore the faithful fallback (mirrors the SDK's `mode`/`ttl` shape) so the option type-checks identically on every supported openai version, and add a runtime guard: setting `prompt_cache_options` on openai < 2.45 now raises a clear ChatClientInvalidRequestException instead of forwarding an unusable option to the SDK. This keeps the option non-silent for all users regardless of type checker, without forcing an openai upgrade. Adds tests covering the guard for both clients. * Gate system/developer breakpoint shape on a real mapping value The system/developer branch switched to list-form content whenever prompt_cache_breakpoint was set to any non-None value, but the option is only attached when the value is a mapping. A malformed value (e.g. a string) therefore changed the message shape without adding a breakpoint. Decide the shape from the built part instead, matching the user-role path.
2026-07-23 00:43:19 +03:00
parent ddb0622f9c
commit 61802723ff
7 changed files with 439 additions and 26 deletions
@@ -22,6 +22,7 @@ This folder contains OpenAI provider samples for the generic clients in
 | [`client_basic.py`](client_basic.py) | Basic non-streaming and streaming responses sample with an explicit `gpt-5.4-nano` model and API key. |
 | [`client_image_analysis.py`](client_image_analysis.py) | Analyze images with the responses client. |
 | [`client_image_generation.py`](client_image_generation.py) | Generate images from text prompts. |
+| [`client_prompt_caching.py`](client_prompt_caching.py) | Explicit prompt cache breakpoints and `prompt_cache_options` on GPT-5.6 models. |
 | [`client_reasoning.py`](client_reasoning.py) | Reasoning-focused sample for models such as `gpt-5`. |
 | [`client_streaming_image_generation.py`](client_streaming_image_generation.py) | Streaming image generation sample. |
 | [`client_verbosity.py`](client_verbosity.py) | GPT-5 `verbosity` option (`low`/`medium`/`high`) with default and per-call overrides. |
@@ -0,0 +1,86 @@
+# Copyright (c) Microsoft. All rights reserved.
+
+import asyncio
+
+from agent_framework import Content, Message
+from agent_framework.openai import OpenAIChatClient, OpenAIChatOptions
+from dotenv import load_dotenv
+
+load_dotenv()
+
+"""
+OpenAI Chat Client Prompt Caching Example
+
+Demonstrates explicit prompt cache breakpoints on GPT-5.6 and later models. Cache
+writes are billed on these models, so marking exactly where a reusable prefix ends
+lets you control what gets cached.
+
+Two knobs work together:
+
+- ``prompt_cache_options`` on ``OpenAIChatOptions`` sets the request-wide policy.
+  ``{"mode": "explicit"}`` disables the automatic breakpoint on the latest message,
+  so only the breakpoints you place are used for cache reads and writes.
+- ``Content.additional_properties["prompt_cache_breakpoint"]`` marks the end of the
+  reusable prefix on a specific content part.
+
+The content before a breakpoint must be at least 1024 tokens long to be cached.
+Running the same prefix twice shows the cache hit through
+``usage_details["cache_read_input_token_count"]`` on later responses.
+
+Environment variables:
+    OPENAI_API_KEY — OpenAI API key
+
+See: https://developers.openai.com/api/docs/guides/prompt-caching#prompt-cache-breakpoints
+"""
+
+# A stable block of context that is reused across requests, for example a product
+# catalog, a policy document, or long system guidance. Repeated here to clear the
+# 1024-token minimum a cache breakpoint requires.
+STABLE_CONTEXT = (
+    "You are a support assistant for the Contoso appliance store. "
+    "Always answer briefly, quote the relevant catalog section, and never invent "
+    "model numbers. If a question is out of scope, say so and point the customer "
+    "to support@contoso.example. "
+) * 40
+
+
+def build_messages(question: str) -> list[Message]:
+    """Build a request with a cache breakpoint at the end of the stable prefix."""
+    return [
+        Message(
+            role="user",
+            contents=[
+                Content.from_text(
+                    STABLE_CONTEXT,
+                    additional_properties={"prompt_cache_breakpoint": {"mode": "explicit"}},
+                )
+            ],
+        ),
+        Message(role="user", contents=[Content.from_text(question)]),
+    ]
+
+
+async def main() -> None:
+    print("\033[92m=== OpenAI Chat Client Prompt Caching Example ===\033[0m\n")
+
+    client = OpenAIChatClient[OpenAIChatOptions](model="gpt-5.6-luna")
+    options: OpenAIChatOptions = {"prompt_cache_options": {"mode": "explicit"}}
+
+    questions = ["Do you sell refrigerators?", "What is the return policy contact?"]
+    for turn, question in enumerate(questions, start=1):
+        response = await client.get_response(build_messages(question), options=options)
+        usage = response.usage_details or {}
+        cached = usage.get("cache_read_input_token_count", 0)
+        print(f"Turn {turn}: {question}")
+        print(f"  Answer: {response.text}")
+        print(f"  Cached input tokens: {cached}\n")
+        if turn < len(questions):
+            # A freshly written cache entry becomes readable shortly after the request
+            # completes; the brief pause keeps the next turn from racing this one.
+            await asyncio.sleep(2)
+
+    print("The first turn writes the prefix to the cache; later turns read it back.")
+
+
+if __name__ == "__main__":
+    asyncio.run(main())