antirez--ds4/misc/RESPONSE_API.md

Responses API Continuation Plan

This note tracks the design for fixing /v1/responses tool-call continuation and the KV-cache behavior around it.

Problem

The current implementation handles /v1/responses too much like a stateless chat-completions replay API. After a tool call, it tries to canonicalize the live KV checkpoint so it matches the next rendered prompt byte-for-byte.

That is wrong for the live Responses protocol path.

In a live Responses tool loop, the next request is not an unrelated prompt. It is a continuation bound to the previous model output by tool-call ids. The live KV already contains the true assistant turn, including hidden reasoning that the client may not replay. Rebuilding the session to match only the visible client transcript is both slow and less faithful to the state the model actually produced.

Observed symptom:

tool checkpoint canonicalization needs rebuild ... common=16846 live=16994 canonical=16937
tool checkpoint canonicalized ... via=rebuild

This caused a long pause because DS4 rebuilt roughly the whole context before returning the streamed tool-call response tail.

Protocol Model

There are two useful id classes:

• previous_response_id / conversation: response-level server-side state. DS4 does not currently persist this. If a non-null value arrives, return an error asking the client to replay full input.
• call_id: tool-call binding. A function_call_output or hosted-tool output with this id is the continuation of the previous assistant tool call.

The server should distinguish two modes:

• Live continuation: the request contains tool outputs for tool-call ids still known by the live server state. Continue from the live KV state and append the new tool-result suffix.
• Stateless replay: the request includes the full history. Render and match the best prefix, using exact DSML replay from tool-memory when possible, just like chat-completions style replay.

Unknown tool-output ids are only valid if the request also includes the matching prior function-call item in the replayed history. If neither live state nor full history can explain the id, return an error.

Core mental model:

• The Responses API is designed so a server can avoid matching an already-known prefix. If the request is tied to live server state by response/conversation state or by immediately returning tool outputs for live call ids, DS4 should continue from the live KV instead of proving that the client-visible replay tokenizes to the same prefix.
• Prefix matching is only the fallback for stateless replay, cold start, server restart, branch/edit, or multi-client cases where the server no longer has the relevant live state.
• The fallback should reuse the techniques learned for chat continuation: exact token-prefix match first, then rendered string-prefix match with suffix retokenization, then disk string-prefix checkpoints. The adaptation for Responses is that reasoning/tool-call stateless replay must include the reasoning state the protocol requires, not just the visible transcript.

Official OpenAI Docs Findings

Sources checked:

• Conversation state: https://developers.openai.com/api/docs/guides/conversation-state
• Function calling: https://developers.openai.com/api/docs/guides/function-calling
• Responses API reference: https://platform.openai.com/docs/api-reference/responses

Findings that affect the implementation:

• The Responses API has two explicit server-side state mechanisms: previous_response_id and conversation.
  • previous_response_id chains a new response to a stored prior response.
  • conversation prepends persisted conversation items and then appends new input/output items after the response completes.
  • They are mutually exclusive in the API reference.
• Stateless/manual Responses state is also valid, but the client must preserve the model output items. The function-calling guide shows:
  • Start with an input list.
  • Call responses.create().
  • Append response.output to the same input list.
  • Append function_call_output items with the matching call_id.
  • Send the resulting input list back.
• For reasoning models, the function-calling guide explicitly says that any reasoning items returned in model responses with tool calls must also be passed back with tool-call outputs.
• The Responses API reference exposes include: ["reasoning.encrypted_content"], described as enabling reasoning items to be used in stateless multi-turn conversations when store=false or ZDR applies.

Implication for DS4:

• A request containing only a function-call output is not enough to recreate stateless reasoning context unless DS4 still has live state for that call id.
• A full stateless replay should include the prior function/custom tool call item and, for reasoning/tool-call turns, the reasoning item or an equivalent opaque reasoning-state item.
• DS4 currently does not implement durable previous_response_id or conversation state, so non-null values should remain rejected unless we add a real response/conversation store.
• The current Codex-style store=false / include=[] flow can still work as a live local server fast path, but if the live state is gone DS4 should reject and ask for a full replay that includes the missing reasoning state, rather than silently rebuilding a prompt with hidden reasoning removed.

Current Code Facts

• Exact DSML replay via tool_memory / rax is still valid and working. It fixes the visible tool-call block by replaying the sampled DSML for known call ids.
• The bad path is canonicalize_tool_checkpoint() for Responses tool calls when reasoning summaries are not emitted. It drops hidden reasoning and can trigger full rebuilds.
• live_text_prefix_prompt() is still useful for ordinary replay/cache matching, but it is not enough for the live Responses continuation case because the client-visible text can omit hidden reasoning that exists in the live KV.
• ds4_session_save_snapshot() / ds4_session_load_snapshot() exist, but they should not be the primary continuation mechanism for this protocol path.

Implementation Plan

1. Track live Responses continuation state.

   • Remember the latest Responses assistant tool-call turn after generation.
   • Store the generated call ids and enough rendered visible text to recognize the next request.
   • Keep the live KV as authoritative; do not canonicalize away hidden reasoning for this path.
   • Treat this as an optimization for the current live single-server session, not as a substitute for previous_response_id / conversation.
   • Prefer this path for Codex/pi tool loops when integration tests show those clients send tool outputs that are directly bound to the previous live call ids.

2. Parse Responses tool outputs with validation.

   • Collect function/custom/hosted tool-output ids during parse_responses_input().
   • Collect function/custom/hosted call ids present in the replayed history.
   • For reasoning-mode tool-call turns, distinguish a true full replay from a visible-only replay. A matching prior call id without the reasoning item is enough to render visible history, but not enough to reconstruct hidden model state if live continuation is unavailable.
   • Reject tool-output ids that are neither known live ids nor present as prior calls in the same request history.

3. Add a live continuation fast path before normal cache matching.

   • If the request is a Responses request and its first new semantic item is a tool output for the remembered live call ids, build only the suffix that must be appended to the current live KV.
   • Tokenize that suffix independently from the live token prefix, as the existing text-prefix path does, to avoid BPE boundary mistakes.
   • Set prompt_for_sync to live_tokens + suffix_tokens, with cached = live_tokens->len.

4. Keep stateless replay behavior.

   • If the request is not a live continuation, render the full prompt and use the existing token-prefix, live text-prefix, and disk text-prefix cache paths.
   • Exact DSML tool replay remains available to improve matching.
   • If the request is a stateless reasoning/tool-call replay without reasoning items or equivalent opaque reasoning state, reject instead of pretending the visible transcript is complete.

5. Remove invalid Responses canonicalization.

   • Do not call canonicalize_tool_checkpoint() for Responses live tool-call continuations.
   • Remove the Responses-specific reasoning-dropping behavior in build_tool_checkpoint_suffix().
   • Keep or narrow toolless thinking canonicalization only where the next request is truly a stateless replay and the live hidden reasoning cannot be continued by protocol ids.

6. Improve errors.

   • Non-null previous_response_id / conversation: return a clear 400 error unless DS4 later implements persistent response state.
   • Unknown tool-output id without a matching prior function call in the same request: return a clear 400 error explaining that DS4 needs full history.
   • Reasoning/tool-call replay that lacks reasoning state and is not live: return a clear 400 explaining that DS4 needs either live continuation or a full Responses replay with reasoning items / encrypted reasoning content.

Tests

Automated tests:

• Responses live tool continuation keeps hidden reasoning in live KV and appends only the tool-result suffix.
• Responses live continuation does not call tool checkpoint canonicalization.
• Unknown tool-output id with no prior function call is rejected.
• Unknown tool-output id with a prior function call in replayed history is accepted as stateless replay only when the replay has enough reasoning state for the selected mode.
• Reasoning/tool-call stateless replay without reasoning state is rejected if live continuation is unavailable.
• Normal chat-completions exact DSML replay still works.
• Disk KV tool-map restore still works for stateless replay.
• BPE boundary handling remains correct for live-prefix plus text suffix.

Live tests:

• Codex multi-turn session with repeated tool calls.
  • Expected: after each tool call, next request reports a memory-token continuation or equivalent live continuation, not a rebuild.
  • Expected: no long pause after finish=tool_calls.
  • Verify the actual client behavior: whether Codex sends previous_response_id, full response.output, function-call outputs only, reasoning items, or some mixture.
• Pi agent multi-turn session with tools.
  • Expected: tool calls execute and the agent recovers normally from tool outputs.
  • Expected: no unexpected cache misses.
  • Verify the actual client behavior independently from Codex. Do not assume both clients use the same Responses replay style.
• Stateless replay request with full history and known call ids.
  • Expected: exact DSML replay via rax and best-prefix cache matching.
• Bad continuation request with only a tool output for an unknown id.
  • Expected: HTTP 400 with a direct "replay full history" style message.

Current Progress

• Confirmed from trace that the pause is synchronous checkpoint canonicalization and rebuild after a parsed tool call.
• Confirmed rax exact DSML replay is not the failing part: tool_replay: mem=4 disk=0 canonical=0 missing_ids=0.
• Confirmed the current Responses parser rejects non-null previous_response_id / conversation, which is the right behavior until server-side response persistence exists.
• Checked official OpenAI documentation. The plan was updated to account for the documented requirement that reasoning-model tool-call loops preserve reasoning items in stateless manual replay.
• Implemented live Responses continuation state and removed Responses tool-call checkpoint canonicalization from the main path.
• Added normal server logs for accepted live Responses continuations: responses live continuation match=... cached=... prompt=.... Tool-output continuations now also report the matched id count, for example match=visible-prefix ids=1.
• Added Responses-visible disk checkpoint keys. The payload is still the exact live KV with hidden reasoning, but the file name / lookup text is the visible transcript the client can replay. This is required when switching between Codex, pi, and synthetic clients, because only one live KV session can be resident at a time.

QA Log

All tests in this section were run on the local M3 Max server with Metal, --ctx 65536, and /v1/responses clients configured for deepseek-v4-flash.

Automated checks:

• make ds4_test && ./ds4_test --server && make ds4-server: pass.
• Added unit coverage for unknown tool-output ids, reasoning-required stateless replay, Responses live-tail rendering, and final-answer visible suffix handling.

Official protocol checks:

• previous_response_id != null: returns HTTP 400 with previous_response_id is not supported; replay full input instead.
• conversation != null: returns HTTP 400 with conversation is not supported; replay full input instead.
• Tool output for an unknown call_id: returns HTTP 400 with unknown tool output call_id ...; replay full Responses history.
• Stateless replay with a prior tool call but no reasoning item: returns HTTP 400 with Responses replay is missing reasoning state....
• Stateless replay with a prior tool call and an explicit reasoning item: returns HTTP 200 and generates from the replayed history.

Codex live behavior:

• Codex sends store=false, include=[], prompt_cache_key=<session id>, and full visible replay.
• Exact token matching fails at hidden thinking, as expected: live_prompt_common stops where live KV has hidden reasoning and the visible replay has </think>.
• Live visible continuation now catches this path: cache_source: responses-visible, with short suffix prefill after tool outputs and after normal resumed user turns.
• Repeated Codex turns completed with tool execution and no canonicalization needs rebuild pause.

Pi live behavior:

• Pi uses a custom local provider with api: openai-responses.
• Pi sends reasoning.summary=auto and include: ["reasoning.encrypted_content"].
• Pi replays reasoning summaries for assistant tool-call turns, but not for final assistant text answers. The visible-prefix state must therefore include tool-call reasoning summaries when present, but omit final-answer hidden reasoning.
• Pi first turn, tool result, resumed user turn, and resumed tool result all completed. Accepted live continuations print: responses live continuation match=visible-prefix ...; tool-output turns print ids=1 in the normal server log.

Interleaved session switching:

• Sequence tested: Codex turn, pi turn, Codex resume, pi resume, Codex resume, plus synthetic curl traffic between agent turns.
• On switching clients, the live KV miss is expected and logged in orange.
• Before replacing live state, DS4 stores the outgoing Responses session as key=responses-visible.
• Returning to Codex after pi loaded the Codex checkpoint from disk: kv cache hit text tokens=11772 ..., then prefills only the new suffix.
• Returning to pi after Codex loaded the pi checkpoint from disk: kv cache hit text tokens=1654 ..., then prefills only the new suffix.
• Returning to Codex after synthetic curl traffic again loaded the latest Codex checkpoint from disk: kv cache hit text tokens=12211 ....
• The earlier failure mode was reproduced before the fix: Codex resume after pi received Responses replay is missing reasoning state... because the guard ran before disk recovery and the disk key was raw hidden text. The fix is to key Responses checkpoints by visible transcript and accept only KV_EXT_RESPONSES_VISIBLE disk hits for missing-reasoning replay.

Remaining risks / follow-up:

• DS4 still does not implement durable previous_response_id or conversation; rejecting non-null values remains intentional.
• reasoning.encrypted_content is not decrypted. It is useful for real OpenAI stateless clients, but DS4 currently relies on live state or visible-key disk checkpoints for hidden reasoning recovery.
• The tool-output-only continuation path is unit-tested, but the observed Codex and pi clients both used full visible replay in their normal runs.
• Tool-output-only continuation was also exercised directly with curl: first request produced call_3c4891795bf43aca507f4a987707b259, second request sent only function_call_output for that id, and the server accepted it as responses live continuation match=tool-output-ids ids=1.
• Code review follow-up tightened a race-shaped edge case: if a request contains only tool outputs, it is marked as requiring the live call-id state. If that state no longer matches by worker execution time, DS4 now returns a 400 asking for full input instead of cold-prefilling a prompt that starts with a naked tool result. Full visible replays in thinking mode also remain marked as reasoning-sensitive even if live state existed during parse, because the live KV can be replaced before execution.
• Disk cache hit logs now include the key kind: key=responses-visible for visible-transcript keys that restore hidden KV, and key=token-text for ordinary rendered-token text keys.
• make ds4_test && ./ds4_test --server && make ds4-server: pass after the stricter tool-output-only check and log-key change.
• A separate /tmp/ds4_test_asan build with -fsanitize=address,undefined also passes --server. Leak detection is not supported by Apple ASan on this platform, so the run covered address/UB checks but not leak reporting.

Restart / disk recovery test:

• Stopped the old server and started a fresh ds4-server with the existing /tmp/kvcache-responses-switch3 directory.
• Resumed the existing Codex session 019e23a9-821c-7e40-8c00-122646d7fda7 after process restart.
• Expected behavior occurred immediately: kv cache hit text tokens=12367 ... key=responses-visible, followed by a short suffix prefill and a live Responses tool-output continuation with ids=1.
• After unrelated pi, curl, and opencode traffic replaced live KV, resuming Codex again hit disk with key=responses-visible at the later checkpoint and then continued the tool result live. This confirms the cross-client/session path is not only an in-process memory shortcut.
• Shutdown persistence was also tested: stopping the server logged reason=shutdown key=responses-visible for the latest Codex live state. After a fresh restart, resuming the same Codex session hit that file: kv cache hit text tokens=12897 ... key=responses-visible, then completed the next tool call and tool-result turn normally.

Additional live clients:

• Pi was resumed from the saved session file and completed a real bash tool turn. In this invocation pi sent a shorter prompt for the new turn, so DS4 cold-prefilled that request and then accepted the tool result through responses live continuation match=visible-prefix ids=1.
• opencode was run for two real shell-tool turns against the configured ds4/deepseek-v4-flash provider. This path uses the OpenAI-compatible chat API rather than /v1/responses, and it completed both the initial tool call and the resumed tool call. That is useful shared-path regression coverage: chat/completions tool checkpoint canonicalization still works while Responses skips it.

More protocol negatives:

• Non-null previous_response_id: still HTTP 400.
• Non-null conversation: still HTTP 400.
• Tool output with an unknown call_id: still HTTP 400.
• Stateless replay of a prior tool call plus tool output, without a reasoning item in thinking mode: HTTP 400.
• The same stateless replay shape with an explicit reasoning item: HTTP 200.
• The short tool-output-only happy path was re-tested after the stricter guard: first curl request generated a bash function_call, second request sent only the matching function_call_output, and DS4 logged responses live continuation match=tool-output-ids ids=1.
• The stale-live-state failure path was also tested with real queued requests: create a live bash function_call, start a long unrelated request that replaces the live KV, then submit the old tool-output-only request while the long request is still running. When the queued tool-output-only request executed, DS4 returned HTTP 400 with Responses tool output requires live call state; replay full input instead. This verifies the new guard prevents cold-prefilling an orphan tool result.

Post-merge stress pass, 2026-05-14:

• Merged the responses-api branch into main while preserving its branch commits, then re-applied the exact sampled DSML tool-checkpoint gate on top of the merged Responses live-continuation path.
• Replaced eager tool-less thinking checkpoint rebuilds with a visible-transcript live binding. A no-tool thinking answer now remembers the visible transcript that clients replay, while keeping the sampled hidden reasoning in KV. The next chat/completions or Anthropic request can continue with cache_source: thinking-visible if its prompt extends that visible prefix.
• Unit/build checks after the change: make ds4_test, ./ds4_test --server, and make ds4-server all passed.
• Live stress server: ./ds4-server -m gguf/DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix.gguf --ctx 100000 --kv-disk-dir /tmp/ds4-kv-stress --kv-disk-space-mb 8192 --trace /tmp/ds4-response-stress-trace.txt.
• Pi, using the OpenAI-compatible chat/completions path, completed a multi-tool session and a resumed turn. The trace showed normal live-prefix reuse: requests after the first cold prompt used cache_source: memory-token with only short suffix prefills.
• Codex CLI, using /v1/responses, completed one tool-heavy turn and a resumed turn. Responses continuations used cache_source: responses-visible; the first resume after the final answer logged ids=0 and continued from the remembered visible prefix, then tool-result turns logged ids=1.
• Claude Code, using /v1/messages, completed real read/write/bash tool loops. One apparent bad miss after a Read call was investigated in the rendered trace: the client changed its tool schema block between requests (NotebookEdit disappeared and LSP appeared), so the low common prefix was a real prompt change, not a DSML replay failure. Disk recovery limited the replay to the nearest checkpoint (cached_tokens=22528 for a 24308-token prompt).
• opencode, using OpenAI-compatible chat/completions, completed a tool session and a resumed tool session. Most tool-result turns used live memory; a few used cache_source: memory-text because the exact token stream and rerendered byte prefix were text-identical but had different tokenization around a boundary, which is the intended text-prefix fallback.
• A direct no-tool chat/completions test exercised the new thinking-visible binding: first request logged thinking live checkpoint remembered, second request logged thinking live continuation match=visible-prefix cached=54 prompt=66, with no rebuild and only the new suffix prefilling.
• No trace line in this run showed thinking checkpoint canonicalization needs rebuild or a tool checkpoint rebuild caused by exact sampled DSML replay.