fix(proxy): stop a lone surrogate turning a thinking body into a 500 (#3134)

## What

`serialize_body_canonical` uses `ensure_ascii=False`, so a lone
surrogate anywhere in the body raises `UnicodeEncodeError` at
`.encode("utf-8")`.

This is reachable input, not a hypothetical:
- `"\ud800"` is **valid JSON** — `json.loads` accepts it happily
- a tool result carrying truncated UTF-16 or sliced binary produces one

Both forwarders resolve outbound bytes **outside** their
connection-retry loop (`streaming.py:1131`, `server.py:2170`), so the
exception escapes as an **unretried 500**.

## Why now

#3124 made this newly load-bearing. Before it, a mutated
thinking-bearing body returned the client's bytes verbatim and **never
reached canonical serialization at all**. Now it does — so the largest,
most tool-result-heavy population in Claude Code traffic depends on this
not raising.

Reproduced against `main`:

```
serialize_body_canonical RAISES: UnicodeEncodeError: 'utf-8' codec can't
  encode character '\ud800' in position 91: surrogates not allowed
select_outbound_body RAISES: UnicodeEncodeError: ...
```

## The fix

Fall back to the escaped encoding on `UnicodeEncodeError`.

**Why this and not passthrough.** Falling back to the client's original
bytes would silently drop every mutation — including the handler's
`stream` flip — and diverge from `outbound_body_is_client_bytes`, which
cannot predict a serialization failure without doing the serialization.
That reintroduces the #2952 buffered/streamed mismatch. The escaped form
keeps all mutations on the wire.

It encodes the **identical parsed values**, so upstream reconstructs
exactly the same request and the signed thinking blocks round-trip
untouched (asserted in the test). Only the byte-level encoding differs,
costing one cache miss on a request that would otherwise have failed
outright. Normal bodies are unaffected — the fast path is unchanged and
still emits compact non-ASCII.

## Test

`test_lone_surrogate_in_thinking_body_serializes_instead_of_raising` —
asserts no raise, `source == "canonical"`, mutation preserved, and the
signed block round-tripping to exactly the client's values.

Local: 78 passed across `test_proxy_byte_faithful_forwarding.py` +
`test_ccr_buffered_stream_signed_thinking.py`; 191 passed across all
serialization-touching tests. ruff + mypy clean.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Tejas Chopra <tejas@Tejass-MacBook-Pro.local>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>

headroom/proxy/body_forwarding.py

@@ -56,8 +56,30 @@ def get_python_forwarder_mode() -> PythonForwarderMode:
 
 
 def serialize_body_canonical(body: dict[str, Any]) -> bytes:
-    """Re-serialize a request body deterministically with cache-stable formatting."""
-    return json.dumps(body, separators=(",", ":"), ensure_ascii=False).encode("utf-8")
+    """Re-serialize a request body deterministically with cache-stable formatting.
+
+    ``ensure_ascii=False`` keeps the bytes compact and cache-stable, but it also
+    means a lone surrogate anywhere in the body raises here. That is reachable
+    input, not a hypothetical: ``"\\ud800"`` is valid JSON, ``json.loads``
+    accepts it happily, and a tool result carrying truncated UTF-16 or sliced
+    binary produces one. Both forwarders resolve outbound bytes *outside* their
+    connection-retry loop, so the exception escapes as a 500 with no retry.
+
+    #3124 made that newly load-bearing: mutated thinking-bearing bodies used to
+    return the client's bytes verbatim and never reached this function at all,
+    so the largest, most tool-result-heavy population in Claude Code traffic now
+    depends on it not raising.
+
+    The escaped form is the right degradation -- it encodes the identical parsed
+    values, so upstream reconstructs exactly the same request, and every mutation
+    still reaches the wire (important: the caller's ``stream`` flip rides on
+    these bytes). Only the byte-level encoding differs, costing one cache miss on
+    a request that would otherwise have failed outright.
+    """
+    try:
+        return json.dumps(body, separators=(",", ":"), ensure_ascii=False).encode("utf-8")
+    except UnicodeEncodeError:
+        return json.dumps(body, separators=(",", ":"), ensure_ascii=True).encode("utf-8")
 
 
 def has_signed_thinking_blocks(body: dict[str, Any]) -> bool:

tests/test_proxy_byte_faithful_forwarding.py

@@ -1893,3 +1893,52 @@ def test_unparseable_original_cannot_prove_preservation(
     monkeypatch.setenv("HEADROOM_THINKING_PRESERVING_MUTATIONS", "1")
     assert thinking_blocks_survived_mutation(_tb_body(), b"{not json") is False
     assert thinking_blocks_survived_mutation(_tb_body(), None) is False
+
+
+def test_lone_surrogate_in_thinking_body_serializes_instead_of_raising():
+    """A lone surrogate must not turn a mutated thinking body into a 500.
+
+    ``"\\ud800"`` is valid JSON, so ``json.loads`` accepts it and a tool result
+    carrying truncated UTF-16 produces one. Before #3124 a mutated
+    thinking-bearing body returned the client's bytes verbatim and never reached
+    canonical serialization; now it does, and both forwarders resolve outbound
+    bytes outside their retry loop, so a raise here escapes as an unretried 500.
+    """
+    import json
+
+    from headroom.proxy.body_forwarding import select_outbound_body
+
+    lone_surrogate = chr(0xD800)
+    original = {
+        "messages": [
+            {
+                "role": "assistant",
+                "content": [
+                    {
+                        "type": "thinking",
+                        "thinking": f"reasoning {lone_surrogate}",
+                        "signature": "sig",
+                    }
+                ],
+            }
+        ]
+    }
+    original_bytes = json.dumps(original, ensure_ascii=True).encode("utf-8")
+    mutated = json.loads(original_bytes)
+    mutated["messages"].append({"role": "user", "content": "compressed"})
+
+    outbound = select_outbound_body(
+        body=mutated,
+        original_body_bytes=original_bytes,
+        body_mutated=True,
+        forwarder_mode="byte_faithful",
+    )
+
+    # The relaxation still applies (the thinking block is untouched) and the
+    # mutation reaches the wire rather than being discarded or crashing.
+    assert outbound.source == "canonical"
+    assert not outbound.dropped_mutations
+    reparsed = json.loads(outbound.content)
+    assert reparsed == mutated
+    # The signed block round-trips to exactly the values the client sent.
+    assert reparsed["messages"][0] == original["messages"][0]