* feat(claude_code_exec): add claude code optimizer backend with SDK trace support
Register claude_code_exec as a full optimizer/target backend (issue #233).
--backend claude_code_exec now defaults both roles to claude_code_exec so
reflection sees the agent's complete session, and the SDK message stream is
parsed into structured trace steps persisted as claude_trace_steps.txt and
injected into the analyst prompt.
- model/claude_code_backend.py (new): chat_optimizer/chat_optimizer_messages on
run_claude_code_chat, reasoning_effort threaded through, retry loop that
surfaces non-JSON structured replies as RuntimeError, token tracking.
- model/codex_harness.py: parse/format/persist claude trace steps (text,
tool_call, tool_result; drops init/thinking_tokens; 200-char tool_result cap;
total truncation) + effort override on run_claude_code_chat.
- trainer.py/reflect.py: inject Claude Trace Steps gated behind
REFLACT_CLAUDE_TRACE_TO_OPTIMIZER, set by the trainer only for claude_code_exec
targets with model.claude_trace_to_optimizer (mirrors codex gate; default true).
- config.py/default.yaml/docs: model.claude_trace_to_optimizer key + flatten
mapping + config.md rows.
- backend_config.py + model/__init__.py: register backend, route chat dispatch,
token summary, reasoning effort, deployments.
- scripts/train.py, eval_only.py: symmetric default + accurate comments.
- tests: tests/test_claude_code_backend.py (10 tests: parsing, dispatch, effort,
retry, trainer/reflect gating); test_role_backend_resolution.py updated to the
symmetric default.
Verified: 58 unit tests pass; integration smoke on searchqa improved best-on-val
0.7500 -> 0.9375 with 80 claude_trace_steps.txt written; all output files valid
UTF-8 (no GBK mojibake).
* fix(claude_code_exec): address #233 review feedback
chat_template_kwargs is a vLLM/SGLang extension. OpenAI, Azure, and strict
OpenAI-compatible gateways reject the unknown body field with HTTP 400, and
non-Qwen vLLM models served with it can emit <think> output with no <answer>
tag (acc=0.000). c31c50b fixed that by only emitting the field when thinking
was enabled, which closed#28 but left no supported way to send an explicit
enable_thinking: false -- the request in #90/#109.
The protocol has three states, so make the setting three-state:
server_default (default) -> omit chat_template_kwargs
enabled -> send enable_thinking: true
disabled -> send enable_thinking: false
server_default keeps every existing deployment on exactly the bytes it sends
today, so #28 stays fixed, while disabled gives #90 the explicit false it asks
for. The legacy enable_thinking boolean keeps its historical wire meaning
(true -> send true, false -> omit), so no config changes behavior; setting
both keys to conflicting values raises rather than silently picking a winner.
Unknown tokens raise too -- a typo must not silently flip a reproducibility
control.
Because server_default delegates a result-affecting choice to the server's
chat template, the backend warns once per role when it is used, and the
resolved per-role mode is recorded in the run's config.json under
resolved_qwen_thinking_modes.
Also settles the docs contradiction between "local vLLM endpoint" and
"OpenAI-compatible": qwen_chat speaks the OpenAI protocol and reaches both
self-hosted servers and hosted gateways, which is exactly why the wire policy
cannot be inferred and must be explicit.
Closes#90
The minimax_chat backend hardcoded a single global OpenAI-compatible base
URL, so there was no supported way to target the China-region service.
Add a region-to-base-URL table with global_en and cn_zh entries, select the
region from MINIMAX_REGION or the new model.minimax_region setting, and keep
an explicitly configured base URL as the override. Document both regional
base URLs and cover the resolution order with tests.
Co-authored-by: octo-patch <266937838+octo-patch@users.noreply.github.com>
Resolve Copilot roles before model defaults and omit inherited OpenAI deployment sentinels for copilot_exec. Preserve explicit target model selections and add entry-point regression coverage.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 0e8472e4-56ad-4daf-80b4-1c0ed0258133
Only apply a high-level backend label to inherited default role values. Preserve non-default YAML roles and explicit CLI role overrides, with direct regression coverage.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 0e8472e4-56ad-4daf-80b4-1c0ed0258133
Preserve the shared message contract, fail fast for unsupported tool calls, harden the Copilot CLI tool boundary, and resolve partial role overrides independently. Correct the CLI-authenticated documentation and command example.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 0e8472e4-56ad-4daf-80b4-1c0ed0258133
Addresses review: the reconfigure block was duplicated in both entry points
and would drift. Move it to skillopt/utils/console.force_utf8_stdout_stderr()
and call it from train.py and eval_only.py.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Never raise from stream reconfigure (wrap in try/except); a stream that
exposes an incompatible reconfigure() no longer aborts startup.
- Skip streams already encoded as UTF-8 so redirected output is not needlessly
re-encoded, while cp1252 consoles and cp1252 file redirects are still fixed.
- Drop the unused 'script' parametrization on the arrow-output test.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Adds two backends. `copilot_chat` drives the Copilot CLI as a chat model and
can fill either role, so `--backend copilot` selects it for BOTH optimizer and
target -- the CLI carries its own sign-in, which makes that the only fully local
configuration: a complete train/eval loop with no cloud API key.
`copilot_exec` is the separate target-only execution harness, alongside the
existing codex/claude/cursor harnesses.
Verified end to end on SearchQA with no credentials configured: baseline eval,
rollout, reflect, aggregate, select, update and gate all execute against the
local CLI.
Safety: chat calls disable built-in MCP servers and custom instructions so the
model sees only the prompt SkillOpt sends, and never pass --allow-all-tools.
Unlike the other exec harnesses, `copilot_exec` does NOT grant unattended tool
use by default -- it requires an explicit `copilot_exec_allow_all_tools`
opt-in, because a file-edit rollout is the only case that needs it.
Two caveats worth knowing before use: the CLI is an agent rather than a
completions endpoint, so expect roughly 20-40 s per call; and it reports no
token counts, so usage totals are zero for these backends.
Depends on the --backend resolution fix: without it, --backend copilot is
discarded whenever the base config sets both role backends.
Progress output contains arrow and box-drawing characters (e.g. the
'[2/6 REFLECT] failure=0->0 groups' line and the banner rules). On a Windows
console that defaults to cp1252, writing them raises UnicodeEncodeError and
kills the process partway through a run -- after rollouts and reflect calls
have already been paid for.
Both entry points now reconfigure stdout/stderr to UTF-8 with errors='replace'
at import time, guarded by hasattr so redirected or exotic streams are left
alone.
configs/_base_/default.yaml ships optimizer_backend: openai_chat and
arget_backend: openai_chat. Both entry points only resolved a high-level
--backend label when a role was missing, so for any run using the shipped
defaults the label was silently discarded and the run executed on openai_chat.
skillopt-train --config configs/searchqa/default.yaml --backend cursor
...
[model config] backend=cursor_exec optimizer=... (openai_chat) target=... (openai_chat)
train.py guarded on "is either role unset?"; eval_only.py used
cfg.setdefault(), which is equally a no-op once the key exists. A role left at
the default openai_chat now counts as unset so the label wins, while a role the
operator explicitly pointed elsewhere still takes precedence.
The trainer's resolution moves to a module-level _resolve_role_backends() so it
is testable -- it previously sat inline inside Trainer.train().
Add missing configuration setup in scripts/eval_only.py to properly
support the minimax_chat backend, which was entirely omitted.
Fix the following coverage gaps in eval_only.py:
- Add minimax CLI arguments
- Include the minimax config mappings in _MAP
- Update the backend parsing logic
- Call configure_minimax_chat
The eval-only tool skipped configuring some of the backend types, that
the training did configure. Because of this, the eval is silently
fell back to a local endpoint that wasn't actually configured, and
all evaluations runs failed.
Replicate the backend setup based on the trainer's code, and eval-only
can run with the qwen_chat backends.
Co-authored-by: Qwen-Coder <noreply@qwen.ai>
- Skill optimization framework with training loop analogy
- 11 benchmarks, 4 model backends (Azure OpenAI, Claude, Codex, Qwen)
- WebUI for browser-based training control
- Pluggable architecture for extending benchmarks and backends
Nuplum