* feat(claude_code_exec): add claude code optimizer backend with SDK trace support
Register claude_code_exec as a full optimizer/target backend (issue #233).
--backend claude_code_exec now defaults both roles to claude_code_exec so
reflection sees the agent's complete session, and the SDK message stream is
parsed into structured trace steps persisted as claude_trace_steps.txt and
injected into the analyst prompt.
- model/claude_code_backend.py (new): chat_optimizer/chat_optimizer_messages on
run_claude_code_chat, reasoning_effort threaded through, retry loop that
surfaces non-JSON structured replies as RuntimeError, token tracking.
- model/codex_harness.py: parse/format/persist claude trace steps (text,
tool_call, tool_result; drops init/thinking_tokens; 200-char tool_result cap;
total truncation) + effort override on run_claude_code_chat.
- trainer.py/reflect.py: inject Claude Trace Steps gated behind
REFLACT_CLAUDE_TRACE_TO_OPTIMIZER, set by the trainer only for claude_code_exec
targets with model.claude_trace_to_optimizer (mirrors codex gate; default true).
- config.py/default.yaml/docs: model.claude_trace_to_optimizer key + flatten
mapping + config.md rows.
- backend_config.py + model/__init__.py: register backend, route chat dispatch,
token summary, reasoning effort, deployments.
- scripts/train.py, eval_only.py: symmetric default + accurate comments.
- tests: tests/test_claude_code_backend.py (10 tests: parsing, dispatch, effort,
retry, trainer/reflect gating); test_role_backend_resolution.py updated to the
symmetric default.
Verified: 58 unit tests pass; integration smoke on searchqa improved best-on-val
0.7500 -> 0.9375 with 80 claude_trace_steps.txt written; all output files valid
UTF-8 (no GBK mojibake).
* fix(claude_code_exec): address #233 review feedback
chat_template_kwargs is a vLLM/SGLang extension. OpenAI, Azure, and strict
OpenAI-compatible gateways reject the unknown body field with HTTP 400, and
non-Qwen vLLM models served with it can emit <think> output with no <answer>
tag (acc=0.000). c31c50b fixed that by only emitting the field when thinking
was enabled, which closed#28 but left no supported way to send an explicit
enable_thinking: false -- the request in #90/#109.
The protocol has three states, so make the setting three-state:
server_default (default) -> omit chat_template_kwargs
enabled -> send enable_thinking: true
disabled -> send enable_thinking: false
server_default keeps every existing deployment on exactly the bytes it sends
today, so #28 stays fixed, while disabled gives #90 the explicit false it asks
for. The legacy enable_thinking boolean keeps its historical wire meaning
(true -> send true, false -> omit), so no config changes behavior; setting
both keys to conflicting values raises rather than silently picking a winner.
Unknown tokens raise too -- a typo must not silently flip a reproducibility
control.
Because server_default delegates a result-affecting choice to the server's
chat template, the backend warns once per role when it is used, and the
resolved per-role mode is recorded in the run's config.json under
resolved_qwen_thinking_modes.
Also settles the docs contradiction between "local vLLM endpoint" and
"OpenAI-compatible": qwen_chat speaks the OpenAI protocol and reaches both
self-hosted servers and hosted gateways, which is exactly why the wire policy
cannot be inferred and must be explicit.
Closes#90
The minimax_chat backend hardcoded a single global OpenAI-compatible base
URL, so there was no supported way to target the China-region service.
Add a region-to-base-URL table with global_en and cn_zh entries, select the
region from MINIMAX_REGION or the new model.minimax_region setting, and keep
an explicitly configured base URL as the override. Document both regional
base URLs and cover the resolution order with tests.
Co-authored-by: octo-patch <266937838+octo-patch@users.noreply.github.com>
The warning was a DeprecationWarning, which no normal CLI user would see:
skillopt-train is a console script for scripts.train:main, so the warning is
raised from an imported module rather than from __main__, and Python's default
filters end in ignore::DeprecationWarning. FutureWarning has no such filter.
The CLI flag was also the only path checked, and it is the least dangerous one.
A retired key left in a config file was dropped in silence, since flatten_config
no longer maps it and the trainer no longer reads it, and --cfg-options had the
same hole. All three now warn and name the one that supplied it, for structured
and legacy flat configs alike. An override is reported once rather than twice,
because load_config merges --cfg-options into the config before this check runs.
The check therefore moves below _load: it needs the merged config to see a key
that arrived from a file.
The option was flattened, exposed as --max_analyst_rounds and printed in
the trainer's config banner, but nothing ever read it: the analyst call
count follows from the rollout results, gradient.minibatch_size and
gradient.failure_only. Dropping it also keeps the config.json written
for each run honest about what the run actually used.
The CLI flag is still parsed so existing launch scripts do not fail on
an unrecognised argument, and now warns. It is skipped when CLI
arguments are mapped into the config: an argument with no structured
path would otherwise be filed under env, and env keys are passed
through to the trainer.
Resolve Copilot roles before model defaults and omit inherited OpenAI deployment sentinels for copilot_exec. Preserve explicit target model selections and add entry-point regression coverage.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 0e8472e4-56ad-4daf-80b4-1c0ed0258133
Only apply a high-level backend label to inherited default role values. Preserve non-default YAML roles and explicit CLI role overrides, with direct regression coverage.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 0e8472e4-56ad-4daf-80b4-1c0ed0258133
Preserve the shared message contract, fail fast for unsupported tool calls, harden the Copilot CLI tool boundary, and resolve partial role overrides independently. Correct the CLI-authenticated documentation and command example.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 0e8472e4-56ad-4daf-80b4-1c0ed0258133
Addresses review: the reconfigure block was duplicated in both entry points
and would drift. Move it to skillopt/utils/console.force_utf8_stdout_stderr()
and call it from train.py and eval_only.py.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Never raise from stream reconfigure (wrap in try/except); a stream that
exposes an incompatible reconfigure() no longer aborts startup.
- Skip streams already encoded as UTF-8 so redirected output is not needlessly
re-encoded, while cp1252 consoles and cp1252 file redirects are still fixed.
- Drop the unused 'script' parametrization on the arrow-output test.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Adds two backends. `copilot_chat` drives the Copilot CLI as a chat model and
can fill either role, so `--backend copilot` selects it for BOTH optimizer and
target -- the CLI carries its own sign-in, which makes that the only fully local
configuration: a complete train/eval loop with no cloud API key.
`copilot_exec` is the separate target-only execution harness, alongside the
existing codex/claude/cursor harnesses.
Verified end to end on SearchQA with no credentials configured: baseline eval,
rollout, reflect, aggregate, select, update and gate all execute against the
local CLI.
Safety: chat calls disable built-in MCP servers and custom instructions so the
model sees only the prompt SkillOpt sends, and never pass --allow-all-tools.
Unlike the other exec harnesses, `copilot_exec` does NOT grant unattended tool
use by default -- it requires an explicit `copilot_exec_allow_all_tools`
opt-in, because a file-edit rollout is the only case that needs it.
Two caveats worth knowing before use: the CLI is an agent rather than a
completions endpoint, so expect roughly 20-40 s per call; and it reports no
token counts, so usage totals are zero for these backends.
Depends on the --backend resolution fix: without it, --backend copilot is
discarded whenever the base config sets both role backends.
Progress output contains arrow and box-drawing characters (e.g. the
'[2/6 REFLECT] failure=0->0 groups' line and the banner rules). On a Windows
console that defaults to cp1252, writing them raises UnicodeEncodeError and
kills the process partway through a run -- after rollouts and reflect calls
have already been paid for.
Both entry points now reconfigure stdout/stderr to UTF-8 with errors='replace'
at import time, guarded by hasattr so redirected or exotic streams are left
alone.
configs/_base_/default.yaml ships optimizer_backend: openai_chat and
arget_backend: openai_chat. Both entry points only resolved a high-level
--backend label when a role was missing, so for any run using the shipped
defaults the label was silently discarded and the run executed on openai_chat.
skillopt-train --config configs/searchqa/default.yaml --backend cursor
...
[model config] backend=cursor_exec optimizer=... (openai_chat) target=... (openai_chat)
train.py guarded on "is either role unset?"; eval_only.py used
cfg.setdefault(), which is equally a no-op once the key exists. A role left at
the default openai_chat now counts as unset so the label wins, while a role the
operator explicitly pointed elsewhere still takes precedence.
The trainer's resolution moves to a module-level _resolve_role_backends() so it
is testable -- it previously sat inline inside Trainer.train().
F16: persist last_model_key in sleep state and warn at cycle start when the backend/model changed since the previous night (skill text may not transfer). F12: correct docs to say replay isolation varies by backend. F08: emit a DeprecationWarning when API keys are passed via train.py CLI args, pointing to env vars / managed identity. Adds tests for the state roundtrip, the warning conditions, and the CLI deprecation warning.
Addresses 8 further Copilot inline findings across 04d3b65/4c2aa21:
- pytest_runs: drop the trivially-overwritable .count sidecar; derive the count
from nonce-tagged log lines (per-run os.urandom nonce). Documented honestly as
tamper-EVIDENT, not tamper-proof, since an unsandboxed agent runs as the same
OS user; harness_test_passes (parent re-runs the tests) remains the
authoritative unforgeable gate.
- Refuse SKILLOPT_HOST_AUTH=1 together with SKILLOPT_SANDBOX: host ~/.claude is
not mounted, so the credential symlinks would dangle and auth silently fail.
- Raise on an unknown --scenario instead of returning an empty score=0 result
that looks like a real evaluation.
- POSIX guard: the bash shims + claude/git shell-out are POSIX-only; raise a
clear error on non-POSIX hosts rather than failing obscurely.
- CLI: catch git CalledProcessError / ValueError / RuntimeError so missing
git/claude, bad SHA, and unknown scenarios exit non-zero with a message
instead of dumping a traceback.
- Clarify that superpowers_version is a reporting label; the checkout is
controlled solely by pinned_sha (--sha).
- Smoke sanitizer: also redact /tmp workspace paths and soften the "no host
paths" claim to best-effort.
Tests: 49 focused, full suite 309 passed / 6 skipped.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Addresses remaining maintainer + Copilot review blockers on #134.
- Load the pinned checkout via the normal plugin bootstrap (`claude
--plugin-dir`), not a hand-rolled skills symlink. A per-run session marker is
injected into using-superpowers/SKILL.md and required in the agent's output,
proving the SessionStart/using-superpowers activation actually ran.
- Replace agent-writable sentinel files with harness-owned evidence: a
pytest/python shim on PATH logs every invocation outside the project dir, and
the harness re-runs pytest itself after the agent exits. Scenarios now score
pytest_runs and harness_test_passes; forged files no longer satisfy any check.
- Stop reusing host credentials by default. ~/.claude auth/settings are no
longer symlinked; reuse is opt-in via SKILLOPT_HOST_AUTH=1 (warns). Fail
closed (NO_AUTH) when neither a key nor host-auth is available.
- Add OS-level isolation, opt-in via SKILLOPT_SANDBOX=bwrap|docker.
- Prompt on stdin + --output-format text, matching backend.py CLI usage.
- Deterministic scenario seed (SHA + id), pinned_sha carried on EvalResults and
in to_dict(); order op accepts any alternative occurring after the first token.
- Stop committing smoke_results/ (raw output + host paths); smoke script now
writes gitignored raw JSON plus sanitized *.summary.txt excerpts to share.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Fix not_contains to split on pipe (all alternatives must be absent)
- Add regression tests for false completion claim detection
- Scrub host env: only PATH/TERM/LANG/ANTHROPIC_API_KEY, no credentials
- Remove unconditional --dangerously-skip-permissions (opt-in via SKILLOPT_UNSAFE=1)
- Include raw output in JSON for smoke test evidence
- Fix smoke script: fail on errors, preserve raw output
Runnable SearchQA splits should remain disjoint. A duplicate manifest id across train, val, or test previously collapsed into the wanted-id set and reused the same row in multiple output splits without warning.
Constraint: Preserve manifest order and output schema for valid manifests.
Rejected: Deduplicate automatically | hiding split overlap would make evaluation contamination harder to notice.
Confidence: high
Scope-risk: narrow
Reversibility: clean
Directive: Fail fast on split-manifest integrity problems instead of repairing them silently.
Tested: uv run --with pytest pytest tests/test_materialize_searchqa.py -q
Tested: uv run --with ruff ruff check scripts/materialize_searchqa.py tests/test_materialize_searchqa.py
Not-tested: Loading the live Hugging Face dataset.
Add missing configuration setup in scripts/eval_only.py to properly
support the minimax_chat backend, which was entirely omitted.
Fix the following coverage gaps in eval_only.py:
- Add minimax CLI arguments
- Include the minimax config mappings in _MAP
- Update the backend parsing logic
- Call configure_minimax_chat
The eval-only tool skipped configuring some of the backend types, that
the training did configure. Because of this, the eval is silently
fell back to a local endpoint that wasn't actually configured, and
all evaluations runs failed.
Replicate the backend setup based on the trainer's code, and eval-only
can run with the qwen_chat backends.
Co-authored-by: Qwen-Coder <noreply@qwen.ai>
Split failure reflections into SKILL_DEFECT (body edit) vs EXECUTION_LAPSE
(protected appendix note that re-emphasizes an existing rule, never edited
by step-level analysts). Toggle: optimizer.use_skill_aware_reflection
(default false; baseline byte-identical when off).
- optimizer/appendix.py: protected APPENDIX region (inject/extract/append
with dedup), mirrors the slow_update protected-field pattern
- optimizer/skill_aware.py: analyst prompt augmentation, appendix_notes
parsing, threshold-gated LLM consolidation, and a process-wide runtime
switch (configure_skill_aware_reflection) set once by the trainer
- gradient/reflect.py: augment error/success analyst prompts at runtime;
None-sentinel kwargs resolve from the global switch, so env adapters
need no per-benchmark wiring (works for all envs, present and future)
- optimizer/skill.py: generalize the protected-region check to
(slow_update, appendix); edits inside any protected region are skipped
- engine/trainer.py: inject appendix at init, flush per-step
EXECUTION_LAPSE notes after the gate settles, optional consolidation
- tests: regression suite incl. toggle-off byte-identical guarantee and
env-independent global-switch resolution (6/6 passing + live smoke)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
PR #26 added a MiniMax chat backend but left three loose ends that
silently dropped any YAML / CLI configuration of minimax_* keys: only
the environment-variable path worked.
- skillopt/config.py: add 6 model.minimax_* entries to _FLATTEN_MAP so
the keys declared in configs/_base_/default.yaml actually survive
flatten_config() (mirroring the existing model.qwen_chat_* block).
- skillopt/engine/trainer.py: import configure_minimax_chat and call
it alongside configure_qwen_chat, so cfg-supplied credentials,
temperature, max_tokens, and enable_thinking reach the backend. Also
apply cfg["minimax_model"] via set_target_deployment when the active
target backend is minimax_chat.
- scripts/train.py: add 6 --minimax_* CLI flags + the corresponding
_CLI_TO_YAML entries, add 'minimax' / 'minimax_chat' to the --backend
choices, auto-route to target_backend=minimax_chat, and pick the
right default target_model for the new backend.
Default behavior on existing backends (openai, claude, qwen, codex,
claude_code_exec) is unchanged; all 8 shipped configs continue to load
with gate_metric falling back to 'hard' for paper reproduction.
Remove sealqa, babyvision, mathverse, mmrb, swebench envs and configs.
Remove deep_probe, deep_reflect, meta_reflect modules and prompts.
Remove download_babyvision script.
These are not part of the core released benchmarks.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Skill optimization framework with training loop analogy
- 11 benchmarks, 4 model backends (Azure OpenAI, Claude, Codex, Qwen)
- WebUI for browser-based training control
- Pluggable architecture for extending benchmarks and backends
Nuplum