Register a VERL policy loss that gives each rollout equal weight across its expanded training rows. Normalize advantages by rollout token count and trained batch size, and load the custom loss in Ray actor workers.
Add focused coverage for registration, normalization, PPO aggregation, and input validation.
* example/async_swe_smith: online GRPO training example for SWE-smith
Add an end-to-end async GRPO example that trains a SWE agent on the
SWE-smith dataset via agl-lite, plus supporting infrastructure:
- examples/async_swe_smith: agent, chat template, k8s job template,
image pre-pull, FSDP + Megatron trainers, Qwen3-30B-A3B / Qwen3.5-9B
launch scripts, README, and a smoke test.
- agl_lite/verl/trainer.py: per-step GRPO group / zero-advantage group
counts and a per-stage perf/mfu/actor_infer metric.
- examples/calc_x/comparison: async-vs-sync runner scripts.
Co-Authored-By: Claude Opus 4 (1M context) <noreply@anthropic.com>
* feat(verl): record per-rollout pod lifecycle timing
Capture server-authoritative running_at/finished_at while polling and
emit, per training step, a wandb Table of per-rollout submitted/running/
finished timestamps plus queue-wait/run-duration scalar aggregates. The
queue wait (running_at - submitted) exposes how long pods sit waiting
when jobs are launched in CPU-limited batches.
Co-Authored-By: Claude Opus 4 (1M context) <noreply@anthropic.com>
* fix(setup_verl): install matching CUDA 13.0 toolchain for cu130 flash-attn build
The cu130 variant builds flash-attn from source against torch's CUDA 13.0
runtime, but the system CUDA toolkit is often 12.x, so nvcc rejects the
build. Install a CUDA 13.0 pip toolchain (nvcc/crt/nvvm/cccl/runtime) pinned
to >=13.0,<13.1 and point the build at it via CUDA_HOME/PATH/LIBRARY_PATH/
LD_LIBRARY_PATH/CPATH. Pinning to 13.0.x keeps nvcc's version aligned with
torch's CUDART (13000); a mismatched minor (e.g. 13.3) trips cccl's
"CUDA compiler and CUDA toolkit headers are incompatible" check.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* smaller cpu request
* feat(async_swe_smith): add Qwen3-8B sync-rollout trainer launch script
Mirrors the async async_swe_smith_qwen3_8b run (wandb fxaj7bcw) but flips
agentlightning.async_rollout.enabled=False for synchronous rollouts, keeping
all other hyperparameters identical (TP=2, GRPO n=8, train_batch_size=32,
ppo_micro_batch_size_per_gpu=2, lr=1e-6, clip 0.2/0.28, max_model_len=32768).
rollout.mode stays async (vLLM OpenAI server mode for tool calling). Sizing
knobs are env-overridable (AGL_GPU_MEM_UTIL, AGL_ROLLOUT_TP, etc.).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* example/async_swe_smith: add Qwen3-8B async runner; enable chunked prefill
Add run_qwen3_8b_async.sh (async counterpart of run_qwen3_8b_sync.sh):
identical training hyperparameters, only agentlightning.async_rollout.
enabled=True, for the sync-vs-async speedup A/B. Also flip the example's
vLLM enable_chunked_prefill to True.
Co-Authored-By: Claude Opus 4 (1M context) <noreply@anthropic.com>
* example/async_swe_smith: lower agent pod memory limit to 2Gi
Co-Authored-By: Claude Opus 4 (1M context) <noreply@anthropic.com>
* example/async_swe_smith: expose train/async batch sizes as env-overridable knobs
Surface data.train_batch_size (32) and async_rollout.async_train_batch_size
(48) in run_qwen3_8b_async.sh via AGL_TRAIN_BATCH_SIZE / AGL_ASYNC_TRAIN_BATCH_SIZE,
passed through SIZING_OVERRIDES. Also bump default total_epochs 2 -> 4.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* example/async_swe_smith: disable chunked prefill + val-before-train; add requirements.txt
Co-Authored-By: Claude Opus 4 (1M context) <noreply@anthropic.com>
* update config
* smaller poll interval for less cpu pressure
* example/async_swe_smith: auto-create AGL_NAMESPACE before controller start
The controller does not create its target namespace; with a non-default
AGL_NAMESPACE the ConfigMap step would fail on a missing namespace. Ensure
it idempotently before launch so per-user namespace isolation works.
Co-Authored-By: Claude Opus 4 (1M context) <noreply@anthropic.com>
* example/async_swe_smith: add openai-preinstalled job template variant
Uses the :openai SWE-smith images (openai library baked in) and drops the runtime 'pip install openai', removing the per-rollout cold-start cost.
* async_swe_smith: use openai-preinstalled job template for training
Point the async trainer at job-template-openai.yaml (":openai" images
with the openai library prebuilt) so rollout pods skip the runtime
pip install openai and start the agent directly.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* example/async_swe_smith: add Qwen3-30B-A3B Megatron/R3 async runner
Conservative single-node 4x B200 defaults (n=4, gpu_mem_util=0.6,
train_batch=16, full param/optim/grad offload + recompute) sized to fit
30B-A3B; all knobs env-overridable. Set vLLM moe_backend=triton.
Co-Authored-By: Claude <noreply@anthropic.com>
* async_swe_smith: support explicit pre-split train/val datasets
Add --train-dataset-path / --val-dataset-path (default to
train_datasets.jsonl / val_datasets.jsonl) so the trainer consumes
pre-split, pre-curated datasets as-is: no FAIL_TO_PASS curation and no
train/val split. Falls back to the legacy single-file + split path when
those files are absent. Adds --max-val-instances to bound validation
eval time.
Large local datasets (subset0 / train / val .jsonl) are excluded via
.git/info/exclude rather than .gitignore, so this drops the tracked
subset0 ignore line.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* async_swe_smith: support explicit pre-split train/val datasets in megatron entrypoint
Mirror the load_split_file path from train_smith_agent.py (27ec14d) into the
Megatron+R3 entrypoint: add --train-dataset-path/--val-dataset-path/
--max-val-instances and consume the pre-split files as-is (no FAIL_TO_PASS
curation, no internal train/val split) when both exist, else fall back to the
single-file split. Also add --ci so the run script's smoke flag is a real arg
instead of leaking into the hydra config overrides.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* async_swe_smith: add rollout correction (IS/RS) knobs to megatron runner
Wire algorithm.rollout_correction.* (bypass_mode / rollout_is / rollout_rs
and thresholds) into the Qwen3-30B-A3B megatron runner as env-overridable
sizing overrides, mirroring uni-agent's train_qwen3_moe_rc.sh defaults.
loss_mode is left at the actor default (verl 0.8.0 has no gspo).
Co-Authored-By: Claude <noreply@anthropic.com>
* async_swe_smith: match fsdp rollout sizing in megatron runner
Bump default ROLLOUT_N 4->8, TRAIN_BATCH_SIZE 16->32, ASYNC_TRAIN_BATCH_SIZE
24->48 to mirror the fsdp distributed run's rollout workload, enabling an
apples-to-apples engine comparison at equal generation volume.
Co-Authored-By: Claude <noreply@anthropic.com>
* async_swe_smith: standardize wandb experiment naming
Name runs as swe_smith_{sync|async}_{model}_{backend} so FSDP and Megatron
runs are distinguishable in wandb. Mode is read from rollout.mode, model
from the basename of the model path; backend is fsdp / megatron per script.
Co-Authored-By: Claude <noreply@anthropic.com>
* async_swe_smith: add pure-rollout collection harness
Collect reward (all-0/all-1 GRPO degeneration), completion-token length, and
turn-count stats over the train/val datasets without running a VERL trainer.
- serve_vllm_qwen3_8b.sh: stand up Qwen3-8B vLLM (128K ctx, chunked prefill,
TP=1, CUDA graph, hermes tools, return-tokens-as-token-ids) and register it
with the server proxy via POST /api/models.
- enqueue_rollouts.py: POST each dataset row N times (GRPO group) to
/api/rollouts with the openai job template + is_train flag.
- rollout_stats.py: poll finished rollouts, aggregate from reward/model_request/
agent_output events, group by data_id, report train/val separately.
- job-template-openai.yaml: SMITH_MAX_TURNS=1000 so turns (not the 40-cap) bound
self-completion.
- rollout_test.md: design doc.
Co-Authored-By: Claude <noreply@anthropic.com>
* async_swe_smith: retry reward post until it succeeds
Reward events are training signal, so a single timeout silently dropped
the sample. Retry with capped exponential backoff until the post lands.
Co-Authored-By: Claude <noreply@anthropic.com>
* async_swe_smith: full-run coverage for pure-rollout stats harness
The rollout_stats collector previously polled GET /api/rollouts?state_in=
succeeded&failed&limit=10000. That list route returns matches[:limit] in
insertion order with no offset, and each rollout is ~77KB (job_template is
embedded), so for the 164k-rollout SWE-smith run stats would silently plateau
at ~10k terminal rollouts (~6% of the run) — never seeing later completions.
Server: add an append-only completion log (_terminal_order, appended on
terminal state transitions in patch_rollout) and a cursor-paginated, lightweight
projection endpoint GET /api/rollouts/terminal?after=&limit= returning only
rollout_id/state/data_id/is_train + a next_after cursor. Because the log is
ordered by completion, an index cursor never misses out-of-order completions
and needs no full rescan of the store.
Collector: drain the cursor instead of re-listing; O(page_size) work per poll,
no 10k cap. Atomic snapshot writes (tmp + os.replace), per-rollout event-fetch
retry that skips unreadable rollouts without stalling the cursor, and a
--page-size flag. Snapshot now reports processed/total_terminal/backlog.
serve_vllm_qwen3_8b.sh: Qwen3-8B native ctx is 40960 and vLLM 0.20.2 dropped
--rope-scaling, so --max-model-len 131072 was rejected outright. Add YaRN via
--hf-overrides (factor 4.0 over original 32768) when extending beyond native
context — the documented 128K goal now actually serves.
Tests: cover the new endpoint (completion-order pagination, cursor advance,
projection fields, non-terminal exclusion); reset _terminal_order between tests.
.gitignore: ignore the generated rollout_stats.json output.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* async_swe_smith: checkout bug branch so the agent sees the injected bug
The SWE-smith image defaults /testbed to clean `main`, so the agent saw fixed
code, had nothing to fix, and evaluate() false-passed (reward 1.0 on a 0-byte
patch). Now the agent checks out the bug branch HEAD (`Remove F2P Tests`):
buggy source with the FAIL_TO_PASS tests removed, so it cannot read the tests
to reverse the fix. At evaluation time the F2P test files are restored from the
parent `Bug Patch` commit, keeping the agent's edits, so the real
FAIL_TO_PASS/PASS_TO_PASS suite runs against the fix.
Co-Authored-By: Claude <noreply@anthropic.com>
* Align SWE-smith agent loop with mini-swe-agent
Replace OpenAI tool-calling with mini-swe-agent's text/bash action space:
- one bash code block per turn (parse_action); submit via the
COMPLETE_TASK_AND_SUBMIT_FINAL_OUTPUT marker instead of a submit tool
- swebench-style system/instance prompts adapted for /testbed
- head/tail observation elision + format-error nudges (max_format_errors)
- execute commands via 'bash -c' with pagers/progress bars silenced
Keeps AGL glue (events, eval-meta, bug-commit checkout, F2P restore,
git-diff patch capture, FAIL_TO_PASS/PASS_TO_PASS eval) and the
context-overflow / transient-error loop contracts. Adds SMITH_MAX_FORMAT_ERRORS.
* async_swe_smith: distinguish eval timeout from real failure + rollout tooling
- smith_agent: evaluate() now flags eval_timeout (pytest rc=124 / "timed out"),
surfaced in the reward event; fix p2p_ok counting None status as PASSED (which
let timeouts false-pass as resolved). Set temperature=1.0 explicitly; cap
SMITH_MAX_TURNS at 100 in job-template-openai.
- rollout_stats: tolerate transient connection errors (RemoteDisconnected).
- add monitor_controller.sh (periodic health probe), trace.md (4-rollout trace
example), reward_table.md (per-instance reward/patch snapshot).
Co-Authored-By: Claude <noreply@anthropic.com>
* async_swe_smith: add SWE-smith --f2p_only eval mode (on by default)
Mirror swesmith/harness/grading.py: when f2p_only, restrict evaluation to the
test files containing FAIL_TO_PASS. F2P is kept whole; PASS_TO_PASS is filtered
to only the P2P tests in those same files — same-file regressions still count,
but the many unrelated cross-file P2P tests (the eval-time/timeout driver) are
dropped. Resolution still needs F2P AND filtered P2P to pass. Toggled by
SMITH_F2P_ONLY (default on).
Co-Authored-By: Claude <noreply@anthropic.com>
* async_swe_smith: add rollout-collection runbook + full f2p_only result table
- plan.md: end-to-end runbook for launching a local pure-rollout collection
batch (cold-start vLLM/server/controller, enqueue, monitor, gold-patch
comparison), with the known pitfalls and how to run the second half.
- reward_table.md: full 367-instance agent-vs-gold patch comparison under
official f2p_only (all-0 98.1%, eval timeouts 0/1468).
Co-Authored-By: Claude <noreply@anthropic.com>
* async_swe_smith: lightweight branch switch to avoid checkout timeouts
git checkout --force <branch> stats/rewrites the whole /testbed working tree
(thousands of files for repos like pandas), which times out under high pod
concurrency (node at 6x oversubscription) and fails jobs. Instead, diff
main..branch at the object level to get just the changed files and apply only
those: modified/added paths via checkout, deleted F2P tests via git rm. All
git calls now go through a retry helper. Measured 1234ms -> 32ms on pandas;
working-tree state is identical to the old full checkout.
Co-Authored-By: Claude <noreply@anthropic.com>
* feat(swe-smith): Qwen3.5-9B pure-rollout eval + 8B vs 9B reward table
- 4x vLLM Qwen3.5-9B (TP=1, round-robin) pure-rollout over subset0
- gen_compare_table.py / gen_reward_table_9b.py reward-table builders
- reward_table.md: 8B(all-0 98%) vs 9B(~47%), 116/230 improved
- baseline 8B table snapshot; trainer wandb table trimmed; async sizing
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* swe-smith: per-call max_tokens 16384, vLLM ctx 128k for 9B rollout
- smith_agent default AGL_MAX_TOKENS 4096->16384
- job-template-openai: AGL_MAX_TOKENS=16384 env
- vLLM restarted max_model_len 131072 (4x, registered)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* fix shuffling issue & zero sample issue
* async_swe_smith: align reward with official XFAIL grading + 128k train ctx
Count XFAIL as pass (F2P/P2P) to match swesmith/harness/grading.py; set
data/trajectory max lengths to 65536 = 128k. Add event-driven fail/starve
pod watcher.
Co-Authored-By: Claude <noreply@anthropic.com>
* swe-smith: reward_table 9B 321/367, add per-rollout total token length
- gen_compare_table: 9B total-len column (avg) + p50/p90/max summary
- 9B vs 8B: all-0 98%->52%, improved 147, regress 0
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* swe-smith: annotate units in reward_table (patch=bytes, total=tokens)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* async_swe_smith: tee rollout pod logs to hostPath for post-mortem
Mount host /agl-logs and tee stdout/stderr to agl-rollout-<id>.log so a
killed/evicted pod's trace survives the TTL reaper. Both templates.
Co-Authored-By: Claude <noreply@anthropic.com>
* async_swe_smith: bump pod memory to 4Gi to stop OOMKills
Long 128k-ctx trajectories were killed at 2Gi. Raise request+limit to 4Gi.
Co-Authored-By: Claude <noreply@anthropic.com>
* refactor: promote async SWE-smith example
Rename async_swe_smith to swe_smith, update startup docs/scripts, and prepare local :openai rollout images from minikube-loaded tags.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* feat: log trace merge mismatches to wandb
Record capped unmerged trajectory triplets as a wandb table and expose mismatch row metrics from the rollout adapter.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* async_swe_smith: add delete.jsonl — instances with missing bug branches
35 instances across 11 repos (mostly Go) whose images lack the
origin/<instance_id> branch, so checkout fails with "unknown revision"
and the rollout dies before turn 1. Listed for dataset filtering.
Co-Authored-By: Claude <noreply@anthropic.com>
* async_swe_smith: wait out gateway pause instead of burning a turn
On 429 'gateway paused' (async weight sync), the agent now retries in place
every 5s up to SMITH_GATEWAY_WAIT_S (default 600s) without incrementing the
turn or appending an empty assistant message. Previously the pause was
treated as a generic error, burning turns and polluting the trajectory.
Co-Authored-By: Claude <noreply@anthropic.com>
* async_swe_smith: harden fetch_eval_meta against transient server errors
RemoteDisconnected escaped the except (it is http.client.HTTPException, not
URLError) and crashed the agent; timeouts were swallowed but left instance_id
empty, wasting a full rollout until checkout failed. Catch connection-reset /
HTTP exceptions, retry 4x with backoff for the bursty transient failures, and
abort immediately when no instance_id can be fetched instead of running on.
Co-Authored-By: Claude <noreply@anthropic.com>
* async_swe_smith: force pytest serial in eval to stop OOMKills
The eval pytest command passed no -n flag, so a testbed project's own config
(e.g. scipy's `-n auto`) spawned workers = HOST core count (dozens), ignoring
the pod's cpu:1 / 4Gi cgroup. ~70 python workers @ ~112MB each exhausted the
4Gi limit and triggered cgroup OOMKill mid-rollout — verified live: OOMed pods
had 70+ python procs, memory.current pinned at 4094/4096Mi. Append `-p no:xdist`
to disable the plugin (overrides any ini `-n auto`; no-op when xdist absent).
Confirmed fixed: new pods now run 1 python proc instead of 70.
Co-Authored-By: Claude <noreply@anthropic.com>
* async_swe_smith: update reward_table.md — Qwen3.5-9B pure-rollout (620 instances)
Snapshot of the in-progress Qwen3.5-9B pure-rollout collection over
train_datasets.jsonl (is_train=false, n=4 GRPO groups), throttled to 50
in-flight. 620 instances / 2490 rollouts terminal so far; all-0 544, all-1 51,
mixed 25; 0 eval timeouts (SMITH_EVAL_TIMEOUT=3000, SMITH_MAX_TURNS=100).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* async_swe_smith: cap pytest at -n4, bump pod mem 6Gi, throttle job rate
Eval-time OOMKills traced to testbed `-n auto` reading host cores (~70 xdist
workers) and busting the cgroup. Pin -n4 (~450MB) with serial fallback when
xdist is absent. Raise pod memory to 6Gi and lower max_jobs_per_minute to 100.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix: harden SWE-smith rollout checkout
* docs: document SWE-smith Qwen3.5 sync run
* swe_smith: point hostPath logs at host dir after kubeadm migration
minikube is replaced by a kubeadm single-node cluster, so pods run directly
on host docker. hostPath /agl-logs (minikube VM path) now points at the real
host dir /home/v-zhiwenzhou/agl-logs — no minikube mount needed. Also lower
openai template pod memory to 4Gi.
Co-Authored-By: Claude <noreply@anthropic.com>
* swe_smith: update reward_table.md — Qwen3.5-9B pure-rollout (7300 instances)
Snapshot of the Qwen3.5-9B pure-rollout collection over the verified + full
train datasets (is_train=false, n=4 GRPO groups), throttled to 50 in-flight,
deduped by data_id. 7300 instances / 29174 rollouts: all-0 4510 (61.8%),
all-1 1667 (22.8%), mixed 1123 (15.4%); 50 eval timeouts
(SMITH_EVAL_TIMEOUT=3000, SMITH_MAX_TURNS=100).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* swe_smith: store reward table as SQLite db, drop markdown
Switch the reward table from reward_table.md to reward_table.db (SQLite).
Column names match the old markdown detail table exactly: instance, rollouts,
"r=0", "r=1", "分类", "best patch_size", "avg turns", "超时".
gen_reward_db.py is incremental — it caches each terminal rollout by id in a
rollout_events table and only fetches events for NEW rollouts, so a refresh
takes ~7s instead of ~10min at 30k+ rollouts. reward_table is rebuilt from
rollout_events (is_train=0, grouped by data_id).
Snapshot: 30905 rollouts / 7718 instances (all-0 4606, all-1 1893, mixed 1219).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* swe_smith: retry idempotent /proxy/pause with 5min timeout; pod mem 6Gi
/proxy/pause only sets state.paused=True (idempotent, lock-guarded), so retrying
is safe. When the server is saturated with in-flight rollouts the pause POST can
be slow to get a connection and the default 30s timeout is not enough — give it
300s and 5 retries with backoff. Also bump openai template pod memory to 6Gi
(kernel OOM traced anon-rss ~4.1GB busting the 4Gi limit).
Co-Authored-By: Claude <noreply@anthropic.com>
* swe_smith: self-documenting timestamped run logs in launcher
Redirect each role's stdout+stderr to a timestamped file
(/tmp/agl_logs/{role}_{model}_{YYYYmmdd-HHMMSS}.log) with a stable
{role}_latest.log symlink, so repeated server/trainer restarts no longer
overwrite each other's logs. A fixed name like /tmp/train_9b_async.log
silently loses the previous run's crash trace on relaunch.
Opt out with AGL_LOG_TO_FILE=0; override dir/name via AGL_LOG_DIR/AGL_LOG_FILE;
auto-skipped for --ci smoke tests.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* fix: dedupe triplet model requests by prompt tokens
* feat: add retry logic to non-GET HTTP requests to harden the trainer
Make every non-GET request in the trainer process resilient to transient
failures so a single HTTP error can no longer crash training mid-step:
- register_model (POST /api/models): retry with backoff (idempotent upsert)
- _resume_gateway (POST /proxy/resume): retry with backoff (idempotent)
- _create_rollouts (POST /api/rollouts): pre-assign client rollout_ids and
retry; server-side enqueue is now idempotent (existing id returns the
existing rollout, events untouched), so retries never duplicate rollouts
- delete_model / _delete_rollout: best-effort, swallow errors
- add DELETE /rollouts/{id} (idempotent, cascades to events) and delete
completed rollouts from the managers to keep server-side state bounded
Also switch AglLiteSyncClient's retrying GET to print over logging for
consistency with the rest of the codebase.
Co-Authored-By: Claude <noreply@anthropic.com>
* swe_smith: prevent git-history reward hacking in smith_agent
SWE-smith testbed repos ship the full git history, so the agent can recover
the injected-bug fix (and the deleted FAIL_TO_PASS tests) via
`git checkout <pre-bug-sha> -- <src>` / `git show` / `git log -p`. On the
Qwen3.5-9B run this inflated val/reward to ~0.80 (96% of resolved val rollouts
used the git hack; ~0% solved cleanly).
Agent-side mitigation:
- relocate_git(): move /testbed/.git out of the worktree after checkout (O(1)
same-fs rename); the harness still reaches it via _git_base()
(--git-dir/--work-tree) for restore_f2p_tests + capture_patch.
- _forbidden_action(): reject any agent command that invokes git or reads git
metadata (.git / --git-dir / the relocated dir), returned as an observation;
wired into run_agent_loop before _run.
- _agent_env(): strip SMITH_HIDDEN_GIT_DIR and any var leaking the path so the
relocated dir is invisible in the pod env.
- Drop the git-based submission step from the prompt (harness captures the
patch independently) and note that git is disabled.
Verified on 50 random val instances (base Qwen3.5-9B, isolated pods):
relocation 50/50 (O(1)), 0 leaks, 0 false-positive blocks; submission and
eval/patch-capture work via --git-dir; blocked git attempts recover to manual
edits without hindering legitimate solves.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* feat: log rollout trajectories as wandb artifacts
* feat: add rollout DELETE endpoint and auto-cleanup in rollout managers
Add DELETE /rollouts/{rollout_id} (idempotent, cascades to events) and a
retrying GET on AglLiteSyncClient. Sync manager deletes each round's
completed rollouts; async manager deletes non-carry-over groups on
completion, keeping server-side state bounded during training.
Co-Authored-By: Claude <noreply@anthropic.com>
(cherry picked from commit 149400351dc1fe16c2bd98a0a8eae34d3931da85)
* feat: add retry logic to non-GET HTTP requests to harden the trainer
Make every non-GET request in the trainer process resilient to transient
failures so a single HTTP error can no longer crash training mid-step:
- register_model (POST /api/models): retry with backoff (idempotent upsert)
- _resume_gateway (POST /proxy/resume): retry with backoff (idempotent)
- _create_rollouts (POST /api/rollouts): pre-assign client rollout_ids and
retry; server-side enqueue is now idempotent (existing id returns the
existing rollout, events untouched), so retries never duplicate rollouts
- delete_model / _delete_rollout: best-effort, swallow errors
- add DELETE /rollouts/{id} (idempotent, cascades to events) and delete
completed rollouts from the managers to keep server-side state bounded
Also switch AglLiteSyncClient's retrying GET to print over logging for
consistency with the rest of the codebase.
Co-Authored-By: Claude <noreply@anthropic.com>
(cherry picked from commit 137506f293f5287ee62dcacdab358c43f9c40825)
* fix: align rollout cleanup retry merge
* swe_smith: update reward_table.db — 24836 instances (85% of verified)
Qwen3.5-9B pure-rollout over train_dataset_verified.jsonl (is_train=false, n=4,
throttled 50 in-flight, deduped by data_id). 99647 rollouts / 24836 instances:
all-0 13789 (55.5%), all-1 6575 (26.5%), mixed 4472 (18.0%).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* swe_smith: lower pod memory request to 2Gi, keep 6Gi limit
request=limit=6Gi over-reserved: the scheduler pinned ~83 pods against the
node's 510Gi allocatable while pods actually use ~1-2GB, leaving CPU at 16%
and 150Gi RAM idle. Drop request to 2Gi so more pods schedule; keep limit at
6Gi so long-trajectory pods (kernel-observed anon-rss ~4.1GB) don't OOM.
Co-Authored-By: Claude <noreply@anthropic.com>
* feat: route rollouts deterministically by rollout_id hash for prefix caching
Replace round-robin server selection with a stable sha256(rollout_id) mod
pool-size mapping so every request from a rollout lands on the same endpoint,
maximizing prefix-cache hits. Sort the pool by endpoint to keep the mapping
independent of registration order.
Co-Authored-By: Claude <noreply@anthropic.com>
* Log compact rollout trajectory artifacts
* swe_smith: lower pod memory request to 1Gi, keep 6Gi limit
Co-Authored-By: Claude <noreply@anthropic.com>
* swe_smith: Qwen3.5-9B run defaults + tighten token caps, drop internal plan.md
- run_qwen3_8b_async.sh: default model Qwen3-8B->Qwen3.5-9B, max_model_len
32768->65536, expose AGL_ROLLOUT_N, tie ppo_mini_batch_size to train_batch_size,
async overrides (val_before_train=False, test_freq=-1), run-name qwen35_9b_async.
- job-template-openai.yaml: AGL_MAX_TOKENS 16384->12288, add SMITH_OBS_CHAR_CAP=6000
to keep agent trajectories shorter under the 65536 context window.
- remove examples/swe_smith/plan.md (internal planning notes, not shipped code).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Update SWE-smith training settings
* swe_smith: final Qwen3.5-9B reward table + 5245 mixed instances
Collection complete over train_dataset_verified.jsonl (is_train=false, n=4,
throttled 50 in-flight, deduped by data_id via reward db). Final:
29976 instances / 120283 rollouts — all-0 17109 (57.1%), all-1 7622 (25.4%),
mixed 5245 (17.5%). 82.5% are degenerate (zero GRPO advantage); the 5245 mixed
instances are the trainable set.
- reward_table.db: final SQLite snapshot (30MB).
- mixed_instance_ids.txt: the 5245 mixed data_ids (instances with GRPO signal).
- enqueue_throttled.py: throttled enqueuer with --skip-from-db (dedup that
survives a server restart — needed after the in-memory store was wiped).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* swe_smith: prune unused scripts, add log/reward analysis tooling
- Remove no-longer-used enqueue/gen/monitor/serve scripts and stale docs.
- job-template-openai: point rollout-logs hostPath at /agl-logs (minikube 9p
mountpoint) so pod logs reach the host under the docker driver.
- Add filter.md (full->verified dataset filtering rationale), scan_abnormal_pods.sh
(k8s abnormal-pod probe), stat_pod_logs.sh (bulk pod-log anomaly stats), and
report_qwen35_9b.md (Qwen3.5-9B rollout report: 34% resolve, 83% ctx overflow).
Co-Authored-By: Claude <noreply@anthropic.com>
* swe_smith: remove unused pod monitoring/log scripts
Co-Authored-By: Claude <noreply@anthropic.com>
* Add SWE Verify checkpoint evaluation
* Add B200 Qwen3.5-9B vLLM pure-decode roofline benchmark
Measure single-card B200 decode performance vs theoretical roofline for
Qwen3.5-9B (hybrid GatedDeltaNet + full-attention). Includes:
- theoretical roofline (mem/compute roofs, ridge, TPOT floor)
- batch sweep + context sweep via latency-subtraction (pure decode isolation)
- reusable bench + plot scripts, CSV data, roofline.png, README + summary
Key: ~24k tok/s/card decode ceiling (21% of compute roof); batch-1 at 51%
of bandwidth roof (hybrid recurrent-state overhead); empirical ridge ~batch
32-64; context 128->64k drops 4x but <=4k near-lossless (hybrid KV advantage).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* roofline: document measured HBM BW / BF16 TFLOPS in README
Roof charts & theory keep datasheet spec (8.0 TB/s, 2250 TFLOPS) per roofline
convention. Add README section 1.4 with empirically-measured achievable ceilings
(HBM 7.11 TB/s = 89% of spec; BF16 dense 1623 TFLOPS = 72% of spec) plus two
reusable microbenchmark scripts, so real attainment can be cross-referenced
(batch-1 = 57% of measured BW roof; batch-256 = 29% of measured compute roof).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add per-step + per-minute rollout stats collector with long-tail metrics
Instrumentation for the async-GRPO SWE-smith run (Qwen3.5-9B, 4xB200) to
quantify per-vLLM-call long tails and their effect on rollout throughput.
- collect_rollout_stats.py: 60s collector writing two tables from vLLM
Prometheus histograms (e2e latency, response/prompt tokens).
- rollout_step_stats: per-call avg prompt/resp len, per-card out tokens +
throughput, max response, and per-call long-tail percentiles
(resp_tok/e2e_lat p50/p90/p99 + tail_ratio_lat_p99_p50).
- gen_window_throughput: per-~60s throughput time-series inside each gen
window, showing tail-segment collapse.
- rollout_stats.db: 18 step rows + 177 per-minute rows.
- README documents schema + findings: per-call e2e p99/p50 ~12-18x every
step (driven by ~21x response-token tail); within-window throughput
collapses 88-99.7% from peak; effective throughput ~62% of peak.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* swe_smith: reward==1 long-turn penalty + val temperature 0.7; prune scratch files
- smith_agent.py: add length_penalized_reward() that penalizes ONLY solved
(reward==1) trajectories: reward = 1 - λ·clip((n_turns-T0)/(max_turns-T0),0,1);
run_agent_loop now returns (submitted, turns_used). Tunable via env
SMITH_LEN_PEN_T0 (default 55) / SMITH_LEN_PEN_LAMBDA (default 0.2). Reports
raw_value + n_turns on the reward event for monitoring.
- job-template-openai.yaml: surface SMITH_LEN_PEN_T0=55 / SMITH_LEN_PEN_LAMBDA=0.2.
- tests: cover the penalty (solved-only, within-budget, ramp-to-cap, degenerate
span) and unpack the new run_agent_loop tuple.
- server.yaml: default_proxy.val.temperature 0 -> 0.7 (validation samples at 0.7).
- README: add `export FLA_TILELANG=0` to the run recipe.
- Remove scratch/dev files (enqueue_rollouts.py, rollout_stats.py, gen_*.py,
monitor_controller.sh, run_qwen3_8b_async.sh, serve_vllm_qwen3_8b.sh,
watch_failpods.sh, reward_table.db, reward_table_8b_baseline.md, rollout_test.md,
delete.jsonl).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* swe_smith: update mixed_instance_ids.txt to 5407 (add 162 rescued from 128k rerun)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* swe_smith: block network-based reward hacking (NetworkPolicy + agent guards)
Agents were downloading upstream correct source (curl raw.githubusercontent,
pip install <target pkg>, wget, urllib) to overwrite the buggy file — ~26% of
rollouts — after git was already blocked. Two layers now cut this off:
- Network root fix: deploy Calico VXLAN (replacing flannel, which cannot enforce
NetworkPolicy) + default-deny egress that only allows the agl-lite server IP,
so pods have no public internet. See canal.md for the full deploy/verify/
rollback runbook. Adds canal-policy-only.yaml, networkpolicy-egress-lockdown.yaml,
and an app=agl-rollout label on the job template pod.
- Code backstop: extend smith_agent.py _forbidden_action to reject curl/wget/
pip-install/urllib and test-harness file writes (conftest/pytest.ini/
sitecustomize), and align the prompt (drop "you may install it").
Also enables fused kernels in the train configs (unrelated perf tweak).
Co-Authored-By: Claude <noreply@anthropic.com>
* swe_smith: gate long-turn penalty to training only; README -> mixed dataset
- smith_agent.py: length_penalized_reward() now takes is_train and only shapes
reward when is_train AND solved. Validation reward stays the true, unshaped
metric (it drives checkpoint selection). main() derives is_train from
AGL_OPENAI_BASE_URL (/mode/train/ marker; fail-safe: never penalize val) and
logs mode. Fixes the earlier bug where the penalty applied to val too.
- tests: pass is_train explicitly; add test_length_penalty_skips_validation
asserting val reward is never reshaped (even a long solved rollout keeps 1.0).
- README: point the active Qwen3.5-9B recipe at train_dataset_mixed.jsonl (6343
instances = mixed_dataset minus the 3 network-dependent repos dspy/pydantic/
MONAI whose eval needs runtime pip install and breaks once pods are offline).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* swe_smith: add prompt-length penalty (plan B) + align train config to bh103or9
Penalize context bloat on SOLVED training rollouts: track the largest
server-reported usage.prompt_tokens per rollout (no local tokenizer) and
subtract up to 0.1 as the longest prompt grows from 50K to 64K tokens.
Gated on raw solved status so it stacks with the long-turn penalty (plan A)
and never reshapes validation reward. Tunable via SMITH_PROMPT_PEN_*.
Align train config to the bh103or9 run: model Qwen3.5-9B, gpu_mem_util 0.8,
max_num_batched_tokens 8192, ppo_max_token_len_per_gpu 16384.
Co-Authored-By: Claude <noreply@anthropic.com>
* add max_ppo_update_times
* Merge rollout-level advantage computation (cherry-pick ee78dbe)
Add optional rollout-level advantage (algorithm.enable_rollout_level_advantage)
and its module + tests, wired into AglLiteRayPPOTrainer alongside the existing
compute_advantage path. Cherry-picked from SiweiPro's ee78dbe to take only the
advantage change without the unrelated swe_verify work on that branch.
Co-Authored-By: SiweiPro <18474108006@163.com>
Co-Authored-By: Claude <noreply@anthropic.com>
* swe_smith: turn penalty T0=80, enable max_ppo_update_times=2 + bypass_mode
- Long-turn penalty now starts at 80 turns (was 55), max penalty 0.2 at the
100-turn cap.
- Set agentlightning.max_ppo_update_times=2 to bound PPO updates per step.
- Enable rollout_correction.bypass_mode: reuse rollout log-probs as old_log_prob
for importance sampling, skipping the old-logprob forward pass.
Co-Authored-By: Claude <noreply@anthropic.com>
* swe_smith: untrack examples/swe_smith/infra (local-only tooling)
Roofline benchmarks + rollout-stats tooling are local infra, not part of the
training example. Remove from the tree and exclude locally so they stay off origin.
Co-Authored-By: Claude <noreply@anthropic.com>
* swe_smith: untrack reward_table.db (large local-only artifact)
30MB reward cache is a local build artifact, not source. Remove from the tree
and exclude locally so it stays off origin.
Co-Authored-By: Claude <noreply@anthropic.com>
* swe_smith / swe_verify: untrack local-only files (docs, dataset curation, network policies, comparison scripts, swe_verify example)
Move these to .git/info/exclude so they stay off origin while remaining local:
- examples/swe_verify/ (7 files, plus tests/examples/test_swe_verify.py to avoid CI ImportError)
- examples/swe_smith/README.md
- examples/swe_smith/canal.md, filter.md, report_qwen35_9b.md (internal notes)
- examples/swe_smith/canal-policy-only.yaml, networkpolicy-egress-lockdown.yaml
(cluster-specific network policies)
- examples/swe_smith/mixed_instance_ids.txt (local dataset curation, 5407 rows)
- examples/calc_x/comparison/ (local benchmark scripts)
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4 (1M context) <noreply@anthropic.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: hzy46 <362583303@qq.com>
Co-authored-by: SiweiPro <18474108006@163.com>
* feat(is): add bypass-mode rollout importance-sampling support
Capture chosen-token rollout logprobs end to end and feed them into VERL
bypass mode so old_log_probs = rollout_log_probs, avoiding actor recompute.
- proxy: inject logprobs=True on train-mode requests
- events: extract finite chosen-token logprobs (chat + completions schemas);
never fail the triplet query on bad logprobs, report status/error instead
- rollout_bridge: thread response_log_probs through triplets/trace/trajectory,
pad/truncate in lockstep with response_mask, fill masked/dropped positions
with finite 0.0, drop rows with missing/invalid/length-mismatched logprobs
(with metrics), emit float32 rollout_log_probs only when present
- trainer: import apply_bypass_mode; in _async_train_step, when bypass_mode is
enabled, require finite rollout_log_probs and skip _compute_old_log_prob
* feat(science_world): enable bypass-mode rollout correction (IS)
Add algorithm.rollout_correction with bypass_mode + ppo_clip so the
science_world example trains using rollout logprobs (RS/IS off for first pass).
When test_freq fires inside async_fit(), validation runs through the legacy
_rollout(is_train=False) path which ends with abort_all_requests(). Without
a subsequent update_weights() call, vLLM 0.7.1+ engines can stay paused and
the next step's resume_generation() does not recover them. Observed as the
step-32→33 deadlock: all rollout replicas hang at 20:42:17Z, training enters
a 16-hour zombie state with succeeded=82, timeout_rollouts growing by 256
per step, reward=0, grad_norm=0, and global_seqlen=32 (placeholder-only).
train→train avoids this because _async_train_step always ends with
sleep_replicas() + update_weights(global_steps), which forces a vLLM
engine state refresh while vLLM is offloaded. val→train must do the same
explicitly. Snapshot/restore of the async carry-over bridge state
(_validate_preserving_async_carry_over) was already in place; the missing
piece was the engine reset itself.
The reset must call sleep_replicas() BEFORE update_weights(): the reset
runs while vLLM is still awake holding weights + KV cache on GPU, and
FSDP's state_dict() unshard collides with vLLM's resident memory during
DTensor redistribute() — observed OOM at step 32→33 on 40 GB A100s with
GPU 0 at 39.49 GB used / 29.88 MiB free, vLLM TP worker holding 26.94 GiB,
FSDP unshard needing ~12.48 GiB. Mirror the train→train cycle pattern
(sleep_replicas() then update_weights()) so vLLM frees its weights + KV
cache before FSDP gathers params for IPC.
Tests:
- positive: drive async_fit through one train step plus a test_freq-
triggered validation; assert sleep_replicas runs between val_rollout
and the engine-reset update_weights(global_steps) call.
- negative: assert no spurious engine reset (neither sleep_replicas nor
extra update_weights) fires on non-val steps — a stray reset would
leave vLLM paused without a wake_up partner.
Verified the positive test fails on the pre-fix code with the intended
assertion message.
Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- tune .env.example, train_sw_agent.py, run.sh for 8x A100
- split train/val temperature in sw_agent.py
- expand README with the new sizing/temperature guidance
- LocalReconciler: inject AGL_IS_TRAIN=1|0 into rollout subprocess env so
the agent can branch behavior (e.g. sampling temperature) on train vs val
Yuge Zhang