A-tier (contradicted the code): - RUNTIME_API.md: document the undo/patch-undo/retry/snapshot-restore endpoints that shipped but were marked deferred; drop product-name brand residue (DeepSeek engine/client/tools -> Codewhale) - AGENT_RUNTIME.md: rebrand shims removed in v0.9.0, file decomposition landed (ui.rs ~3.9k, main.rs stub), verifier contract path, v0.9.1 release precondition met (current line v0.9.7) - public-surface-facts.json + web test: defaultActive = the six DEFAULT_ACTIVE_NATIVE_TOOLS names (todo_write, not tool_search) - MODES.md: toolbox list matches the seven model-facing names; retired /set removed from active slash surfaces (legacy note kept) - KEYBINDINGS.md: Alt-P/A/Y = Plan/Work/Yolo, not Plan/Work/Operate B-tier (clearly stale): - ACCESSIBILITY.md: brand legacy first line; /settings set examples -> /config <key> <value> --save; section title - SKILLS.md: MAX_AVAILABLE_SKILLS_CHARS 12 000 -> 2 400 - FLEET.md / WORKFLOW_AUTHORING.md: Fleet rings 5/2 -> 8/3 - TERMUX.md: self-update requests only the codewhale-android-arm64 asset - OPERATIONS_RUNBOOK.md: RUST_LOG target deepseek_cli -> codewhale_tui - CATALOG_REFRESH.md: broken MODEL_PROVIDER_AUDIT.md -> PROVIDERS.md - CONFIGURATION.md: provider list 28 -> 40; /note primary path - MCP.md: "Running DeepSeek as an MCP Server" brand -> Codewhale; /mcp subcommand list (import/recommendations/add recommended) - web tools page: todo_write out of the Deferred group into its own schema row ARCHITECTURE.md updates (LLM client layer, sandbox backends, github.rs removal) that landed in the working tree are included. Verified: facts JSON valid, no broken .md links in docs, every claim cross-checked against code; web tests not run (node_modules absent).
37 KiB
Agent Fleet
Agent Fleet is the local-first control plane for durable multi-worker runs. It
is not a separate execution engine: a fleet worker is a headless
codewhale exec run that the fleet launches and tracks durably. See
AGENT_RUNTIME.md for how sub-agents, exec, and the fleet
converge on one durable runtime. In product language, a user may still "open a
sub-agent"; in architecture language, durable nested work should be a
fleet-backed worker with a role.
Use Fleet rather than short-lived agent fanout whenever the work
needs retry, sleep/restart survival, remote execution, receipts, or a ledgered
audit trail. The initial CLI surface is:
For a guided start-to-monitor walkthrough that combines Fleet task specs with Workflow authoring, see Fleet + Workflow Tutorial.
codewhale fleet init
codewhale fleet run tasks.json --max-workers 4
codewhale fleet status
codewhale fleet inspect <worker-id>
codewhale fleet logs <worker-id>
codewhale fleet artifacts <worker-id>
codewhale fleet interrupt <worker-id>
codewhale fleet restart <worker-id>
codewhale fleet resume <run-id>
codewhale fleet stop --all
codewhale fleet resume <run-id> is the restart-recovery verb: it replays the
ledger, reconciles any in-flight lease whose worker stopped heartbeating
(retrying within the task's budget, else failing and escalating per the alert
policy), and prints the post-resume status. It launches no new work and is
idempotent, so it is safe to run after a manager exit, laptop sleep, or runtime
restart.
Fleet state is stored under the workspace in .codewhale/fleet.jsonl. Worker
logs and adapter logs are stored under .codewhale/fleet/ and
.codewhale/fleet-host/.
Interactive and persistent status
/fleet status and codewhale fleet status are the same command on two
surfaces. Both read the durable .codewhale/fleet.jsonl ledger for the
workspace, through one shared control-plane contract, and both report the same
verb id (fleet.status), read-vs-write authority, persistence scope, and
receipt. When the workspace has no ledger they say so with a typed reason
(no_fleet_ledger) instead of rendering an empty-looking "all clear" — and
neither creates the ledger as a side effect of reading it.
The current interactive session's sub-agents are a different set, and now have their own name:
/fleet workers(or/subagents, orn) shows sub-agents attached to the current TUI session. It does not read the persistent ledger./fleet list|status|interrupt|resumeandcodewhale fleet list|status|interrupt|resumeact on the durable ledger.codewhale fleet restart <worker-id>is CLI-only: it re-leases the task and then drives the manager loop to completion./fleet restartdoes not silently do a smaller thing — it reportssurface_not_supportedand names the CLI command.
Before v0.9.2, /fleet status showed session sub-agents. That reading is gone;
/fleet workers replaces it.
The contract behind this — descriptors, availability reasons, exact-identity
targets, receipts, typed unknowns, and bounds — is documented in
docs/COMMAND_CONTROL_PLANE.md.
Authoring agent profiles (/fleet setup)
/fleet setup (also /fleet setup edit / new) opens an in-TUI wizard for
authoring a reusable agent-team profile. Bare /fleet and the
roster/roles/profiles/party aliases open the roster (the saved profiles).
/fleet workers opens the current-session worker view; /subagents is a
compatibility shortcut for that view. For durable run history, use
/fleet status or the shell command codewhale fleet status described above —
they are the same command.
The wizard is progressive: you make one focused choice at a time — a role,
then a model (inherit, or a concrete model from any configured
provider, not only the one the parent session is currently using), then
where the profile lives, and finally a review of the full posture
(route, thinking, permissions, tools, and review policy). The header shows
"Saves to: …" on every step — the choice you still have to make, or the exact
resolved file once you have made it. Nothing is written until you activate the
save control on the review step.
The Destination step is a focused two-option list (arrows move, Enter or Space chooses; Tab never changes the destination):
- This project writes
<workspace>/.codewhale/agents/<role>.toml. It applies to this project only and takes precedence over a Personal profile with the same id. When project profiles are disabled for the session (--no-project-config) or the workspace folder is unavailable, the option is shown disabled with that reason; the wizard never falls back to Personal on its own. - Personal writes
$CODEWHALE_HOME/agents/<role>.tomland is available in every project on this machine, except where a project has its own profile with the same id.
For the highlighted option the step shows the exact file, whether saving would
create a new file or replace an existing one, and the precedence
consequence for the roster. The review step repeats those facts under
"Saves to" and names the final action by its effect — Save to this
project, Save as Personal profile, or Replace …. Replacing an
existing file needs a second Enter on the save control. Tab / Shift+Tab (or
←/→) move focus between the save control, Change destination, and
Back; s is a secondary shortcut back to the Destination step. Reopening a
saved member from /fleet starts from what is on disk: its route, thinking
tier, and the scope it was saved in. Thinking (inherit, off, low,
medium, high, max, or auto) is adjusted on the review step with t.
Profile scope controls where a role definition is reusable; it does not widen the authority of a running operation. To coordinate several nearby repositories, start Codewhale from their shared parent directory so that parent is the workspace. Explicit trusted external paths or Full Access can still change what tools may reach; workers inherit the active trust and permission posture, never the profile's storage scope.
Picking a concrete model pins its provider explicitly: the saved profile records both
model and provider fields, so the route it names doesn't depend on
whichever provider happens to be active when the profile is later loaded.
Pressing Enter ("start") on the review step previews the exact starter
profile TOML inline on that same screen; nothing is written until you save it.
The provider field may be a built-in provider id such as openrouter or a
user-named OpenAI-compatible provider configured under [providers.<name>]
such as lm-studio; the launch path preserves that id and fails closed if the
provider is not configured.
Profiles are also how the model-facing agent tool selects a route since the
v0.9.9 schema slim (#5324, #5123): the advertised surface no longer carries
model or thinking — a child either runs as a profile (whose saved route
and thinking tier it uses exactly) or inherits the operator's model. Removed
fields stay parse-accepted for saved transcripts, ACP/MCP clients and Fleet
configs; see docs/SUBAGENTS.md for the advertised 12-field list and the
compat list.
When a provider is configured, the review step also offers model-assisted drafting behind an explicit preview-before-save gate:
- Press
mto have your first configured model draft the profile. The draft arrives sanitized and bounded — permissions stay at the fleet floor (no shell, no trust, approval required) regardless of what the model proposes. - Drafting is not saving. The exact rendered TOML preview renders
inline on the review step (not in a separate scrollable viewer), so nothing
is saved until you press
gor Enter to save (or pressmagain to redraft). Saving writes the profile to the project or personal scope shown in the preview.
Naming: Modes, Workflow, and Fleet
These names describe different layers, not competing systems. Plan and Act are the everyday work modes. Operate accepts ordinary messages and keeps the parent's normal tool surface under the same approval, sandbox, shell, ask-rule, and repository protections as Act. It prefers background Fleet workers for independent, parallel, isolated, or long-running work, but does not require a worker for every executable step. Workflow is an optional orchestration overlay for work that needs ordering, gates, shared budgets, replay, or deterministic fan-in.
The short public vocabulary is:
-
Fleet = who does the work: the configured workers, roles, models, hosts, and trust boundaries.
-
Workflow = what order the work follows: phases, gates, budgets, replay, and fan-in.
-
Lane = one running Workflow instance and its live progress.
-
Runtime = where and how a Lane executes: local or remote process, provider route, sandbox, and API boundary.
-
Workflow is the repeatable plan and user-facing orchestration overlay: a script/IR that decides which phases and agents run next, keeps intermediate results out of the main conversation, and can be inspected or rerun. A Workflow run should have a visible progress view and a clear active header state instead of feeling like a hidden background task.
-
Fleet is the durable sub-agent configuration and execution substrate: slots, profiles, per-slot models, tool posture, local/SSH hosts, trust policy, leases, heartbeats, logs, receipts, and status APIs.
-
High fan-out is a behavior of a Workflow run, not a separate system: when a phase needs many workers at once, Workflow dispatches them as a Fleet-backed run (durable workers, receipts, goal re-dispatch) rather than reviving prompt-only sub-agent fanout.
-
Fan-in is explicit: when the user needs one combined result, an owner aggregates, verifies, and synthesizes the worker receipts. Independent tasks may finish separately; dispatch is never presented as completion.
UI guidance: keep the main transcript calm. A Workflow run should appear as a compact progress card plus work-bar rows (the strip above the transcript, or a side rail) with phase names, worker counts, receipts, and nested indentation for child workers. Use the whale mark sparingly as an active header/status signal; avoid repeating emoji-heavy rows for every worker.
Exact Fleets and the Reasoning Router
An exact Fleet freezes the provider, model, reasoning policy, and permission
ceiling of every worker before a Workflow starts. Save it as
fleets/<name>.toml in the workspace or under $CODEWHALE_HOME. Models cannot
replace those assignments at runtime:
name = "release"
schema = "exact"
schema_revision = 1
reasoning_router = "luna-low"
[[members]]
id = "implementer"
role = "builder"
provider = "zai"
model = "glm-5.2"
reasoning = "auto"
permissions = "read_write"
[[members]]
id = "advice"
role = "consultant"
provider = "openai"
model = "gpt-5.6"
reasoning = "high"
permissions = "read_only"
The optional Reasoning Router is a reusable service, not a Fleet member. Save
one profile at routers/<name>.toml in either search root and reference it from
any number of Fleets:
name = "luna-low"
schema = "reasoning_router"
schema_revision = 1
provider = "openai"
model = "gpt-5.6-luna"
call_reasoning = "low"
At runtime it may choose only the reasoning tier for an already-frozen worker
route. It cannot change the worker, provider, model, role, tools, or permissions.
The Router call itself is capped at off or low; more expensive values are
rejected. A manually selected worker reasoning tier makes no Router call. Route
and reasoning receipts name the worker model and, when used, the Router's exact
provider/model so the operator can see which model did which job. If the same
bare Router or Fleet name exists in both roots, qualify it as
workspace/<name> or codewhale_home/<name> instead of relying on shadowing.
Each member's permissions preset is a ceiling, never a grant: it is
intersected with the live session posture, so a read_write member inside a
read-only session runs read-only. The intersection becomes the child's real tool
envelope — permissions = "none" leaves it with no tools at all, and a member
without a network tool loses every reaching network surface — web.run,
fetch_url, web_search, github, the mcp* families, and the reaching
rlm actions — rather than merely being refused at call time. The one
deliberate exception is the canonical Web family's search/fetch actions:
they are the read-only web surface a read-only member is entitled to
(parity with an ordinary scout), so a network-denied member keeps exactly
Web {search, fetch}. It stays read-only in fact, not just in name: a
URL-addressed fetch is refused at dispatch, and nothing else on the surface
is granted. A member that cannot write loses the mutating file tools, and —
when it kept shell = "full" so it can run checks — the raw shell as well,
keeping the bounded verification tools
(run_tests, run_verifiers) it exists for: an arbitrary shell command mutates
a workspace just as surely as write_file, so leaving it would make
write = false untrue. That verification surface is bounded only in its
default form, so the unbounded arguments go with the shell: a write-denied
member may run the built-in gates, but not run_verifiers with an explicit
commands array or run_tests with a raw args string, both of which spawn
operator-supplied programs and are the raw shell by another name.
Two of these denials are narrower than a tool name, because two tools reach past
their name. rlm loads a url by calling the fetch tool inside the process,
under its own name, and rlm's eval action runs Python against a live kernel
— sockets and filesystem both. So the family is denied per action, not
wholesale:
- No network tool removes
rlm_openandrlm_eval, by the legacy alias and by therlm {action: ...}spelling alike. This is deliberately narrower than the capability it protects:rlm_openpicks its source from input fields (file_path,content,url,session_object), not from the action name, and the action-policy seam resolves names rather than field shapes — it cannot prove a source is local before the tool runs. Rather than leave a URL-shaped hole, a network-denied member loses RLM loading entirely, including the purely localfile_pathform, and keeps only the bounded metadata actions (session_objects,configure,close). - No write removes
rlm_evalonly. Loading a large local file into a kernel and reading it is analysis, not mutation, so the rest of the family stays.
Beyond those names, a network-denied member is refused at call time whenever any
tool is handed a URL-bearing field (url, urls, endpoint, target, …)
holding an http/https/ws/wss/ftp address. A URL appearing in file
content or a search pattern is data, not a destination, and is untouched.
A member's role picks the worker
posture (and system prompt) when that role already fits inside the ceiling —
reviewer, verifier, consultant, planner — and a domain-specific role
such as auditor falls back to the narrowest posture the ceiling allows.
Built-in role defaults withhold only what the role intends: read-only roles
never write the workspace, every role keeps network reads, planner may run
read-only shell probes, and builder/worker/custom inherit the parent's
effective write/network/shell posture as their ceiling (see the role table in
docs/SUBAGENTS.md). The session's permission posture — Ask, Auto-Review,
Full Access — then gates each worker call exactly as it gates the parent
(docs/MODES.md, "Children"). Tasks
cannot override any of it: model, thinking,
subagent_type, allowed_tools, and write_authority are rejected on an exact
Fleet rather than silently ignored.
Reasoning receipts record the requested tier and the tier the provider was
actually asked for. Those differ whenever a route cannot express the requested
one — CodeWhale's route normalizer sends high for a requested low on most
routes, and Z.AI's GLM routes express only thinking on/off — so the receipt
reports the real request rather than the label that was selected. The value a
call actually carries is spelled by that route's own normalizer, not by the tier
label: an OpenAI Codex route is asked for xhigh, not max, and cannot be
asked for off at all.
A receipt also keeps a member's semantic role and its permission posture
apart. member_role is what the operator named and what gates key on;
posture_role (present only when it differs) is the built-in tool surface the
clamped ceiling permits — so a member named auditor running under the scout
posture is displayed as auditor and enforced as scout, and neither fact is
substituted for the other.
A Workflow start fails closed on anything decidable locally: an unresolvable
provider or model, a missing credential, a client that cannot be built for a
member's route, or an auto member with no usable Reasoning Router. Per-task
validation that the spawn boundary would refuse anyway — notably a write-capable
member with no declared write_roots/exact_files/coordination_contracts —
is checked before the Router is called, so an invalid task never spends a
routing request. If a spawn fails after a Router decision, the receipt is
still recorded: the tokens were spent, and any cross-provider disclosure already
happened.
Manager-owned Workflow fan-in
When parallel work must return one combined answer, use a manager-owned
Workflow instead of a flat agent fan-out:
- Cast one manager (operator or workflow orchestrator).
- Fan out child tasks through
workflow(task(),parallel(),pipeline(),phase()) or a single manager session that owns the children. - Wait for child receipts or completion events.
- Aggregate and verify load-bearing claims before treating them as facts.
- Synthesize one result the operator can depend on.
Raw agent fan-out is appropriate only for independent, fire-and-forget work
where no single fan-in result is required. If results must be merged, compared,
or verified, route through workflow so the manager owns fan-in.
Workflow on Fleet
The intended high-capability path is agent-authored. When the main agent decides a task needs more durable coordination than turn-by-turn sub-agent calls, it drafts a Workflow script/IR, presents the run plan according to the active permission mode, and the runtime compiles it into typed Fleet work.
Fleet remains the sub-agent config surface. It owns slot count, role profiles,
saved route pins or inheritance, tool posture, launch concurrency, and the ledger.
Workflow owns only the orchestration plan: branch, sequence, loop, expand,
review, and reduce decisions. The workflow script must not get direct shell,
filesystem, network, provider-secret, cancellation, or TUI authority; workers
perform real work as codewhale exec processes.
Default Workflow-to-Fleet validation is intentionally bounded:
- 1,000 total worker agents per Workflow run;
- 16 live worker agents at once; larger populations queue (block) on the host's per-run concurrency gate until a live slot frees, then route through Fleet;
- 8 recursive Fleet rings as the opt-in ceiling (default user configuration: 3);
- bounded loops only (
max_iterationsrequired); - bounded dynamic expansion only (
max_childrenplus a template required).
These are population limits, not a demand to launch everything at once. A
1,000-agent Workflow should still drain through the configured Fleet worker
pool.
Recommended model layouts, such as a DeepSeek Pro orchestrator with Flash
workers in the first ring and cheaper workers farther out, are presets only.
Every slot can inherit the active model or carry an explicit model override.
Inheritance is literal: the model you select in /model is the operator
(the pinned first row in /fleet roster), and any worker whose task spec and
roster profile pin no model runs on that session model. Task-level model and
profile model overrides still win; route receipts record which source
applied (task.model, agent_profile.model, or run.model).
The setup UI should render this as an expanding grid: an orchestrator plus a small number of visible sub-agent slots, with Right/Enter drilling into a slot's next recursive ring rather than trying to show the whole tree at once.
Task Spec
codewhale fleet run accepts JSON or TOML. A minimal JSON spec:
{
"name": "local smoke",
"tasks": [
{
"id": "lint",
"name": "Lint",
"instructions": "Run the lint check and report failures.",
"expected_artifacts": ["log"]
}
]
}
Workers are optional. If omitted, Codewhale creates local worker slots up to
--max-workers.
Task specs are typed in Rust and keep verification data separate from worker transcripts. A task can declare:
id,name,description,objective, andinstructionsworkerrole, tool profile, tools, and required capabilitiesworkspaceroot, required files, writable paths, and environment allowlistinput_files, extracontext,budget,timeout_seconds, andretry_policyexpected_artifacts,scorer,tags, and free-formmetadata
Workers write bounded artifact files under .codewhale/fleet/ and ledger only
the artifact refs: kind, path, checksum, MIME type, and size. Receipts record
pass, fail, partial, skip, or timeout; failed receipts may also mark
the source as transport, task, or verifier. codewhale fleet status
surfaces those failure-source counts separately.
Deterministic built-in scorers are exit_code, file_exists, regex_match,
and json_path. Specs may also declare command,
code_whale_verifier_prompt, or manual; those record a partial receipt until
an explicit verifier pass completes.
Using Role Presets
Tasks can reference a role name, and the fleet manager fills in defaults
from the role registry. Built-in roles (smoke-runner, reviewer, builder,
read-only) are always available; define your own in [fleet.roles].
{
"name": "smoke check",
"tasks": [
{
"id": "lint",
"name": "Lint check",
"instructions": "Run lint and report failures.",
"worker": { "role": "smoke-runner" },
"expected_artifacts": ["log"]
}
]
}
The task inherits the role's tool profile, budget, and timeout. You can override any field in the task spec:
{
"id": "deep-review",
"name": "Deep review",
"instructions": "Review the entire crate for soundness issues.",
"worker": {
"role": "reviewer",
"tools": ["cargo", "rg", "git"],
"capabilities": ["rust"]
},
"input_files": ["crates/**/*.rs"],
"budget": { "max_tokens": 32000 },
"expected_artifacts": ["log", "report"],
"scorer": { "kind": "regex_match", "path": ".codewhale/fleet/report.md", "pattern": "finding|all clear" }
}
Multi-Task Run Example
A single fleet run can dispatch several independent tasks in parallel:
{
"name": "CI gate",
"tasks": [
{
"id": "check",
"name": "Compile check",
"instructions": "Run cargo check --workspace and report errors.",
"worker": { "role": "builder" },
"expected_artifacts": ["log"],
"scorer": { "kind": "exit_code" }
},
{
"id": "clippy",
"name": "Clippy lint",
"instructions": "Run cargo clippy --workspace and report warnings.",
"worker": { "role": "reviewer", "tools": ["cargo", "cargo-clippy"] },
"expected_artifacts": ["log"],
"scorer": { "kind": "exit_code" }
},
{
"id": "security",
"name": "Secret audit",
"instructions": "Search for plaintext secrets and report any matches.",
"worker": { "role": "read-only", "tools": ["rg"] },
"input_files": ["crates/**/*.rs"],
"expected_artifacts": ["log", "report"],
"retry_policy": { "max_attempts": 1 }
}
]
}
Alerts
Fleet alerting is disabled by default. A caller must supply an enabled alert config before anything is sent. Routes match typed fleet event classes, not log strings:
stalerestart_exhaustedneeds_humanbudget_exceededverifier_failedrun_completed
Adapter config stores environment variable names, not secret values. Send-time
code resolves those names from the environment or a future secrets provider.
Ledger records store only audit labels such as slack, webhook, or
pagerduty; task specs persisted in the ledger redact webhook URLs and routing
keys.
Example alert config shape:
{
"enabled": true,
"dry_run": true,
"routes": [
{
"events": ["stale", "restart_exhausted", "verifier_failed"],
"adapter": "ops-slack"
},
{
"events": ["restart_exhausted"],
"adapter": "pager"
}
],
"adapters": {
"ops-slack": {
"kind": "slack",
"webhook_env": "CODEWHALE_FLEET_SLACK_WEBHOOK",
"channel": "#codewhale-fleet"
},
"pager": {
"kind": "pager_duty",
"routing_key_env": "CODEWHALE_FLEET_PAGERDUTY_ROUTING_KEY",
"severity": "critical"
}
}
}
Use dry-run to inspect a redacted adapter payload without sending:
codewhale fleet alert-dry-run \
--event stale \
--run-id fleet-demo \
--worker-id fleet-demo-local-1 \
--task-id release-triage \
--reason "worker heartbeat stale since 2026-06-13T02:00:00Z" \
--adapter slack
The payload includes the run id, worker id, task id, status, short reason, and
safe inspection commands such as codewhale fleet status and
codewhale fleet inspect <worker-id>. Endpoints, webhook secrets, and
PagerDuty routing keys are shown as <redacted:env:...>.
Status Surfaces
codewhale fleet status shows compact counts for queued, running, completed,
partial, failed, restarted, escalated, cancelled, stale, and verifier/transport
failure sources. inspect shows the worker state plus the current task
objective, role, host, heartbeat, latest event, artifact refs, latest error, and
alert state. logs prints bounded log artifact contents, and artifacts lists
artifact refs without embedding large payloads.
The Runtime API exposes the same ledger-backed projection behind the existing runtime auth middleware:
GET /v1/fleet/runs
GET /v1/fleet/runs/{run_id}
GET /v1/fleet/runs/{run_id}/workers
GET /v1/fleet/workers/{worker_id}
POST /v1/fleet/workers/{worker_id}/interrupt
POST /v1/fleet/workers/{worker_id}/restart
POST /v1/fleet/runs/{run_id}/stop
Action endpoints call the same manager controls as the CLI and record their decisions in the fleet ledger.
Manager-Agent Runbook
Manager agents should treat Fleet operations as typed, ledgered control-plane
work. Start with codewhale fleet status, then inspect one run or worker with
codewhale fleet inspect <worker-id>, logs, and artifacts. Use direct
reads of .codewhale/fleet.jsonl, host logs, or remote files only when the
typed CLI/API surface cannot provide the required evidence.
Classify the worker before taking action:
transient failure: stale heartbeat, host timeout, interrupted transport, retryable provider/network error, or an adapter status that can plausibly recover without changing the task.task failure: the worker completed but produced an incorrect result, domain failure, missing required artifact, or explicit task-level error.verifier failure: the worker result exists, but the scorer/verifier failed, timed out, or disagrees with the receipt.needs-human: missing authority, secret request, destructive operation, repeated restart exhaustion, ambiguous product decision, or conflicting evidence that the manager cannot resolve from typed artifacts.
Choose one typed action:
- Restart a worker only when the failure is transient, retry budget remains,
the task is idempotent or retry-safe, and no permission or secret boundary is
involved:
codewhale fleet restart <worker-id>. - Interrupt or stop only when the current task is unsafe to continue or the
operator explicitly asks for cancellation:
codewhale fleet interrupt <worker-id>orcodewhale fleet stop --all. - Do not restart pure task failures by default; preserve artifacts and hand the receipt to the task owner unless the task spec says retrying can produce new evidence.
- For verifier failures, inspect scorer inputs and artifact refs first. If the verifier cannot be corrected through typed fleet actions, escalate for human review.
- For
needs-human, draft an escalation instead of sending it unless alert config explicitly authorizes sending.
Safe Slack or PagerDuty draft:
Codewhale fleet needs attention
Run: <run-id>
Worker: <worker-id>
Task: <task-id or unknown>
Classification: <transient failure | task failure | verifier failure | needs-human>
Reason: <one sentence, no secrets>
Latest typed evidence: codewhale fleet inspect <worker-id>; codewhale fleet artifacts <worker-id>
Safe log excerpt: <3 lines max or "see artifact <ref>">
Requested decision: <restart approval | verifier review | task owner review | permission decision>
Post-run summaries should include the run id, workers checked, classification, typed action taken or drafted, expected ledger effect, artifact refs reviewed, and next owner. Keep summaries bounded; link artifact refs instead of copying full logs or transcripts.
The bundled fleet-manager skill mirrors this runbook for manager agents. It
is a first-party system skill and should be discoverable through the normal
skill registry after system skills are installed or refreshed.
Host Adapters
The host adapter boundary supports local child processes and explicit SSH workers. Adapters expose the same operations: start, read status, read bounded logs, interrupt, restart, stop, and cleanup.
Local workers run as child processes with stdin closed and stdout/stderr written
to bounded fleet host logs. They inherit only a small safe base environment
such as PATH and explicitly allowlisted variables.
SSH workers run through the system ssh client with BatchMode=yes and a
bounded connect timeout. Remote environment variables are sent with OpenSSH
SendEnv; values are not embedded in the local ssh argv or fleet logs.
Example SSH worker spec:
{
"id": "builder-1",
"name": "Builder 1",
"host": {
"kind": "ssh",
"host": "builder.example.com",
"user": "codewhale",
"port": 22,
"identity": "~/.ssh/codewhale_fleet",
"working_directory": "/srv/codewhale/work",
"env_allowlist": ["CODEWHALE_PROFILE"],
"codewhale_binary": "/usr/local/bin/codewhale"
},
"capabilities": ["local", "linux", "tests"],
"max_concurrent_tasks": 1
}
Defaults are intentionally conservative:
- no hosted control plane or cloud provisioning is enabled;
- SSH requires an explicit host, working directory, and Codewhale binary path;
- secret-like environment names such as
TOKEN,SECRET,PASSWORD,API_KEY, andPRIVATE_KEYare rejected from adapter allowlists; - secrets should remain in Codewhale config providers or remote host config, not in task instructions, argv, or fleet logs.
Security and Trust Boundaries
Agent Fleet enforces a trust-level model that separates workers into four tiers. The trust level determines what a worker can access (secrets, network, workspace writes) and how it must prove its identity before being granted those privileges.
Trust Levels
| Level | Access | Requires |
|---|---|---|
sandbox |
No network, no secrets, writes only to .codewhale/fleet/ |
Nothing — default for new workers |
local |
Workspace reads, gated writes, configured secrets | Local process (same uid) |
remote-verified |
Network access, bounded capability grants, configured secrets | SSH host-key verification or equivalent attestation |
operator |
Full access to all secrets, unrestricted writes, any action | Operator-owned machine |
The default trust level is sandbox. Operators must explicitly raise trust for
SSH or container workers through the security policy.
Security Policy
A fleet run may carry an optional security_policy block that defines the
default trust level, which secrets workers may resolve, what capabilities are
granted, and a ceiling on the maximum trust level:
{
"security_policy": {
"default_trust_level": "sandbox",
"allowed_secrets": [
{"key": "GH_TOKEN", "source": "env"},
{"key": "CODEWHALE_API_KEY", "source": "keyring"}
],
"capability_grants": [
{
"capability": "network",
"scope": "github.com",
"reason": "PR review needs GitHub API access"
}
],
"max_trust_level": "remote_verified",
"require_identity_verification": true
}
}
When a run has no explicit security_policy, workers inherit conservative
defaults: sandbox trust, no secrets, no capability grants, and no identity
verification requirement.
Secret References
Secrets are never stored as plaintext in task specs, alert configs, or worker
definitions. Instead, every secret is a FleetSecretRef — a key name plus an
optional source hint that tells the fleet manager where to resolve the value:
{"key": "GH_TOKEN", "source": "env"}
Supported sources:
"env"— resolve from a process environment variable"keyring"— resolve from the OS keyring (macOS Keychain, Windows Credential Manager, Linux Secret Service)"file"— resolve from~/.codewhale/secrets/- absent — try all sources in default order (store first, then env)
Secret refs are redacted in logs and ledger entries: <secret:env.GH_TOKEN>.
Worker Authentication
Workers authenticate to the fleet manager using one of three methods:
- None — local workers sharing the same uid (default)
- SSH key — with optional host-key fingerprint pinning and known-hosts
verification. The
host_key_fingerprintfield (SHA256:...) pins the expected server key, preventing MITM attacks on first connection. - Token — a bearer token resolved from a
FleetSecretRef, useful for remote workers behind a fleet proxy. - mTLS — mutual TLS with a client certificate and a secret-backed private key.
SSH workers should always set host_key_fingerprint in production:
{
"id": "builder-1",
"name": "Builder 1",
"trust_level": "remote_verified",
"host": {
"kind": "ssh",
"host": "builder.example.com",
"user": "codewhale",
"port": 22,
"identity": "~/.ssh/codewhale_fleet",
"host_key_fingerprint": "SHA256:aLGqZo1M6c...",
"known_hosts": "~/.ssh/known_hosts",
"working_directory": "/srv/codewhale/work",
"env_allowlist": ["CODEWHALE_PROFILE"],
"codewhale_binary": "/usr/local/bin/codewhale"
},
"capabilities": ["local", "linux", "tests"],
"max_concurrent_tasks": 1
}
Alert Channel Secrets
Alert channels (Slack, generic webhook, PagerDuty) use FleetAlertEndpoint
instead of raw URLs. The webhook URL can be provided inline for non-sensitive
endpoints, or as a secret reference:
{
"kind": "slack",
"webhook": {
"url_ref": {"key": "CODEWHALE_FLEET_SLACK_WEBHOOK", "source": "env"},
"secret_ref": {"key": "CODEWHALE_FLEET_SLACK_SIGNING_SECRET", "source": "keyring"}
}
}
The secret_ref field provides an optional HMAC secret for webhook payload
signing, never stored in plaintext.
Config File
The [fleet] table in config.toml sets global trust policy defaults:
[fleet]
default_trust_level = "sandbox"
require_identity_verification = true
max_trust_level = "operator"
[fleet.exec]
# Recursion depth shares ONE axis with standalone sub-agents — a fleet worker
# IS a headless sub-agent. 0 blocks child agents (the root worker still runs);
# 3 is the default; explicit config clamps to the shared safety ceiling.
max_spawn_depth = 3
These defaults apply to fleet runs that don't carry their own security_policy.
Per-run policies always override the config defaults.
Capability Grants
Capability grants are additive, scoped permissions that authorize specific actions. By default, workers get no grants (least privilege). Common grants:
"network"with scope"github.com"— allow outbound HTTP to GitHub"git-push"— allowgit pushto remotes"provider-secrets"— allow accessing provider API keys"release"— allow release-related operations (tagging, publishing)"workspace-write"with scope"crates/tui/**"— allow writes within a path
Environment Sanitization
The host adapter layer enforces environment sanitization at worker start:
- Only
HOME,PATH, and platform-specific vars (SYSTEMROOT,COMSPEC) are injected into worker processes by default - Environment allowlists reject any key containing
SECRET,TOKEN,PASSWORD,PASSWD,API_KEY,CREDENTIAL, orPRIVATE_KEY - SSH workers only send explicitly allowlisted variables via OpenSSH
SendEnv - Secret values are never embedded in worker argv, task instructions, or fleet logs — only secret refs appear, and they are always redacted