Files
triggerdotdev--trigger.dev/apps/supervisor
Saadi Myftija d0b2d79b3b
⚒️ Publish Worker (v4) / build (supervisor) (push) Has been cancelled
fix(supervisor): cancel pending delayed snapshots when the run completes or disconnects (#3894)
The compute suspend flow delays snapshots by `snapshotDelayMs` (~30s) so
short-lived waitpoints skip the snapshot entirely, with the intent that
a run continuing before the delay expires cancels the pending snapshot.
But the only `cancel()` call site was the `/continue` action, which
runners only invoke when restoring from an already-taken snapshot — so
pending snapshots were never cancelled (zero `snapshot.canceled` events
ever emitted in prod). When a run resumed and completed inside the
window, the stale snapshot fired ~30s later anyway, pausing the VM 6–13s
mid warm-start long-poll; the frozen guest couldn't fire its abort timer
or send a FIN, causing stalls and run-engine driven retries.

### Change

- Cancel the pending snapshot on `attempt.complete` — after the platform
accepts the completion, before the HTTP reply (so it can't reorder with
the runner's next `/suspend`).
- Cancel on `runDisconnected` (crash, exit, or run replaced on the
socket).
- Both cancels are guarded by a runnerId match (new
`TimerWheel.peek()`): a stale duplicate runner for a reassigned run must
not cancel the fresh runner's pending snapshot. A missing runnerId falls
through to an unconditional cancel (the pre-existing `/continue`
behavior is unchanged).

Waitpoint suspensions keep the runner socket connected and the attempt
incomplete, so neither hook touches a snapshot that is still wanted.

Known limitation (fail-safe direction): `socket.data.runnerId` is frozen
at the websocket handshake, so after a same-supervisor restore the
disconnect-path guard refuses the cancel. The `attempt.complete` path
uses the runner's current header id and is unaffected.
2026-06-11 18:29:54 +02:00
..

Supervisor

Dev setup

  1. Create a worker group
api_url=http://localhost:3030
wg_name=my-worker

# edit this
admin_pat=tr_pat_...

curl -sS \
    -X POST \
    "$api_url/admin/api/v1/workers" \
    -H "Authorization: Bearer $admin_pat" \
    -H "Content-Type: application/json" \
    -d "{\"name\": \"$wg_name\"}"

If the worker group is newly created, the response will include a token field. If the group already exists, no token is returned.

  1. Create .env and set the worker token
cp .env.example .env

# Then edit your .env and set this to the token.plaintext value
TRIGGER_WORKER_TOKEN=tr_wgt_...
  1. Start the supervisor
pnpm dev
  1. Build CLI, then deploy a test project
pnpm exec trigger deploy --self-hosted

# The additional network flag is required on linux
pnpm exec trigger deploy --self-hosted --network host

Worker group management

Shared variables

api_url=http://localhost:3030
admin_pat=tr_pat_... # edit this
  • These are used by all commands

Create a worker group

wg_name=my-worker

curl -sS \
    -X POST \
    "$api_url/admin/api/v1/workers" \
    -H "Authorization: Bearer $admin_pat" \
    -H "Content-Type: application/json" \
    -d "{\"name\": \"$wg_name\"}"
  • If the worker group already exists, no token will be returned

Set a worker group as default for a project

wg_name=my-worker
project_id=clsw6q8wz...

curl -sS \
    -X POST \
    "$api_url/admin/api/v1/workers" \
    -H "Authorization: Bearer $admin_pat" \
    -H "Content-Type: application/json" \
    -d "{\"name\": \"$wg_name\", \"projectId\": \"$project_id\", \"makeDefaultForProject\": true}"
  • If the worker group doesn't exist, yet it will be created
  • If the worker group already exists, it will be attached to the project as default. No token will be returned.

Remove the default worker group from a project

project_id=clsw6q8wz...

curl -sS \
    -X POST \
    "$api_url/admin/api/v1/workers" \
    -H "Authorization: Bearer $admin_pat" \
    -H "Content-Type: application/json" \
    -d "{\"projectId\": \"$project_id\", \"removeDefaultFromProject\": true}"
  • The project will then use the global default again
  • When removeDefaultFromProject: true no other actions will be performed