Saadi Myftija
002b8458d5
⚒️ Publish Worker (v4) / build (supervisor) (push) Has been cancelled
### Problem Firestarter's `didWarmStart: true` means the response was written to a long-poll socket — not that the runner received it. A silently dead poller (no FIN, e.g. a VM torn down mid-poll) leaves the dispatched run stuck in `PENDING_EXECUTING` until the run engine's heartbeat redrive, and each redrive burns a queue redelivery toward `TASK_RUN_DEQUEUED_MAX_RETRIES`. ### Change After a warm-start hit, the supervisor retains the `DequeuedMessage` (TimerWheel, default 10s), then probes the existing `getLatestSnapshot` API. If the run is still on the exact dequeued snapshot, no runner ever acted — it falls through to the regular cold-create path. Recovery: ~10s + cold start, no new APIs, no CLI changes. - **Double-start safe**: `startRunAttempt` runs under a per-run lock and 409s stale snapshot ids, so a reviving runner and the fallback workload can't both execute; the loser exits before running anything. - **Probe errors → do nothing**: healthy runners legitimately act late during platform brownouts (nested attempt-start retries), so falling back on uncertainty would stampede duplicates. The heartbeat redrive stays as the backstop (also covers supervisor restarts dropping timers). - **Off by default**: `TRIGGER_WARM_START_VERIFY_ENABLED` (+ `TRIGGER_WARM_START_VERIFY_DELAY_MS`, 1–60s, default 10s). Disabled = complete no-op. Works for all workload managers (compute/k8s/docker) since it hooks the shared dequeue path. - Emits `warmstart.verify` wide events (`outcome: delivered | fallback | probe_error`), making the silent-loss rate directly measurable.
Supervisor
Dev setup
- Create a worker group
api_url=http://localhost:3030
wg_name=my-worker
# edit this
admin_pat=tr_pat_...
curl -sS \
-X POST \
"$api_url/admin/api/v1/workers" \
-H "Authorization: Bearer $admin_pat" \
-H "Content-Type: application/json" \
-d "{\"name\": \"$wg_name\"}"
If the worker group is newly created, the response will include a token field. If the group already exists, no token is returned.
- Create
.envand set the worker token
cp .env.example .env
# Then edit your .env and set this to the token.plaintext value
TRIGGER_WORKER_TOKEN=tr_wgt_...
- Start the supervisor
pnpm dev
- Build CLI, then deploy a test project
pnpm exec trigger deploy --self-hosted
# The additional network flag is required on linux
pnpm exec trigger deploy --self-hosted --network host
Worker group management
Shared variables
api_url=http://localhost:3030
admin_pat=tr_pat_... # edit this
- These are used by all commands
Create a worker group
wg_name=my-worker
curl -sS \
-X POST \
"$api_url/admin/api/v1/workers" \
-H "Authorization: Bearer $admin_pat" \
-H "Content-Type: application/json" \
-d "{\"name\": \"$wg_name\"}"
- If the worker group already exists, no token will be returned
Set a worker group as default for a project
wg_name=my-worker
project_id=clsw6q8wz...
curl -sS \
-X POST \
"$api_url/admin/api/v1/workers" \
-H "Authorization: Bearer $admin_pat" \
-H "Content-Type: application/json" \
-d "{\"name\": \"$wg_name\", \"projectId\": \"$project_id\", \"makeDefaultForProject\": true}"
- If the worker group doesn't exist, yet it will be created
- If the worker group already exists, it will be attached to the project as default. No token will be returned.
Remove the default worker group from a project
project_id=clsw6q8wz...
curl -sS \
-X POST \
"$api_url/admin/api/v1/workers" \
-H "Authorization: Bearer $admin_pat" \
-H "Content-Type: application/json" \
-d "{\"projectId\": \"$project_id\", \"removeDefaultFromProject\": true}"
- The project will then use the global default again
- When
removeDefaultFromProject: trueno other actions will be performed