nicktrn
3fba04573d
The dequeue brake released the moment its signal became unreadable. `refresh()` caught any error from `source.read()` and set the verdict to `null`, which `computeEngaged()` treats as not-engaged — so a few failed reads dropped an engaged brake, silently, with no log and no metric. That handling was symmetric while the risk is not. A source that has stopped answering correlates with the pressure the brake exists for, so releasing on read failure gives up protection at exactly the wrong moment; holding too long only costs throughput. Now a failed read keeps the last verdict instead of discarding it. The verdict then ages normally, so the existing `maxVerdictAgeMs` check becomes the grace window and still bounds how long a dead source can hold the brake — a permanently unreachable source releases it rather than pinning dequeuing forever. Because `computeEngaged()` only consults staleness for an *engaged* verdict, a released one is unaffected and stays released. The default grace moves from 15s to 120s, comparable to how long the brake normally stays engaged. One guard worth calling out: holding is only safe when something bounds it, so when `maxVerdictAgeMs` is unset the previous discard behaviour is kept. Otherwise an unbounded hold could pin the brake indefinitely. Read failures were previously invisible — the catch block neither logged nor counted. Adds a `read_failures_total` counter, plus an error log on the transition into failure rather than once per tick, since the refresh loop runs every second. The post-release ramp needs no change: it anchors off the engaged-to-released transition, so a grace-window release still ramps back up instead of snapping to full rate, which is what you want after a blind period. Tests cover holding while reads fail, releasing past the max age, and the existing unbounded-config paths are unchanged.
Supervisor
Dev setup
- Create a worker group
api_url=http://localhost:3030
wg_name=my-worker
# edit this
admin_pat=tr_pat_...
curl -sS \
-X POST \
"$api_url/admin/api/v1/workers" \
-H "Authorization: Bearer $admin_pat" \
-H "Content-Type: application/json" \
-d "{\"name\": \"$wg_name\"}"
If the worker group is newly created, the response will include a token field. If the group already exists, no token is returned.
- Create
.envand set the worker token
cp .env.example .env
# Then edit your .env and set this to the token.plaintext value
TRIGGER_WORKER_TOKEN=tr_wgt_...
- Start the supervisor
pnpm dev
- Build CLI, then deploy a test project
pnpm exec trigger deploy --self-hosted
# The additional network flag is required on linux
pnpm exec trigger deploy --self-hosted --network host
Worker group management
Shared variables
api_url=http://localhost:3030
admin_pat=tr_pat_... # edit this
- These are used by all commands
Create a worker group
wg_name=my-worker
curl -sS \
-X POST \
"$api_url/admin/api/v1/workers" \
-H "Authorization: Bearer $admin_pat" \
-H "Content-Type: application/json" \
-d "{\"name\": \"$wg_name\"}"
- If the worker group already exists, no token will be returned
Set a worker group as default for a project
wg_name=my-worker
project_id=clsw6q8wz...
curl -sS \
-X POST \
"$api_url/admin/api/v1/workers" \
-H "Authorization: Bearer $admin_pat" \
-H "Content-Type: application/json" \
-d "{\"name\": \"$wg_name\", \"projectId\": \"$project_id\", \"makeDefaultForProject\": true}"
- If the worker group doesn't exist, yet it will be created
- If the worker group already exists, it will be attached to the project as default. No token will be returned.
Remove the default worker group from a project
project_id=clsw6q8wz...
curl -sS \
-X POST \
"$api_url/admin/api/v1/workers" \
-H "Authorization: Bearer $admin_pat" \
-H "Content-Type: application/json" \
-d "{\"projectId\": \"$project_id\", \"removeDefaultFromProject\": true}"
- The project will then use the global default again
- When
removeDefaultFromProject: trueno other actions will be performed