nicktrn
2c82d4c4d1
Adds an in-process backpressure signal that pauses dequeuing when the
Kubernetes cluster is saturated, so work overflows cheaply in the queue
instead of piling up as unschedulable pods. Saturation is read by
scraping the apiserver's total pod-object count
(`apiserver_storage_objects{resource="pods"}`) and applying an
engage/release threshold with hysteresis - a single lightweight
aggregate scrape, not a pod listing.
Backpressure sources are now evaluated independently and OR'd: each
source has its own enable and dry-run flag, and the supervisor engages
if any enabled source trips. This adds the pod-count source alongside
the existing one without changing it, and is extensible to more sources
later. Off by default.
The scrape uses the in-cluster kubeconfig over `https` so TLS verifies
against the cluster CA (the fetch-options helper attaches the CA as an
`https.Agent`, which the global `fetch` ignores - that path silently
dropped the CA). Enabling the pod-count source requires the supervisor's
service account to be granted `get` on the `/metrics` non-resource URL;
that RBAC and the per-deployment env wiring are operator-side and live
elsewhere.
New config (pod-count source):
`TRIGGER_DEQUEUE_BACKPRESSURE_POD_COUNT_ENABLED` (default false),
`_POD_COUNT_DRY_RUN` (default true), `_POD_COUNT_ENGAGE` /
`_POD_COUNT_RELEASE` (hysteresis thresholds), `_POD_COUNT_REFRESH_MS`
(scrape interval, default 5s). The existing source's flags are
unchanged.
Observability: a `supervisor_cluster_pod_count` gauge, and the pod-count
monitor's metrics are namespaced (`supervisor_backpressure_pod_count_*`)
so the existing backpressure metrics keep their names.
Supervisor
Dev setup
- Create a worker group
api_url=http://localhost:3030
wg_name=my-worker
# edit this
admin_pat=tr_pat_...
curl -sS \
-X POST \
"$api_url/admin/api/v1/workers" \
-H "Authorization: Bearer $admin_pat" \
-H "Content-Type: application/json" \
-d "{\"name\": \"$wg_name\"}"
If the worker group is newly created, the response will include a token field. If the group already exists, no token is returned.
- Create
.envand set the worker token
cp .env.example .env
# Then edit your .env and set this to the token.plaintext value
TRIGGER_WORKER_TOKEN=tr_wgt_...
- Start the supervisor
pnpm dev
- Build CLI, then deploy a test project
pnpm exec trigger deploy --self-hosted
# The additional network flag is required on linux
pnpm exec trigger deploy --self-hosted --network host
Worker group management
Shared variables
api_url=http://localhost:3030
admin_pat=tr_pat_... # edit this
- These are used by all commands
Create a worker group
wg_name=my-worker
curl -sS \
-X POST \
"$api_url/admin/api/v1/workers" \
-H "Authorization: Bearer $admin_pat" \
-H "Content-Type: application/json" \
-d "{\"name\": \"$wg_name\"}"
- If the worker group already exists, no token will be returned
Set a worker group as default for a project
wg_name=my-worker
project_id=clsw6q8wz...
curl -sS \
-X POST \
"$api_url/admin/api/v1/workers" \
-H "Authorization: Bearer $admin_pat" \
-H "Content-Type: application/json" \
-d "{\"name\": \"$wg_name\", \"projectId\": \"$project_id\", \"makeDefaultForProject\": true}"
- If the worker group doesn't exist, yet it will be created
- If the worker group already exists, it will be attached to the project as default. No token will be returned.
Remove the default worker group from a project
project_id=clsw6q8wz...
curl -sS \
-X POST \
"$api_url/admin/api/v1/workers" \
-H "Authorization: Bearer $admin_pat" \
-H "Content-Type: application/json" \
-d "{\"projectId\": \"$project_id\", \"removeDefaultFromProject\": true}"
- The project will then use the global default again
- When
removeDefaultFromProject: trueno other actions will be performed