Saadi Myftija
a3c387697e
⚒️ Publish Worker (v4) / build (supervisor) (push) Has been cancelled
**Background** Runs with `large-1x` or `large-2x` machine presets are disproportionally affected by scheduling delays during peak times. This is in part caused by the fact that the worker pool is shared for all runs, meaning large runs compete with smaller runs for available capacity. Because large runs require significantly more CPU and memory, they are harder for the scheduler to bin-pack onto existing nodes, often requiring a node with a significant amount of free resources or waiting for a new node to spin up entirely. This effect is amplified during peak times when nodes are already densely packed with smaller workloads, leaving insufficient contiguous resources for large runs. Also, large runs make up a small percentage of the total runs. **Changes** This PR adds Kubernetes node affinity settings to separate large and standard machine workloads across node pools. - Controlled via `KUBERNETES_LARGE_MACHINE_POOL_LABEL` env var (disabled when not set) - Large machine presets (large-*) get a soft preference to schedule on the large pool, with fallback to standard nodes - Non-large machines are excluded from the large pool via required anti-affinity - This ensures the large machine pool is reserved for large workloads while allowing large workloads to spill over to standard nodes if needed
Supervisor
Dev setup
- Create a worker group
api_url=http://localhost:3030
wg_name=my-worker
# edit this
admin_pat=tr_pat_...
curl -sS \
-X POST \
"$api_url/admin/api/v1/workers" \
-H "Authorization: Bearer $admin_pat" \
-H "Content-Type: application/json" \
-d "{\"name\": \"$wg_name\"}"
If the worker group is newly created, the response will include a token field. If the group already exists, no token is returned.
- Create
.envand set the worker token
cp .env.example .env
# Then edit your .env and set this to the token.plaintext value
TRIGGER_WORKER_TOKEN=tr_wgt_...
- Start the supervisor
pnpm dev
- Build CLI, then deploy a test project
pnpm exec trigger deploy --self-hosted
# The additional network flag is required on linux
pnpm exec trigger deploy --self-hosted --network host
Worker group management
Shared variables
api_url=http://localhost:3030
admin_pat=tr_pat_... # edit this
- These are used by all commands
Create a worker group
wg_name=my-worker
curl -sS \
-X POST \
"$api_url/admin/api/v1/workers" \
-H "Authorization: Bearer $admin_pat" \
-H "Content-Type: application/json" \
-d "{\"name\": \"$wg_name\"}"
- If the worker group already exists, no token will be returned
Set a worker group as default for a project
wg_name=my-worker
project_id=clsw6q8wz...
curl -sS \
-X POST \
"$api_url/admin/api/v1/workers" \
-H "Authorization: Bearer $admin_pat" \
-H "Content-Type: application/json" \
-d "{\"name\": \"$wg_name\", \"projectId\": \"$project_id\", \"makeDefaultForProject\": true}"
- If the worker group doesn't exist, yet it will be created
- If the worker group already exists, it will be attached to the project as default. No token will be returned.
Remove the default worker group from a project
project_id=clsw6q8wz...
curl -sS \
-X POST \
"$api_url/admin/api/v1/workers" \
-H "Authorization: Bearer $admin_pat" \
-H "Content-Type: application/json" \
-d "{\"projectId\": \"$project_id\", \"removeDefaultFromProject\": true}"
- The project will then use the global default again
- When
removeDefaultFromProject: trueno other actions will be performed