Saadi Myftija
6d6ed471d1
⚒️ Publish Worker (v4) / build (supervisor) (push) Has been cancelled
This PR makes our image builds deterministic and reproducible by ensuring that identical source code always produces the same image layers and image digest. This means that deployments where nothing has changed will no longer invalidate the image cache in our worker cluster nodes, thus avoid making the cold starts for runs worse. **Context** New deployments currently increase the cold start times for runs, as they generate a new image which needs to be pulled in the worker cluster where runs are executed. It happens also when the source code for the deployment has not changed due to non-deterministic steps in our build system. This addresses the latter issue by making builds reproducible. **Main changes** - Avoided baking `TRIGGER_DEPLOYMENT_ID` and `TRIGGER_DEPLOYMENT_VERSION` in the image, we now pass these via the supervisor instead. - Used `json-stable-stringify` for consistent key ordering in the files we generate for the build, e.g., `package.json`, `build.json`, `index.json`. - Removed `metafile.json` from the image contents as it is not actually used in the container. This is only relevant for the `analyze` command. - Added `SOURCE_DATE_EPOCH=0` and `rewrite-timestamp=true` to Docker builds to normalize file timestamps. - Removed some `timings` and `outputHashes` from build outputs and manifests. The builds are now reproducible for both native build server and Depot paths. This should also lead to better image layer cache reuse in general.
Supervisor
Dev setup
- Create a worker group
api_url=http://localhost:3030
wg_name=my-worker
# edit this
admin_pat=tr_pat_...
curl -sS \
-X POST \
"$api_url/admin/api/v1/workers" \
-H "Authorization: Bearer $admin_pat" \
-H "Content-Type: application/json" \
-d "{\"name\": \"$wg_name\"}"
If the worker group is newly created, the response will include a token field. If the group already exists, no token is returned.
- Create
.envand set the worker token
cp .env.example .env
# Then edit your .env and set this to the token.plaintext value
TRIGGER_WORKER_TOKEN=tr_wgt_...
- Start the supervisor
pnpm dev
- Build CLI, then deploy a test project
pnpm exec trigger deploy --self-hosted
# The additional network flag is required on linux
pnpm exec trigger deploy --self-hosted --network host
Worker group management
Shared variables
api_url=http://localhost:3030
admin_pat=tr_pat_... # edit this
- These are used by all commands
Create a worker group
wg_name=my-worker
curl -sS \
-X POST \
"$api_url/admin/api/v1/workers" \
-H "Authorization: Bearer $admin_pat" \
-H "Content-Type: application/json" \
-d "{\"name\": \"$wg_name\"}"
- If the worker group already exists, no token will be returned
Set a worker group as default for a project
wg_name=my-worker
project_id=clsw6q8wz...
curl -sS \
-X POST \
"$api_url/admin/api/v1/workers" \
-H "Authorization: Bearer $admin_pat" \
-H "Content-Type: application/json" \
-d "{\"name\": \"$wg_name\", \"projectId\": \"$project_id\", \"makeDefaultForProject\": true}"
- If the worker group doesn't exist, yet it will be created
- If the worker group already exists, it will be attached to the project as default. No token will be returned.
Remove the default worker group from a project
project_id=clsw6q8wz...
curl -sS \
-X POST \
"$api_url/admin/api/v1/workers" \
-H "Authorization: Bearer $admin_pat" \
-H "Content-Type: application/json" \
-d "{\"projectId\": \"$project_id\", \"removeDefaultFromProject\": true}"
- The project will then use the global default again
- When
removeDefaultFromProject: trueno other actions will be performed