Commit Graph

152 Commits

Author SHA1 Message Date
Saadi Myftija ff80742ab7 fix(ci): use workflow_dispatch instead of repository_dispatch for publish.yml (#2791)
Mixed up the trigger in #2790
2025-12-16 16:53:22 +01:00
Saadi Myftija 11366e658c chore(ci): add manual trigger to the image publishing workflow (#2790)
Useful to retry failures manually.
2025-12-16 16:34:57 +01:00
Oskar Otwinowski 7f7f993587 feat(webapp): improve adjacent runs navigation and timeline performance (#2776)
- Add replace prop to LinkButton to use history replacement for adjacent
run navigation
- Preserve span and tab params when navigating between adjacent runs
- Disable animations for completed spans in timeline to improve
performance
- Include spanId in runs list navigation for better context preservation
- Direct link to task test page when filtering by single task with no
runs
- Fix minor styling issue with run friendlyId display padding
2025-12-11 17:26:15 +00:00
Saadi Myftija b71bf89444 chore(releases): minor improvements to the release workflow (#2764)
Changes in this PR:
- Arbitrary refs are now allowed when triggering the release workflow
manually (ref must be on the main branch).
- Release summary is now displayed in the GH job output; makes for a
nicer experience when approving the release workflow.
2025-12-09 11:29:48 +01:00
Saadi Myftija d8f5853457 fix(releases): use contents: write permission in the release workflow (#2761)
Needed to push tags
2025-12-05 17:22:24 +01:00
Saadi Myftija a52566d9cc feat(releases): add a manual trigger for the package release workflow (#2759)
The manual trigger is currently enabled only for the prerelease job.
This PR adds it for the normal release flow too, as it is useful to
retrigger failed release workflows.
2025-12-05 17:00:02 +01:00
Saadi Myftija 07a1d04d52 fix(releases): add missing npm upgrade step (#2758)
Accidentally removed this in a previous PR.
2025-12-05 16:41:45 +01:00
Saadi Myftija beb52b9800 chore(releases): adjust gh workflow title (#2755)
Just a tiny workflow label change
2025-12-05 13:33:35 +01:00
Saadi Myftija 3401a1d0a9 feat(releases): require approval for package releases (#2753)
**Background**
Currently the changeset PR creation and the publishing is handled by the
same workflow. This is not ideal:
- The build steps are executed on every run of the pipeline, even though
they're only needed for the publish case.
- The PR creation workflow does not need permissions to publish to npm,
only the release path needs them.
- Adding an approval step is painful as we'd need to also approve each
changeset PR creation workflow run.

**Changes in this PR**
- Separated the changeset PR creation into its own workflow and minimum
permission set.
- Added a GH environment with an approval step for the package
publishing workflow (also for prereleases).
- New publish workflow runs will not cancel in-progress runs; helps
avoid partial failures in publishing.

These changes also enable hardening the npm OIDC setup by tying it to a
GH environment that requires approval.
2025-12-05 13:25:25 +01:00
Saadi Myftija 748ae658f7 fix(releases): use npm 11.x for OIDC support in the release workflow (#2742)
Support for OIDC requires npm v11.5.1 or newer.
2025-12-04 12:21:21 +01:00
Saadi Myftija da4c753b68 fix(releases): add missing GH token in prerelease workflow (#2739)
The `changeset version` command needs the GH token too.
2025-12-04 11:27:11 +01:00
Saadi Myftija 652d95c7eb feat(releases): add a prerelease workflow (#2737)
Adds a manual trigger to the `release.yml` workflow for publishing
prerelease versions. Needs to be in the same workflow file due to a NPM
limitation on how OIDC claims are checked.

Currently there is a validation step on the ref for the prerelease: it
must be merged to the main branch. We can revisit this in the future in
case we find it too limiting.
2025-12-04 10:17:05 +01:00
Saadi Myftija c5f7a8daf7 fix(releases): explicitly use no frozen lockfile in the install step (#2733) 2025-12-03 16:28:06 +01:00
Saadi Myftija 8b0f51b317 chore(releases): automatically update the lockfile in changeset PRs (#2732)
This is a step which we currently need to do manually and it's rather
painful. The lockfile update is necessary due to cross references in our
packages.

Added it as a separate job instead of a step to start from fresh
workspace, as the state that the `changeset` step leaves the workdir is
not explicitly clear to the reader.
2025-12-03 16:04:30 +01:00
Saadi Myftija 331882f59c chore(releases): add package version in changeset PRs (#2730)
* Add the release version to changeset PRs

* Add missing id-token permission, needed for oidc

* Remove a couple of unnecesary steps

* Reference the `changeset-release/main` branch explicitly
2025-12-03 14:42:40 +01:00
Eric Allam 61b338bea7 chore(repo): upgrade repo to pnpm@10 to prevent executing dep scripts on install (#2712)
* chore: migrate pnpm lockfile to v9 format via pnpm@9

* Upgrade to pnpm 10.23.0

* update the dockerfile and added a few deps to bundle in remix app
2025-11-27 16:26:19 +00:00
Eric Allam 15fef916f6 feat(build): update prisma extension to work with generated clients and rust-free clients (#2689)
* prisma extension fixes WIP

* More prisma stuff

* more prisma stuff

* remove changelog

* upgrade github workflows to use node 20.19 because installing prisma@7 breaks with lower versions

* Don't use generate for the prisma reference projects

* make sure it works if no mode is passed in
2025-11-19 10:48:15 +00:00
Saadi Myftija 1cc62230ab feat: introduce organization access tokens (#2391)
* Create schema and migration for organization access tokens

* Add helpers for creating and authenticating OATs

* Adapt the auth service to also accept OATs

* Accept OATs in the whoami v2 endpoint

* Enable deployments with the CLI using OATs

* Avoid reading env variables directly in the token utils

* Remove duplicate cli token utils

* Validate ENCRYPTION_KEY length when parsing env vars

* Make token utils a server-only module

* Disallow revoking already revoked OATs

* Simplify generics in authenticateRequest

* Use 32 bytes mock encryption key in the test setup

* Update dummy encryption key values in tests and templates

* Add a column in the OATs table to differentiate between user and system generated

* Simplify args for v3ProjectPath

Co-authored-by: Matt Aitken <matt@mattaitken.com>

* Add index on org id and createdAt

* Avoid storing the encrypted oat token and its obfuscated version in the DB at all

It is a safer approach. Also we do not need to ever read the decrypted token value after creation.

* Fix prisma update condition

* Add token type to the OAT table index

* Accept OATs in the mcp auth flow

* Simplify env auth flow around the /projects endpoints

---------

Co-authored-by: Matt Aitken <matt@mattaitken.com>
2025-08-27 13:23:45 +02:00
nicktrn 7b54c3527e fix(otel): prevent infinite retry loops on unicode hex escape errors (#2337)
* prevent infinite retry loop for unicode errors

* structured logs for prisma events

* preserve all prisma event fields

* either use structured logs or stdout, never both

* split runs repo tests

* decrease test shards to 8
2025-08-02 16:54:11 +01:00
Eric Allam 09d0e80804 Add sentry error reporting (#2309)
* Sentry WIP

* Configure sentry for uploading and releasing during the publish webapp step

* Delete source maps after uploading

* Forward logger.error calls to sentry through Logger.onError

* Couple tweaks to the dockerfile
2025-07-24 15:09:50 +01:00
Matt Aitken 630e9556b0 Bulk actions 2.0 (and switch all run listing to ClickHouse) (#2264)
* useSearchParams has

* useSearchParams has

* useSearchParams has

* Consistent way to get the run filters

* Consistent way to get the run filters

* Consistent way to get the run filters

* Initial work on the new bulk actions

* Initial work on the new bulk actions

* Initial work on the new bulk actions

* WIP actions and filtering

* WIP actions and filtering

* WIP actions and filtering

* Empty filter arrays are set to undefined

* Empty filter arrays are set to undefined

* Empty filter arrays are set to undefined

* WIP prisma schema

Removed extra runtimeEnvironmentId

* WIP prisma schema

Removed extra runtimeEnvironmentId

* WIP prisma schema

Removed extra runtimeEnvironmentId

* Migrations

* Migrations

* Migrations

* BulkActionGroup changed some columns around

* BulkActionGroup changed some columns around

* BulkActionGroup changed some columns around

* New badge variant, removed unused ones

* New badge variant, removed unused ones

* New badge variant, removed unused ones

* Bulk action button

* Bulk action button

* Bulk action button

* Make the next runs page the default now

* Make the next runs page the default now

* Make the next runs page the default now

* Improved the RadioButton style

* Improved the RadioButton style

* Improved the RadioButton style

* Remove the old bulk action bar

* Remove the old bulk action bar

* Remove the old bulk action bar

* More UI progress

* More UI progress

* More UI progress

* Lots of UI changes to the Runs page

* Lots of UI changes to the Runs page

* Lots of UI changes to the Runs page

* Fixed period filter resetting everything

* Fixed period filter resetting everything

* Fixed period filter resetting everything

* Improved the Switch secondary style

* Improved the Switch secondary style

* Improved the Switch secondary style

* Buggy filter fixes

* Buggy filter fixes

* Buggy filter fixes

* Improved the filter display and fixed a bug with search param from object

* Improved the filter display and fixed a bug with search param from object

* Improved the filter display and fixed a bug with search param from object

* Clear button is minimal

* Clear button is minimal

* Clear button is minimal

* Using a presenter now

* Using a presenter now

* Using a presenter now

* Bulk actions are created, but not actually processed (yet)

* Bulk actions are created, but not actually processed (yet)

* Bulk actions are created, but not actually processed (yet)

* Bulk replay/cancel is working

* Bulk replay/cancel is working

* Bulk replay/cancel is working

* Multiple fixes, added bulk column to PG

* Multiple fixes, added bulk column to PG

* Multiple fixes, added bulk column to PG

* Bulk action run filtering working using CH

* Bulk action run filtering working using CH

* Bulk action run filtering working using CH

* Replay setting the bulk id on the runs

* Replay setting the bulk id on the runs

* Replay setting the bulk id on the runs

* Properly cap the time when doing a bulk action

* Properly cap the time when doing a bulk action

* Properly cap the time when doing a bulk action

* If the bulk action isn't recent, add it to the dropdown anyway

* If the bulk action isn't recent, add it to the dropdown anyway

* If the bulk action isn't recent, add it to the dropdown anyway

* Blank version of the bulk actions page

* Blank version of the bulk actions page

* Blank version of the bulk actions page

* Individually selected runs working

* Individually selected runs working

* Individually selected runs working

* Use selected mode if runs are checked

* Use selected mode if runs are checked

* Use selected mode if runs are checked

* Added the modal

* Added the modal

* Added the modal

* Marked the old bulk actions stuff as deprecated

* Marked the old bulk actions stuff as deprecated

* Marked the old bulk actions stuff as deprecated

* Renamed bulk action file

* Renamed bulk action file

* Renamed bulk action file

* Bulk run filter with the name and a default

* Bulk run filter with the name and a default

* Bulk run filter with the name and a default

* WIP on bulk actions page

* WIP on bulk actions page

* WIP on bulk actions page

* Updated panel, added new truncated id component

* Updated panel, added new truncated id component

* Updated panel, added new truncated id component

* Style improvements to the radio buttons

* Style improvements to the radio buttons

* Style improvements to the radio buttons

* Added an option action completion email

* Added an option action completion email

* Added an option action completion email

* Adds a blank state for the bulk actions page

* Adds a blank state for the bulk actions page

* Adds a blank state for the bulk actions page

* Nicer completed email

* Nicer completed email

* Nicer completed email

* Don't open the bulk action panel if there are no runs

* Don't open the bulk action panel if there are no runs

* Don't open the bulk action panel if there are no runs

* Runs blank state and bulk action accordion

* Runs blank state and bulk action accordion

* Runs blank state and bulk action accordion

* Updates secondary/small switch style

* Updates secondary/small switch style

* Updates secondary/small switch style

* Pagination buttons no longer split in twain (WIP)

* Pagination buttons no longer split in twain (WIP)

* Pagination buttons no longer split in twain (WIP)

* Aborting working

* Aborting working

* Aborting working

* Bulk action live reloading

* Bulk action live reloading

* Bulk action live reloading

* ListPagination works correctly in all states

* ListPagination works correctly in all states

* ListPagination works correctly in all states

* Run page, show friendlyId instead of number

* Run page, show friendlyId instead of number

* Run page, show friendlyId instead of number

* Bulk action help open by default if you have none

* Bulk action help open by default if you have none

* Bulk action help open by default if you have none

* Extra status filtering step because of replication delay

* Extra status filtering step because of replication delay

* Extra status filtering step because of replication delay

* Wider bulk action onboarding

* Wider bulk action onboarding

* Wider bulk action onboarding

* More sensible widths on the bulk action side panel

* More sensible widths on the bulk action side panel

* More sensible widths on the bulk action side panel

* Border color tweak to the RadioButton

* Border color tweak to the RadioButton

* Border color tweak to the RadioButton

* Improved the accordion component hover states

* Improved the accordion component hover states

* Improved the accordion component hover states

* Updates the bulk action blank state images to the latest UI

* Updates the bulk action blank state images to the latest UI

* Updates the bulk action blank state images to the latest UI

* Added R and C shortcuts back in

* Added R and C shortcuts back in

* Added R and C shortcuts back in

* Fix for selecting a single run

* Fix for selecting a single run

* Fix for selecting a single run

* Improved exit icon, added shortcut to modal

* Improved exit icon, added shortcut to modal

* Improved exit icon, added shortcut to modal

* Tidy imports

* Tidy imports

* Tidy imports

* Tidy imports

* Tidy imports

* Tidy imports

* Tidy imports

* Tidy imports

* Tidy imports

* Tidy imports

* Fix for grid layout when 1 page of bulk actions visible

* Fix for grid layout when 1 page of bulk actions visible

* Fix for grid layout when 1 page of bulk actions visible

* Removed the ... on the abort button

* Removed the ... on the abort button

* Removed the ... on the abort button

* Removed the ... on the abort button

* Animate the progress bar

* Set TZ="UTC" in the env example

* Filter summary in the bulk inspector

* Improves the pagination styling

* Improves the pagination styling

* Delete old bulk action routes

* Removed old Postgres RunListPresenter

* Retry any replication error where the message contains "timeout"

* Increase wait to make test less flaky

* The test was using run id instead of friendly id

* Safer array access

* Remove error log if there's a bad status

* Nicer frontend type safety with the bulk action and mode

* Switched a log to a debug log

* Retry replication unless the error is a known non-retry error

Flip the strategy to retry by default

* Make ClickHouse required

* Backfill run replication admin API endpoint

* Set a CLICKHOUSE_URL for unit tests

---------

Co-authored-by: James Ritchie <james@trigger.dev>
2025-07-14 15:50:01 +01:00
nicktrn 0c71dc74df feat: add node 22 and bun runtime support with version display (#2254)
* update node-22 image

* update bun image

* disable io_uring

* fix fallback bun path

* prevent duplicate warnings

* add runtime and version to deployments

* runtime icons

* fallback to nodejs

* prevent empty table cell menu

* log if local build on deploy

* pass io_uring env var to child

* denormalize runtime and version, display on run details

* add changesets

* disable pr checks for changeset commits..

* add runtime data to deployed bg workers
2025-07-10 16:43:10 +01:00
nicktrn fa6675eb00 Helm chart improvements (#2207)
* add helm deps

* configurablecli tag for setup commands

* enabled -> deploy

* extraEnv -> extraEnvVars

* update docs

* minio -> s3

* improve persistence config

* set cli tag to v4-beta

* chart and image bump

* fix tests

* lock chart

* workflow hotfix

* quote for niels

* make probes configurable

* fix webapp and supervisor templates

* bump chart
2025-06-28 12:59:29 +01:00
nicktrn 6535cf918b v4 helm chart and docs (#2195)
* v4 helm chart

* ignore packaged chart archives

* chart publish prep

* add helm chart release workflow

* enable draft gh release

* stateful services should be statefulsets

* expose internal otel config

* podAnnotations for all services

* postgresql.extraArgs

* postgresql -> postgres

* add clickhouse vars as extraEnv examples

* support external secrets

* improve clickhouse config

improve clickhouse config

* disable internal otel config by default

* allow nil internal config

* fix gh release permission

* fix validation step for ci

* secrets.enabled

* simplify secret keys

* bump chart version

* fix contents permission

* remove redundant print steps

* remove helm release heading

* optimise clickhouse config

* support wait-for-it.sh

* fix ch overrides

* add low-resource ch overrides to compose setup

* supervisor bootstrap config

* disable otel trace logging

* retain shared volume

* enable clickhouse for compose setup

* disable trace logging

* bump chart

* add extraManifests

* telemetry config

* add repo namespace config

* scope cluster role to namespace to support multiple installs

* external s3 config

* values tweaks

* helm docs

* bump chart

* set contents read permission

Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>

* add object store root credentials warning

* run init containers as uid 1000

* pass ch user creds with explicit flag

* improve ingress anotation handling

* remove duplicate colume claim block

* trim htpasswd auth

* lock curl images used for tests

* add v4-beta package version warning

* make schema and sslmode configurable

* package tag info -> warning

* improve secrets config example

* update docs

* bump version

* bump webapp reqs to account for ch

* make ch creds more easily configurable

* publish ch http port

* example point at latest beta chart

* small docs tweaks

---------

Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
2025-06-25 23:57:51 +01:00
nicktrn f603725393 Feat: unified deploys for self-hosted and cloud users incl. multi-platform support (#2138)
* remove registry proxy

* remove --self-hosted flag

* automatically set network build flag

* update syncEnvVars debug log

* improve switch command

* always display deploy errors if they exist

* fix stuck deploy command after finalize error

* webapp-driven deploys, multi-platform support, lots of fixes

* add worker deployment migration

* rename image platform env var

* only try to sync parent env vars for preview deployments

* add KEEP_TMP_DIRS

* supervisor: docker api version lock, auth, multi-platform

* set image ref on create, validate digest

* use metadata for digest, fix local multi-platform builds

* print git meta branch before commit

* improve push and load flag handling

* make runs after local builds compatible with load and push

* small improvement for platform overrides

* add image platform to dequeued message

* remove deprecated init request body fields

* fix fail deployment id param

* remove build debug logs

* pass report merge with no tests

* structured run debug logs

* add required env var for tests

* should not be an error log

* add changeset
2025-06-04 15:54:10 +01:00
nicktrn a61951049f Enhance webapp version with build info (#2146)
* improve app version output

* set build info

* fix typo

* always show additional build info when self-hosting

* Update apps/webapp/app/routes/_app.orgs.$organizationSlug.settings/route.tsx

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>

* Update apps/webapp/app/components/navigation/OrganizationSettingsSideMenu.tsx

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>

* fix build timestamp name

---------

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
2025-06-04 15:49:07 +01:00
nicktrn e84d8cfa1d Fix: lock vitest version in CI (#2139) 2025-06-02 15:14:33 +01:00
nicktrn 7c791dd519 Improve unit test workflow performance (#2096)
* shard unit tests

* temp enable for all pushes

* fix test workflow

* update to latest vitest and only add to root package.json

* additionally use default reporter

* gather reports before uploading

* split up slow replication tests

* split up unit tests workflow

* move workflows to parent dir

* use new paths in parent workflow

* prevent artifact clashes

* we always need to create the reports dir

* speed up merge reports

* gather reports even when tests fail

* fix artifact patterns

* increase shards

* disable push trigger again

* improve dequeue snapshot test reliability
2025-05-23 12:22:14 +01:00
Eric Allam 41a3bdec68 Add id-token permission to the publish webapp workflow (#2088) 2025-05-21 14:30:03 +01:00
Eric Allam cc562c58fa Remove unused publish job from the release github workflow to fix permissions error (#2085) 2025-05-21 13:19:49 +01:00
Dan e08e493b3c Potential fix for code scanning alert no. 2: Workflow does not contain permissions (#2078)
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
2025-05-19 21:17:35 +01:00
Dan f667d61d4c Potential fix for code scanning alert no. 3: Workflow does not contain permissions (#2077)
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
2025-05-19 21:16:58 +01:00
Dan 325ef04c7f Potential fix for code scanning alert no. 6: Workflow does not contain permissions (#2076)
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
2025-05-19 21:15:42 +01:00
Dan 79f0a6a662 Potential fix for code scanning alert no. 42: Workflow does not contain permissions (#2074)
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
2025-05-19 21:14:26 +01:00
Dan ad5027b5fd Potential fix for code scanning alert no. 1: Workflow does not contain permissions (#2079)
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
2025-05-19 20:59:50 +01:00
Dan f0809e6ad6 Potential fix for code scanning alert no. 38: Workflow does not contain permissions (#2075)
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
2025-05-19 20:48:58 +01:00
Eric Allam a8e38d9231 chore: Skip unit tests dependency when running publish image workflow (#2054) 2025-05-14 19:31:50 +01:00
nicktrn fa5360a99d Feat: supervisor ARM builds (#2046)
* v4 worker multi-arch builds

* v4-beta tag for webapp

* add oidc permission

* deps fetcher needs python

* node-gyp needs complete toolchain

* fix headers

* bust pnpm cache

* add setuptools

* switch back to old cache id

* push worker images

* pass the correct tags
2025-05-14 11:23:41 +01:00
Mendy Landa 21c0dc619e feat: playwright extension (#1764)
* feat: playwright build extension

* feat: add playwright task

* full dep list & added some comments

* fix v3-catalog

* log image size for local builds

* detect playwright version

* add changesets

* add docs

* additional checks

* fix unit test workflow for forks

* use middleware and update docs

* update jsdoc

---------

Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
2025-05-13 12:22:01 +01:00
nicktrn 053389d9b3 Fix testcontainers cleanup (#2023)
* update testcontainers packages

* increase cleanup timeout and add better logs

* small tweaks

* decrease docker network size so we can have more of them

* add a test flow to check this all works

* await all engine.quit calls

* add docker diagnostics

* reduce number of test runs

* improve network attachment output

* add setup logs

* log redis setup errors

* add cleanup log type

* stop redis container if setup fails

* disable ipv6

* tidy things up a bit

* fix unit tests workflow

* decrease container cleanup timeout

* fix types

* fix webapp typecheck
2025-05-06 08:56:34 +01:00
nicktrn 15816e9955 Fix long retry delays (#2020)
* make testcontainers wait until container has stopped

* require unit tests for publishing again

* add failing test case

* make it pass

* add retry threshold ms env var and use it
2025-05-02 16:57:21 +01:00
nicktrn a10e2a8b16 Small fixes in prep for v4 self-hosting (#2018)
* fix entitlement validation when client doesn't exist

* add supervisor to publish workflow

* fix v4 publish workflow name
2025-05-02 14:33:20 +01:00
nicktrn fa0e2bf17c Fix v4 build workflow (#1894) 2025-04-07 17:34:07 +01:00
Eric Allam 6d819c6726 chore: publish docker even if the unit tests fail for now (#1867) 2025-04-02 14:20:16 +01:00
Eric Allam 5a725675d3 Fix the unit tests CI action 2025-03-21 15:06:55 +00:00
nicktrn e297c7f7f0 Add supervisor build pipeline (#1772)
* dockerignore node_modules in subdirectories

* image tag action should handle re2 tags

* add supervisor containerfile

* add publish worker re2 workflow

* fix copypasta

* require branch check
2025-03-06 18:22:36 +00:00
nicktrn e97704d904 Run Engine 2.0 (WIP) (#1575)
* bump worker version

* Suggested glossary for the RunEngine, TBC

* Removed BatchTaskRun changes from this branch, they were done in main

* Set the BatchTaskRun status to completed when all runs are completed

* When dequeuing respect passed in maxResources

* Ported over the new run props: idempotencyKeyExpiresAt, versions, oneTimeUseToken, maxDurationInSeconds

* Didn’t hit save… the new props when triggering tasks passed through

* Idempotency expiration + waitpoint edge case

* WIP on creating checkpoint, parking for now

* fix worker routes

* upgrade webapp node types to support generic event emitter

* separate event bus handler singleton and run failure alerts

* duration waits

* fix execution snapshot debug spans

* task waits

* fix event bus types

* temporary fix for react hook run handle type

* disable run notifications for now

* convert any typecasts to expect errors to more easily fix later

* fix webapp types after node types upgrade

* updateEnvConcurrencyLimits across marqs and the runqueue

* Pass proper values into the run engine

* RunQueue settings and removed unused rebalancing workers

* Remove rebalancing prop

* Tidied more things up

* Update/remove queue limits for MARQS and RunQueue

* taskQueue/concurrencyLimit changes ported back into the RunEngine

* Reworked completing waitpoints to improve performance and reduce race conditions

* Improved test robustness

* Down to a single run lock only when a run is totally unblocked and ready to continue

* warm starts, worker notifications, wait fixes

* Fix for Run Engine poll interval env var

* Expect the waitpoint to be completed quickly

* If a run is locked then it’s too late to expire it

* Added VALKEY_ env vars and plugged them into the run engine

* Extracted and updated the guard queue function so it can be used when batching

* Added logging and universal concurrency changes to trigger task v1

* Added notes back in

* Bump @trigger.dev/worker to 3.3.7

* reportInvocationUsage for the runAttemptStarted event

* improve execution snapshot span debug span start times

* Unfriendly IDs

* update lockfile

* Created a shared determineEngineVersion function

* disable unfinished commands

* save new cli config to different location, misc fixes

* add basic engine version check via current deploy

* new run engine will default to node 22 runtime

* block some actions for projects on previous run engine

* fix worker group tests

* fix triggerAndWait test

* one typescript version to rule them all

* redlock type patch

* fix type issues caused by ts-reset

* improve cleanup scripts

* add missing socket.io dep

* fix run notification handler type

* fix worker group test again

* generate prisma client for e2e tests

* remove worker group tests for now

* prevent image pull rate limits during unit tests

* increase timeout for queue concurrency limit test

* generate prisma client for preview release

* same node types everywhere

* Updated engine readme, removed legacy system notes

* use default machine preset from platform package

* worker instances plural in schema

* disable pnpm update notifications

* return worker group details from connect call

* add workers admin route

* fix heartbeat route return type

* move deployment labels to core apps

* refactor run controller env schema

* Add firstAttemptStartedAt to TaskRun

* RunEngine 2.0 batch trigger support (#1581)

* Make it clear when BatchTriggerV2Service is used

* Copy of BatchTriggerV2Service

* WIP batch triggering

* Allow blocking a run with multiple waitpoints at once. Made it atomic

* Removed unused param

* New batch service

* Pass through the parentRunId and resumeParentOnCompletion

* Use the new batch service, and correct trigger task version

* Force V1 engine if using BatchTriggerV2Service, we’ve already done the check at this point

* Removed the $transaction and early exit if nothing changed

* Adedd a simple batch task to the hello world reference catalog

* Fix for batch waits not working

* Added parentRunId in a couple more places

* Removed waitForBatch log

* Added another parentRunId

* Expanded the example to include all the different triggers

* More changes to blocking to support continuing after idempotent completed runs

* Fix for the wrong type when blocking a run

* remove @map

* optimise worker auth query

* add engine version header to core api client requests

* remove unique constraint for default group id

* consolidate migrations

* the first managed worker becomes the global default

* Debug events off by default, added an admin toggle to show them

* worker group name can't be an empty string

* add exec helper to core

* move machine resources to core

* add pre-dequeue callback to determine max resources

* optionally skip dequeue

* bump worker package

* move worker to core

* fix ReadableStream type error

* fix another type issue

* update a few more tsconfigs

* add metadata changes introduced in #1563

* Run Engine 2.0 trigger idempotency (#1613)

* Return isCached from the trigger API endpoint

* Fix for the wrong type when blocking a run

* Render the idempotent run in the inspector

* Event repository for idempotency

* Debug events off by default, added an admin toggle to show them

* triggerAndWait idempotency span

* Some improvements to the reference idempotency task

* Removed the cached tracing from the SDK

* Server-side creating cached span

* Improved idempotency test task

* Create cached task spans in a better way

* Idempotency span support inc batch trigger

* Simplified how the spans are done, using more of the existing code

* Improved the idempotency test task

* Added Waitpoint Batch type, add to TaskRunWaitpoint with order

* Pass batch ids through to the run engine when triggering

* Added batchIndex

* Better batch support in the run engine

* Added settings to batch trigger service, before major overhaul

* Allow the longer run/batch ids in the filters

* Changed how batching works, includes breaking changes in CLI

* Removed batch idempotency because it gets put on the runs instead

* Added `runs` to the batch.retrieve call/API

* Set firstAttemptStartedAt when creating the first attempt

* Do nothing when receiving a BATCH waitpoint

* Some fixes in the new batch trigger service… mostly just passing missing optional params through

* Tweaked the idempotency test task for more situations

* Only block with a batch if it’s a batchTriggerAndWait… 🤦‍♂️

* Added another case to the idempotency test task: multiple of the same idempotencyKey in a single batch

* Support for the same run multiple times in the same batch

* Small tweaks

* Make sure to complete batches, even if they’re not andWait ones

* Export RunDuplicateIdempotencyKeyError from the run engine

* Latest lockfile

* Trigger with a machine (old run engine)

* RE2, allow setting machine when triggering

* Fix for new glob patterns

* add max run count to dequeue from version route

* add worker instance name env var and header

* queue consumer pre skip callback

* poll for more runs after final execution errors

* fix dequeue search param schema

* add shortcut to debug switch

* expose run engine timeouts as env vars

* make warm start durations configurable

* add optional status to json reply helper

* fix preSkip hook, add debug logs

* BLOCKED_BY_WAITPOINTS -> SUSPENDED

* exit controller when run suspended

* check if already replied before http reply

* run controller will wait for next run after the current one is suspended

* cancel run button shortcut

* minimal event repository environment type

* fix update metadata call

* run suspension and misc fixes wip

* change debug shortcut to shift + D

* Started work on the Dev supervisor

* Formatting

* Fix for bad imports

* Before rebuilding SSE

* Presence updating from the CLI working via SSE

* add worker notification debug logs

* send run:stop when exiting run phase

* skip current snapshot poll on worker notification

* add more logs and route to submit run debug logs

* add worker and runner ids to snapshots

* improve run notification debug logs

* add workload debug log route

* misc run controller fixes and refactor

* prevent parallel execution of critical functions

* update bun to 1.2.1

* WIP with dev dequeuing

* Method to convert friendlyIds to non-friendly, do nothing with actual ids

* Set the engine on BackgroundWorker, lazily upgrade projects to engine V2

* Runs with ttls were getting immediately expired… oops.

* Pass the Waiting for deploy reason through, so we have it on the execution snapshots

* Fixed the logic for getting the right background worker for a run

* Use the correct ID when dequeuing…

* determineEngineVersion is now fully functional

* Rate limiter ignores the dev endpoints

* Retrieving a batch gives you the runIds

* Set a unique version for the RE2 BatchTaskRun

* add provisional changeset

* The start of dev run execution is working

* First dev run working

* Moved the dev run controller closer to what Nick did with the managed one

* export exec output type

* Heartbeat fix: don’t heartbeat if _isHeartbeating == false

* Dev runs get notifications, some dev bug fixes

* Improved logging or dequeuing

* We need to dequeue runs from the latest version too, for triggerAndWait

* Ported Eric’s validateWorkerManifest with nicer errors

* When flattening an idempotency key if part is undefined, return undefined

* Dev logging fixes

* Remove sigterm listener

* Deprecating workers. Don’t specify a BackgroundWorker when dequeuing an environment

* Deleted some old files. Renamed “managed” to “deploy”

* When a build finishes, always copy the build dir (otherwise the first one gets trampled on by the 2nd)

* Dev master queues should work differently

* Deleting old workers

* Added debounce function to core

* Improvement to canceling

* WIP on debounce canceling on socket disconnection

* Added environment data to execution snapshots

* Dev runs that have stalled get “Canceled” with a reason explaining why

* Show CLI messaged when a connection to the platform is lost/restored

* Fix TriggerTask after merge

* Add trigger task v2 max attempts, replace some findUniques

* Port the new queue logic to the run engine

* More fixes post-merge

* We weren’t setting a `retryConfig` up for the tests… it’s now required

* Start the Redis worker inside the Run Engine… 🤦‍♂️

* Trying to make the testcontainers more reliable

* Added keyPrefix: "engine:”

* Badly placed bracket in trigger task

* Better Redis namespacing

* Fix for expired run not getting removed from the queue

* Don’t create a redis client in the testcontainers, return the redisOptions instead

* Cleanup redis client in the run lock tests

* Fix for the RunQueue not supporting keyPrefix

* Updated more of the RunQueue scripts rebalancing

* Trying to make Redis more robust in the tests…

* Improved test resiliciency more

* Fix for delays (checkpoint check)

* Increase the timeout slightly to fix ttl test

* Added priority support when triggering

* More wip trying to make test containers more reliable

* batchTriggerAndWait test is still failing… some wip to try fix it

* Fixed redis tests now we’re not providing a client

* Separate Redis clients for the run engine worker/queue/runlock

* Made the wait for duration test more resilient

* Added idempotencyKeyExpiresAt to Waitpoints

* Waitpoint timeouts and idempotency expiry

* Use finishWaitpoint, removed extra worker job

* Added waitpoint idempotency tests

* Creating resume tokens is working

* Some improvements to the resume tokens

* Moved resumeTokens to just be wait functions 🥳

* Delete old RuntimeManagers

* Wait for token is working

* Better test for the wait tokens

* Improved the test task some more

* Hide the accessories in the span inspector

* WIP on waitpoint inspector

* WIP on complete waitpoint form

* Span overview panel can be changed based on the entity type

* Improved the waitpoint display

* WIP on completing waitpoint form

* Use the existing CodeBlock for the tip

* Style improvements

* Complete waitpoint

* All waitpoint sidebar variants

* Waits now use a pause icon

* Durations waits use the API to create/block with a waitpoint, not the runtime

* Fix for engine.blockRunWithWaitpoint required org id

* Removed old wait code from the run controllers/task run process

* Form action for skipping a datetime waitpoint

* Move testDockerCheckpoint to a separate core package export (it can’t be bundled on the client)

* Fix for glitchy hourglass animation

* Completed waitpoints display better

* Increase Redis maxRetriesPerRequest to 20 (default)

* Completing and skipping waitpoints is working

* Remove the database prisma dev command, since we need to use create only now. Updated docs

* Added skip timeout, reworked the UI

* Tweaked spacing

* Added payload limit to waitpoint token completion from dashboard

* Test idempotency works on wait.for and wait.until

* Moved the worker-actions to /engine/ from /api/

* Moved dev engine endpoints to /engine/ from /api/

* Separate /engine/ rate limiter

* Added parallel wait prevention, it’s working for duration waits but not well for triggerAndWait yet

* WIP post-merge conflicts

* Set taskEventStore column in the new engine

* Remove duplicate keys

* Post-merge fixes

* Fix for span merge layout

* Use executedAt instead of firstAttemptStartedAt

---------

Co-authored-by: Matt Aitken <matt@mattaitken.com>
2025-03-05 14:40:14 +00:00
Eric Allam edf0c1024f Publish docker image when internal-packages change (#1750) 2025-03-04 11:18:29 +00:00
James Ritchie d6c869eb8f Docs/community packages (#1714)
* typo

* Adds feature list to the sveltekit plugin page

* Removed Migrations section in side menu

* Tidy up the table links

* Creates a community packages section

* Small tweak

* Adds info and links to community packages

* Updated mintlify broken links version in workflow file

* fixed broken link
2025-02-18 13:13:07 +00:00
Eric Allam a2c2d920b7 Use redis worker for run heartbeats and alerts (#1669)
* Move the task run heartbeats to RedisWorker

* Move alerts to redis worker, improving redis worker

* Fix typecheck errors

* Use single threaded tests for redis worker

* Enable/disable the redis workers independently

* Remove preview release from PR checks
2025-02-06 10:21:57 +00:00