Compare commits

...

822 Commits

Author SHA1 Message Date
github-actions[bot] 36ced50c04 Release 4.0.0-v4-beta.24
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 0s
🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker-v4 (push) Has been skipped
🚀 Publish Trigger.dev Docker / units (push) Failing after 4s
* chore: Update version for release (v4-beta)

* Release 4.0.0-v4-beta.24

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Eric Allam <eallam@icloud.com>
2025-07-22 10:42:59 +01:00
Eric Allam 684e02c8c8 fix: prevent large root/parent metadata updates from endlessly retrying (#2290)
* fixing metadata WIP

* WIP

* fix: prevent large root/parent metadata updates from endlessly retrying

* Fixed other calls to handleMetadataPacket
2025-07-21 23:04:44 +01:00
Matt Aitken 4b9f33082e AI run filtering (#2285)
* Queue in run table and filtering

* Debounce the filter changes

* Remove console log

* Added machine filtering

* Added version filtering

* Filter by version in the db

* Removed duplicate classes

* Version filtering hasFilters consistency

* Added queues and machines to the bulk action summary

* runs.list filtering for queue and machine

* Fix for machine errors

* Input field now has accessory instead of shortcut

* First experiments with the UI

* Got the fake filtering working

* AI filtering is working pretty well 

* Started working on tool calling

* Tool calling is working

* Styling progress

* Working on the error

* Errors work, improved the styling

* Nice glow effect

* Tweak the darkness of the text field

* Re-ordered the UI, set AI settings to use system prompt and telemetry

* Refactored to make it testable

* Added basic evals

* Better time inputs and evals

* Removed some code comments

* Remove unused useSearchParam change

* Tidy imports

* If no OpenAI API key send json back

* Fix for merge conflict with duplicate query filters

* Another conflict resolved

* Another merge conflict resolved

* Pass the model in, allow changing it
2025-07-21 17:53:43 +01:00
Matt Aitken a90b73c7ca Filter runs by queue, machine, version (#2277)
* Queue in run table and filtering

* Debounce the filter changes

* Remove console log

* Added machine filtering

* Added version filtering

* Filter by version in the db

* Removed duplicate classes

* Version filtering hasFilters consistency

* Added queues and machines to the bulk action summary

* runs.list filtering for queue and machine

* Fix for machine errors
2025-07-21 16:21:46 +01:00
Eric Allam ff018718f7 Added experimental_devProcessCwdInBuildDir config option (#2269)
* Added experimental_devProcessCwdInBuildDir config option

Added experimental_devProcessCwdInBuildDir config option to opt-in to new process.cwd behavior when executing tasks in the dev CLI. Currently process.cwd maps to the "root" of your trigger.dev project (the directory that contains your trigger.config.ts file). Setting experimental_devProcessCwdInBuildDir to true changes process.cwd to instead be the temporary build directory inside of the .trigger directory.

This makes it so the files added via the additionalFiles extension can be read using the same path in dev and deployed tasks.

* Remove claude code reference because it breaks windows tests 😡
2025-07-21 16:06:25 +01:00
Eric Allam 07980f8ebd fix: allow updating run metadata up to 1 hour after completion (#2288) 2025-07-21 16:05:29 +01:00
Eric Allam 8d5c86fea0 v4: simplified release concurrency system and status changes (#2284)
* WIP

* Make release concurrency system extremely simple, everything just releases all the time

* update the deadlock detection to use the new lockedQueueReleaseConcurrencyOnWaitpoint column

* WIP new release concurrency system

* Remove releaseConcurrency and releaseConcurrencyOnWaitpoint

Also removed deadlock detection, and added environment burst concurrency

* Added new DEQUEUED status

Cleaned up the API run statuses, including now detecting new clients and not breaking older clients by adding an API version header to all requests

* Introduce the new "current dequeued concurrency set"

* Remove QUEUED_EXECUTING because we no longer "eagerly" release before checkpointing

* Remove waitpoint test for QUEUED_EXECUTING

* Add isWaiting

* Add changeset

* Use createdAt for ordering realtime runs instead of number

* Clarify the envCurrentDequeuedKey usage

* mock the db.server file to fix the tests

* Updated changset "EXECUTED" -> "EXECUTING"

---------

Co-authored-by: Matt Aitken <matt@mattaitken.com>
2025-07-21 15:45:27 +01:00
Eric Allam a9b0b4f7c9 Add beta.23 changelog entry (#2286) 2025-07-21 14:34:59 +01:00
nicktrn 4dee2c360e fix(helm): allow electric db url var to be set from secret (#2282)
* correctly use db url from secret for electric

* bump app and chart version

* bump electric version
2025-07-21 13:57:23 +01:00
James Ritchie 4565f19d3a Copyable admin area table fields and autofocus search (#2280) 2025-07-18 18:48:56 +01:00
github-actions[bot] 0796df2aa9 Release 4.0.0-v4-beta.23
🚀 Publish Trigger.dev Docker / units (push) Failing after 0s
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 3s
🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker-v4 (push) Has been skipped
* Flatten undefined properties correctly (#2276)

* chore: Update version for release (v4-beta)

* Release 4.0.0-v4-beta.23

---------

Co-authored-by: Eric Allam <eric@trigger.dev>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Matt Aitken <matt@mattaitken.com>
2025-07-18 11:53:43 +01:00
Eric Allam 6bf580ac21 Flatten undefined properties correctly (#2276) 2025-07-17 18:15:57 +01:00
James Ritchie 3ad4b8b32d Add machine to run list (#2275)
* Access machinePreset from the run list presenter

* New icons for machine presets

* New icon + name combo label for the machine preset

* Adds new “Machine” column to the runs list

* Make a separate component for the machine tooltip info

* add machinePreset to the span presenter

* Show the Machine in the Details tab in the Run inspector

* Show an admin only separator

* Fix docs icon in the button

* Small padding tweak

* Move the Machine nearer the costs

* Don't cast the machine preset

* Remove typecast, better to have a bad label if we add a new machine and don't update thos

---------

Co-authored-by: Matt Aitken <matt@mattaitken.com>
2025-07-17 14:26:52 +01:00
Dan f685c6637d Added Mastra + Trigger example (#2278)
* Added mastra agent docs

* Added links in the ai agents section

* Caught typos all thanks to coderabbit. Thanks coderabbit
2025-07-17 13:58:59 +01:00
Dan 1dbcf661f3 Added openai agents sdk for typescript playground docs (#2194)
* Added openai agents sdk typescript playground docs and updated guardrails

* Removed broken link and import

* Typo fix
2025-07-17 13:39:18 +01:00
Matt Aitken 6f6852621d Bundle p-map (#2274) 2025-07-16 16:36:36 +01:00
nicktrn aa3a56a85d fix(docker): apply max old space size env var to webapp (#2273) 2025-07-16 16:36:05 +01:00
Matt Aitken 5e999c8992 Use p-map to speed up bulk actions (#2272) 2025-07-16 15:52:42 +01:00
Matt Aitken 569cecb675 The email counts should come from the completed bulk action… not this iteration (#2271) 2025-07-16 13:46:48 +01:00
Saadi Myftija c0b6c41afc Disable payload editing for run replays with large payloads (#2268)
* Use sexy scrollbars for templates and recent runs popover content

@Nick this is for you!!1

* Fix TabButton component disabled state

* Disable payload tab when replaying runs with large payloads
2025-07-15 21:54:43 +01:00
Matt Aitken 35c7a1ecb0 Bulk action 2 fixes (#2267)
* Fix for filtering by run id

Simplified to have a single filter for runId which accepts an array. This unifies the filtering by multiple runs (used by the waitpoint page) and the individual run filtering. It fixes a bug with bulk actions

* Truncate long bulk action titles
2025-07-15 14:43:19 +01:00
Saadi Myftija a64875a297 fix: regression affecting replayed runs with superjson payloads (#2266)
* Fix regression that broke superjson behavior for the replay flow

* Ignore undefined type metadata when overriding superjson payloads

Addresses https://github.com/triggerdotdev/trigger.dev/issues/1968

* Add a few unit tests for replaceSuperJsonPayload
2025-07-15 12:30:33 +01:00
Matthieu Mordrel 8690878a86 Updated generatedCopy to generatedCopy.text (#2247)
I believe the generatedCopy is the object that the AI SDK returns and we want to pass the text of the object rather in our template literal.
2025-07-14 15:58:55 +01:00
Saadi Myftija b78b3dceb2 feat: link the original run from replayed runs (#2262)
* Add ID of the replayedFrom run to the TaskRun schema

* Propagate the replayedFrom run ID in the replay flow

* Link the replayed run in the run details pane
2025-07-14 15:57:54 +01:00
Matt Aitken 630e9556b0 Bulk actions 2.0 (and switch all run listing to ClickHouse) (#2264)
* useSearchParams has

* useSearchParams has

* useSearchParams has

* Consistent way to get the run filters

* Consistent way to get the run filters

* Consistent way to get the run filters

* Initial work on the new bulk actions

* Initial work on the new bulk actions

* Initial work on the new bulk actions

* WIP actions and filtering

* WIP actions and filtering

* WIP actions and filtering

* Empty filter arrays are set to undefined

* Empty filter arrays are set to undefined

* Empty filter arrays are set to undefined

* WIP prisma schema

Removed extra runtimeEnvironmentId

* WIP prisma schema

Removed extra runtimeEnvironmentId

* WIP prisma schema

Removed extra runtimeEnvironmentId

* Migrations

* Migrations

* Migrations

* BulkActionGroup changed some columns around

* BulkActionGroup changed some columns around

* BulkActionGroup changed some columns around

* New badge variant, removed unused ones

* New badge variant, removed unused ones

* New badge variant, removed unused ones

* Bulk action button

* Bulk action button

* Bulk action button

* Make the next runs page the default now

* Make the next runs page the default now

* Make the next runs page the default now

* Improved the RadioButton style

* Improved the RadioButton style

* Improved the RadioButton style

* Remove the old bulk action bar

* Remove the old bulk action bar

* Remove the old bulk action bar

* More UI progress

* More UI progress

* More UI progress

* Lots of UI changes to the Runs page

* Lots of UI changes to the Runs page

* Lots of UI changes to the Runs page

* Fixed period filter resetting everything

* Fixed period filter resetting everything

* Fixed period filter resetting everything

* Improved the Switch secondary style

* Improved the Switch secondary style

* Improved the Switch secondary style

* Buggy filter fixes

* Buggy filter fixes

* Buggy filter fixes

* Improved the filter display and fixed a bug with search param from object

* Improved the filter display and fixed a bug with search param from object

* Improved the filter display and fixed a bug with search param from object

* Clear button is minimal

* Clear button is minimal

* Clear button is minimal

* Using a presenter now

* Using a presenter now

* Using a presenter now

* Bulk actions are created, but not actually processed (yet)

* Bulk actions are created, but not actually processed (yet)

* Bulk actions are created, but not actually processed (yet)

* Bulk replay/cancel is working

* Bulk replay/cancel is working

* Bulk replay/cancel is working

* Multiple fixes, added bulk column to PG

* Multiple fixes, added bulk column to PG

* Multiple fixes, added bulk column to PG

* Bulk action run filtering working using CH

* Bulk action run filtering working using CH

* Bulk action run filtering working using CH

* Replay setting the bulk id on the runs

* Replay setting the bulk id on the runs

* Replay setting the bulk id on the runs

* Properly cap the time when doing a bulk action

* Properly cap the time when doing a bulk action

* Properly cap the time when doing a bulk action

* If the bulk action isn't recent, add it to the dropdown anyway

* If the bulk action isn't recent, add it to the dropdown anyway

* If the bulk action isn't recent, add it to the dropdown anyway

* Blank version of the bulk actions page

* Blank version of the bulk actions page

* Blank version of the bulk actions page

* Individually selected runs working

* Individually selected runs working

* Individually selected runs working

* Use selected mode if runs are checked

* Use selected mode if runs are checked

* Use selected mode if runs are checked

* Added the modal

* Added the modal

* Added the modal

* Marked the old bulk actions stuff as deprecated

* Marked the old bulk actions stuff as deprecated

* Marked the old bulk actions stuff as deprecated

* Renamed bulk action file

* Renamed bulk action file

* Renamed bulk action file

* Bulk run filter with the name and a default

* Bulk run filter with the name and a default

* Bulk run filter with the name and a default

* WIP on bulk actions page

* WIP on bulk actions page

* WIP on bulk actions page

* Updated panel, added new truncated id component

* Updated panel, added new truncated id component

* Updated panel, added new truncated id component

* Style improvements to the radio buttons

* Style improvements to the radio buttons

* Style improvements to the radio buttons

* Added an option action completion email

* Added an option action completion email

* Added an option action completion email

* Adds a blank state for the bulk actions page

* Adds a blank state for the bulk actions page

* Adds a blank state for the bulk actions page

* Nicer completed email

* Nicer completed email

* Nicer completed email

* Don't open the bulk action panel if there are no runs

* Don't open the bulk action panel if there are no runs

* Don't open the bulk action panel if there are no runs

* Runs blank state and bulk action accordion

* Runs blank state and bulk action accordion

* Runs blank state and bulk action accordion

* Updates secondary/small switch style

* Updates secondary/small switch style

* Updates secondary/small switch style

* Pagination buttons no longer split in twain (WIP)

* Pagination buttons no longer split in twain (WIP)

* Pagination buttons no longer split in twain (WIP)

* Aborting working

* Aborting working

* Aborting working

* Bulk action live reloading

* Bulk action live reloading

* Bulk action live reloading

* ListPagination works correctly in all states

* ListPagination works correctly in all states

* ListPagination works correctly in all states

* Run page, show friendlyId instead of number

* Run page, show friendlyId instead of number

* Run page, show friendlyId instead of number

* Bulk action help open by default if you have none

* Bulk action help open by default if you have none

* Bulk action help open by default if you have none

* Extra status filtering step because of replication delay

* Extra status filtering step because of replication delay

* Extra status filtering step because of replication delay

* Wider bulk action onboarding

* Wider bulk action onboarding

* Wider bulk action onboarding

* More sensible widths on the bulk action side panel

* More sensible widths on the bulk action side panel

* More sensible widths on the bulk action side panel

* Border color tweak to the RadioButton

* Border color tweak to the RadioButton

* Border color tweak to the RadioButton

* Improved the accordion component hover states

* Improved the accordion component hover states

* Improved the accordion component hover states

* Updates the bulk action blank state images to the latest UI

* Updates the bulk action blank state images to the latest UI

* Updates the bulk action blank state images to the latest UI

* Added R and C shortcuts back in

* Added R and C shortcuts back in

* Added R and C shortcuts back in

* Fix for selecting a single run

* Fix for selecting a single run

* Fix for selecting a single run

* Improved exit icon, added shortcut to modal

* Improved exit icon, added shortcut to modal

* Improved exit icon, added shortcut to modal

* Tidy imports

* Tidy imports

* Tidy imports

* Tidy imports

* Tidy imports

* Tidy imports

* Tidy imports

* Tidy imports

* Tidy imports

* Tidy imports

* Fix for grid layout when 1 page of bulk actions visible

* Fix for grid layout when 1 page of bulk actions visible

* Fix for grid layout when 1 page of bulk actions visible

* Removed the ... on the abort button

* Removed the ... on the abort button

* Removed the ... on the abort button

* Removed the ... on the abort button

* Animate the progress bar

* Set TZ="UTC" in the env example

* Filter summary in the bulk inspector

* Improves the pagination styling

* Improves the pagination styling

* Delete old bulk action routes

* Removed old Postgres RunListPresenter

* Retry any replication error where the message contains "timeout"

* Increase wait to make test less flaky

* The test was using run id instead of friendly id

* Safer array access

* Remove error log if there's a bad status

* Nicer frontend type safety with the bulk action and mode

* Switched a log to a debug log

* Retry replication unless the error is a known non-retry error

Flip the strategy to retry by default

* Make ClickHouse required

* Backfill run replication admin API endpoint

* Set a CLICKHOUSE_URL for unit tests

---------

Co-authored-by: James Ritchie <james@trigger.dev>
2025-07-14 15:50:01 +01:00
Eric Allam 6db1b71202 fix: prevent stuck batchTriggerAndWait by finalizing runs when they are cancelled (#2265) 2025-07-14 14:29:14 +01:00
Eric Allam 70b4b1257e fix: Logging large objects is now much more performant and uses less memory (#2263)
* Logging large objects performance fixes

* Add changeset

* docs: new otel attribute count defaults

* Scope the switch/case statement with braces

* Add tests for nested map and sets

---------

Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
2025-07-14 13:16:04 +01:00
Saadi Myftija 87c21ab50a feat: enable overriding run options when replaying (#2260)
* Rearrange the layout of the replay modal

* Add run options to the replay modal

* Handle payload and metadata correctly in the shared json editor

* Apply run options in replays

* Fix clear button issue in json editor

Memoization of the clear function caused some problems. Removing it should not cause performance issues.

* Update replay modal hint

* Move machine and version fields to the top for visibility

* Use the same field ordering in the test page

* Reload queues and versions on env override

* Adapt json editor to fill full height

* Clean up a few excessive ternaries

* Avoid ui jump on env selection

* Switch to sexy scrollbars for scheduled tasks in the test page
2025-07-11 16:54:20 +02:00
Matt Aitken 4c635f72a2 Fixed queues SSE live reloading (#2259) 2025-07-11 12:53:28 +01:00
Eric Allam 46dad7dc76 fix: prevent processKeepAlive OOM error process reuse (#2261)
* Improve TaskRunProcess health detection so we don't try and reuse an unhealthy process

This was happening after the process was killed internally, like by an OOM error

* Add changeset
2025-07-11 10:50:05 +01:00
nicktrn 4fdf23b38b fix(deploy): fail in CI with descriptive message and instructions if no access token set (#2258)
* don't try interactive login in ci, link to docs

* add ci note to cli deploy docs

* add ci note to github actions

* add changeset

* make cron test less strict
2025-07-10 17:28:12 +01:00
nicktrn 0c71dc74df feat: add node 22 and bun runtime support with version display (#2254)
* update node-22 image

* update bun image

* disable io_uring

* fix fallback bun path

* prevent duplicate warnings

* add runtime and version to deployments

* runtime icons

* fallback to nodejs

* prevent empty table cell menu

* log if local build on deploy

* pass io_uring env var to child

* denormalize runtime and version, display on run details

* add changesets

* disable pr checks for changeset commits..

* add runtime data to deployed bg workers
2025-07-10 16:43:10 +01:00
Eric Allam 6ff0b923e1 New internal idempotency implementation for trigger and batch trigger (#2256)
* Introduce request idempotency to prevent duplicate triggers

* Implement request idempotency on trigger

* Use x-trigger-request-idempotency-key header instead

Plus cleanup shared logic in the routes

* Add changeset

* Oops, lets not hardcode a 408

* A couple of improvements
2025-07-10 16:42:32 +01:00
nicktrn 1a64013825 fix(helm): external config and secret support (#2245)
* fix postgres and redis external config, incl redis tls

* external secret support for postgres, redis, clickhouse, plus fixes

* s3: full existing secret support, external fixes

* bump chart version

* add subchart links

* update external config docs

* tidy up validation and helper
2025-07-10 10:08:00 +02:00
nicktrn f456f68f88 fix(docs): force latest buildx version in sample gh action (#2252) 2025-07-09 19:41:02 +02:00
Saadi Myftija 28b6be2491 feat: introduce run templates for reusing test run configs (#2253)
* Add new prisma model for task run templates

* Create run templates in a new service

* Add modal to create run templates in the test page

* Show templates list and apply values when selected

* Hide template creation time in the dropdown list, only show date

* Enable deleting run templates

* Show success toast on template creation

* Validate template label length

* Improve the template creation success indicator

* Use formAction consistently to differentiate submissions

* Type formAction for better editor support

* Prettify run template payload and metadata

* Add triggerSource, concurrencyKey and ttl to run templates
2025-07-09 19:09:25 +02:00
Saadi Myftija dbcf632b4e fix: add missing field descriptions in the test page (#2251)
* Switch to sexy scrollbars in the options pane

* Add missing field descriptions

* Add a run options link to the docs

* Make the copy for the field descriptions more concise

* Fix tooltip issue in the Select component
2025-07-09 16:25:06 +02:00
Saadi Myftija 3cfde48bfd feat: expose all run options in the test run page (#2227)
* Implement a new primitive UI component for picking durations

* Implement a new component to input run tags

* Expose all run options in the test run page

* Add subtle animations when adding/removing run tags in the test page

* Add a new resource endpoint for fetching queues

* Fetch usable queues for the selected task

* Fix width display issue in the select component

* Enable locking a run to a version from the test page

* Disable entering max attemps <0

* Validate tags

* Add recent runs popover

* Only show latest version for development environments

* Update run options when selecting a recent run

* Rearrange the test page layout

* Add subtle animation to the duration picker segments on focus

* Improve queue selection dropdown styling

* Fix disabled state issue for the SelectTrigger component

* Disable version selection field for dev envs

* Add usage hints next to the run option fields

* Add machine preset to the run options list

* Allow arbitrary queue inputs for v1 engine runs

* Show truncated run ID instead of run numbers for recent runs

Run numbers will soon get deprecated due to contention issues

* Fix duplicate queue issue

* Extract common elements across the standard and scheduled test task forms

* Apply values from recent runs to scheduled tasks too

* Add additional run options for scheduled tasks

* Use a slightly smaller font size for run option labels

* Disallow commas in the run tag input field

* Switch to a custom icon for recent runs button

* Flatten the load function test task result object

* Avoid redefining machine presets, use zod schema instead

* Fix ClockRotateLeftIcon jsx issues

* Remove recent runs button tooltip as it causes nesting errors

* Adjust the page layout to make it clear which task is currently selected

* Inline the tab group with the copy/clear buttons
2025-07-09 10:40:10 +01:00
Eric Allam ba2e0cc1c6 Prevent circular references from breaking run logs in task outputs (#2249) 2025-07-09 10:28:12 +01:00
Eric Allam b113e5a99c Fix non-MFA login (#2246)
* Fix non-MFA login

* fixed typecheck errors
2025-07-08 14:59:31 +01:00
Eric Allam b119a52e08 Implement MFA (#2244)
* Adds a new route for logging in with mfa

* New path for security page

* Adds “Security” link to account side menu

* Update the Switch component to allow label positions left and right

* Optionally hide the Close button in the Dialog title bar

* Installs `qrcode` react package for generating QR codes.

* CopyButton component now takes children

* New Security route for setting up MFA

* Adds new OTP package for the chadcn InputOTP component

* Adds new InputOTP chadcn component

* Adds InputOTP chadcn component to the MFA login screen

* InputOTP component supports variant styles

* Improvements to form handling

* Show a confirmation modal before you can disable MFA

* Revert redirect back to the dashboard for now

* Implement MFA enabling and disabling

* Refactor and cleanup mfa management code

* More cleanup

* Handle errors in the management action

* Implement mfa login flow

* recovery code input should be password

* Implement rate limiting on the mfa validation endpoint

* Better error ux

* Implement mfa emails and apply James' updates

* Use latest @better-auth/utils

* Improvements via CodeRabbit review

---------

Co-authored-by: James Ritchie <james@trigger.dev>
2025-07-08 13:44:47 +01:00
nicktrn e02c90a4ea Update self hosting images to beta.22 (#2241)
* docker: set new cli tag env var

* helm: update app and chart versions
2025-07-08 09:33:27 +01:00
Eric Allam 493ca11c43 Add ability to disable the prisma query in the healthcheck endpoint (#2242) 2025-07-07 16:41:55 +01:00
Eric Allam b36d81a622 Add v4 changelog to v4 docs (#2240)
* Add v4-beta changelog docs

* Add beta.22 changelog entry
2025-07-06 11:38:10 +01:00
Eric Allam 275ce9f290 Release 4.0.0-v4-beta.22 (#2238)
🚀 Publish Trigger.dev Docker / units (push) Failing after 0s
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 4s
🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker-v4 (push) Has been skipped
2025-07-04 16:15:15 +01:00
github-actions[bot] 5cff5c45e9 chore: Update version for release (v4-beta) (#2159)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2025-07-04 16:13:20 +01:00
Eric Allam 47e583c19d Properly scope the batch list to the selected environment (#2236)
Prevents showing batch runs from environments that are not in the selected environment
2025-07-04 14:41:10 +01:00
nicktrn 1019c9cdea Switch trigger Helm chart RBAC from cluster to namespace scope and bump version (#2232)
* replace supervisor cluster role with ns-scoped role

* fix cross-ns binding

* bump chart version
2025-07-04 13:54:25 +01:00
Eric Allam 469a8df0cf Prevent redis worker from enqueuing pending jobs (#2235) 2025-07-04 13:53:32 +01:00
Eric Allam 922c7cb3a6 Increase batch trigger processing job visibility timeout (#2234) 2025-07-04 13:43:26 +01:00
nicktrn 0fcb9705ae Document new otel limits (#2196)
* update webapp env vars

* update limits
2025-07-04 13:41:54 +01:00
nicktrn 74fdfa93c5 Show project details when running the whoami command (#2231)
* add project details to whoami

* findFirst

* skip project check if no orgs found

* add changeset
2025-07-04 13:40:41 +01:00
Eric Allam bdaa2ed15a Move batch trigger processing jobs to a dedicated redis worker (#2233) 2025-07-04 12:16:05 +01:00
nicktrn afc9a6ade1 Add ECR support for remote builds (#2224)
* create repo if doesn't exist

* fresh auth token for each deploy

* optional assume role

* log when machine overrides enabled

* make test repo namespace configurable

* assume role fix and env var changes

* improve ecr check

* improve tag parsing

* tag parsing tests

* track if repo created and fix test

* missing tryCatch for sts call
2025-07-03 14:25:30 +01:00
Eric Allam 7bb7e7aedc When sharding, use the where clause in the shard key to distribute requests more evenly (#2229) 2025-07-03 14:21:58 +01:00
Eric Allam e90235b317 fix processKeepAlive disabled performance regression (#2228)
* processKeepAlive disabled performance fix

When processKeepAlive was disabled, the eagerly created TaskRunProcess being discarded and a whole new TaskRunProcess created at execution time.

* Don't use a TaskRunProcess if it's being killed, fixes retryImmediately
2025-07-03 14:21:49 +01:00
James Ritchie b7134dd769 Docs: Adds 2 references to the node.js versions we use for v3 and v4 (#2118)
* Adds 2 snippets to reference node.js version for v3 and v4

* Updated the node versions we support

* Remove specific node version from title
2025-07-02 13:08:49 +01:00
nicktrn 6fea802505 helm: set default webapp container and add extraVolumes (#2222)
* set default webapp container

* bump version

* add webapp extraVolumes and extraVolumeMounts

* bump version
2025-07-02 13:03:02 +01:00
Eric Allam 09ef7625ff Merge pull request #2219 from triggerdotdev/metadata-serialization
Serialize metadata to prevent invalid data from breaking run completions
2025-07-02 12:49:08 +01:00
Eric Allam 5cf37e9f8a Merge pull request #2189 from triggerdotdev/increased-and-configurable-attribute-limits
Increase otel attribute limits and make them configurable
2025-07-02 12:48:38 +01:00
Eric Allam e7b6c2fab6 Merge pull request #2183 from triggerdotdev/process-keep-alive
v4: Experimental process keep alive
2025-07-02 12:39:43 +01:00
nicktrn d619e79f74 helm: increase default postgres resource preset (#2221)
* switch to small preset for postgres

* lint

* bump version
2025-07-02 11:57:14 +01:00
Eric Allam 38a94b3b70 Add changeset 2025-07-02 11:30:13 +01:00
Eric Allam 0c2af6d87c Increase otel attribute limits and make them configurable 2025-07-02 11:30:13 +01:00
Eric Allam 0deb2c640d remove change from process-keep-alive 2025-07-02 11:30:13 +01:00
Eric Allam 21c70ad673 Serialize metadata to prevent invalid data from breaking run completions 2025-07-02 11:30:13 +01:00
Eric Allam 0793c2b0f4 address some nick comments in the review 2025-07-02 11:30:13 +01:00
Eric Allam 8dc90d61b4 increase run engine test timeouts 2025-07-02 11:30:13 +01:00
Eric Allam 57f3a89a86 Fix timeline span events tests 2025-07-02 11:30:13 +01:00
Eric Allam ceb737e9f1 Fix broken webapp test 2025-07-02 11:30:13 +01:00
Eric Allam 632d6b91e9 Fix taskSlug extraction 2025-07-02 11:30:13 +01:00
Eric Allam b548561220 Remvoe the sometimes incorrect import file name 2025-07-02 11:30:13 +01:00
Eric Allam aecee4d35f Don't reset task lifecycle hooks between executions 2025-07-02 11:30:13 +01:00
Eric Allam 9ab590bead Better handle snapshot changes with the new persistent task run process,
Also don’t re-import the task file if the task is already in the resourceCatalog
2025-07-02 11:30:13 +01:00
Eric Allam 2cc1453f25 Always set the cold/warm variant on attempt spans, even in dev 2025-07-02 11:30:13 +01:00
Eric Allam 82f3db6825 Only bootstrap on the cold start execution, and record how many executions in the attempt span 2025-07-02 11:30:13 +01:00
Eric Allam f0bc4ad2bd Don't delete env vars with falsy values 2025-07-02 11:30:13 +01:00
Eric Allam 4d15c8f7cf Better reset in the prod usage manager 2025-07-02 11:30:13 +01:00
Eric Allam 7f3b1c1735 Ensure the usage interval is cleared when resetting 2025-07-02 11:30:13 +01:00
Eric Allam 707bc09dac Make sure maxExecutionsPerProcess is a positive integer 2025-07-02 11:30:13 +01:00
Eric Allam a0f2c982f9 Make sure to clear existing metadata flush timer when resetting 2025-07-02 11:30:13 +01:00
Eric Allam bffd18a6b8 Properly reset listener arrays by setting length to 0 2025-07-02 11:30:13 +01:00
Eric Allam 2018b41d22 TRIGGER -> TRIGGERDOTDEV 2025-07-02 11:30:13 +01:00
Eric Allam c41d63497b Handle immediate retries better 2025-07-02 11:30:13 +01:00
Eric Allam 56146afcb6 TaskRunProcessProvider cleanup is async 2025-07-02 11:30:13 +01:00
Eric Allam 020565c009 Handle dev process pool shutdown errors 2025-07-02 11:30:13 +01:00
Eric Allam 58d56f8381 Use hostname instead of host 2025-07-02 11:30:13 +01:00
Eric Allam 5b49b420cb Add changeset 2025-07-02 11:30:13 +01:00
Eric Allam 3fb68b784a couple of tweaks for testing stuff 2025-07-02 11:30:13 +01:00
Eric Allam a414e65efe handle env vars that have been removed between executions 2025-07-02 11:30:13 +01:00
Eric Allam 38fe854559 Make refreshing a task run process work correctly 2025-07-02 11:30:13 +01:00
Eric Allam c24f3a56cc Reset deployed run worker and also get the warm start stuff to work properly 2025-07-02 11:30:13 +01:00
Eric Allam 8798352be3 Super-warm starts in deployed tasks proof 2025-07-02 11:30:13 +01:00
Eric Allam 1a1e70ae3d WIP deployed processKeepAlive 2025-07-02 11:30:13 +01:00
Eric Allam 19bdf3b6ea scope the pool by background version, allow configuring pool params in dev/prod 2025-07-02 11:30:13 +01:00
Eric Allam 427c450391 Fixed metrics, reusing a process counts as a warm start 2025-07-02 11:30:13 +01:00
Eric Allam 9374aea401 Split resource and span attributes 2025-07-02 11:30:13 +01:00
Eric Allam fd3db5f77b Reset execution environment between runs 2025-07-02 11:30:13 +01:00
Eric Allam e2049fcfbf Initial dev process keep alive. moved resource attributes to span attributes because resource attributes are global and immutable 2025-07-02 11:30:13 +01:00
Eric Allam 1be4fcbb82 Fixes the jitter option for the scanConcurrencySets cron job (#2220) 2025-07-02 11:29:54 +01:00
Eric Allam 124435e5e7 Eliminate race conditions in continueRunIfUnblocked to prevent stuck runs (#2217)
* Eliminate race conditions in continueRunIfUnblocked to prevent stuck runs

* Use assertNever for typecheck friendliness
2025-07-01 16:34:12 +01:00
Matt Aitken 4a79b3b35b Fixed retrying on run replication inserts on connection errors (#2216) 2025-07-01 13:35:13 +01:00
Boon Kai 0da6a19987 docs: Update triggering.mdx import for triggering from inside another task (#2215)
When triggering from inside another task, `runs` and `batch` should be imported from `@trigger.dev/sdk/v3`
2025-07-01 08:03:02 +01:00
Eric Allam 08d84eb596 Fix v4 concurrency sweeper flaky test (#2213) 2025-06-30 16:37:52 +01:00
Matt Aitken f38d35e9f6 ClickHouse replication improvements (retrying, strip bad unicode chars) (#2205)
* Add retry logic for insert operations

Add a generic retry mechanism for task run and payload inserts to handle
transient connection errors. The new #insertWithRetry method retries up to
three times with exponential backoff and jitter on retryable connection
errors such as connection resets or timeouts. Errors are logged and
recorded in tracing spans to improve observability and robustness of the
replication service.

* Replication settings are configurable

* Log out the runIds for failed batches

* Detecting bad JSON in run replication and ignoring it

* Reproduced split unicode error

* Move output file

* Massively improved the performance

* Minor performance improvements

* Unskip tests

* Remove unused test in CH package

* Fix for the ClickHouse UI explorer

* RunReplication keepAlive defaults to false

* Add concurrency_key and bulk_action_group_ids to ClickHouse task runs

* ClickHouse package doesn't need to be built anymore for the webapp

* Set the concurrency_key from the run replication service
2025-06-30 16:32:02 +01:00
Eric Allam 1b4dabb8ba v4: current concurrency sweeper (#2206)
* v4: current concurrency sweeper

* Fix webapp tests

* Ensure only a single instance performs concurrency sweeping by using redis-worker cron jobs

* Improved the mark phase

* Ensure cron jobs get rescheduled even if the handler throws an error

* Better property names
2025-06-30 16:22:19 +01:00
nicktrn aa24a9aae6 Fix external registry config and other helm tweaks (#2212)
* fix both internal and external registry host

* add next steps to self-hosting overview

* separate into webapp and registry ingress

* fix external registry config check

* fix localhost registry warning

* bump chart version
2025-06-30 14:21:00 +01:00
nicktrn dc568e7f3e Self-hosting: improved worker and bucket bootstrap (#2209) 2025-06-29 10:42:52 +01:00
nicktrn fa6675eb00 Helm chart improvements (#2207)
* add helm deps

* configurablecli tag for setup commands

* enabled -> deploy

* extraEnv -> extraEnvVars

* update docs

* minio -> s3

* improve persistence config

* set cli tag to v4-beta

* chart and image bump

* fix tests

* lock chart

* workflow hotfix

* quote for niels

* make probes configurable

* fix webapp and supervisor templates

* bump chart
2025-06-28 12:59:29 +01:00
Eric Allam 8a86e7b41d Remove deprecated schedule trigger jobs when registering with the schedule engine (#2204)
* Remove deprecated schedule trigger jobs when registering with the schedule engine

* Use a tryCatch just to be absolutely sure
2025-06-27 12:32:02 +01:00
nicktrn 119bfc0068 Fix update command version detection (#2199)
* fix root package.json resolution

* add changeset
2025-06-27 10:06:36 +01:00
nicktrn 7a6d9dc225 v4 self-hosting docs updates (#2202)
* quote version strings

* Update architecture diagram to include kubernetes

---------

Co-authored-by: James Ritchie <james@trigger.dev>
2025-06-26 16:53:10 +01:00
Eric Allam 0f3179f493 Provide realtime skipColumns option via untamperable public access tokens (#2201)
* Provide realtime skipColumns option via untamperable public access tokens

* Fixed realtime client tests

* Create red-rings-marry.md
2025-06-26 15:13:26 +01:00
Eric Allam b87e1c4630 Improve clickhouse client debug logging (#2197)
* Improve clickhouse client debug logging

* Stop hardcoding ScheduleEngineWorker log level

* Add DLQ debug log message
2025-06-26 15:12:54 +01:00
Eric Allam 14058d557f fix: queues no longer unpaused on deploy (#2198) 2025-06-26 15:08:52 +01:00
nicktrn 6535cf918b v4 helm chart and docs (#2195)
* v4 helm chart

* ignore packaged chart archives

* chart publish prep

* add helm chart release workflow

* enable draft gh release

* stateful services should be statefulsets

* expose internal otel config

* podAnnotations for all services

* postgresql.extraArgs

* postgresql -> postgres

* add clickhouse vars as extraEnv examples

* support external secrets

* improve clickhouse config

improve clickhouse config

* disable internal otel config by default

* allow nil internal config

* fix gh release permission

* fix validation step for ci

* secrets.enabled

* simplify secret keys

* bump chart version

* fix contents permission

* remove redundant print steps

* remove helm release heading

* optimise clickhouse config

* support wait-for-it.sh

* fix ch overrides

* add low-resource ch overrides to compose setup

* supervisor bootstrap config

* disable otel trace logging

* retain shared volume

* enable clickhouse for compose setup

* disable trace logging

* bump chart

* add extraManifests

* telemetry config

* add repo namespace config

* scope cluster role to namespace to support multiple installs

* external s3 config

* values tweaks

* helm docs

* bump chart

* set contents read permission

Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>

* add object store root credentials warning

* run init containers as uid 1000

* pass ch user creds with explicit flag

* improve ingress anotation handling

* remove duplicate colume claim block

* trim htpasswd auth

* lock curl images used for tests

* add v4-beta package version warning

* make schema and sslmode configurable

* package tag info -> warning

* improve secrets config example

* update docs

* bump version

* bump webapp reqs to account for ch

* make ch creds more easily configurable

* publish ch http port

* example point at latest beta chart

* small docs tweaks

---------

Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
2025-06-25 23:57:51 +01:00
Eric Allam aaf40c3be3 Fix "Duplicate path found during parsing JSON object" clickhouse error (#2193) 2025-06-24 14:34:32 +01:00
James Ritchie 35dedee1eb Update number of allowed tags (#2187) 2025-06-24 10:38:59 +01:00
Eric Allam 485f71c8d4 Improve schedule engine performance (#2191)
* Improve schedule engine performance

* Convert an info log to a debug log
2025-06-23 16:55:18 +01:00
nicktrn ec2175fec8 Some docs tweaks (#2190)
* add v4 tag to playwright extension

* add self-hosting architecture section

* add v4-beta tag warning
2025-06-23 10:25:07 +01:00
Eric Allam fb9abe5289 fix: Incorrect/missing logs on new schedule engine runs (#2185)
* Fix incorrect logs on new schedule engine triggered taskss

Also added the ability to recover schedules in the schedule engine via an Admin API endpoint in the new schedule engine

* Fixed schedule recovery failing test
2025-06-19 15:37:50 +01:00
Eric Allam f2db1b8aa5 Create dedicated alerts worker (#2184)
* Add a new alerts worker that just handle alerts

* fallback to graphile worker when a tx is required
2025-06-19 11:54:49 +01:00
nicktrn 3c93783e8b remove nodetype label requirement (#2182) 2025-06-18 11:34:35 +01:00
nicktrn 55f41a2168 Ensure webapp container does not fetch pnpm at runtime (#2181)
* install pnpm during build

* install pnpm for node user as well
2025-06-18 11:31:36 +01:00
Matt Aitken 4ca6506655 Delay reschedule fix (v3 & v4) (#2180)
* Fix for v3 rescheduling bug…

* v4 rescheduling working
2025-06-17 18:22:15 +01:00
Matt Aitken 9f9e74b0b6 Fix hydration error (#2179)
* Remove unused hook

* The Impersonation button was causing DOM hydration issues
2025-06-17 11:17:17 +01:00
Eric Allam ea6f66bdfc Improved schedule engine (#2174) 2025-06-17 06:48:05 +01:00
Eric Allam a42c9acf44 v4: Improved run locking (#2173) 2025-06-17 06:45:59 +01:00
nicktrn f7f8bc28c5 Docker self-hosting improvements (#2177)
* shallow clone

* fix image tag locking example

* electric should use db url env var

* add htpasswd note

* use local driver with log rotation by default

* make worker urls more easily configurable

* configure dev otel endpoint via .env

* increase min recommended worker specs

* move worker url section
2025-06-16 22:22:02 +01:00
Matt Aitken 5f6fd8e235 Tasks page didn't include Delayed runs (#2176)
* Delayed runs weren’t shown in the Tasks page mini graph

* The Tasks page Queued count didn’t include Delayed runs (since ClickHouse switchover)
2025-06-16 14:14:09 +01:00
James Ritchie 33e1008d83 Adds a docker error entry to common problems docs page (#2172) 2025-06-14 11:11:24 +01:00
Matt Aitken a0815c8e3e Clickhouse performance improvements (#2175)
* Runs filter by org id and add created at to ordering

* CopyableText can accept an alternative value for copying

* The runs table now shows the ID instead of run number

* Paginating back/forwards fix

* The task stats need org id and project id too
2025-06-14 11:10:28 +01:00
James Ritchie 8a5e7f34b1 Added more limits and information to the limits page (#2171) 2025-06-13 10:32:44 +01:00
Matt Aitken c3dca5835d Fix for the Kapa widget not working with SSR (#2170)
* Move all the logic into a single component, preparing for client only

* Use ClientOnly

* Fix for search param not working
2025-06-12 22:17:22 +01:00
James Ritchie 20a8e7c6c1 Updated kapa sdk to 0.1.3 (#2169) 2025-06-12 16:21:34 +01:00
Eric Allam cc5514e57b Add status-first TaskRun index to speed up uncommon status filters (#2168) 2025-06-12 15:02:40 +01:00
Eric Allam ab432a2d7d Fix realtime re-subscribing stale data issue (#2162)
* Fix realtime re-subscribing stale data issue

Fixes an issue with realtime when re-subscribing to a run, that would temporarily display stale data and the changes. Now when re-subscribing to a run only the latest changes will be vended

* removed logs
2025-06-12 14:58:01 +01:00
James Ritchie e12c82b444 Use the Kapa AI SDK instead of the Kapa AI widget (#2113)
* Install the kapa sdk

* WIP using the SDK for the Kapa Ask AI widget

* Removes old kapa from root

* Now rendering everything inside the dialog component

* Fixes min-height of dialog content

* Remove kapa from root

* prevents kapa using reCaptcha

* Adds more functionailty with temporary UI placement for now

* Reset conversation button

* Adds a new sparkle list icon

* Adds some example questions as a blank state

* Animate in the example questions

* use “marked” package to render markdown

* Improve some animations

* Submit a question from the URL param

* adds custom scroll bar styling

* fixes modal to correct height after re-opening it

* Add button to stop generating answer mid-stream

* Adds buttons states to show submitting, generating, submittable

* Adds a helpfull sentence in the blank state

* Show a message if the chat returns an error

* Adds reset chat and feedback buttons to the bottom of an answer

* Makes sure you can give feedback in the different states of chat

* Adds a suble background to the dialog

* Fix a button inside button error

* Improve the shortcut esc key on dialog and sheet component

* Fix classname error

* organize imports

* Use our custom focus-visible

* Move the Tooltip for the button into the AskAI component

* Improved error message

* Organize imports

* Animated the modal gradient

* Small layout improvements

* Adds most asked questions from Kapa

* border glow tweak

* AskAI component is now a hook that can take a question

* remove kapa script

* Add a delay before the modal opens when usign the URL params

* Remove old component

* Update to the latest Kapa version

* Rephrased error message

* Use correct types for conversation

* Fixed types for addFeedback

* Adds DOMPurify package

* removed unused const

* Removed unnecessary platform specification

* Reset the timeout when the ai panel pops up

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>

* Fix for coderabbit bad commit

* Clean up imports

---------

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
2025-06-12 14:09:03 +01:00
Dan 808c299008 Added deep research example (#2166)
* Added docs

* Improved docs and added video

* Added upgrade badge
2025-06-11 16:19:25 +01:00
Eric Allam 498b9a21af Improved TaskRun environment indexes (#2164)
### PR: Optimize **TaskRun** indexes for hot-path queries

**What changed**

| Object                          | Type                                                            | Purpose                                                                                                    |
| ------------------------------- | --------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- |
| `taskrun_runtime_id_desc_idx`   | **BTREE** `(runtimeEnvironmentId, id DESC) INCLUDE (createdAt)` | Eliminates explicit sort for the “latest task runs” query (`ORDER BY id DESC`) while remaining index-only. |
| `taskrun_runtime_createdat_idx` | **BTREE** `(runtimeEnvironmentId, createdAt DESC) INCLUDE (id)` | Accelerates the filter-only path that scans by `createdAt >= …` without any ordering requirement.          |
| `taskrun_createdat_brin`        | **BRIN** on `createdAt` (`pages_per_range = 128`)               | Lets the planner skip whole blocks older than the time window for both queries at < 100 MB cost.           |
| *(cleanup)*                     | **DROP** `TaskRun_runtimeEnvironmentId_createdAt_id_idx`        | Retires the 3-column index once the new ones are built.                                                    |

**Key details**

* All indexes created **CONCURRENTLY** to avoid write blocking.
* `fillfactor = 90` on b-trees for balanced space vs. future growth.
* Net disk usage drops **≈ 15–20 GB** while each query now gets a purpose-built access path.

**Why**

* Remove planner Sort nodes for the top-N “latest runs” view.
* Speed up environment-filtered range scans.
* Shrink index bloat and improve cache efficiency.
2025-06-11 09:24:11 +01:00
Eric Allam 48e1e7250b v4: fix stuck batch issue when processing happens before transaction commits (#2163)
* v4: fix stuck batch issue when processing happens before transaction commits

* Fix flaky test
2025-06-10 17:02:34 +01:00
Eric Allam 3f2b748a80 Delete the secure param before passing it to the ClickHouse client (#2161) 2025-06-10 13:55:19 +01:00
Eric Allam a060ceef0b Fix clickhouse migrations by adding the secure=true query param (#2160) 2025-06-10 13:45:24 +01:00
Eric Allam e6fb32194e Add environment filter to declarative schedules (#2148) 2025-06-10 12:11:11 +01:00
Eric Allam b38405cb88 Realtime and task run performance improvements (#2158)
* Add createdAt filter to realtime subscribing with tags

* Filter realtime colums and expose ability to skip some columns

* Add sharding support for electric

* Use unkey cache for the created at filter caching

* Remove 2 unused indexes on TaskRun

* Run list now filters by a single runtime environment

* Remove project ID indexes

* Use clickhouse in task list aggregation queries instead of pg (keep pg for self-hosters)

* WIP clickhouse powered runs list
stuff

* Improve the query to get the latest tasks for the task list presenter

* Update the usage task list to use clickhouse

* Implement next runs list powered by clickhouse

* Add new index for TaskRun for the runs list, by environment ID

* Add runTags gin index

* Handle possibly malicious inputs

* Ignore claude settings

* Better handling not finding an environment on the schedule page

* Use ms since epoch in test, not seconds

* Remove unused function

* Fix test

* Use an env var for the realtime maximum createdAt filter duration (defaults to 1 day)

* Fixed the query builder to correct the group by / order by order

* Make sure runs.list still works

* Create small-birds-arrive.md
2025-06-10 12:11:01 +01:00
James Ritchie e617c14456 Adds a troubleshooting entry for react.email not being bundled correctly (#2117) 2025-06-09 16:43:32 +01:00
James Ritchie 15fd731f3d Fixes embeded video width (#2121) 2025-06-09 16:40:59 +01:00
Eric Allam 4c7742c258 Enable docs AI contextual menu (#2157) 2025-06-09 10:04:56 +01:00
nicktrn e72820c080 Docs improvements for v4 self hosting (#2156)
* add switch command

* couple of changes
2025-06-07 19:51:47 +01:00
nicktrn 7a34c1102b Feat: v4 self-hosting (#2155)
* self-hosting stuff goes in /hosting

* add v4 tags

* add main compose file

* draft overview

* add webapp env vars

* overview tweaks

* add supervisor env vars

* move old docker guide

* new sidebar structure

* update github actions docs

* docker draft

* use env vars for s3 creds

* this might just work

* split into multiple files

* update guide

* document machine overrides

* split legacy docs into different section

* some fixes

* some tweaks

* add login and init instructions

* don't cursorignore .env.example
2025-06-07 00:57:47 +01:00
Eric Allam 2b3d54aff5 Make removing the httpEndpointEnvironmentId column migration correctly idempotent (#2154) 2025-06-06 11:52:14 +01:00
github-actions[bot] bb1f0a711a Release 4.0.0-v4-beta.21 (#2149)
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 4s
🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker-v4 (push) Has been skipped
🚀 Publish Trigger.dev Docker / units (push) Failing after 3s
* chore: Update version for release (v4-beta)

* Release 4.0.0-v4-beta.21

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
2025-06-06 10:50:28 +01:00
nicktrn 47b19a49d4 A few small v4 fixes (#2153)
* remove unused env vars

* actually use ALERT_REPLY_TO_EMAIL for alerts (non-breaking)

* increase fallback branch limit to 100M

* more unused env vars and example

* hide usage page when self-hosted

* fix for init flow

* set default or concurrency to 300 to match 100 on env

* fail fast when registry env vars are empty strings
2025-06-06 10:15:07 +01:00
Eric Allam 5301239e09 chore: remove v2 models from schema (#2107)
* Round 1 of v2 model removals

* Round 2

* Remove trigger http endpoint env

* Round 3

* Round 4

* Round 5

* Round 6

* Round 7

* Round 8

* Round 9

* Round 10

* Removed the remainder v2 code from the webapp

* GitButler WIP Commit

---------

Co-authored-by: GitButler <gitbutler@gitbutler.com>
2025-06-05 22:57:29 +01:00
Eric Allam 261dce102a v4: fix batchTriggerAndWait completion issues by processing batch chunks sequentially (#2152)
* v4: fix batchTriggerAndWait completion issues by processing batch chunks sequentially

* Remove unnecessary test
2025-06-05 17:56:56 +01:00
Eric Allam 575413c414 fix: moved dequeuing from a worker queue out of lua to be able to use blpop (#2151) 2025-06-04 22:04:30 +01:00
nicktrn e7795a06ad Fix: fixes and prerequisites for v4 self-hosting (#2150)
* remove pgadmin

* remove V3_ENABLED

* v3 is always enabled

* enfore docker machine presets by default

* rename autoremove env var

* prefix more k8s-specific env vars

* same prefix for all docker settings

* improve profile switcher copy

* supervisor can load token from file

* optional webapp worker group bootstrap

* fix error message

* fix app origin fallback for otlp endpoint

* use pnpm cache for webapp docker builds

* increase default org and env concurrency limit to 100

* optional machine preset overrides

* improve s3 pre-signing errors

* fix DOCKER_ENFORCE_MACHINE_PRESETS bool coercion

* shard unit tests

* fix for s3-compatible services

* optional object store region

* Update apps/supervisor/src/workerToken.ts

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>

* fix DEPLOY_REGISTRY_HOST example

* fix platform mock

* remove remaining v3Enabled refs

* fix error type.. bad bot

---------

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
2025-06-04 18:02:38 +01:00
Eric Allam 2b3ea692fe v4: dequeue performance improvements (split concurrency from dequeue) (#2127)
* WIP

* Run queue now works with the worker queue / master queue split

* Acking should also cause the master queue to be processed

* Convert run engine tests and run engine to use runQueue changes

* Include the util files in the test tsconfig

* coordinator target should be es2020 as well

* providers target 2020

* Fix the triggerTask tests in the webapp

* v4 now working with the new worker queues, and added the legacy master queue migration stuff

* report worker queue lengths via opentelemetry metrics

* Adding lock metrics

* Release concurrency bucket metrics

* • Updated RunQueue.removeEnvironmentQueuesFromMasterQueue() method signature to take runtimeEnvironmentId instead of masterQueue parameter
• Added automatic master queue shard calculation using this.keys.masterQueueKeyForEnvironment(runtimeEnvironmentId, this.shardCount) 
• Updated RunEngine wrapper method to use new runtimeEnvironmentId parameter
• Updated DeleteProjectService to call the method once per environment instead of once per master queue
• Simplified API by encapsulating master queue sharding logic within RunQueue class

* metrics now working, configure the run queue settings, additional metrics for run engine and redis-worker

* Fix CodeRabbit suggestions

* return undefined from dequeueFromWorkerQueue, not null

* Remove message from worker queue in certain circumstances when acking

* Update log

* Ensure master queue consumers cannot stop from a processing error, and make the consumer interval configurable via an env var

* Change how the run queue master queue consumers are disabled internally

* Fixed tests

* process the queue on nack

* Fix more tests

* Fix priority tests

* Fixed dequeueing test
2025-06-04 17:18:46 +01:00
nicktrn f603725393 Feat: unified deploys for self-hosted and cloud users incl. multi-platform support (#2138)
* remove registry proxy

* remove --self-hosted flag

* automatically set network build flag

* update syncEnvVars debug log

* improve switch command

* always display deploy errors if they exist

* fix stuck deploy command after finalize error

* webapp-driven deploys, multi-platform support, lots of fixes

* add worker deployment migration

* rename image platform env var

* only try to sync parent env vars for preview deployments

* add KEEP_TMP_DIRS

* supervisor: docker api version lock, auth, multi-platform

* set image ref on create, validate digest

* use metadata for digest, fix local multi-platform builds

* print git meta branch before commit

* improve push and load flag handling

* make runs after local builds compatible with load and push

* small improvement for platform overrides

* add image platform to dequeued message

* remove deprecated init request body fields

* fix fail deployment id param

* remove build debug logs

* pass report merge with no tests

* structured run debug logs

* add required env var for tests

* should not be an error log

* add changeset
2025-06-04 15:54:10 +01:00
nicktrn a61951049f Enhance webapp version with build info (#2146)
* improve app version output

* set build info

* fix typo

* always show additional build info when self-hosting

* Update apps/webapp/app/routes/_app.orgs.$organizationSlug.settings/route.tsx

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>

* Update apps/webapp/app/components/navigation/OrganizationSettingsSideMenu.tsx

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>

* fix build timestamp name

---------

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
2025-06-04 15:49:07 +01:00
nicktrn 7ee0992a2f Runtime agnostic SDK configuration via env vars (#2132)
* use std-env in getEnvVar

* add changeset
2025-06-04 15:48:52 +01:00
nicktrn eee0fb2198 Fix tar-fs dependabot alerts v3 (#2145)
* update testcontainers

* override tar-fs used by testcontainers
2025-06-03 14:29:54 +01:00
nicktrn 1b62b348a9 Fix tar-fs dependabot alerts (#2143)
* update dockerode

* override tar-fs for remix dev
2025-06-03 14:10:35 +01:00
github-actions[bot] a2a5204ac5 chore: Update version for release (v4-beta) (#2144)
* chore: Update version for release (v4-beta)

* Release 4.0.0-v4-beta.20

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Matt Aitken <matt@mattaitken.com>
2025-06-03 13:40:04 +01:00
Matt Aitken d4a9cefd78 syncVercelEnvVars fix for syncing the wrong preview branch env vars (#2141)
* syncVercelEnvVars fix for syncing the wrong preview branch env vars

* syncVercelEnvVars() fix for syncing the wrong preview branch env vars
2025-06-03 13:15:03 +01:00
nicktrn 9fca9e0508 Release 4.0.0-v4-beta.19 (#2140) 2025-06-02 15:32:25 +01:00
Matt Aitken d6dd2c9210 Preview branches docs (#2130)
* Draft preview branches docs

* More wip on docs

* Explain what the GH action does

* Added manual deploy/archive instructions

* Added dashboard section to the docs

* More wip on the preview branch docs

* Added preview/branch to the CLI deploy docs

* Add preview branch note to triggering docs

* Added preview branch notes to the API keys docs

* Added preview stuff to the Context docs

* CLI preview archive command

* Add a better explanation and details about env vars

* Fix for weird reference to the dashboard
2025-06-02 15:31:48 +01:00
github-actions[bot] 01e7ef3d04 chore: Update version for release (v4-beta) (#2110)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2025-06-02 15:21:16 +01:00
nicktrn e84d8cfa1d Fix: lock vitest version in CI (#2139) 2025-06-02 15:14:33 +01:00
Matt Aitken 26b0523cd0 Support passing “production” when deploying (#2137) 2025-06-02 14:18:36 +01:00
nicktrn 84d8fb575f Fix syncing parent env vars for non-preview deployments (#2131)
* only try to sync parent env vars for preview deployments

* add changeset
2025-06-02 10:40:06 +01:00
Matt Aitken 80d449bfdb Preview branch alerts (#2136)
* Preview branch alerts

* Show “Preview” not the branch for new alert modal
2025-06-02 10:37:37 +01:00
Matt Aitken ff157e51a1 If the run is already PENDING_EXECUTING, log an error and ack but don’t fail the run (#2129) 2025-05-30 11:44:40 +01:00
Matt Aitken a027e6a1bb Set the branch header for the SDK realtime functions (#2128) 2025-05-30 09:27:47 +01:00
Matt Aitken 5fdc7e66af Preview branch fixes (#2126)
* SDK: auto-detect the branch using the VERCEL_GIT_COMMIT_REF env var

* Waitpoint token completion with HTTP is now working
2025-05-29 21:23:11 +01:00
Matt Aitken ea853b243e Remove onsole.log… oops (#2125) 2025-05-29 14:39:36 +01:00
Matt Aitken 12f4653cd6 Fix long preview branch names (#2124)
* Truncate long branch names, don’t let the icon shrink

* Truncate and show tooltip for long branch names
2025-05-29 14:36:24 +01:00
Matt Aitken 47f4726515 Remove log that actually appears to users (#2123) 2025-05-29 13:47:19 +01:00
Matt Aitken 7ecaf2ce53 Sync env vars (parent env vars) (#2120)
* Prefer branch variables, but load preview envs too

* WIP with vercel preview env var syncing

* Return success if either child or parent env vars are synced

* Removed log of env vars

* Remove another log
2025-05-29 12:18:25 +01:00
nicktrn 7b463710f5 Fix metrics format (#2119)
* ensure eof marker is always at the end

* strip empty lines from prisma metrics
2025-05-29 09:52:42 +01:00
nicktrn fda9565eb0 Optionally disable run debug logs (#2116)
* disable run debug logs by default

* lightweight webapp health check

* disable debug logs for dev runs

* disable run debug logs for supervisor client

* add changeset
2025-05-29 09:27:08 +01:00
nicktrn 82e7484da8 Feat: import timings and bundle size analysis (#2114)
* cursor should ignore .env files

* fix for duplicate builds when starting dev

* output metafile in dev

* attach metafile to background worker

* attach import timings to worker manifest

* warn during dev if imports take more than 1s

* add analyze command

* update disable warnings flag message

* add changeset
2025-05-28 15:01:25 +01:00
Eric Allam 0f66023b54 v4: metadata collapsing now preserves operation order for correctness (#2115)
* v4: metadata collapsing now preserves operation order for correctness

* Add changeset
2025-05-28 14:52:06 +01:00
Eric Allam 021d6d8f20 Improve efficiency of cancelling in progress TaskEvent records in v3 and v4 (#2112) 2025-05-28 11:06:19 +01:00
Matt Aitken 068c024477 Preview branches (#2086)
* Initial preview migrations

* Modified the staging endpoint to create preview environments

* Added isBranchableEnvironment to RuntimeEnvironment

* Staging = yellow Preview = orange

* Changed the env sort order

* Set isBranchableEnvironment correctly. Create preview for new projects

* Very basic branch menu

* Creating branches from the dashboard

* Fix for string icons on project delete page

* Don’t show branch API keys

* WIP on the manage branches page

* RuntimeEnvironment added projectId index

* Only create the parentEnvironmentId column if it doesn’t exist already

* Improved the limit wording

* Add search to the branch list

* contains in both places

* Many style improvements

* Branch dropdown and v4 badge

* Arching/unarchive branches working in the dashboard

* Tidied imports

* Change preview slug from `prev` to `preview`

* Use correct color for side menu preview branch icon

* Upsert the branch and use the shortcode as a unique constraint

* Upserting working with nice messages in the dashboard

* Better errors when upserting branches

* Button shortcut, don’t allow event to propagate

* Better duplicate error message

* Filter out archived branches from the env selector

* Archiving/creating tweaked some more

* Add an archived banner to the app, fixes for archived branches and upsells

* Fixed pagination

* Disable editing schedules, pausing queues, testing tasks

* Don’t allow replaying if the env is archived

* When deploying detect the correct environment

* Get the projectClient when there’s a branch

* createGitMeta function, most code from the vercel CLI repo

* Deploy, getting the correct environment client

* Added git column to WorkerDeployment

* Add GitMeta to core schemas

* Create branch when deploying

* WIP on branch support in the API

* Delete old createTaskRunAttempt fn

* apiAuth remove export from internal functions

* Rename env var to “TRIGGER_PREVIEW_BRANCH”

* Add TRIGGER_PREVIEW_BRANCH to resolved env vars for runs

* First preview deploy and run working

* Set the preview branch in the main SDK

* Added git links to the preview branches table

* Better errors when replaying/testing archived branches

* Don’t dequeue archived environments

* Env var resolution with parent environment

* Hello world default machine small-2x to save my memory

* Fix for more env var functions

* Only return non-archived envs

* Switch to controlled state for the checkboxes

* Uncheck everything when PREVIEW is checked

* WIP on branch UI

* Show the preview branch label on the env vars list

* Fix for overriding env vars

* Adding preview branch env vars working

* Progress on new env vars

* Only allow selecting a single branch

* Layout fix when there are errors

* Set the defaultValue so there are some fields

* Conform fix for team invite page

* Archived environments don’t run scheduled tasks

* Added Git data to deployments

* Added git data to the deployment inspector

* Don’t allow upserting schedules when archived

* Deduplicate and blacklist some env vars

* Fix for wrong conform function being used

* Show a better error if all vars were blacklisted

* Added environment variable search (by key and value)

* Improved preview branch icon

* Replay now supports branches

* Schedule page render branches properly

* Show the env icon in bottom-left of the test page

* When editing older schedules (that have multi-env) show preview branches correctly

* Fix for incorrect disallowed branch name character

* Extract and improve the directory verification code

* WIP for CLI preview archive command

* Improved the preview branch action buttons

* Redirect to the project if we don’t find a matching env

* Archiving branch via the CLI working

* Fix for archiving branches

* Public access token test task

* JWTs working are with preview branches

* Add branch and git data to the Run ctx

* Updated GitMeta functions to work in CI

* Added pullRequestState

* Archive when deploying if the PR is closed/merged

* Fix for the changesets guide

* Fix for CLI dev bug introduced

* CLI promote now supports preview branches

* Add PR title. Reordered them and added tooltips

* syncEnvVars working with branches

* Added preview branch support to syncVercelEnvVars()

* Detect the branch from Vercel env var (set during build)

* Allow passing a branch in

* Use process.env.VERCEL_TOKEN as well… this used in Vercel CI

* Temp delete

* Improved regenerate api key modal

* Added Accordion component (with styles)

* Redesigned the API keys page

* Revert "Temp delete"

This reverts commit 177b92cd935a6161456bde65d01294e23ecfd47f.

* Changeset

* Fixed docs link

* The new branch panel closes when a branch is created

* Update apps/webapp/app/services/upsertBranch.server.ts

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>

* Removed findUniques from WorkerGroupTokenService

* Made the parentEnvironmentId migrations safe

* Latest lockfile

* Update packages/cli-v3/src/commands/workers/build.ts

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>

* Move isValidGitBranchName to a separate file

* Move the sanitize fn too

* removeBlacklistedVariables moved to a separate file

* Moved deduplicateVariableArray to a separate file…

* Fix broken sanitizeBranchName import

* Another import fix…

* Improved blacklisted error message

* SImplified migration to use `ADD COLUMN IF NOT EXISTS "parentEnvironmentId" TEXT`

---------

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
2025-05-28 11:02:56 +01:00
Eric Allam b74c000fb6 Improve metadata flushing efficiency by collapsing operations (fix #2104) (#2106)
* Improve metadata flushing efficiency by collapsing operations (fix #2104)

* Add changeset

* A couple of style tweaks
2025-05-27 14:08:15 +01:00
Eric Allam 0410316e14 Enable prisma metrics and add them to the /metrics endpoint (#2111)
* Enable prisma metrics and add them to the /metrics endpoint

* Add support for bearer token auth on metrics endpoint
2025-05-27 13:28:45 +01:00
Eric Allam c571c5ef32 v4: Can now set project ref using the TRIGGER_PROJECT_REF env var (#2109)
* v4: Can now set project ref using the TRIGGER_PROJECT_REF env var

* Add changeset

* Improve the name of the resolvEnvVars function and move setting NODE_ENV into the dev supervisor

* Removed moved function
2025-05-27 12:51:04 +01:00
Eric Allam e84ede9599 Upgrade to electric@1.0.13 (#2089)
* Upgrade to electric@1.0.13

* Add ELECTRIC_INSECURE env var to test container
2025-05-23 16:59:53 +01:00
Eric Allam 3fc5630be7 fix: calculating next schedule should always just be from now (#2102) 2025-05-23 16:55:52 +01:00
Eric Allam 13d378f6df v4: batches can only complete when all their runs have been created (#2101)
* v4: batches can only complete when all their runs have been created

* Fix flaky runs replication test
2025-05-23 16:53:50 +01:00
Eric Allam 1a6a9ca97a perf fix: calculateNextScheduledTimestamp speed improvement (#2099)
* Simulate "next steps" when calculating the next timestamp to prevent millions of iterations on dev schedues

* Add additional tests

* Fix typescript issue
2025-05-23 16:24:33 +01:00
Dan 6cb57b803c Added openai agent sdk guardrails docs (#2100) 2025-05-23 16:09:12 +01:00
nicktrn 7c791dd519 Improve unit test workflow performance (#2096)
* shard unit tests

* temp enable for all pushes

* fix test workflow

* update to latest vitest and only add to root package.json

* additionally use default reporter

* gather reports before uploading

* split up slow replication tests

* split up unit tests workflow

* move workflows to parent dir

* use new paths in parent workflow

* prevent artifact clashes

* we always need to create the reports dir

* speed up merge reports

* gather reports even when tests fail

* fix artifact patterns

* increase shards

* disable push trigger again

* improve dequeue snapshot test reliability
2025-05-23 12:22:14 +01:00
github-actions[bot] cf3735976e Release 4.0.0-v4-beta.18 (#2095)
* chore: Update version for release (v4-beta)

* Release 4.0.0-v4-beta.18

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
2025-05-22 21:59:11 +01:00
nicktrn a9a40b5184 Fix dev indexing (#2094)
* save build manifest for dev runs again

* add changeset
2025-05-22 21:54:00 +01:00
github-actions[bot] 0f928e224b Release 4.0.0-v4-beta.17 (#2071)
* chore: Update version for release (v4-beta)

* Release 4.0.0-v4-beta.17

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
2025-05-22 19:51:18 +01:00
Eric Allam a306545952 Always find debug logs when in the span sidebar (#2093) 2025-05-22 14:22:12 +01:00
Eric Allam 1bb1fc4c8f v4 run engine debug logs now filtered out in SQL by default (#2092) 2025-05-22 10:44:06 +01:00
nicktrn 1ca37f5745 Expose esbuild keepNames and minify options (experimental) (#2091)
* Expose esbuild `keepNames` option (experimental)

* Expose esbuild `minify` option (experimental)

* test new experimental options
2025-05-22 10:42:48 +01:00
nicktrn 8112c12092 Improve email whitelisting and extend to GitHub auth (#2090)
* make github auth respect email whitelist

* display github login errors

* fix smtp secure env vars

* prevent sending magic link if email not allowed

* fix MARQS_DISABLE_REBALANCING and RUN_ENGINE_DEBUG_WORKER_NOTIFICATIONS
2025-05-22 10:30:00 +01:00
nicktrn c8d252e300 Add experimental externals detection (#2083)
* add auto detect externals

* stop spinner on esbuild errors

* test problematic packages

* fix braces types

* improve detection 10x

* ignore sentry stub

* improve main package json detection

* rename to experimental_autoDetectExternal
2025-05-21 16:40:37 +01:00
nicktrn d89f740d18 Write esbuild metafile to output dir (#2087)
* write esbuild metafile to output dir

* add changeset
2025-05-21 16:17:10 +01:00
Eric Allam 41a3bdec68 Add id-token permission to the publish webapp workflow (#2088) 2025-05-21 14:30:03 +01:00
Eric Allam a9a937dd25 Implement example of streamText fallback using an AI data stream (#2068) 2025-05-21 13:22:06 +01:00
Eric Allam 14e081e17d release concurrency system reliability improvements (#2081)
* Refill release concurrency tokens when a run is cancelled

* Improved release concurrency accounting system + a sweeper to auto-refill tokens for snapshots that are no longer the latest snapshot on a run (e.g. the run has moved to a new snapshot state)

* Fix order of arguments to the releasings sweeper

* Add a heartbeat for SUSPENDED snapshots, where when stalled will attempt to continue the run if unblocked
2025-05-21 13:21:17 +01:00
Eric Allam 500a173d67 Set default values on the invite and new env var forms to fix an issue after updating conform (#2084) 2025-05-21 13:20:16 +01:00
Eric Allam cc562c58fa Remove unused publish job from the release github workflow to fix permissions error (#2085) 2025-05-21 13:19:49 +01:00
Eric Allam 0334df70ad Prevent socket hangup error by removing the aggressive clickhouse keep alive ttl env var default (#2082) 2025-05-21 13:15:15 +01:00
Dan e08e493b3c Potential fix for code scanning alert no. 2: Workflow does not contain permissions (#2078)
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
2025-05-19 21:17:35 +01:00
Dan f667d61d4c Potential fix for code scanning alert no. 3: Workflow does not contain permissions (#2077)
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
2025-05-19 21:16:58 +01:00
Dan 325ef04c7f Potential fix for code scanning alert no. 6: Workflow does not contain permissions (#2076)
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
2025-05-19 21:15:42 +01:00
Dan 79f0a6a662 Potential fix for code scanning alert no. 42: Workflow does not contain permissions (#2074)
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
2025-05-19 21:14:26 +01:00
Dan ad5027b5fd Potential fix for code scanning alert no. 1: Workflow does not contain permissions (#2079)
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
2025-05-19 20:59:50 +01:00
Dan f0809e6ad6 Potential fix for code scanning alert no. 38: Workflow does not contain permissions (#2075)
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
2025-05-19 20:48:58 +01:00
Eric Allam c63559f55d v4: fix race condition when continuing run when blocked at the same time (#2073)
* v4: fix race condition when continuing run when blocked at the same time

* Attempt to fix flaky e2e test

* Make waitpoint race condition test less flaky
2025-05-19 16:50:36 +01:00
Eric Allam 8471169b30 Fix various dependabot alerts (#2063)
* Upgrade posthog-node to clear axios vulns

* Upgrade @slack/web-api to use a secure version of axios

* Update parse-duration to fix security vulns

* Mitigate against the ws DoS vuln

* Upgrade body-parser to 1.20.3 in the webapp

* Upgrade react-use to 17.5.1 to remove the fast-loops transitive dep

* Remove unused babel dev deps and config file that's no longer used

* Upgrade prismjs to 1.30.0 and bundle parse-duration now

* upgrade express to 4.20.0 to fix issue with XSS when redirecting

* Upgrade @conform/zod to 0.9.2
2025-05-19 12:01:51 +01:00
nicktrn 558a39c644 Remove supervisor docker binary requirement and other tweaks (#2062)
* don't require docker binary

* more structured logs

* remove docker type dep

* add core changeset

* use implicit DOCKER_HOST instead

* disable resource monitor for now

* support attaching docker runners to multiple networks

* warn if dequeue interval > idle dequeue interval

* verbose logs

* add changeset
2025-05-17 12:33:41 +01:00
Eric Allam b4ab06600b Create AGENTS.md file (#2070) 2025-05-17 07:54:04 +01:00
Eric Allam c8e8bc9306 engine v2: Fix selectTopEnvs missing queues (#2069) 2025-05-17 07:53:48 +01:00
Eric Allam a69621bdcc Allow self-hosted deploys locally by pointing to docker.host.internal on macOS (#2064) 2025-05-16 15:20:48 +01:00
Eric Allam ad4daa3301 Fix run replication payloads/outputs trying to send invalid types like BigInt to clickhouse (#2065) 2025-05-16 11:02:30 +01:00
Eric Allam bb7173b4ab Improve run replication logging (#2060)
* Remove the tcp buffer monitors

* Improve run replication logging
2025-05-15 14:12:01 +01:00
Eric Allam 2aee3debc0 Docs: 3rd party log exporters (#2058) 2025-05-15 10:50:15 +01:00
Eric Allam d1e40643fc Allow creating and monitoring run replication services with different settings (#2055)
* Allow creating and monitoring run replication services with different settings

* Fix test
2025-05-14 20:40:47 +01:00
Eric Allam a8e38d9231 chore: Skip unit tests dependency when running publish image workflow (#2054) 2025-05-14 19:31:50 +01:00
github-actions[bot] 2ae0e22d81 Release 4.0.0-v4-beta.16 (#2053)
* chore: Update version for release (v4-beta)

* Release 4.0.0-v4-beta.16

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
2025-05-14 16:26:28 +01:00
nicktrn 206a2c3245 Fix init.ts detection for sentry plugin users (#2051)
* fix init.ts detection when using the sentry plugin

* add changeset
2025-05-14 16:20:05 +01:00
Eric Allam eb3929880f runs replication leader lock expiration fix (#2050)
* runs replication leader lock expiration fix

* Allow configuring the container image --max-old-space-size using NODE_MAX_OLD_SPACE_SIZE

* Ability to configure the clickhouse keep alive settings

* Add some logging because we might not be able to do telemetry
2025-05-14 15:49:48 +01:00
github-actions[bot] 0661ee5fdd Release 4.0.0-v4-beta.15 (#2040)
* chore: Update version for release (v4-beta)

* Release 4.0.0-v4-beta.15

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
2025-05-14 12:33:37 +01:00
Eric Allam 2deef00d48 Improvements to the runs replication service to prevent buffer exhaustion (#2047) 2025-05-14 11:35:51 +01:00
nicktrn fa5360a99d Feat: supervisor ARM builds (#2046)
* v4 worker multi-arch builds

* v4-beta tag for webapp

* add oidc permission

* deps fetcher needs python

* node-gyp needs complete toolchain

* fix headers

* bust pnpm cache

* add setuptools

* switch back to old cache id

* push worker images

* pass the correct tags
2025-05-14 11:23:41 +01:00
Eric Allam 7ace1c3e25 Fix run replication telemetry by passing in a tracer (#2045) 2025-05-14 09:52:09 +01:00
Eric Allam a815633712 Add external log exporters and fix missing external trace exporters in deployed tasks (#2038)
* Add external log exporters and fix missing external trace exporters in deployed tasks

* Generate the external traceID correctly and exporter 3rd party logs with the external traceID as well
2025-05-13 16:54:24 +01:00
Eric Allam 84a0336bdd Fix #1955: schemaTask parse payload failures handled gracefully (#2043) 2025-05-13 16:47:21 +01:00
Eric Allam 7be07b4d0e allow configuring the replication leader lock settings (#2042) 2025-05-13 14:17:36 +01:00
nicktrn bb96e848de Fix init.ts import for deployed workers (#2041)
* import init file in managed worker

* refactor for easier diff

* managed worker: attach configured incorrectly error details

* add changeset

* disable pre

* we also didn't rewrite the output path

* Revert "disable pre"

This reverts commit 89afe44f6c42d4bd7b5a51f7fda1caec6cab8a24.
2025-05-13 13:27:49 +01:00
Rudra Sankha Sinhamahapatra 39f03085b4 feat: ffmpeg v7 (#1777)
* adding ffmpeg v7

* Renaming id ffmpeg to ffmpeg7

* added space

* fixing formatting issue

* fix unit test workflow for forks

* merge ffmpeg v7 into existing extension

* verify md5 checksum

* fix dns resolution

* fix v7 install

* remove redundant code

* add changeset

* mark fluent-ffmpeg as external

* add docs

* Revert "mark fluent-ffmpeg as external"

This reverts commit 74cd317ae69c7453a22d3fa412ff00b402840b87.

---------

Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
2025-05-13 12:27:35 +01:00
Mendy Landa 21c0dc619e feat: playwright extension (#1764)
* feat: playwright build extension

* feat: add playwright task

* full dep list & added some comments

* fix v3-catalog

* log image size for local builds

* detect playwright version

* add changesets

* add docs

* additional checks

* fix unit test workflow for forks

* use middleware and update docs

* update jsdoc

---------

Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
2025-05-13 12:22:01 +01:00
James Ritchie da0ef73724 Docs: Removes an issue listed in ‘known issues’ in the v4 migration guide (#2039) 2025-05-13 10:14:27 +01:00
Eric Allam 65da20c225 feat: replicate task runs to clickhouse to power dashboard improvements (#2035)
* WIP clickhouse package with test containers setup

* More clickhouse client setup now with otel and real tests, and the v1 of raw run events

* Add some additional columns to raw_run_events_v1

* WIP runs dashboard service

* Create a new run engine event bus event for the runs dashboard to hook into

* Track run events in the run engine

* make sure engine v1 runs get synced to CH

* Update the attemptNumber of v3 task runs

* Restructure the run events to be more sparse

* emit more stuff

* Setup replication package

* scaffold the replication package

* replication wip

* resolve conflicts

* more replication stuff

* Add ability to drop the replication slot completely on teardown

* Use the new single replacingmergetree task events table for replication

* get it working

* insert payloads into their own table only on insert and then join

* prepare for using clickhouse cloud and now running ch migrations during boot in the entrypoint.sh

* Handover WIP and tests

* Testing the replication service

* Remove the runs dashboard stuff that we aren't using anymore

* Added a test for large payloads

* hacky typecheck fix

* Fix new internal package typecheck issues and start adding telemetry to the replication service

* tracing over spans, some other improvements

* Improvements to the runs replication service, now ready for testing

* Some fixes and cleanups

* Don't need this code anymore

* move transaction types into the runs replication service

* only send spans where there are transaction events

* A couple of suggested tweaks
2025-05-12 22:12:36 +01:00
James Ritchie ccc9764383 Active incident status panel in the side menu (#2033)
* WIP adding a side menu panel to display incident statuses

* Fixes re-rendering bug and copy tweak

* cache the betterstack response using unkey

* Style the button to fit the panel colors
2025-05-09 15:52:17 +01:00
Matt Aitken 584722d48b [v4] Complete a waitpoint and then get affected runs (#2034)
* Complete waitpoint before getting affected runs. Added more logging

* Added another error log
2025-05-09 15:46:41 +01:00
nicktrn 1b1ad16d94 Improve worker management admin route (#2032)
* improve route to add worker group, handles existing groups gracefully

* add option to remove default worker group from project

* separate project id field

* update supervisor readme with route changes
2025-05-09 10:43:31 +01:00
Matt Aitken 37aadfc80b We can use TaskRun error instead of attempt. Faster and works across v3/v4 (#2031) 2025-05-09 09:36:40 +01:00
github-actions[bot] 6736cb5ae9 Release 4.0.0-v4-beta.14 (#2029)
* chore: Update version for release (v4-beta)

* Release 4.0.0-v4-beta.14

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Matt Aitken <matt@mattaitken.com>
2025-05-08 11:12:25 +01:00
Matt Aitken 87630bc64f Fix for TS issue with old DB type (#2030) 2025-05-07 20:51:36 +01:00
Matt Aitken d23fa38a0a Waitpoint token callback URLs (#2025)
* Initial commit with a plan for what we’re going to do

* Some initial types and improved plan

* Add Waitpoint resolver

* Add resolver + status index

* Remove type + status index

* Only drop if exists

* Remove type index

* Update waitpoint list presenter to use resolver

* Added resolver to the engine

* Made the existing waitpoint list presenter more flexible

* Initial implentation ofr wait.forHttpCallback()

* Added the callback endpoint (no API rate limit)

* schema version

* Added jsdocs, removed schema version because of errors

* Show callback URL if it’s set

* Dashboard pages and panels

* Remove todos

* Added temporary icon

* Added a blank state

* Some tweaks and added a Replicate example

* Implement unwrap() for httpCallback

* Added unwrap to wait.forToken() as well

* Improved jsdocs

* Added docs

* Added unwrap to the token docs

* Show a dash if there are no tags

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>

* Make the timeout error safer

* Fixed migrations… should use id desc not createdAt desc

* Fixed page title

* Fixed migration so it only adds them if they don’t exist. This allows us to manuall run in cloud first

* Respect the max content length by getting the length of the body

* Added more docs details about the callback format

* Remove code comment

* Improved the error

* Added a hash to the HTTP callback URLs

* Add the apiKey to the API input type to fix TS error

* Return the error responses. They were being caught and not preserved

* The content-length header is required. Deal with an empty body

* Removed unused types

* Added some new span icons

* Reworked http callback to be a create call then just use wait.forToken()

* Added a changeset

* Updated the docs

* Updated the wait overview docs

* Simplify to just a call

* WIP stripping right back to waitpoints just having a URL associated with them…

* More deletions

* Remove missing icon

* Updated the changeset

* Add URL to the token return types

* Remove wait for http callback page

* Updated docs

* More tidying

* Type and import fix

* Remove unused import

* Some type fixes for the retrieve

---------

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
2025-05-07 19:55:29 +01:00
James Ritchie 4ec0395424 Fixes a console error introduced when updating the loading animation (#2019) 2025-05-07 18:40:04 +01:00
James Ritchie e94558a0d5 Stop indexing the test-cloud domain (#2026) 2025-05-07 18:39:38 +01:00
James Ritchie 6f4af89adb Docs: Adds Yarn ‘Plug’n’Play” conflicts fix to troubleshooting page (#2028) 2025-05-07 18:38:40 +01:00
James Ritchie 3e45f785a0 Dashboard: Org menu improvements for admins (#1969)
* Adds path for admin route

* Updates impersonation banner and show new org button if only 1 org

* Simpler UI for displaying admin access
2025-05-07 18:38:00 +01:00
James Ritchie 04b9973f20 Shows the tooltip date time on DateTImeAccurate component (#2021) 2025-05-07 18:35:48 +01:00
github-actions[bot] 8da3043111 Release 4.0.0-v4-beta.13 (#2027)
* chore: Update version for release (v4-beta)

* Release 4.0.0-v4-beta.13

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
2025-05-07 15:10:50 +01:00
nicktrn 2b586c87a0 Fix controller waitpoint resolution, suspendable state, and snapshot race conditions (#2006)
* remove dead code

* rename managed to shared runtime manager

* rename to resolve waitpoint for clarity

* add resolver id helper

* store and correctly resolve waipoints that come in early

* fix ipc message type change

* branded type for resolver ids

* add fixme comments

* remove more unused ipc schemas

* fix entitlement validation when client doesn't exist

* restore hello world reference workspace imports

* runtime manager debug logs

* prefix engine run logs

* managed run logger accepts nested props

* runtime suspendable state and improved logs

* require suspendable state for checkpoints, fix snapshot processing queue

* add terminal link as cli module so we can more easily patch it

* apply cursor patch

* add license info

* remove terminal-link package and add deprecation notice

* remove old patch

* remove terminal-link from sdk

* rename snapshot module

* add cli test tsconfig

* add run logger base type

* add snapshot manager tests

* fix cli builds

* improve QUEUED_EXECUTING test

* changeset

* make testcontainers wait until container has stopped

* require unit tests for publishing again

* avoid mutation during iteration when resolving pending waitpoints

* improve debug logs and make them less noisy

* always update poller snapshot id for accurate logs

* detach task run process handlers

* check for env overrides in a few more places and add verbose logs

* log when poller is still executing when we stop it

* add supervisor to publish workflow

* always print full deploy logs in CI

* Revert "avoid mutation during iteration when resolving pending waitpoints"

This reverts commit 87b0ce1e5b.

* disable pre

* print prerelease script errors

* Revert "disable pre"

This reverts commit 9403409637.

* misc fixes

* better debug logs

* add snapshots since methods and route

* prep for snapshots since

* improve deprecated execution detection

* update supervisor and schema

* properly log http server errors

* detect restore after failed snapshot fetch

* run and snapshot id can be overridden

* fix restore detection

* fix deprecation checks, move into snapshot manager

* less logs

* rename snapshot manager stop

* restore detection was moved into snapshot manager

* fix notifier logs

* make runtime manager status a debug log

* no need to attach runtime status twice

* findUnique -> findFirst

* sort snapshots by created at everywhere
2025-05-07 13:02:32 +01:00
nicktrn 053389d9b3 Fix testcontainers cleanup (#2023)
* update testcontainers packages

* increase cleanup timeout and add better logs

* small tweaks

* decrease docker network size so we can have more of them

* add a test flow to check this all works

* await all engine.quit calls

* add docker diagnostics

* reduce number of test runs

* improve network attachment output

* add setup logs

* log redis setup errors

* add cleanup log type

* stop redis container if setup fails

* disable ipv6

* tidy things up a bit

* fix unit tests workflow

* decrease container cleanup timeout

* fix types

* fix webapp typecheck
2025-05-06 08:56:34 +01:00
Eric Allam 43a4139424 docs: onCancel (#2024)
* docs: onCancel

* Add availability note
2025-05-03 21:40:03 +01:00
github-actions[bot] 7b509849e9 chore: Update version for release (v4-beta) (#1996)
* chore: Update version for release (v4-beta)

* Release 4.0.0-v4-beta.12

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Eric Allam <eallam@icloud.com>
2025-05-03 21:26:12 +01:00
Eric Allam 13f750a18b v4: implement onCancel callbacks (#2022)
* v4: implement onCancel callbacks

* chat example

* Add changeset

* A few improvements
2025-05-03 20:50:11 +01:00
nicktrn 15816e9955 Fix long retry delays (#2020)
* make testcontainers wait until container has stopped

* require unit tests for publishing again

* add failing test case

* make it pass

* add retry threshold ms env var and use it
2025-05-02 16:57:21 +01:00
nicktrn a10e2a8b16 Small fixes in prep for v4 self-hosting (#2018)
* fix entitlement validation when client doesn't exist

* add supervisor to publish workflow

* fix v4 publish workflow name
2025-05-02 14:33:20 +01:00
Matt Aitken 423209661b Give helpful hint if you pass a bad directory when deploying (#2013) 2025-05-02 10:54:26 +01:00
James Ritchie 73f363f9d5 Changes the spinner to white now we have purple primary buttons (#2014) 2025-05-02 10:50:22 +01:00
James Ritchie 4456184f37 Fix task and schedule icons squishing (#2017) 2025-05-02 10:47:57 +01:00
James Ritchie 2448242be3 Small improvement to the loading animation so it moves left to right only (#2012) 2025-05-01 19:24:03 +01:00
James Ritchie 9c3aeb3efb Adds friendly formatting for max duration values (#2011) 2025-05-01 19:22:56 +01:00
James Ritchie df0bce3a8f Code blocks have an optional text-wrap toggle (#2009)
* Code blocks have an optional text-wrap button

* Wrap “words”, not “all”

* wrapping is default false

* Change the wording in the tooltip
2025-05-01 17:44:17 +01:00
James Ritchie 0fe85eaec4 Highlight queues when concurrency limit reached and new table col (#2008)
* Combines the Running and Concurrency limit cols into 1

* Display a badge when a queue is at the concurrency limit

* Colors the Running/Limit column text amber if the concurrency limit is hit

* Turns the “Running” big number amber and shows “At concurrency limit” text

* BigNumber now handles big values using formatNumber and formatNumberCompact

Also includes some responsive improvements to make sure things wrap when it gets tight

* Adds a new col for showing how the queue is limited

* Reinstates a threshold for making very big numbers compact

* Added border to make the search bar not float
2025-05-01 17:40:54 +01:00
Matt Aitken d2dde0a844 formatDateTimeISO() fix (#2010)
* Fix for the DateTime ISO function

* Fix implementation and added unit tests

* Preserve milliseconds properly
2025-05-01 17:35:55 +01:00
Dan aacec569d0 Added Python MarkItDown docs (#1977)
* Added Python MarkItDown docs

* Simplified the task

* Update example to use fetch and improved sidebar titles

* Fixed link
2025-05-01 14:18:12 +01:00
nicktrn 0df7af4403 Add idle dequeue interval to supervisor (#2007)
* supervisor uses new idle interval after empty dequeues and errors

* make supervisor heartbeat interval configurable
2025-05-01 10:53:19 +01:00
James Ritchie 553a03580f Show a tooltip when hovering over any DateTime (#2003)
* Adds a tooltip to display the UTC, timezone and local time for DateTime

* Made it safer for SSR

* Makes the copy function into a hook and separate button + adds copy dateTime button to the date tooltip

* Adds an extra-small size copy button

* Show the offset (UTC +1) next to your local time

* Optionally don’t show the date tooltip. Defaults to true (always show)

* Tidy imports

* Refactor the tooltip content to remove duplicated markup

* Coderabbit suggestion to include support for minutes for some timezones like Nepal
2025-05-01 10:00:42 +01:00
Matt Aitken af019473cd Get data for the last day of the month for usage graphs (#1999) 2025-05-01 09:53:06 +01:00
Matt Aitken cb47200d34 Added search to the Queues page (#2000)
* Added search to the queues page

* Fix for empty state copy
2025-05-01 09:33:03 +01:00
James Ritchie 98fd8045e9 Tooltip to explain dev schedules not counting towards your limit (#2004) 2025-05-01 09:32:13 +01:00
James Ritchie 44184a61c7 Updates the secret key text to be more explicit what you have to do to start triggering tasks (#2005) 2025-05-01 09:31:37 +01:00
nicktrn c0807ad0d0 Display terminal links in cursor (#1998)
* add terminal link as cli module so we can more easily patch it

* apply cursor patch

* add license info

* remove terminal-link package and add deprecation notice

* remove old patch

* remove terminal-link from sdk

* changeset
2025-05-01 08:53:11 +01:00
James Ritchie cf9460afee Post new sign up reasons to slack, not plain (#1995)
* Post new sign up reasons to slack, not plain

* Makes the Slack env vars optional

---------

Co-authored-by: Matt Aitken <matt@mattaitken.com>
2025-04-30 14:02:02 +01:00
Matt Aitken 7fe111abe9 AI dashboard widget + CLI AI generate fix suggestions (#1925)
* Made a better AI icon and improved how it looks on the create new schedules inspector

* WIP adding kapa ai to the app

* WIP adding a new Ask AI button to the side menu

* WIP using the react example from the docs

* Align the AI button in the bottom bar

* Kapa widget now works

* Trigger the Kapa modal from the custom button

* Fix imports

* Adds Ask AI shortcut to Shortcuts panel

* Adds a new enter shortcut key

* Adds a prop so you can optionally hide the shortcut key

* Latest

* Moved the Kapa/Help stuff into a component, out of root

* WIP using onModalClose

* Creates a wrapper provider to block shortcuts while kapa modal is open (has bugs)

* Fixes button alignment

* Hide the shortcut key at the button layer

* Fix for enable/disable shortcut keys globally

* Kapa is working

* You can bring up the shortcut keys without opening the help panel

* TODO remove listeners

* remove imports and fix invalid tailwind class

* style kapa widget as best i can

* Remove Kapa event listeners

* Allow passing in a query

* Open the AI widget if there’s a URL param

* Much cleaner implementation for Kapa

* Trying to auto-open the Kapa widget when the page loads

* Delay opening the widget because it was causing issues

* Improved Kapa widget colors

* Added an AI help link to the CLI

* We don’t need this anymore

* Exit with 1, indicating an error. This is important for CI

* Removed old auto-opening code

* Added some code comments to explain some of the annoying stuff

---------

Co-authored-by: James Ritchie <james@trigger.dev>
2025-04-30 11:11:42 +01:00
github-actions[bot] 525285da7f chore: Update version for release (v4-beta) (#1994)
* chore: Update version for release (v4-beta)

* Release 4.0.0-v4-beta.11

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Matt Aitken <matt@mattaitken.com>
2025-04-29 16:48:57 +01:00
Eric Allam 07365cd405 realtime: fix streams missing chunks when streams are longer than 5 minutes and receive 408 Request Timeout errors. Also now support multiple client streams being sent to a single stream key (#1993)
* realtime: fix streams missing chunks when streams are longer than 5 minutes and receive 408 Request Timeout errors. Also now support multiple client streams being sent to a single stream key

* Safely release the reader lock
2025-04-29 12:42:38 +01:00
Matt Aitken 83d0e8710d Fix for Schedules list page slow loading (#1992)
* Fix for Schedules list page slow loading

Getting BackgroundWorkerTask was very slow (Prisma was getting every single one…)

* Same fix for the upserting of schedules in the dashboard
2025-04-29 12:40:56 +01:00
Matt Aitken 280a6d0297 V4 dequeue test fixes (#1991)
* Fix priority tests by dequeuing in sequence

* Expect this test to fail until we support parallel dequeuing of an env with streaming

* Fix one and expect fail from the other

* Fix for batch trigger test dequeuing

* Fixed pending version test
2025-04-29 12:40:22 +01:00
Matt Aitken f579afb4cc V4 dequeue performance (return faster) (#1989)
* Loop through tenants only once, return as soon as we have enough.

* Hello world deadlock task set back to triggerAndWait
2025-04-28 14:49:38 +01:00
James Ritchie 3d744c6ecc Mergent migration guide (#1988)
* Adds a migration guide for converting mergent tasks to trigger.dev

* Adds link to migration guide in the docs side bar
2025-04-28 14:18:09 +01:00
github-actions[bot] 5691480dac Release 4.0.0-v4-beta.10 (#1984)
* chore: Update version for release (v4-beta)

* Release 4.0.0-v4-beta.10

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
2025-04-28 13:34:34 +01:00
nicktrn 09a859d056 Fix managed run controller edge cases (#1987)
* parse disconnect description

* decrease snapshot poller logs

* remove another useless log

* attempt number change detection

* ensure no executions or child processes are being reused

* prevent snapshot change handler from running after execution stopped

* prevent creating zombie intervals on reset

* add changeset

* add enabled poller check before getting latest snapshot

* make attempt number update more explicit, improve logs

* remove deprecated heartbeat service

* ..also remove the import
2025-04-28 13:22:30 +01:00
Saadi Myftija 7bdbbdc433 Handle rate limit errors from Slack on new alert creation (#1986) 2025-04-28 11:07:59 +01:00
Matt Aitken 8b0e4f2367 Fix for completing batches (when there are idempotent runs) (#1985) 2025-04-25 18:41:20 +01:00
Eric Allam e837500486 feat: v4 deadlock detection (#1970)
* Locked task runs will now require queues and tasks to be in the locked version

* Client errors caught in a run function now will skip retrying

* Extracted out the trigger queues logic

* extract validation, idempotency keys, payloads to concerns

* Extracted out a bunch of more stuff and getting trigger tests to work

* Add queue and locked version tests

* Deadlock detection WIP

* more deadlock detection

* Only detect deadlocks when the parent run is waiting on the child run

* Improve the error experience around deadlocks

* A couple tweaks to make CodeRabbit happy and fixing the tests in CI

* Fixed failing test

* Changeset

* wip

* Make sure to scope queries to the runtime env
2025-04-25 12:56:54 +01:00
github-actions[bot] 60fcf5fdac Release 4.0.0-v4-beta.9 (#1980)
* chore: Update version for release (v4-beta)

* Release 4.0.0-v4-beta.9

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
2025-04-25 12:43:32 +01:00
Eric Allam 3a62ee36d9 v3: prevent envs with excess queues from degrading dequeue performance (#1982)
* v3: WIP maximum queues per env implementation

* Add a queue distribution test

* Add edge case tests

* Set the marqs maximumQueuePerEnvCount
2025-04-24 18:02:49 +01:00
Matt Aitken bb606a1fcd Don’t allowed canceled delayed runs to be put into the queue (#1981) 2025-04-24 15:10:47 +01:00
Matt Aitken e9c6cc71f0 New batch id length is longer (#1944) 2025-04-24 13:38:15 +01:00
Eric Allam 30f93521c8 fix: default machine config indexing now works (#1979) 2025-04-24 13:38:03 +01:00
Matt Aitken 18f77660e0 Runlock telemetry (#1974)
* Added error throwing tests to runlock

* Added some more run lock tests

* Added customConditions so the test can use the Logger

* Fix for JSON error

* Added a span to the run lock with a name
2025-04-24 13:37:33 +01:00
Matt Aitken aa1027969b Increase v4 visibility timeouts & don't try continue finished runs (#1978)
* Increase v4 visibility timeouts

* Added some logging for #continueRunIfUnblocked: run has no checkpoint

* If a run is finished or pending cancel, don’t try and continue it
2025-04-24 13:36:18 +01:00
James Ritchie 32ff569844 Fixes bug on Safari preventing Queues and Schedules tables from scrolling (#1973) 2025-04-24 11:40:47 +01:00
James Ritchie dfca04aaea Docs: Updates the run function params syntax + adds new run.list() example (#1976) 2025-04-24 11:31:22 +01:00
Matt Aitken 672a6b8be7 RedisWorker clean up orphaned items when dequeuing (#1975)
* Remove setting the invisibility timeout because it’s already done in the dequeue Lua script

* Remove orphaned queue items when dequeuing

* Added an ack to the visibility timeout test
2025-04-24 11:15:42 +01:00
github-actions[bot] 56ec0523b5 Release 4.0.0-v4-beta.8 (#1972)
* chore: Update version for release (v4-beta)

* Release 4.0.0-v4-beta.8

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
2025-04-23 19:30:49 +01:00
nicktrn 73214d760e Prevent large outputs from overwriting each other (#1971)
* large outputs now get unique keys again

* replace other attempt id occurences

* add changeset
2025-04-23 19:21:21 +01:00
James Ritchie 853a7ef7e4 Updates button copy and layout tweaks (#1941) 2025-04-23 16:33:46 +01:00
HUORT Louis 0726620525 Switch to docker compose v2 (#1692)
* refactor: docker compose migration

* fix compose download link

* set static name for electric container

---------

Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
2025-04-23 14:29:26 +01:00
github-actions[bot] cfb77159d7 Release 4.0.0-v4-beta.7 (#1954)
* chore: Update version for release (v4-beta)

* Release 4.0.0-v4-beta.7

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
2025-04-23 13:34:26 +01:00
Saadi Myftija 2dbc085bb3 Fix alert creation bug related to the Slack integration (#1967)
* Fix alert creation bug related to the Slack integration

* Handle token_expired error from slack gracefully
2025-04-23 11:11:17 +01:00
Saadi Myftija 26aad58abd Add fuzzy matching to the tasks filter (#1960)
* Add a custom react hook to enable fuzzy searching in item lists

* Use fuzzy filtering in the tasks view list

* Use fuzzy filtering in the test tasks list

* Remove the old tasks filtering react hook
2025-04-22 16:34:56 +01:00
Saadi Myftija b62c17bec1 Show more recent payloads in the test task page (#1961) 2025-04-22 16:24:15 +01:00
Saadi Myftija fab565256e Allow empty payloads in test runs (#1962)
* Allow empty payloads in test runs

* Expose a basic linter configs in the code editor component

* Touch up the linting error elements in the code editor component
2025-04-22 16:23:52 +01:00
nicktrn cedd932f99 Fix queued snapshot status handler (#1963)
* handle queued status change gracefully

* add resource monitor to references

* add resource monitor example

* improve runtime manager debug logs

* fix resource monitor example

* add changeset
2025-04-22 15:32:16 +01:00
Matt Aitken e379fc15be Fix redis worker debounce (#1966)
* Added some Redis worker debounce tests (one failing that reproduces a bug)

* Added some tests for acking

* Added a deduplicationKey to prevent acking when items are queued

* The worker passes the deduplicationKey back in for acking

* Improved logs and removed events from test
2025-04-22 15:24:39 +01:00
Dan c0172d428d added cursor rules docs page (#1965) 2025-04-22 15:23:47 +01:00
Dmitry Sabanin 2c19693cfc Fixes runLimiter check on #dequeueRuns (#1953)
* Fixes runLimiter check on #dequeueRuns

This was causing issues in the dev environment with logs being spammed with `[DevSupervisor] dequeueRuns. Run limit reached, trying again later` and making the worker unresponsive and lose connection.

* Create polite-impalas-care.md

---------

Co-authored-by: Eric Allam <eallam@icloud.com>
2025-04-19 21:45:27 +01:00
github-actions[bot] 64ca9dc9a8 Release 4.0.0-v4-beta.6 (#1951)
* chore: Update version for release (v4-beta)

* Release 4.0.0-v4-beta.6

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
2025-04-18 16:50:22 +01:00
Eric Allam 39e535772f fix: realtime streams from tasks now retry if they receive a 408 timeout error from the realtime/trigger.dev server (#1952) 2025-04-18 16:20:13 +01:00
Eric Allam 006951aa53 Set a maximum number of times the dequeue loop can run (#1950) 2025-04-18 13:36:52 +01:00
nicktrn 0ebecbe574 Multiple queue consumers and dev engine URL changesets (#1949)
* add engine url changeset

* add queue consumer changeset
2025-04-18 11:36:29 +01:00
nicktrn aa74260646 Dev command engine URL now configurable via the webapp (#1948)
* return dev engine url from config call

* set and use engine url from dev config reply
2025-04-18 11:11:11 +01:00
nicktrn f683fe8d2d Multiple queue consumers per supervisor (#1947)
* multiple queue consumer in the same supervisor instance

* disable pre

* Revert "disable pre"

This reverts commit 4b154398982758d09667531e3a64761da83664eb.
2025-04-18 11:07:49 +01:00
Eric Allam 863ecf4830 fix: prevent unbounded looping while dequeueing by exiting early when no queues have messages to dequeue (#1946) 2025-04-18 09:41:09 +01:00
Matt Aitken bbf397e661 Completed batch waitpoints when we completed the BatchTaskRun (#1945)
* Completed batch waitpoints when we completed the BatchTaskRun

* Try complete the batch faster now it’s being used operationally

* Fix for tests that were using the old engine.unblockRunForCreatedBatch() function
2025-04-17 21:01:58 +01:00
github-actions[bot] 4a8303212c Release 4.0.0-v4-beta.5 (#1943)
* chore: Update version for release (v4-beta)

* Release 4.0.0-v4-beta.5

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Matt Aitken <matt@mattaitken.com>
2025-04-17 17:25:07 +01:00
github-actions[bot] 59eea654ac Release 4.0.0-v4-beta.4 (#1939)
* chore: Update version for release (v4-beta)

* Release 4.0.0-v4-beta.4

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Matt Aitken <matt@mattaitken.com>
2025-04-17 17:05:52 +01:00
Matt Aitken 0513bfadfb Changeset: The envvars.list() and retrieve() functions receive isSecret for each value. Secret values are always redacted. (#1942) 2025-04-17 16:55:51 +01:00
Matt Aitken a7e326c3f3 Remove concurrently, can’t have two in a single file and they’ve already been applied (#1940) 2025-04-17 15:30:37 +01:00
James Ritchie ad088456d4 Changes button spinner to white and Replay Run button changes to secondary variant (#1912)
* Uses the white variant of the spinner instead of dark on the replay run button

* Changes the replay run button on the run page to secondary
2025-04-17 15:09:29 +01:00
Dan e61466c3a5 Added meme generator example (#1929) 2025-04-17 15:08:47 +01:00
Eric Allam 4b42f778ce fix: release concurrency token exhaustion no longer happens on batchTriggerAndWait (#1937) 2025-04-17 15:07:06 +01:00
James Ritchie 2aacf76444 Adds a new entry in the help and feedback modal to request more feedback (#1936) 2025-04-17 15:06:41 +01:00
Matt Aitken 5597291380 Schedules list performance (#1938)
* Order the schedules: most recent first

* TaskSchedule dashboard speed indexes
2025-04-17 15:05:50 +01:00
Matt Aitken 5c9610b807 The mini task page graphs were missing Timed out and Expired (#1915) 2025-04-17 15:05:37 +01:00
Himanshu Sharma e0fd06490a maintain proper context in metadata.root and parent getters (#1917)
* fix(TRI-5140): maintain proper context in metadata.root and parent getters

* Create two-tigers-dream.md

---------

Co-authored-by: Eric Allam <eallam@icloud.com>
2025-04-17 15:04:01 +01:00
Matt Aitken 33e7b6865a Set env vars to be "secret" (#1923)
* WIP on secret env vars

* Editing individual env var values is working

* Sort the env vars by the key

* Deleting values

* Allowing setting secret env vars

* Added medium switch style

* Many style changes to the env var form

* “Copy text” -> “Copy”

* Draw a divider between hidden buttons

* Env var tweaks

* Don’t show Dev:you anymore

* Grouping the same env var keys together

* Styles improved

* Improved styling of edit panel

* Fix bun detection, dev flushing, and init command (#1914)

* update nypm to support text-based bun lockfiles

* add nypm changeset

* handle dev flushing failures gracefully

* fix path normalization for init.ts

* add changesets

* chore: remove pre.json after exiting pre mode

* init command to install v4-beta packages

* Revert "chore: remove pre.json after exiting pre mode"

This reverts commit f5694fde9314114c74a220c2213d19667bca1a6c.

* make init default to cli version for all packages

* Release 4.0.0-v4-beta.1 (#1916)

* chore: Update version for release (v4-beta)

* Release 4.0.0-v4-beta.1

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>

* Both run engines will only lock to versions they can handle (#1922)

* run engine v1 will only lock to v1 deployments

* run engine v2 will only lock to managed v2 deployments

* test: create background worker and deployment with correct engine version

* Add links to and from deployments (#1921)

* link from deployments tasks to filtered runs view

* jump to deployment

* don't add version links for dev (yet)

* Fix current worker deployment getter (#1924)

* only return last v1 deployment in the shared queue consumer

* be explicit about only returning managed deployments

* Add a docs page for the human-in-the-loop example project (#1919)

* Add a docs page for the human-in-the-loop example project

* Order guides, example projects and example tasks alphabetically in the docs list

* Managed run controller revamp (#1927)

* update nypm to support text-based bun lockfiles

* fix retry spans

* only download debug logs if admin

* add nypm changeset

* pull out env override logic

* use runner env gather helper

* handle dev flushing failures gracefully

* fix path normalization for init.ts

* add logger

* add execution heartbeat service

* add snapshot poller service

* fix poller

* add changesets

* create socket in constructor

* enable strictPropertyInitialization

* deprecate dequeue from version

* start is not async

* dependency injection in prep for tests

* add warm start count to all controller logs

* add restore count

* pull out run execution logic

* temp disable pre

* add a controller log when starting an execution

* refactor execution and squash some bugs

* cleanup completed docker containers by default

* execution fixes and logging improvements

* don't throw afet abort cleanup

* poller should use private interval

* rename heartbeat service file

* rename HeartbeatService to IntervalService

* restore old heartbeat service but deprecate it

* use the new interval service everywhere

* Revert "temp disable pre"

This reverts commit e03f4179de6a731c17253b68a6e00bcb7ac1736b.

* add changeset

* replace all run engine find uniques with find first

* Release 4.0.0-v4-beta.2 (#1928)

* chore: Update version for release (v4-beta)

* Release 4.0.0-v4-beta.2

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>

* Remove batch ID carryover for non-batch waits (#1930)

* add failing test case

* do not carry over previous batch id when blocking with waitpoint

* delete irrelevant test

* Delete project (#1913)

* Delete project

- Don’t schedule tasks if the project is deleted
- Delete queues from the master queues
- Add the old delete project UI back in

* Mark the project as deleted last

* Fix for overriding local variable

* Added a todo for deleting env queues

* Remove todo

* Improve usage flushing (#1931)

* add flush to global usage api

* enable controller debug logs

* initialize usage manager after env overrides

* add previous run id to more debug logs

* add changeset

* For secret env vars, don’t return the value

* Added a new env var repository function for getting secrets with redactions

* Test task for env vars

* Delete heartbeat file, merge mess up

---------

Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Saadi Myftija <saadi.myftija@gmail.com>
2025-04-17 14:39:09 +01:00
Matt Aitken 1b758bd894 Only prevent triggering runs if the parent is completed AND it’s a triggerAndWait (#1935) 2025-04-17 14:25:20 +01:00
Eric Allam e501113a73 fix: refill release concurrency token bucket queue when runs resume before checkpoints are created (#1933)
* fix: refill release concurrency token bucket queue when runs resume before checkpoints are created

* Fix the engine package.json tests

* Remove log
2025-04-16 20:18:38 +01:00
github-actions[bot] 543540581d Release 4.0.0-v4-beta.3 (#1932)
* chore: Update version for release (v4-beta)

* Release 4.0.0-v4-beta.3

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
2025-04-16 19:57:42 +01:00
nicktrn 8b182ce8cb Fix stalled run detection (#1934)
* drive run heartbeats from child process

* track last heartbeat

* add changeset

* use v4-specific heartbeat interval env var and decrease to 20s

* delete old heartbeat service

* remove heartbeat handler during execution cleanup
2025-04-16 19:48:51 +01:00
nicktrn 1f6a283ddf Improve usage flushing (#1931)
* add flush to global usage api

* enable controller debug logs

* initialize usage manager after env overrides

* add previous run id to more debug logs

* add changeset
2025-04-16 14:34:20 +01:00
Matt Aitken 10f78cbad4 Delete project (#1913)
* Delete project

- Don’t schedule tasks if the project is deleted
- Delete queues from the master queues
- Add the old delete project UI back in

* Mark the project as deleted last

* Fix for overriding local variable

* Added a todo for deleting env queues

* Remove todo
2025-04-16 13:31:22 +01:00
nicktrn 2957ee9af9 Remove batch ID carryover for non-batch waits (#1930)
* add failing test case

* do not carry over previous batch id when blocking with waitpoint

* delete irrelevant test
2025-04-16 11:01:09 +01:00
github-actions[bot] bcef9696e8 Release 4.0.0-v4-beta.2 (#1928)
* chore: Update version for release (v4-beta)

* Release 4.0.0-v4-beta.2

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
2025-04-15 16:19:04 +01:00
nicktrn 3f97cf4a6b Managed run controller revamp (#1927)
* update nypm to support text-based bun lockfiles

* fix retry spans

* only download debug logs if admin

* add nypm changeset

* pull out env override logic

* use runner env gather helper

* handle dev flushing failures gracefully

* fix path normalization for init.ts

* add logger

* add execution heartbeat service

* add snapshot poller service

* fix poller

* add changesets

* create socket in constructor

* enable strictPropertyInitialization

* deprecate dequeue from version

* start is not async

* dependency injection in prep for tests

* add warm start count to all controller logs

* add restore count

* pull out run execution logic

* temp disable pre

* add a controller log when starting an execution

* refactor execution and squash some bugs

* cleanup completed docker containers by default

* execution fixes and logging improvements

* don't throw afet abort cleanup

* poller should use private interval

* rename heartbeat service file

* rename HeartbeatService to IntervalService

* restore old heartbeat service but deprecate it

* use the new interval service everywhere

* Revert "temp disable pre"

This reverts commit e03f4179de6a731c17253b68a6e00bcb7ac1736b.

* add changeset

* replace all run engine find uniques with find first
2025-04-15 16:08:51 +01:00
Saadi Myftija 0b2eb34ea6 Add a docs page for the human-in-the-loop example project (#1919)
* Add a docs page for the human-in-the-loop example project

* Order guides, example projects and example tasks alphabetically in the docs list
2025-04-14 20:49:47 +01:00
nicktrn 1816115f0f Fix current worker deployment getter (#1924)
* only return last v1 deployment in the shared queue consumer

* be explicit about only returning managed deployments
2025-04-14 14:55:27 +01:00
nicktrn 5f4c6071d8 Add links to and from deployments (#1921)
* link from deployments tasks to filtered runs view

* jump to deployment

* don't add version links for dev (yet)
2025-04-14 14:43:52 +01:00
nicktrn faf5d01b2c Both run engines will only lock to versions they can handle (#1922)
* run engine v1 will only lock to v1 deployments

* run engine v2 will only lock to managed v2 deployments

* test: create background worker and deployment with correct engine version
2025-04-13 21:51:42 +01:00
github-actions[bot] 1d920d5b3f Release 4.0.0-v4-beta.1 (#1916)
* chore: Update version for release (v4-beta)

* Release 4.0.0-v4-beta.1

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
2025-04-11 20:19:03 +01:00
nicktrn cc553d3392 Fix bun detection, dev flushing, and init command (#1914)
* update nypm to support text-based bun lockfiles

* add nypm changeset

* handle dev flushing failures gracefully

* fix path normalization for init.ts

* add changesets

* chore: remove pre.json after exiting pre mode

* init command to install v4-beta packages

* Revert "chore: remove pre.json after exiting pre mode"

This reverts commit f5694fde9314114c74a220c2213d19667bca1a6c.

* make init default to cli version for all packages
2025-04-11 20:11:13 +01:00
Dan adbc7b44c7 Added examples to the intro table and updated react email (#1878)
* Updated intro list and added task coe

* Fixed import name
2025-04-11 14:12:31 +01:00
Dan e7afc5cfee Updated vercel synv env vars example (#1911)
* Added accesstoken / projectId / vercelTeamId

* Moved v4 upgrade up
2025-04-11 13:54:32 +01:00
Dan 31b06ba5e1 Added ‘defaultIntegrations: false’ to the sentry docs (#1889) 2025-04-11 13:53:56 +01:00
nicktrn 63a221abd0 Only include v4 debug logs for admins (#1910)
* fix retry spans

* only download debug logs if admin
2025-04-11 09:07:51 +01:00
Matt Aitken 030c8ae510 Don't show queue time in the run timeline by default (#1909)
* WIP on hiding the queued time

* Fix for position/sizes of spans

* Fix for duration squashing

* Tweaked some styles on the timeline

* Fix for retry spans appearing in the wrong place

* Added “Q” shortcut key

* Fix for in progress span durations
2025-04-10 16:37:39 +01:00
Eric Allam 61232abd8e docs: Upgrade to v4 guide (#1873)
* WIP upgrade to v4 docs

* WIP

* Upgrade to v4 docs

* Add wait for token docs and improve upgrade guide

* Added known issues to upgrade guide, and moved the migration section higher up

* Update to say "There are no known issues at the moment.”

* Docs upgrade guide package version. Need to use exact packages for the CLI to work

---------

Co-authored-by: Matt Aitken <matt@mattaitken.com>
2025-04-09 17:04:08 +01:00
nicktrn e714f3262a Release 4.0.0-v4-beta.0 (#1907)
* fix package versions in references

* update lockfile
2025-04-09 16:12:36 +01:00
github-actions[bot] b949b5d257 chore: Update version for release (v4-beta) (#1755)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2025-04-09 16:07:14 +01:00
Eric Allam bc8901dbf1 chore: configure changesets to enter prerelease mode on v4-beta tag (#1869)
* Add major v4 changeset

* Enter v4 prerelease mode in changesets

* Use v4-beta tag instead of v4
2025-04-09 15:55:34 +01:00
nicktrn 98a3cfbfa1 Clear runner state of previous executions before warm starts (#1905)
* attach all run controller logs to the run

* make controller-level pre-suspend wait duration configurable

* snapshot status should remain EXECUTING for short retry delays

* add suspended process error

* ensure clean slate before waiting for next run

* treat immediate retries as warm starts

* fix for finished runs waiting forever
2025-04-09 15:53:42 +01:00
Matt Aitken db97f80646 Fix for queue indexing not updating concurrencyLimit/removing it (#1906) 2025-04-09 15:53:21 +01:00
James Ritchie 8382274402 Fixes a padding issue on the org menu dropdown (#1903) 2025-04-09 14:00:57 +01:00
nicktrn ebb71932f9 snapshot status should remain EXECUTING for short retry delays (#1902) 2025-04-09 13:23:39 +01:00
James Ritchie f12141b307 Removes hard coded “5” from “5 related runs” in the waitpoint token inspector (#1901) 2025-04-09 12:21:11 +01:00
Matt Aitken 3ce6144339 Added the Attempt span error back (and properties) (#1900) 2025-04-08 22:45:11 +01:00
Matt Aitken 3cbbece473 Run completed at fix and engine API rate limit whitelist (#1899)
* Use completedAt instead of updatedAt

* Disable verbose SSE logs

* Engine rate limiter: whitelist engine worker-actions

* Show the compute column in the runs table

* The compute column should be invocation + compute cost
2025-04-08 20:55:35 +01:00
Matt Aitken 79d6044dd0 Added engine worker immediatePollIntervalMs and limit env vars (#1898) 2025-04-08 15:39:44 +01:00
nicktrn 72d98778eb Fix node options and OOM retries (#1897)
* don't hardcode max old space size in deployment images

* flags: treat underscores as hyphens

* append attempt number to runner name if >1

* improve retry spans for oom errors
2025-04-08 13:07:45 +01:00
Matt Aitken a51b9b9f98 Fix: custom queue releaseConcurrencyOnWaitpoint. (#1896)
* Make sure releaseConcurrencyOnWaitpoint is set on taskQueue

* Allow task queue to have releaseConcurrencyOnWaitpoint (required to fix custom queues)

* Test for releaseConcurrencyOnWaitpoint

* Added a warning if you use the same queue twice with different settings

* Improved the error some more
2025-04-08 12:32:22 +01:00
Logan Mimaroglu 9d261f4986 Update atomic-deployment.mdx (#1895) 2025-04-08 10:08:31 +01:00
nicktrn fa0e2bf17c Fix v4 build workflow (#1894) 2025-04-07 17:34:07 +01:00
Matt Aitken 56a880f1bd Fix dequeue snapshot batch ids (#1893)
* Pass the batchId through from previous snapshots when dequeuing

* Fix and test for resuming batch after checkpoint
2025-04-07 17:22:56 +01:00
James Ritchie b322cf1eda Fix for long file names wrapping instead of truncating on the RunTimeline (#1891) 2025-04-07 16:36:22 +01:00
James Ritchie 86e56144c0 Adds selected state for waitpoint table rows (#1892) 2025-04-07 16:35:58 +01:00
Matt Aitken fc0a331c42 We weren’t copying the batch info when suspending a run (#1890) 2025-04-07 16:00:07 +01:00
nicktrn a9d5646d75 Very short waits count towards compute usage (#1888)
* add duration wait threshold

* improve wording

* Updates the wait overview, wait.for and wait.until pages copy to better explain compute costs

---------

Co-authored-by: James Ritchie <james@trigger.dev>
2025-04-07 14:52:33 +01:00
Matt Aitken 32d2992802 Security update of nanoid from 3.3.4 to 3.3.8 (#1887) 2025-04-07 13:56:37 +01:00
Himanshu Sharma e546028e9b upgrade nanoid to latest version (#1886) 2025-04-07 11:31:06 +01:00
nicktrn 58567bfe44 Add v4 timeline metrics to prod runs (#1884)
* set correct worker name on create

* dev: kill run process on failed cancel attempt

* add missing prod timeline events

* add more debug logs after api call failures

* import events are always visible

* set dequeued at on new runners

* skip fork event test

* fix timeline test for import event
2025-04-04 18:39:13 +01:00
Eric Allam e7bca6fa1b fix: release concurrency system only consumes tokens when releasings are executed successfully (#1883)
* fix: release concurrency system only consumes tokens when releasings are executed successfully

* Add release concurrency metrics sending to otel
2025-04-04 17:20:33 +01:00
Matt Aitken 195c5d758b Update the path for the subcription portal (#1885) 2025-04-04 17:15:07 +01:00
Matt Aitken 36159be544 Warm start UI (#1882)
* Better trace icon

* Better Waitpoint token icon

* Fix for bad jsx

* Warm and cold start icons

* Tooltips now use a <Portal> so they appear on top

* Warm start components

* Added warm start markers to the Run page and inspector

* Fix for getting the correct value from the metadata

* Better trace icon, with fallback to a passed in one

* Removed unused isWarmStart function
2025-04-04 14:20:28 +01:00
nicktrn 7644d92efa Fix restored runner ID and various small enhancements (#1880)
* make worker nodetype configurable

* make max dequeue count configurable from supervisor

* deterministic runner ids

* update runner id on restore

* disable resource monitor

* explicit run controller type imports

* enhance debug logs
2025-04-04 12:35:23 +01:00
Eric Allam e6efb3c61b realtime: allow realtime streams to have it's own redis env vars (#1876) 2025-04-04 12:30:29 +01:00
Eric Allam 7b9ab1781a re2: add ability to selectively disable release concurrency queue consumers via env var (#1881)
* run engine: add ability to selectively disable release concurrency queue consumers via env var

Also added some additional logging

* actually pass the disableConsumers option through to the release concurrency system

* Fixed CLI e2e failing tests
2025-04-04 12:28:39 +01:00
Eric Allam 25c1abac8e v4: eagerly fork child processes before warm start initiates (#1879)
* v4: eagerly fork child process before warm start

Also fixes an issue where the attempt span events weren't coming through in the partial spans

* Making flushing a safer operation

* Add changeset

* Improve attempt metrics when in a warm start, and pass the STYLE_VARIANT for a warm or cold start on attempt spans
2025-04-04 11:07:36 +01:00
Dan 179a9c95c7 Added satori and improved react email (#1877)
* Improved the react email docs and added a video

* Copy

* Added a link to the cursor rules

* Added satori example

* Copy updates

* Added react pdf link
2025-04-03 16:42:54 +01:00
Eric Allam 18ad897171 fix: silence erroneous prisma package.json warnings (#1875) 2025-04-02 22:39:29 +01:00
Matt Aitken 5bdbf47fb3 Run page improvements (#1874)
* When selecting spans, don’t add to the browser history

* Only non-debug events can extend the total duration of the run timeline
2025-04-02 19:51:09 +01:00
nicktrn 7591b6e619 Fix v4 restore race condition (#1870) 2025-04-02 18:05:43 +01:00
Eric Allam cb6d697e6d chore: improve blank state copy (#1872) 2025-04-02 17:44:18 +01:00
Eric Allam 8e5d5cf690 fix: only pass engine version header when inside a task (#1871)
This will allow migrating to the v4 SDK in your application backend before deploying v4 to trigger, and everything will just work
2025-04-02 17:40:07 +01:00
nicktrn dcacc3a9e2 Fixes for v4 waits and restores (#1868)
* remove dummy metrics from heartbeat

* fix heartbeat timeouts..

* reset clock when resolving waitpoints

* optionally set metadata url when scheduling run

* don't use global vitest as it's outdated

* don't run test files in parallel

* handle special graceful shutdown code

* exit codes for success and failure can now be set and overridden

* ensure immediate cleanup in tests

* update lockfile after conflict
2025-04-02 15:27:01 +01:00
Eric Allam 6d819c6726 chore: publish docker even if the unit tests fail for now (#1867) 2025-04-02 14:20:16 +01:00
Eric Allam 3e341701ae v4 engine version selection logic updated (#1866) 2025-04-02 14:04:01 +01:00
Eric Allam 51bb3ee464 v4 ai improvements (#1863)
* AI SDK sql generator and approval example WIP

* Create some nice spans and extract the tools out so we can get the type of the streams

* Enable AI SDK telemetry

* Adding ai.tool and removing toolTask, 3rd party telemetry spans now wil create partials, better ai SDK telemetry icons

* Created a separate d3-chat example, split out from the openai-agents example

* Fixed the tool options being passed to metadata in `ai.tool`

* Add a link to the run

* Slightly improved design

* Add a crawler task using crawl4ai

* Use a tool to get the userId

* Couple of tweaks

* Adding markdown rendering to assistant messages and added an e2b based chart rendering task

* Backup to anthropic

* Add changeset
2025-04-02 12:00:02 +01:00
Eric Allam e539c7c941 v4: allow disabling of the run engine worker via the RUN_ENGINE_WORKER_ENABLED env var (#1865) 2025-04-02 11:39:55 +01:00
Eric Allam 31bef8d7cd fix: update migration with the "Drop/Add" index of SecretStore key (#1864) 2025-04-02 10:43:02 +01:00
Matt Aitken 7ce8bd1acd Queues page: fix for upgrade state, and upgrade docs link (#1862) 2025-04-01 19:20:23 +01:00
James Ritchie 49c067d755 Fixes the avatar squishing if the project name is too long (#1861) 2025-04-01 17:39:10 +01:00
James Ritchie 551b7947f2 New waitpoint token docs page + updates the URL and description for the waitpoint blank state in the app (#1860)
* Delete old wait-for pages & create new wait-for-token page

* Link to new wait-for-token page and update description

* Adds an upgrade to v4 coming soon docs page

* Removes unncessary property from Note component
2025-04-01 17:15:59 +01:00
James Ritchie a5b9151e3a Adds a "dev connected" banner to the top of the Run and Runs list pages (#1855)
* Adds DevPresenceBanner to the Run page

* Adds DevPresenceBanner to the Run list page

* Makes the DevConnection component reusable and moves components to the DevPresence.tsx file

* SideMenu: only show dev presence when project engine === V2

* WIP on disconnected banner on v3 and v4

* v3 dev connection working. Made it slightly red

* Only show the disconnected banner on v3 if the run is not completed

---------

Co-authored-by: Matt Aitken <matt@mattaitken.com>
2025-04-01 14:34:01 +01:00
Himanshu Sharma 28121249bb fix: prevent enqueuing into closed ReadableStream (#1781)
* fix: prevent enqueuing into closed ReadableStream

* Create smart-coins-hammer.md

---------

Co-authored-by: Eric Allam <eallam@icloud.com>
2025-04-01 11:52:41 +01:00
James Ritchie 81941c8910 Fixes a docs link and tweaks some wording for the triggering task blank state (#1856) 2025-04-01 11:33:07 +01:00
nicktrn bed834b589 Misc v4 checkpoint fixes (#1859)
* logs for optional services

* print env vars on startup in debug mode

* routes need to explicitly ask to keep connection alive

* log indicators for now

* make workload api listen host configurable

* expose supervisor metrics and make more configurable

* configurable pull secrets, no defaults

* remove restore route

* run controller to handle queued executing

* fix v3 deploys in v4 project

* update admin worker route

* only start pod cleaner et al in k8s mode

* set new worker group as default if none yet

* make image ref optional

* checkpoint image ref is optional for output as well

* export feature flag const

* one last image ref type fix

* make runner intervals configurable

* ability to set arbitrary env vars on new runners

* set default runtime back to node 21

* move all runner env vars to the same section
2025-04-01 10:08:15 +01:00
Eric Allam 9daa3977fa fix: safe upserting of task queues during worker indexing (#1858) 2025-03-31 16:28:08 +01:00
Eric Allam f320e14f38 cli: add dev lock file to prevent 2 dev processes running at the same time in the same dir (#1854)
* cli: add dev lock file to prevent 2 dev processes running at the same time in the same dir

* Make sure the .trigger dir exists before creating the dev.lock file
2025-03-31 16:27:11 +01:00
James Ritchie 4d5080d097 Fixes missing truncation on the project name text (#1857) 2025-03-31 14:49:37 +01:00
Matt Aitken 2b34dbe4b0 Fix for TaskRunErrorCodes.MAX_DURATION_EXCEEDED error saying ms when it’s seconds (#1848) 2025-03-31 14:48:03 +01:00
nicktrn 394f1decd3 Misc v4 fixes and improved logging (#1831)
* logs for optional services

* print env vars on startup in debug mode

* routes need to explicitly ask to keep connection alive

* log indicators for now

* make workload api listen host configurable

* expose supervisor metrics and make more configurable

* configurable pull secrets, no defaults

* remove restore route

* run controller to handle queued executing

* fix v3 deploys in v4 project

* update admin worker route

* only start pod cleaner et al in k8s mode

* set new worker group as default if none yet
2025-03-31 14:06:17 +01:00
Eric Allam c963dcda57 fix: use default retry settings when no catchError handlers are defined (#1852) 2025-03-31 13:57:02 +01:00
Eric Allam 0e77747d45 fix: runs.retrieve now uses the output stored on the TaskRun table instead of deprecated attempts table (#1853)
* fix: runs.retrieve now uses the output stored on the TaskRun table instead of deprecated attempts table

* Fixed typecheck error
2025-03-31 13:54:46 +01:00
Eric Allam 8977d2546c fix: custom queues defined on tasks on v3 now get correctly created (#1846) 2025-03-28 17:56:26 +00:00
Eric Allam af45e8397a fix: ensure task list and test action use correct dev env (#1845)
* Ensure the task list uses the dev environment of the logged in user

* Ensure the test task action uses the correct dev environment

* Fixed issue with missing case statement scope, wrapping in block
2025-03-28 17:55:05 +00:00
Dan 15b426d5b6 Add realtime / react hooks examples snippet (#1770)
* Added realtime / react hooks examples snippet and added it to all the relevant pages

* Fix link
2025-03-28 15:57:41 +00:00
James Ritchie b29f3ffb2d Updates the onboarding copy now we have multiple example projects (#1842) 2025-03-28 15:56:26 +00:00
James Ritchie f3efdad797 "Dev connecting" icon update + text color fix for the purple login buttons (#1835)
* Updates the large login buttons to the new purple style

* Updates the dev checking connection icon to include 3 dots on the screen

* Fixes svg property names
2025-03-28 15:56:00 +00:00
James Ritchie 182f5bc706 Fixes the disabled state for the Switch component (#1841) 2025-03-28 15:55:03 +00:00
James Ritchie 94c574bfef Fixes svg icon property for React (#1837) 2025-03-28 15:54:39 +00:00
Matt Aitken 5640467ce5 Task queues weren’t setting concurrencyLimit or releaseConcurrencyOnWaitpoint (#1844) 2025-03-28 15:53:56 +00:00
James Ritchie 73b41c28bf Support solo buttons in table row (#1840)
* Removed old condition that handled old v2 tables

* Adds support for a single button to be shown on hover in a table cell

* Makes the “hiddenButtons” support a single button instead of adding a new property

* Format the docs link for Schedules like the other for consistency
2025-03-28 15:53:15 +00:00
Eric Allam f4ef008e2c feat: Python otel support, enriching spans, OpenAI Agents SDK example (#1839)
* D3 demo WIP

* Agent working and completing token successfully

* Sending OpenAI Agent SDK spans through to the platform now works

* A couple of perf tweaks for enriching events
2025-03-28 13:01:59 +00:00
Dan 2a6d8251b2 Added react email example (#1802)
* Added react email docs

* Added jsx note

* Removed troubleshooting
2025-03-28 11:56:02 +00:00
Dan fc7e196a09 Added supabase auth page and updated examples (#1838)
* Added supabase auth page

* Added auth info snippet

* Added snippet and updated examples
2025-03-28 11:55:47 +00:00
James Ritchie 49c43a128b Adds a python logo icon (#1834) 2025-03-28 11:04:16 +00:00
Matt Aitken 174484fb32 Added default time period to Runs, Waitpoint and Batch list pages. Improved dev presence (#1832)
* New time period filter (permanently displayed)

* Batches, and fix for blank state

* Waitpoint token filtering

* Tags query: remove things we’re not using

* Run tag and waitpoint tags use startsWith for faster search

* Fix for run page on wrong env. Added schedule last triggered column

* Removed the Redis pubsub, just use the presence key

* Improve the dev presence responsiveness

* The CLI presence connection recovers when the webapp is restarted

* Dev schedules are now working for v4

* Refactored to make the dev presence stuff

* Got rid of stupid extra /dev, added a connecting state with icon

* Remove unused Redis client
2025-03-28 09:32:42 +00:00
Eric Allam 589f7c3bcc v4: Fix HandleErrorFunction type (#1829)
* v4: Fix HandleErrorFunction type

* Only show the catchError span if there are any catchError hooks
2025-03-28 09:15:33 +00:00
James Ritchie ff6359caf2 Buttons: Updates the primary button style to fit the UI style better (#1828)
* Move to indigo primary button to better fit in with the app UI style

* Updates Pause queue button to secondary

* Updates some blank state CTA buttons to primary
2025-03-27 13:25:34 +00:00
Eric Allam 0e82698548 v4: Waitpoint PAT completion, fix dequeuing without retry config, fixed metadata (#1826)
* WIP

* Completing waitpoints using public access tokens

* fixed example

* adding agent-loops reference, adding useWaitToken react hook, adding a way to throw an error to output from the run, adding an icon option to logger.trace

* Fixed metadata system

* Check if the waitpoint is COMPLETED already in the complete endpoint and return true
2025-03-27 10:04:12 +00:00
James Ritchie be02439d1d Icons: updates and new icons for the side menu, run page and general improvements to the way they work (#1825)
* New format for the functions instead of “onWait() task”

* Make the icon sizes match the hero icons

* Updating icons for the new task hooks

* Adds new icons to the RunIcon.tsx component

* Improves the size of the menu icons

* Fixes the missaligned environment dropdown and dev connected button

* Changes the button colours so they are all unique

* More icon updates

* Adds new variables for main page icons

* Improves icons sizes

* Using new color variables in the side menu and blank states

* Adds preview environment color variable

* Align the env icon in the menu

* Updates the env icons to be the correct size and padding to match hero icons

* Label uses new colors, removes unused cases and new env icons

* Lower case env label

* Removes unused code

* Use full env title

* Organize imports

* More variants of the runs icon to work at smaller sizes

* small padding adjustment

* reformat init function span

* Fix last init formatting

* New colour for settings

* Organize switch statement for icons nicely

* Use new color variable

* Update icon svgs

* Renamed lifecycle hook icon to Function icon

* Renamed function icon file name
2025-03-26 19:12:12 +00:00
James Ritchie d745bb63ae Cursor rules: Copy improvements and reordering sections (#1813)
* Re-ordered some rules for the video

* Adds description back in
2025-03-26 18:38:01 +00:00
Matt Aitken 00586ffaaf Waitpoint tokens page, wait.listTokens() and wait.retrieveToken() (#1824)
* Added waitpoints/tokens to the sidebar

* Added indexes to the Waitpoint time for filtering

* Begun work on `WaitpointTokenListPresenter`, the pag is a copy of the Queues page for now

* MVP of waitpoint token page

* Added status

* Expiry of timeout/ttl

* Improvements to the waitpoint table

* Improved columns and icon

* Changes from the RunTag copy on hover branch

* Fix for nested button error

* Added waitpoint tags to the DB/table

* Applied Eric’s task run tag fix (it’s live on prod in the legacy run engine branch)

* Added tags to waitpoints

* Removed todos that have been done

* Added token support for releaseConcurrency. Also added a ton of JSDocs

* Added releaseConcurrency to the API token endpoint…

* WIP on waitpoint page filters

* Fix for tags filtering

* Waitpoint filters working

* Fix for badly named function

* WaitpointPresenter used from SpanPresenter

* Waitpoint detail panel WIP

* Fix for server client hydration issue with CodeBlock

* Selected waitpoint panel

* Added a blank state

* Added waitpoint docs link

* Fix for animated number going past the target

* Fix for the queue list pagination and upgrade status

* Engine version error for waitpoint token list

* RunTag component doesn’t get squished and hover behaviour is nicer

* Associating runs with waitpoints

* Added triggered icon

* Link directly to the waitpoint

* Fix for TS error on waitpoint retrieve

* Added CopyableText component, used for waitpoint id in the table

* Removed the confetti 🎊

* Deleted some old images

* Moved some schemas/types to core. Use `id` instead of `friendlyId`

* Added wait.listTokens() function. Made some changes to the types to make it nicer

* WIP wait.retrieveToken()

* wait.retrieveToken working

* Added data to retrieve token

* Separate ApiWaitpointPresenter completely

* Added completed time to the waitpoint detail panel

* Fix for the Avatar component having SSR issues. Specify the size in rems and removed the useLayoutEffect

* Fix for applied idempotency key filter dropdown showing the id field

* Use parentheses to make sure the token list query respects idempotency key correctly

* Use the proper logger, and have a decent message with info to track the bug down

* Pass the org title into the Avatar

* Better error when failing to creating a manual waitpoint after X attempts
2025-03-26 18:05:26 +00:00
nicktrn c08c3b488c Update lockfile (#1822) 2025-03-26 10:55:59 +00:00
nicktrn 8c3542cfd6 Small package release tweaks (#1821)
* set core version for redis-worker

* add empty changelog
2025-03-26 10:39:11 +00:00
Eric Allam d6fd7f559e Run engine: Using root queue timestamp to prioritize completing runs (#1818)
* Improve the new run engine Trigger/Batch trigger service class names

* centralize queue timestamp logic in EnqueueSystem, adding queueTimestamp support and propagation

* Fixing the create tags in the new run engine trigger task service
2025-03-26 10:25:49 +00:00
nicktrn 4fe1d49f94 Add v4 pod lifecycle handlers (#1819)
* patch k8s client to allow field selector on informers

* add pod cleaner and tests

* add failed pod handler and tests

* remove supervisor test script for now
2025-03-26 10:22:24 +00:00
nicktrn 5f48b9c7f9 Fix worker deployment engine version (#1820)
* enable re2 cli flag by default

* background worker engine determines deployment type
2025-03-26 10:22:13 +00:00
Eric Allam 82251e21f2 Merge pull request #1817 from triggerdotdev/revamped-lifecycle-hooks
v4: New lifecycle hooks
2025-03-25 16:46:34 +00:00
Eric Allam 6d5d10a45a Support other init file names 2025-03-25 16:33:12 +00:00
Eric Allam 1def8d06c2 Use allSettled so onWait/onResume errors don't break anything 2025-03-25 16:32:17 +00:00
Eric Allam e745fcb663 Remove console.log 2025-03-25 15:58:50 +00:00
Eric Allam 887c5cac5c Add changeset 2025-03-25 15:56:55 +00:00
Eric Allam 268c9f6898 implement onWait/onResume 2025-03-25 15:55:42 +00:00
Eric Allam 3ca5985277 Hooks now all use the new types, and adding some spans 2025-03-25 15:55:42 +00:00
Eric Allam df9cad6667 Get it all working 2025-03-25 15:55:42 +00:00
Eric Allam ba520fcdd8 Make sure and register all the config hooks 2025-03-25 15:55:42 +00:00
Eric Allam 23ae77afff handle max duration timeout errors better 2025-03-25 15:55:42 +00:00
Eric Allam 30705fd59b implement cleanup 2025-03-25 15:55:42 +00:00
Eric Allam d0a5c16c63 Add cleanup hook 2025-03-25 15:55:42 +00:00
Eric Allam 6ba2b0476b more tests 2025-03-25 15:55:42 +00:00
Eric Allam 5edbb77dfe use tryCatch in TaskExecutor 2025-03-25 15:55:42 +00:00
Eric Allam 7f804f381c middleware working and tests passing 2025-03-25 15:55:42 +00:00
Eric Allam 6ab74c47d6 implement and test handleError (now catchError) 2025-03-25 15:55:42 +00:00
Eric Allam f84ddcd6ea test oncomplete 2025-03-25 15:55:42 +00:00
Eric Allam 87824a8ce3 dry up some stuff 2025-03-25 15:55:42 +00:00
Eric Allam 91225da813 test on failure 2025-03-25 15:55:42 +00:00
Eric Allam 2262a42632 more tests and fixes 2025-03-25 15:55:42 +00:00
Eric Allam ef871b0434 starting test executor 2025-03-25 15:55:42 +00:00
Eric Allam 4a8d6932c8 a couple tweaks 2025-03-25 15:55:42 +00:00
Eric Allam 307309f4f7 implement onComplete 2025-03-25 15:55:42 +00:00
Eric Allam b974d8297d use new onFailure 2025-03-25 15:55:42 +00:00
Eric Allam 116788912b use new onStart 2025-03-25 15:55:42 +00:00
Eric Allam 2e6d0d8783 share the hook registration code 2025-03-25 15:55:42 +00:00
Eric Allam 2d16d664ef Add middleware hooks 2025-03-25 15:55:42 +00:00
Eric Allam 6001dfa7de Add middleware types 2025-03-25 15:55:42 +00:00
Eric Allam dc6e659e94 new locals API 2025-03-25 15:55:42 +00:00
Eric Allam e34e520057 more hooks 2025-03-25 15:55:42 +00:00
Eric Allam 597b7baaaa adding imports 2025-03-25 15:55:42 +00:00
Eric Allam e77e8d48d6 handleError 2025-03-25 15:55:42 +00:00
Eric Allam aaf2ed8a20 onWait and onResume 2025-03-25 15:55:42 +00:00
Eric Allam e816ba4382 onComplete 2025-03-25 15:55:42 +00:00
Eric Allam 06f6a04056 onStart 2025-03-25 15:55:42 +00:00
Eric Allam be676fdb33 onFailure 2025-03-25 15:55:42 +00:00
Eric Allam 1777ff4cfb move onStart to the new lifecycle hook system 2025-03-25 15:55:42 +00:00
Eric Allam 17e162310d Improve init lifecycle hook types and fix tabler icons on spans 2025-03-25 15:55:42 +00:00
Eric Allam 2c44e59d07 init.ts at the root of the trigger dir is now automatically loaded 2025-03-25 15:55:42 +00:00
Eric Allam 0215a8cfa8 vibes 2025-03-25 15:55:42 +00:00
Eric Allam f59bad9227 Revamping the lifecycle hooks, starting with init 2025-03-25 15:55:42 +00:00
James Ritchie 2ef60ce51f Side menu: style fixes, small improvements and better Info Panel component props (#1816)
* Adds “plan” to the plan name if it’s a paid plan so they all format correctly

* Fixes menu item having incorrect height

* Adds gap between switch org menu items

* Moves main action button to right side in blank state

* Simpler way to add “plan” to the plan name

* Styled the plan badge a bit nicer

* Project page now matches the other settings page layouts

* Fixes colour of queue icon

* Fixes table blank state not spanning enough cols

* Organize imports

* Unified the blank state panels and improved some copy in them

* Improved the copy in the no deployed tasks blank state

* Make the upgrade icon solid and indigo

* Transition the hover state on the side menu section

* Replaces many props with a single ‘accessory’ React node on the Info Panel component

* Organize imports
2025-03-24 14:44:08 +00:00
Eric Allam c4e1364cf9 Merge pull request #1808 from triggerdotdev/re2-queue-indexing
re2: Queue indexing, queue trigger changes
2025-03-21 16:09:15 +00:00
Eric Allam f1d5886c11 Another attempt at getting CI to pass tests 2025-03-21 15:50:51 +00:00
Eric Allam b72b7c2c52 Fixed typechecks 2025-03-21 15:22:53 +00:00
Eric Allam f46a409f0e Fixed typescript error 2025-03-21 15:06:55 +00:00
Eric Allam 5a725675d3 Fix the unit tests CI action 2025-03-21 15:06:55 +00:00
Eric Allam 671ff26d50 Removing more exportName stuff 2025-03-21 15:06:55 +00:00
Eric Allam 109bfaca00 Remove exportName from the trigger span names 2025-03-21 15:06:55 +00:00
Eric Allam b65f2256e0 Tasks are indexed even if you don't export them 🤩 2025-03-21 15:06:55 +00:00
Eric Allam c87ff2c634 Various fixes for no longer having the exportName 2025-03-21 15:06:55 +00:00
Eric Allam 233a22d59e Move runs sidebar menu item below tasks and alerts into the manage section 2025-03-21 15:06:55 +00:00
Eric Allam e53eed1753 Fixed the types on pauseQueue service 2025-03-21 15:06:55 +00:00
Eric Allam ad08756001 Run tests sequentially in CI, and pass "--run" to vitest to disable watch mode 2025-03-21 15:06:55 +00:00
Eric Allam 8d514b902d Use orderableName on the TaskQueue and add the releaseConcurrencyOnWaitpoint to the SDK and the dashboard 2025-03-21 15:06:55 +00:00
Eric Allam f58dee1aa1 Fix the task run status reason UI 2025-03-21 15:06:55 +00:00
Eric Allam de5e0c8e8b try to fix flaky tests 2025-03-21 15:06:55 +00:00
Eric Allam bee5e75931 Fixed cli e2e tests 2025-03-21 15:06:55 +00:00
Eric Allam f4a1eb1914 Fixed changesets and added one 2025-03-21 15:06:55 +00:00
Eric Allam 02eeefd647 Increase maxRetriesPerRequest in CI 2025-03-21 15:06:24 +00:00
Eric Allam cbe651e7e4 Fix useTaskTrigger queue option params and remove exportName from e2e tests 2025-03-21 15:06:24 +00:00
Eric Allam 941e03b3eb Using the new PENDING_VERSION status now in the UI 2025-03-21 15:06:24 +00:00
Eric Allam 1bec4e1432 Getting the queues indexing to work by passing them to the create background worker request bodies 2025-03-21 15:06:24 +00:00
Eric Allam 7e411ac162 New PENDING_VERSION system which now requires queues to exist at dequeue time 2025-03-21 15:06:24 +00:00
Eric Allam 7a58439728 WIP queue indexing 2025-03-21 15:06:24 +00:00
nicktrn 49a3f72e13 Publish redis-worker and add graceful shutdown manager (#1810)
* add shutdown manager

* update ai test instructions

* add shutdown timeout to redis-worker

* move redis worker to packages

* add unregister method

* prep for publishing package

* fix types

* update ai files

* fix cursor terminal links

* prevent overly friendly ids

* use structured logger

* use unique shutdown handler names

* rework suspend completion

* add trycatch util

* rework suspend restore

* add http server metrics

* add missing prom-client to core

* add prom metrics to redis worker

* bundle redis-worker

* fix esm/cjs interop

* remove proxy from changeset ignore and add supervisor

* add pause to prerelease script for any manual edits

* unregister the correct handler and add early detection

* small change to http handler return

* fix worker tests

* fix shutdown manager tests
2025-03-21 14:53:27 +00:00
Dan adca1997af Added turborepo and prisma docs (#1807)
* Added python links in guides intro

* Added example 1

* Updated intro

* Added example 2

* Typo

* Improved intro
2025-03-20 17:29:06 +00:00
Niels 1eb3fb34e1 Enhance RunTag parsing with robust key-value tag splitting (#1782)
* Enhance RunTag parsing with robust key-value tag splitting

* Clarify RunTag parsing documentation
2025-03-20 13:19:26 +00:00
Eric Allam a8b3c7003f Merge pull request #1804 from triggerdotdev/re2-reserve-concurrency
re2: New release concurrency system
2025-03-19 18:01:01 +00:00
Eric Allam 38e1887495 Fixed failing redis worker tests 2025-03-19 17:58:53 +00:00
Eric Allam e5ea9cb2d7 move the release concurrency queue into the release concurrency system, make it disabled by default, configure the run engine in the webapp with env vars 2025-03-19 17:55:29 +00:00
Eric Allam 28b3ed0496 Implement release concurrency system 2025-03-19 17:55:29 +00:00
Eric Allam 7eaf81abbe Use releaserId in case we don't end up using run IDs 2025-03-19 17:54:47 +00:00
Eric Allam f9c7e95981 Added updated execution states chart and updated readme 2025-03-19 17:54:47 +00:00
Eric Allam b170a62671 fixed the create checkpoint valid snapshot logic 2025-03-19 17:54:47 +00:00
Eric Allam e7c8f94447 implement the QUEUED_EXECUTING dequeuing, and creating a checkpoint while the run is in QUEUED_EXECUTING state by saving the EXECUTING_WITH_WAITPOINTS snapshotId as the previousSnapshotId on the QUEUED_EXECUTING snapshot 2025-03-19 17:54:47 +00:00
Eric Allam 29371e950b Implement checkpoint tests, handle dequeuing QUEUED_EXECUTING runs 2025-03-19 17:54:47 +00:00
Eric Allam 42fb5d05ff Fix more tests 2025-03-19 17:54:47 +00:00
Eric Allam 67d74f066c More tests passing, fixed the heartbeat issue 2025-03-19 17:54:47 +00:00
Eric Allam 7866e95636 waiting for worker system 2025-03-19 17:54:47 +00:00
Eric Allam a4581f1597 ttl system 2025-03-19 17:54:47 +00:00
Eric Allam 363f0668b5 Delayed run system 2025-03-19 17:54:47 +00:00
Eric Allam 63b43ff357 more system work 2025-03-19 17:54:47 +00:00
Eric Allam 717cec879b move startRunAttempt to RunAttemptSystem 2025-03-19 17:54:47 +00:00
Eric Allam 1b61b95c57 WIP run engine systems 2025-03-19 17:54:47 +00:00
Eric Allam 7d11e827d9 Remove reserve concurrency system from run engine 2025-03-19 17:54:47 +00:00
Eric Allam 8c66ec3189 Remove reserve concurrency system from run queue 2025-03-19 17:54:47 +00:00
Eric Allam 063651c7b8 Upgrade vitest in the run-engine package 2025-03-19 17:54:47 +00:00
Eric Allam 2048b72288 go to QUEUE_EXECUTING state if reacquiring concurrency doesn't work 2025-03-19 17:54:47 +00:00
Eric Allam 61c0834aae Implement reserve concurrency clearing when the child run is acked 2025-03-19 17:54:47 +00:00
Eric Allam fd9b0bf676 WIP new reacquire concurrency system 2025-03-19 17:54:47 +00:00
Eric Allam cf3b23862e remove project and task current concurrency tracking 2025-03-19 17:54:47 +00:00
Eric Allam de9e296551 Add configuration for the release concurrency queue 2025-03-19 17:54:47 +00:00
Eric Allam cce402dfe7 If max tokens is 0, then don't do releasings 2025-03-19 17:54:47 +00:00
Eric Allam b7ddf20d62 Correctly use the new release concurrency queue in the run engine 2025-03-19 17:54:47 +00:00
Eric Allam bf41703dfd improve the consumer of the concurrency queue 2025-03-19 17:54:47 +00:00
Eric Allam c8b99728b9 Get all the release concurrency queue tests passing 2025-03-19 17:54:47 +00:00
Eric Allam d11d491215 WIP release concurrency queue 2025-03-19 17:54:47 +00:00
Eric Allam 1362d8296d Add batchId to the delayed enqueueRun call 2025-03-19 17:54:47 +00:00
Eric Allam 2b6ce169be remove unused method 2025-03-19 17:54:47 +00:00
Eric Allam e9fa4ce3a2 Update delays to use a redis worker and work with the new reserve concurrency system 2025-03-19 17:54:47 +00:00
Eric Allam 1a233efd42 the run engine now works with the new reserve concurrency system 2025-03-19 17:54:47 +00:00
Eric Allam 555d9912ec fixed some key producer tests 2025-03-19 17:54:47 +00:00
Eric Allam 880538ff92 improve the dead letter queue stuff 2025-03-19 17:54:47 +00:00
Eric Allam a5b2b39201 ack 2025-03-19 17:54:47 +00:00
Eric Allam 16546f144f dequeue message 2025-03-19 17:54:47 +00:00
Eric Allam 8b3551c9e2 wip reserve concurrency system 2025-03-19 17:54:47 +00:00
Eric Allam 71c7b53891 Merge pull request #1805 from triggerdotdev/queues-page
Queues page and pausing the environment/queues
2025-03-19 17:52:24 +00:00
James Ritchie 4b6f51a110 Adds js to file paths and fixes typos (#1803) 2025-03-19 17:32:03 +00:00
Matt Aitken 4e3ef9f199 Blank state if you have no queues 2025-03-19 17:21:01 +00:00
Matt Aitken 16ff502605 Better scrolling on the Queues page 2025-03-19 17:11:58 +00:00
Matt Aitken 99e722c5c9 Do all the environment concurrency lookups in parallel 2025-03-19 17:01:26 +00:00
Matt Aitken 1d5dadbc24 Remove unused number-flow package 2025-03-19 16:44:34 +00:00
Matt Aitken 110548c354 Don’t use defer, it causes a horrible UI flash with revalidate 2025-03-19 16:42:10 +00:00
Matt Aitken 5c9a305e6b Auto-reload the queue page every 10 seconds 2025-03-19 16:34:06 +00:00
Matt Aitken 46eed03743 Added pause/resume functions to the SDK 2025-03-19 14:59:37 +00:00
Matt Aitken 03fb6e2890 Redirect to the correct page, to keep your place 2025-03-19 14:39:24 +00:00
Matt Aitken 42089a4691 Pausing individual queues working 2025-03-19 14:36:53 +00:00
Matt Aitken 34a178f169 Added upgrade panel 2025-03-19 13:22:31 +00:00
Matt Aitken 8ddaa626b6 WIP on upgrade panel 2025-03-19 13:13:52 +00:00
Matt Aitken 57a57b4b87 InfoBox storybook file 2025-03-19 13:07:19 +00:00
Matt Aitken aa787c2dbb Queues page promise fix 2025-03-19 11:49:20 +00:00
Matt Aitken d8b114465f use TypedAwait 2025-03-19 11:41:46 +00:00
Matt Aitken c74d57f9fd Fix for type issue 2025-03-19 11:41:34 +00:00
Matt Aitken c2038b51a8 Retrieving a queue is working well 2025-03-19 11:18:19 +00:00
Matt Aitken e2da1810b8 WIP on retrieving a queue using the SDK/API 2025-03-19 11:09:16 +00:00
Matt Aitken 4ee85cbe8c Git ignore the react hooks src/package.json 2025-03-19 11:08:56 +00:00
Matt Aitken 2b8c77d8bf Added some JSDocs 2025-03-18 19:53:56 +00:00
Matt Aitken b4a20909f7 Don’t return more than 100 queues per page 2025-03-18 19:53:49 +00:00
Matt Aitken 714e4ecb39 Added queues.list() SDK function 2025-03-18 19:50:15 +00:00
Matt Aitken 323a12499d Merge remote-tracking branch 'origin/main' into queues-page
# Conflicts:
#	apps/webapp/app/components/SetupCommands.tsx
#	apps/webapp/app/components/navigation/SideMenu.tsx
2025-03-18 19:08:25 +00:00
Matt Aitken 50593eb24f Transform the queue type and added tooltips 2025-03-18 19:07:33 +00:00
James Ritchie a6e85f00b2 Side menu and Org dropdown improvements (#1801)
* Organize imports

* Removes animation on the dropdown menu icon

* Improves Settings and Usage button states

* Side menu sections animate with easeInOut

* Incorporates docs button into the text and adds title to the clipboard field

* Icons look better as the small variant

* Improved the typography layout of the org name and small links

* Made the divide line show when the side menu scrolls

* Made the impersonating icon amber so it stands out more

* Adds an optional amber border to the org dropdown
2025-03-18 19:04:51 +00:00
Matt Aitken 629393334d Show 25 queues per page 2025-03-18 18:58:29 +00:00
Matt Aitken 3a4991dd45 Loading improvements 2025-03-18 18:55:56 +00:00
Dan f713b1e056 Added 2 more python examples (#1800)
* Added python image processing and updated docs.json

* Edits to crawl4ai

* Updated commands

* Added pdf form extractor
2025-03-18 16:50:43 +00:00
Matt Aitken d6beb0c745 Separate the presenters out, useful for the SDK 2025-03-18 16:38:59 +00:00
Matt Aitken df53715ee5 Added app version to the org setting menu 2025-03-18 13:31:20 +00:00
Matt Aitken 343e7ec90b Merge remote-tracking branch 'origin/main' into queues-page 2025-03-18 13:16:01 +00:00
Matt Aitken 0173406999 Pausing the environment is working 2025-03-18 13:14:29 +00:00
James Ritchie ec1cec5767 Centering account pages and making them consistent (#1799)
* Small updates to the Settings page

* Remove max-w-md from Members page

* Removing max width from Profile page
2025-03-18 12:57:23 +00:00
Matt Aitken a51f8c56a4 WIP pausing 2025-03-18 09:28:14 +00:00
Matt Aitken 43413c1b41 Reduce the env data that determineEngineVersion needs 2025-03-18 09:12:05 +00:00
Matt Aitken ec4511b629 Add paused status to RuntimeEnvironment and TaskQueue 2025-03-18 09:11:39 +00:00
Matt Aitken bbe30ea625 Move increase limit button to the new spot 2025-03-17 18:54:38 +00:00
Matt Aitken d5ee760b9d Get values for the queue and running 2025-03-17 18:49:19 +00:00
Matt Aitken c6fd1816ba Current badge changed 2025-03-17 17:33:13 +00:00
Matt Aitken 6539c4a267 Added pagination 2025-03-17 16:32:02 +00:00
Matt Aitken a871137e09 WIP on queues page 2025-03-17 15:52:31 +00:00
Matt Aitken fabfb5051b WIP on queues page 2025-03-17 15:07:47 +00:00
Matt Aitken 3fd4470e89 Fix for SSE error when trying to send when the controller has been aborted 2025-03-17 14:29:51 +00:00
Matt Aitken d5561b5400 Disable vscode warning about vitest 2025-03-17 14:29:25 +00:00
Matt Aitken 2ec76d6a35 Ignore .husky 2025-03-17 14:29:01 +00:00
Matt Aitken 9a777099ee Changed CLI clipboard fields to secondary 2025-03-17 13:36:05 +00:00
Matt Aitken 7b862b9438 Major dashboard improvements (environment centric) (#1796)
* Delete the proxy app (was v2)

* Delete RunPresenterElectric

* Select the best proj/org/env

* Storing current proj/env in DB. Initial selection logic working with tasks page

* 2sm needed to be in the Tailwind merge list

* Move the task stream route (although we don’t actually use the env for now)

* Alerts moved from /v3

* API keys page moved from /v3

* Concurrency page moved from /v3

* WIP on side menu sections

* Improved the accordion animation

* Moved schedules from /v3

* More pages moved

* Move pages working

* Run page working

* Schedules working

* Moved deployments

* Alert pages moved

* Delete electric hooks, not used

* Started setting up blank states

* Test page working

* Removed “Select task” from the test page

* Some work on deployment page

* Style tweaks

* Redirect from project root to approriate env

* Improved env selector styling

* Fix for jsx errors

* Better min width on env selector

* Improved the env switching logic

* Added deployments to env routing

* Redirect deployments to the correct env

* Redirect run from proj to env

* JSX icon fix

* Only allow single env schedules from now on

* Remove env var count from the API keys page

* Move improvements and redirects

* Project settings moved

* Fix for scroll area on test page

* Tweaked the test design

* Made recent payloads column narrower

* Improved the test layout some more

* Added org icon, new project selector menu

* WIP on org switching menu

* Org switching is working

* New menu working well, removed old side menu items

* Buttons can now have a component name or an actual component for their icons

* Removed the Projects page, instead redirect appropriately

* Fix for broken blank states

* Minor run table improvements

* Removed unused switcher log and logic

* Concurrency page fix for invalid html, improved layout

* Minor improvements

* Moved the side menu to the project level

* Improved account styling

* Moved org settings pages (with redirects)

* Add current plan to billing side menu link

* Upgrade to get staging from env dropdown

* New env badge on concurrency limits page

* Show Run Engine version in span presenter

* New promote icon

* Concurrency limits page is the sum of engine v1 + v2 queues

* Fix for missing batch import

* Added currentConcurrencyOfEnvQueue function

* Basic avatar setting working

* Avatar setting is working

* You can change the color of your icon

* Avatar improvements

* Bugfix for mising prop

* Removed some old env badges

* Fixed replaying

* Removed EnvironmentLabel

* Old env badge deleted, changed everywhere to the new one

* Fix for Slack integration paths

* Fix for waitpoint completion form moving

* Bulk replay/cancel env fix

* Fix for alert webhook path

* Redirect projects/v3/* to project/*

* Fixes for CLI redirect routes

* Remove welcome email (unused)

* Change how we count schedules towards your limits

* Use new schedules limits when checking a schedule

* Added projectId back in to task queries (indexes)

* WIP dev presence

* CLI modal

* Moved things around and use Context

* Fix for p inside p

* Dev connected status on run page

* Correct dev env (not a teammates)

* Show disconnected message at the end

* Minor tweak on project dropdown icon padding

* Fix for inconsistent date format for presence

* Added a message when pushing to the billing page

* Center the team page

* Project settings page centered

* Improvements to the dev presence
2025-03-17 13:32:31 +00:00
Erin Allison 2a51ae86a2 Suppress external instrumentations for fetch calls from ApiClient (#1788)
Signed-off-by: Erin Allison <erin@eallison.us>
Co-authored-by: Matt Aitken <matt@mattaitken.com>
2025-03-16 16:45:48 +00:00
nicktrn 4fda8a5ee1 Fix prerelease script (#1794)
* clean before building

* add main branch commit protection
2025-03-15 12:20:54 +00:00
nicktrn 7842e9dff0 Warm start and restore improvements (#1793)
* handle warm start service failure on supervisor side

* export zodfetch wrapper

* add warm start client

* rename to keepalive

* add missing warm start header

* make heartbeat and snapshot poll interval configurable

* create warm start client in constructor

* add warm start run debug log

* re-enable checkpoints and improve error messages

* reduce run pod container name cardinality

* move runner id generator into core

* fix runner id import

* log when no checkpoint client and we try to restore

* move run controller constructor to the top

* fix import

* support env overrides after restore
2025-03-14 18:37:53 +00:00
Eric Allam 58961658d8 Merge pull request #1791 from triggerdotdev/cursor-rules-writing-tasks 2025-03-12 18:46:03 +00:00
James Ritchie 9d06c114b6 Fixed some copy 2025-03-12 17:56:38 +00:00
nicktrn 7f2569721e Fix supervisor builds (#1790)
* fix and restructure dockerignore

* switch to using pnpm deploy

* pass webapp node image as build arg

* ensure pnpm is downloaded at build, not runtime
2025-03-12 16:10:37 +00:00
Eric Allam 3a0475930f Merge pull request #1785 from myftija/mcp-server
Add support for running a local MCP server via the `dev` CLI command
2025-03-12 10:53:47 +00:00
Eric Allam 80e981c9f7 Merge pull request #1787 from triggerdotdev/docs/crawl-docs-updates 2025-03-11 21:32:28 +00:00
D-K-P 60c1e2c16e Updated crawl4ai docs 2025-03-11 20:42:08 +00:00
saadi 7d65aa9299 Describe the MCP server feature in the readme 2025-03-11 19:17:46 +01:00
saadi 603c903fbe Add MCP tool to list all available tasks and enable fuzzy task matching
This enables LLMs to figure out which task you are referring to, without needing to specify the full task ID
2025-03-11 18:34:17 +01:00
saadi 2e2632d5cc Add new MCP tool to list the logs for a run
This enables some basic level of debugging capabilities in combinatio with the other MCP tools
2025-03-11 16:20:56 +01:00
saadi ed1f4cea12 Expose a new API endpoint to list events for a given task run 2025-03-11 16:20:12 +01:00
Dan 282cc0629f Added Python crawling example (#1786)
* Added Python crawling (slithering?) example

* Fixed link
2025-03-11 14:16:55 +00:00
saadi b6b25b05f5 Add MCP tool to cancel task runs 2025-03-11 15:11:54 +01:00
saadi 9cd6d40acc Shut down the MCP server when the dev session stops 2025-03-11 14:53:55 +01:00
saadi 2e7b1a7b1c Make the MCP server port configurable via a flag in the dev command 2025-03-11 14:47:11 +01:00
saadi af3af9ca17 Add MCP tool to get a single run
This can be used in combination with the trigger-task tool from LLMs to show details about the run after triggering the task.
2025-03-11 14:33:51 +01:00
saadi 1d7224d2ee Add MCP tool to list runs 2025-03-11 14:22:42 +01:00
saadi ce3e49b4e3 Enable passing custom payload in the trigger-task MCP tool 2025-03-11 14:05:07 +01:00
James Ritchie 25c2fdb30a Adds writing-tasks.mdc for cursor rules 2025-03-11 12:56:54 +00:00
saadi e00afc1332 Use the v2 run engine by default in the SDK API client
V2 is the way to go for the future.
2025-03-11 12:58:11 +01:00
saadi 5c0158e98b Expose a basic version of the MCP server as an option for the dev CLI command
This just adds the main components for the MCP server. It hooks on the
existing `dev` command and can be enabled by passing the `--mcp` flag.

Currently only the `trigger-task` tool is exposed, which enables users
trigger tasks via MCP and see the resulting run.

Still WIP :)
2025-03-11 12:11:33 +01:00
Eric Allam 67115ca880 Contributing guide updates (#1784) 2025-03-11 07:54:03 +00:00
Eric Allam de4ee19d15 Merge pull request #1783 from triggerdotdev/cursor-repo-rules-and-docs
Added initial repo cursor rules and other docs for AI
2025-03-11 06:20:10 +00:00
Eric Allam 5db995856e Added repo cursor rules and other docs for AI 2025-03-10 23:03:27 +00:00
Eric Allam 70ff5de7b9 WIP 2025-03-10 13:26:13 +00:00
Matt Aitken 6cf86d5916 Deleting v2 dashboard/api (#1628)
* Delete v2 Stripe routes

* Delete v2 billing/usage pages

* Delete v2 integration pages

* Delete v2 project pages

* Deleted a load of components and services

* Deleted a load more components, presenters and services

* Deleted another 100 files or so…

* Removed old v2 paths

* Removed named icons from form titles

* Removed more string icons

* Delete NamedIcon

* Fixed some type errors

* Delete endpointApi

* Removed v2 from core/sdk

* Post merge fixes

* added explicit return types

* using the new sdk export without v3

* Delete old v2 file

* Added explicit return types because TS was complaining…

* Don’t export RuntimeEnvironmentType from two core files. Was causing TS issue

* Fix for removal of NamedIcon in new route

* Removed strange eslintrc rule

* Use the new redis client

---------

Co-authored-by: James Ritchie <james@trigger.dev>
2025-03-08 14:53:06 +00:00
Eric Allam d855d55ea0 re2: env based queue selection algo (#1775)
* re2: fix @trigger.dev/core exports

* re2: WIP env based queue selection algo

* more wip

* WIP

* Get run engine tests to pass

* Adding tests for the fair dequeueing strat in the run engine

* Configure the new queue selection strategy in the webapp and get it all building and typechecks passing

* webapp now uses built packages, building redis-worker, run-engine, database, using better tsconfig setups for tests, moving isomorphic code into core/v3/isomorphic

* Fixed webapp typechecks

* dev now depends on build, fixed supervisor typecheck

* Fixed run engine tests

* Fixed e2e tests
2025-03-07 14:30:19 +00:00
nicktrn 38ddd830d7 Add supervisor split service controls (#1774)
* dockerignore node_modules in subdirectories

* image tag action should handle re2 tags

* add supervisor containerfile

* add publish worker re2 workflow

* fix copypasta

* require branch check

* add more granular service control to supervisor session

* fix supervisor api domain for split setups

* remove default workload api domain

* option to disable workload api

* fix bool env var coercion
2025-03-07 12:40:11 +00:00
Matt Aitken 4dbf11aed1 Run Engine 2: More robust attempt failing/retrying (inc. OOM retrying) (#1773)
* Added describe to tests that were missing it

* Added a function to get the maxOldSpaceSize

* Make it easy to take `NODE_OPTIONS` and set the old space flag

* Added a zed task to rebuild the packages

* Moved isOOMRunError and added SIBABRT condition

* Deduplication flags function with tests

* Export flags file

* On TaskRunProcess, set max old space and deduplicate the flags with priority order

* Move retrying logic to a separate function, it was getting very messy

* Created new test file for attempt failures

* Allow setting retry settings for tests

* Some retrying tests, including OOM

* More failure condition tests

* Fix for OOM retrying

* Complete the attempt span if it was an OOM error

* Remove old broken import

* Fixed order of exports

* Fix for docker-provider checkpoints import
2025-03-07 09:07:49 +00:00
nicktrn e297c7f7f0 Add supervisor build pipeline (#1772)
* dockerignore node_modules in subdirectories

* image tag action should handle re2 tags

* add supervisor containerfile

* add publish worker re2 workflow

* fix copypasta

* require branch check
2025-03-06 18:22:36 +00:00
Luca D'Alessandro 3089caff0c docs: suggest updated vercel-wait action for Atomic deploys (#1771) 2025-03-06 17:22:22 +00:00
nicktrn 94fb532f5e Add RE2 supervisor (#1769)
* add new core server only export

* exit after run if warm starts disabled

* add supervisor
2025-03-06 13:20:45 +00:00
James Ritchie 78194e1a36 Corrects the number of schedules on Free tier (#1768) 2025-03-06 13:06:26 +00:00
Eric Allam cf4c4d7b65 Add attempt metrics in dev (prod WIP). Added max concurrent runs setting to dev using p-limit (#1766) 2025-03-06 11:57:01 +00:00
Matt Aitken 722fae566c Reschedule heartbeats and added a test to check it works (#1765) 2025-03-06 10:44:53 +00:00
Matt Aitken 983bb411cc Redis client factory (error handling and good defaults) (#1761)
* @internal/redis package for creating Redis clients with sensible defaults and error handling

* Added @internal/redis to webapp
2025-03-05 18:32:46 +00:00
James Ritchie 4358e2d2ef Clarifies only executing runs count towards concurrency (#1760) 2025-03-05 17:57:42 +00:00
James Ritchie 31c2a23ffc Docs improvements: clearing build cache, concurrently running next + trigger, trouble shooting pino (#1758)
* Adds “Explore by build extension” to the intro page

* Adds instructions for clearing the build cache

* Add a tip for running trigger dev + next dev concurrently

* left align table headers

* Adds trouble shooting for pino

* Adds corepack bug and workaround

* Removes reference to undici as it messes with spans

* Left align table headers

* Adds example for exporting logs to Axiom
2025-03-05 17:35:23 +00:00
Eric Allam 7b10d0b0ee re2: fixed cli e2e tests (#1757) 2025-03-05 16:58:28 +00:00
Eric Allam e89fb92532 re2: dev runs work without worker groups, fixed some type issues (#1756)
* In dev, the worker group is optional when triggering tasks (the master queue is defined by the environment). Also deprecated the TaskEvent.isDebug column and using TaskEventKind.LOG instead for debug events

* Fixed a couple of type issues

* More type fixes
2025-03-05 16:29:41 +00:00
nicktrn e97704d904 Run Engine 2.0 (WIP) (#1575)
* bump worker version

* Suggested glossary for the RunEngine, TBC

* Removed BatchTaskRun changes from this branch, they were done in main

* Set the BatchTaskRun status to completed when all runs are completed

* When dequeuing respect passed in maxResources

* Ported over the new run props: idempotencyKeyExpiresAt, versions, oneTimeUseToken, maxDurationInSeconds

* Didn’t hit save… the new props when triggering tasks passed through

* Idempotency expiration + waitpoint edge case

* WIP on creating checkpoint, parking for now

* fix worker routes

* upgrade webapp node types to support generic event emitter

* separate event bus handler singleton and run failure alerts

* duration waits

* fix execution snapshot debug spans

* task waits

* fix event bus types

* temporary fix for react hook run handle type

* disable run notifications for now

* convert any typecasts to expect errors to more easily fix later

* fix webapp types after node types upgrade

* updateEnvConcurrencyLimits across marqs and the runqueue

* Pass proper values into the run engine

* RunQueue settings and removed unused rebalancing workers

* Remove rebalancing prop

* Tidied more things up

* Update/remove queue limits for MARQS and RunQueue

* taskQueue/concurrencyLimit changes ported back into the RunEngine

* Reworked completing waitpoints to improve performance and reduce race conditions

* Improved test robustness

* Down to a single run lock only when a run is totally unblocked and ready to continue

* warm starts, worker notifications, wait fixes

* Fix for Run Engine poll interval env var

* Expect the waitpoint to be completed quickly

* If a run is locked then it’s too late to expire it

* Added VALKEY_ env vars and plugged them into the run engine

* Extracted and updated the guard queue function so it can be used when batching

* Added logging and universal concurrency changes to trigger task v1

* Added notes back in

* Bump @trigger.dev/worker to 3.3.7

* reportInvocationUsage for the runAttemptStarted event

* improve execution snapshot span debug span start times

* Unfriendly IDs

* update lockfile

* Created a shared determineEngineVersion function

* disable unfinished commands

* save new cli config to different location, misc fixes

* add basic engine version check via current deploy

* new run engine will default to node 22 runtime

* block some actions for projects on previous run engine

* fix worker group tests

* fix triggerAndWait test

* one typescript version to rule them all

* redlock type patch

* fix type issues caused by ts-reset

* improve cleanup scripts

* add missing socket.io dep

* fix run notification handler type

* fix worker group test again

* generate prisma client for e2e tests

* remove worker group tests for now

* prevent image pull rate limits during unit tests

* increase timeout for queue concurrency limit test

* generate prisma client for preview release

* same node types everywhere

* Updated engine readme, removed legacy system notes

* use default machine preset from platform package

* worker instances plural in schema

* disable pnpm update notifications

* return worker group details from connect call

* add workers admin route

* fix heartbeat route return type

* move deployment labels to core apps

* refactor run controller env schema

* Add firstAttemptStartedAt to TaskRun

* RunEngine 2.0 batch trigger support (#1581)

* Make it clear when BatchTriggerV2Service is used

* Copy of BatchTriggerV2Service

* WIP batch triggering

* Allow blocking a run with multiple waitpoints at once. Made it atomic

* Removed unused param

* New batch service

* Pass through the parentRunId and resumeParentOnCompletion

* Use the new batch service, and correct trigger task version

* Force V1 engine if using BatchTriggerV2Service, we’ve already done the check at this point

* Removed the $transaction and early exit if nothing changed

* Adedd a simple batch task to the hello world reference catalog

* Fix for batch waits not working

* Added parentRunId in a couple more places

* Removed waitForBatch log

* Added another parentRunId

* Expanded the example to include all the different triggers

* More changes to blocking to support continuing after idempotent completed runs

* Fix for the wrong type when blocking a run

* remove @map

* optimise worker auth query

* add engine version header to core api client requests

* remove unique constraint for default group id

* consolidate migrations

* the first managed worker becomes the global default

* Debug events off by default, added an admin toggle to show them

* worker group name can't be an empty string

* add exec helper to core

* move machine resources to core

* add pre-dequeue callback to determine max resources

* optionally skip dequeue

* bump worker package

* move worker to core

* fix ReadableStream type error

* fix another type issue

* update a few more tsconfigs

* add metadata changes introduced in #1563

* Run Engine 2.0 trigger idempotency (#1613)

* Return isCached from the trigger API endpoint

* Fix for the wrong type when blocking a run

* Render the idempotent run in the inspector

* Event repository for idempotency

* Debug events off by default, added an admin toggle to show them

* triggerAndWait idempotency span

* Some improvements to the reference idempotency task

* Removed the cached tracing from the SDK

* Server-side creating cached span

* Improved idempotency test task

* Create cached task spans in a better way

* Idempotency span support inc batch trigger

* Simplified how the spans are done, using more of the existing code

* Improved the idempotency test task

* Added Waitpoint Batch type, add to TaskRunWaitpoint with order

* Pass batch ids through to the run engine when triggering

* Added batchIndex

* Better batch support in the run engine

* Added settings to batch trigger service, before major overhaul

* Allow the longer run/batch ids in the filters

* Changed how batching works, includes breaking changes in CLI

* Removed batch idempotency because it gets put on the runs instead

* Added `runs` to the batch.retrieve call/API

* Set firstAttemptStartedAt when creating the first attempt

* Do nothing when receiving a BATCH waitpoint

* Some fixes in the new batch trigger service… mostly just passing missing optional params through

* Tweaked the idempotency test task for more situations

* Only block with a batch if it’s a batchTriggerAndWait… 🤦‍♂️

* Added another case to the idempotency test task: multiple of the same idempotencyKey in a single batch

* Support for the same run multiple times in the same batch

* Small tweaks

* Make sure to complete batches, even if they’re not andWait ones

* Export RunDuplicateIdempotencyKeyError from the run engine

* Latest lockfile

* Trigger with a machine (old run engine)

* RE2, allow setting machine when triggering

* Fix for new glob patterns

* add max run count to dequeue from version route

* add worker instance name env var and header

* queue consumer pre skip callback

* poll for more runs after final execution errors

* fix dequeue search param schema

* add shortcut to debug switch

* expose run engine timeouts as env vars

* make warm start durations configurable

* add optional status to json reply helper

* fix preSkip hook, add debug logs

* BLOCKED_BY_WAITPOINTS -> SUSPENDED

* exit controller when run suspended

* check if already replied before http reply

* run controller will wait for next run after the current one is suspended

* cancel run button shortcut

* minimal event repository environment type

* fix update metadata call

* run suspension and misc fixes wip

* change debug shortcut to shift + D

* Started work on the Dev supervisor

* Formatting

* Fix for bad imports

* Before rebuilding SSE

* Presence updating from the CLI working via SSE

* add worker notification debug logs

* send run:stop when exiting run phase

* skip current snapshot poll on worker notification

* add more logs and route to submit run debug logs

* add worker and runner ids to snapshots

* improve run notification debug logs

* add workload debug log route

* misc run controller fixes and refactor

* prevent parallel execution of critical functions

* update bun to 1.2.1

* WIP with dev dequeuing

* Method to convert friendlyIds to non-friendly, do nothing with actual ids

* Set the engine on BackgroundWorker, lazily upgrade projects to engine V2

* Runs with ttls were getting immediately expired… oops.

* Pass the Waiting for deploy reason through, so we have it on the execution snapshots

* Fixed the logic for getting the right background worker for a run

* Use the correct ID when dequeuing…

* determineEngineVersion is now fully functional

* Rate limiter ignores the dev endpoints

* Retrieving a batch gives you the runIds

* Set a unique version for the RE2 BatchTaskRun

* add provisional changeset

* The start of dev run execution is working

* First dev run working

* Moved the dev run controller closer to what Nick did with the managed one

* export exec output type

* Heartbeat fix: don’t heartbeat if _isHeartbeating == false

* Dev runs get notifications, some dev bug fixes

* Improved logging or dequeuing

* We need to dequeue runs from the latest version too, for triggerAndWait

* Ported Eric’s validateWorkerManifest with nicer errors

* When flattening an idempotency key if part is undefined, return undefined

* Dev logging fixes

* Remove sigterm listener

* Deprecating workers. Don’t specify a BackgroundWorker when dequeuing an environment

* Deleted some old files. Renamed “managed” to “deploy”

* When a build finishes, always copy the build dir (otherwise the first one gets trampled on by the 2nd)

* Dev master queues should work differently

* Deleting old workers

* Added debounce function to core

* Improvement to canceling

* WIP on debounce canceling on socket disconnection

* Added environment data to execution snapshots

* Dev runs that have stalled get “Canceled” with a reason explaining why

* Show CLI messaged when a connection to the platform is lost/restored

* Fix TriggerTask after merge

* Add trigger task v2 max attempts, replace some findUniques

* Port the new queue logic to the run engine

* More fixes post-merge

* We weren’t setting a `retryConfig` up for the tests… it’s now required

* Start the Redis worker inside the Run Engine… 🤦‍♂️

* Trying to make the testcontainers more reliable

* Added keyPrefix: "engine:”

* Badly placed bracket in trigger task

* Better Redis namespacing

* Fix for expired run not getting removed from the queue

* Don’t create a redis client in the testcontainers, return the redisOptions instead

* Cleanup redis client in the run lock tests

* Fix for the RunQueue not supporting keyPrefix

* Updated more of the RunQueue scripts rebalancing

* Trying to make Redis more robust in the tests…

* Improved test resiliciency more

* Fix for delays (checkpoint check)

* Increase the timeout slightly to fix ttl test

* Added priority support when triggering

* More wip trying to make test containers more reliable

* batchTriggerAndWait test is still failing… some wip to try fix it

* Fixed redis tests now we’re not providing a client

* Separate Redis clients for the run engine worker/queue/runlock

* Made the wait for duration test more resilient

* Added idempotencyKeyExpiresAt to Waitpoints

* Waitpoint timeouts and idempotency expiry

* Use finishWaitpoint, removed extra worker job

* Added waitpoint idempotency tests

* Creating resume tokens is working

* Some improvements to the resume tokens

* Moved resumeTokens to just be wait functions 🥳

* Delete old RuntimeManagers

* Wait for token is working

* Better test for the wait tokens

* Improved the test task some more

* Hide the accessories in the span inspector

* WIP on waitpoint inspector

* WIP on complete waitpoint form

* Span overview panel can be changed based on the entity type

* Improved the waitpoint display

* WIP on completing waitpoint form

* Use the existing CodeBlock for the tip

* Style improvements

* Complete waitpoint

* All waitpoint sidebar variants

* Waits now use a pause icon

* Durations waits use the API to create/block with a waitpoint, not the runtime

* Fix for engine.blockRunWithWaitpoint required org id

* Removed old wait code from the run controllers/task run process

* Form action for skipping a datetime waitpoint

* Move testDockerCheckpoint to a separate core package export (it can’t be bundled on the client)

* Fix for glitchy hourglass animation

* Completed waitpoints display better

* Increase Redis maxRetriesPerRequest to 20 (default)

* Completing and skipping waitpoints is working

* Remove the database prisma dev command, since we need to use create only now. Updated docs

* Added skip timeout, reworked the UI

* Tweaked spacing

* Added payload limit to waitpoint token completion from dashboard

* Test idempotency works on wait.for and wait.until

* Moved the worker-actions to /engine/ from /api/

* Moved dev engine endpoints to /engine/ from /api/

* Separate /engine/ rate limiter

* Added parallel wait prevention, it’s working for duration waits but not well for triggerAndWait yet

* WIP post-merge conflicts

* Set taskEventStore column in the new engine

* Remove duplicate keys

* Post-merge fixes

* Fix for span merge layout

* Use executedAt instead of firstAttemptStartedAt

---------

Co-authored-by: Matt Aitken <matt@mattaitken.com>
2025-03-05 14:40:14 +00:00
Matt Aitken c519a5afc7 Convert the number part of the version to a number, from a string (#1754)
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 0s
🚀 Publish Trigger.dev Docker / units (push) Failing after 4s
🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker (push) Has been skipped
⚒️ Publish Worker / build (coordinator) (push) Has been cancelled
⚒️ Publish Worker / build (docker-provider) (push) Has been cancelled
⚒️ Publish Worker / build (kubernetes-provider) (push) Has been cancelled
2025-03-05 08:29:16 +00:00
Eric Allam 5730faea00 Release 3.3.17 (#1752)
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 0s
🚀 Publish Trigger.dev Docker / units (push) Failing after 10s
🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker (push) Has been skipped
2025-03-04 12:47:31 +00:00
github-actions[bot] 58f5955c66 chore: Update version for release (#1710)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2025-03-04 12:40:41 +00:00
James Ritchie 7162f07216 Updates sveltekit community plugin links (#1746) 2025-03-04 12:30:50 +00:00
Eric Allam 8f3a28effb pythonExtension and python runtime improvements (#1734)
* pythonExtension and python runtime improvements

* Adding streaming support

* Use writeFileSync

* Restructure extension docs and add python extension docs

* Fix broken link

* Update docs/config/extensions/overview.mdx

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>

* Update docs/config/extensions/aptGet.mdx

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>

* Update docs/config/extensions/custom.mdx

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>

* Add environment variable support

---------

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
2025-03-04 12:30:26 +00:00
Eric Allam bbd82adbda docs: improve deployment docs and add atomic deploy guide (#1748)
* docs: improve deployment docs and add atomic deploy guide

* Various docs fixes
2025-03-04 12:00:06 +00:00
Eric Allam edf0c1024f Publish docker image when internal-packages change (#1750) 2025-03-04 11:18:29 +00:00
Eric Allam 457273e4cf Fix email alert failures because of newlines in error messages (fixes #1745) (#1749) 2025-03-04 10:54:39 +00:00
Dan c315e38d76 Updated imports (#1747) 2025-03-03 20:21:28 +00:00
Eric Allam e0241819a4 fix: needsPromotion fixed in deploy github actions (#1741) 2025-03-01 10:29:44 +00:00
Dan 796e710439 Added claude thinking chatbot docs (#1743) 2025-03-01 09:57:22 +00:00
Eric Allam c8732becbd Updates @clack/prompts to 0.10.0 to improve CI output when using the spinner (#1740)
* Updates @clack/prompts to 0.10.0 to improve CI output when using the spinner

* Add changeset
2025-03-01 07:13:58 +00:00
Dan 5164935f9c Added X-Frame-Options and CSP (#1690) 2025-02-28 11:06:27 +00:00
James Ritchie e1db12b8ba Expandable code blocks (#1683)
* Removed unnecessary classes from Dialog

* Display large modal with code and copy button

* Fixes button position if chrome is shown

* Removes the duplication of the Highlight component
2025-02-28 09:17:56 +00:00
Eric Allam 187200a1bc Feat: Improved run start timeline visibility (#1732)
* Record cold start and execution metrics on attempt executions. Add cold start metrics as span events on attempt spans and display them in the run dashboard

* Add deployed tasks run timeline metrics

* Add Dequeued event to run timeline and cleanup the run timeline code

* Adds variants to storybook

* WIP adding new span styles

* Added offset progress bar animation

* More storybook states

* Adds support for the full vertical span to show the same state

* Adds error state to timelineLine

* Added additional state

* Added more line styling

* Added progress state to dequeued

* Added another state to storybook

* Fixed classname error

* Updated styles for the span timeline points

* Fixes alignment of timeline follow cursor indicator

* Adds help text tooltip to timeline span type titles

* Fixes type error

* Tweaked wording of tooltips

* Fixed type error (check this)

* Moved isAdmin to a higher level

* removed unused admin props

* Removed unused Admin filter

* Fixed border styling

* made the opacity of the timeline states 30% less

* Undo type cast

* Added a diminished style that’s used for spans (grey progress bar)

* Adds new storybook state

* Fixed timeline state

* Removed state if span isn’t the first

* Changed the timestamp span icon

---------

Co-authored-by: James Ritchie <james@trigger.dev>
2025-02-27 20:47:48 +00:00
Eric Allam 26f9a1e478 Feat: two phase deployment, version pinning (#1739)
* WIP two-phase deployments

* Fix the help text

* Rename TRIGGER_WORKER_VERSION to TRIGGER_VERSION

* Add changeset

* A few naming fixes
2025-02-27 20:44:23 +00:00
Eric Allam 0e5ec8bfbc Only allow a single dev queue consumer to dequeue at once (#1737)
* Only allow a single dev queue consumer to dequeue at once

* Expire dev queue consumer connections keys

* Add missing return
2025-02-27 15:08:47 +00:00
Eric Allam d2ceb825a7 Fix issue where a typescript file with only types causes indexing to fail (#1735)
* Fix issue where a typescript file with only types causes indexing to fail

* Create ninety-cars-peel.md
2025-02-27 15:03:14 +00:00
Dan 376a0c5d04 Update README.md 2025-02-26 15:37:27 +00:00
Victor Duarte 06f60f2720 feat(build): Add support for Python scripts via pythonExtension (#1686)
* add PythonExtension

* remove potential shell injection risk

* Filter out blank lines or comment lines

* fix spelling

* add pythonExtension's `runInline`

* changes to requirements don’t invalidate the entire install layer

* copy script files on-demand

* improve PythonExtension types and logging

* add changeset

* fix broken imports

* Improve security of inline script execution

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>

* Add file existence check for requirementsFile

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>

* update lock file

* Enhance error handling with detailed error information

* Add portable type annotation

* fix error TS18046: 'e' is of type 'unknown'

* export the python extension

* add `pythonExtension` to the catalog

* fix `Cannot find module '@trigger.dev/build/extensions/core' (TS2307)

* replace execa by tinyexec

* Update pnpm-lock.yaml

* add custom traces instead of logging

* The cleanup in the finally block does not fail silently anymore

* move python runtime/extension to independent package

* fix build package readme

* update lock file

* add documentation to python's package

* add missing dependency

* Update little-trains-begin.md

---------

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Co-authored-by: Matt Aitken <matt@mattaitken.com>
Co-authored-by: Eric Allam <eallam@icloud.com>
2025-02-26 13:39:28 +00:00
Gustavo Fior ca05a544d5 add missing imports to overview.mdx (#1731)
added missing imports:

import { auth } from "@trigger.dev/sdk/v3";
import { tasks } from "@trigger.dev/sdk/v3";
2025-02-25 21:00:38 +00:00
Dmitrii Malakhov 8044433320 Typo fix cli-commands-deploy.mdx (#1728) 2025-02-25 08:42:26 +00:00
James Ritchie 412eb81ee1 Explain TTL even better so the LLMs hopefully understand it (#1726) 2025-02-21 16:11:16 +00:00
Eric Allam 62e3238ecc Add configurable redis TTL on realtime streams (#1725) 2025-02-21 16:07:12 +00:00
James Ritchie 5a250cf92d Adds help and fix for “No loader is configured” (#1724) 2025-02-21 10:09:41 +00:00
nicktrn 80f2fc8bbe Add scheduled at timestamp to created run containers (#1722)
* add scheduled at timestamp to k8s provider

* add scheduled at timestamp to docker provider
2025-02-20 17:27:38 +00:00
Eric Allam bf43fd4f8e Use priority offsets in MarQS and schedule future messages using redis worker (#1720)
* Improve the MarQS priority system by moving future messages into the LRE worker and using a priority timestamp offset to define priority in messages

* Add a test to make sure priority offsets don't unfairly favor environments

* requeuing should clear concurrency sets

* Heartbeats should only reschedule existing heartbeat jobs

* Fix type error
2025-02-20 14:06:32 +00:00
nicktrn 7186b1e428 Add support for deferred checkpoints (#1721)
* add ability to delay checkpoints

* optional checkpoint delays for dependency waits

* prevent checkpoint creation for resumed batches

* unpause after checkpoint rejected by platform

* prevent checkpoint creation for resumed task waits

* fix checkpoint cancellation
2025-02-20 12:47:09 +00:00
James Ritchie 99f2caf5c2 Docs/new reference section (#1719)
* Moves API reference to a new dropdown section

* Move toubleshooting higher up

* Separates the reference Overview page into new pages

* Removed Projects api and redirects old page to overview

* typo
2025-02-20 11:30:32 +00:00
James Ritchie e19bf2abaf Docs/more docs improvements (#1718)
* Format the machine override example code

* Adds note to add `fluent-ffmpeg` to external

* Updates onFailure with more clarification

* Adds debugging in vs code to Troubleshooting

* Removed link
2025-02-19 16:06:42 +00:00
Eric Allam 23b4c2a214 Add MarQS requeueMessage, an atomic version of replaceMessage (#1717)
* New MarQS method requeueMessage, an atomic version of replace message

* Remove redundant call to remove the message from the env queue in requeueMessage
2025-02-19 15:11:43 +00:00
Eric Allam 7b1159eb45 MarQS reserve concurrency system & queue priority for resuming/retrying (#1715)
* run engine v1: orgs are no longer considered for concurrency

* Add reserve concurrency concept to allow waiting to resume parent tasks to release concurrency at the env level for child tasks to use (or else there is a deadlock). WIP recursive tasks

* child tasks inherit the queue timestamp from their parent tasks to prioritize completing child tasks based on when their parent started

* handle reserve concurrency with recursive deadlocks

* Finish docs update for concurrency

* Some fixes from badge conflict resolution

* WIP priority queues

* Implement MarQS priority queues

* Fix the migrations
2025-02-19 11:43:34 +00:00
James Ritchie c9f7ea87c3 Docs/various docs improvements (#1716)
* Adds more Prisma documentation

* Adds more info and a diagram to improve idempotencyKeyTTLs

* Adds more clarity to the run with TTL docs section

* Limits page copy improvements

* Adds schema properties to the alert webhooks
2025-02-19 11:13:29 +00:00
James Ritchie d6c869eb8f Docs/community packages (#1714)
* typo

* Adds feature list to the sveltekit plugin page

* Removed Migrations section in side menu

* Tidy up the table links

* Creates a community packages section

* Small tweak

* Adds info and links to community packages

* Updated mintlify broken links version in workflow file

* fixed broken link
2025-02-18 13:13:07 +00:00
James Ritchie 31a43273df Latest lock file 2025-02-18 11:01:14 +00:00
Erin Allison b4d1619732 Remove V2 countdown from sidebar (#1712)
Signed-off-by: Erin Allison <erin@eallison.us>
Signed-off-by: Erin Allison <eallison@andrettikarting.com>
Co-authored-by: James Ritchie <james@trigger.dev>
2025-02-18 10:59:12 +00:00
Vinny ferddy-ogo lagueule 1e194a53e5 Bump next from 14.2.15 to 14.2.21 in /references/nextjs-realtime (#1583)
Bumps [next](https://github.com/vercel/next.js) from 14.2.15 to 14.2.21.
- [Release notes](https://github.com/vercel/next.js/releases)
- [Changelog](https://github.com/vercel/next.js/blob/canary/release.js)
- [Commits](https://github.com/vercel/next.js/compare/v14.2.15...v14.2.21)

---
updated-dependencies:
- dependency-name: next
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-02-17 18:40:09 +00:00
nicktrn 8f3f373002 Improve reliability of completion submission (#1711)
* add timeout support to sendWithAck

* coordinator will retry completion submission

* actually retry

* increase default retries

* something went wrong there, add this back in

* add changeset
2025-02-14 15:14:42 +00:00
nicktrn 36fea7d645 Add support for manual checkpoints (#1709)
* don't crash run on lazy attempt errors and count on retry

* remove abort controller after checkpoint in all cases

* duration checkpoints

* manual checkpoints

* add busybox to coordinator image

* add async toggle to process in background

* remove early returns

* add changeset for manual checkpoint schema
2025-02-14 11:35:10 +00:00
James Ritchie 37723344cd Adds SvelteKit to the frameworks (#1707) 2025-02-13 17:30:38 +00:00
Matt Aitken e61b29ae52 Release 3.3.16 2025-02-13 15:31:55 +00:00
github-actions[bot] 4dbf217edb chore: Update version for release (#1705)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2025-02-13 15:25:08 +00:00
James Ritchie a62cb3e60b Agent docs examples (#1706)
* Uses image cards for the frameworks

* Removes old snippets

* New AI agents side menu section

* WIP adding new ai agent pages

* Better overview page

* More copy added to the agent example pages

* Copy improvements

* Removes “Creating a project” page and side menu section

* Fixes broken links

* Updates to the latest Mintlify version, fixes issues, changes theme

* Adds descriptions to the main dropdown menu items

* Reformatted Introduction docs ‘landing page’

* Retry heartbeat timeouts by putting back in the queue (#1689)

* If there’s a heartbeat error and no attempts we put it back in the queue to try again

* When nacking, return whether it was put back in the queue or not

* Try and nack, if it fails then fail the run

* Consolidated switch statement

* Fail executing/retrying runs

* OOM retrying on larger machines (#1691)

* OOM retrying on larger machines

* Create forty-windows-shop.md

* Update forty-windows-shop.md

* Only retry again if the machine is different from the original

* Kubernetes OOMs appear as non-zero sigkills, adding support for treating these as OOMs

* Complete the original attempt span if retrying due to an OOM

* Revert "Complete the original attempt span if retrying due to an OOM"

This reverts commit 5f652c6212.

* chore: Update version for release (#1666)

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* Release 3.3.14

* Set machine when triggering docs

* Batch queue runs that are waiting for deploy (#1693)

* Detect ffmpeg OOM errors, added manual OutOfMemoryError (#1694)

* Detect ffmpeg OOM errors, added manual OutOfMemoryError

* Create eighty-spies-knock.md

* Improved the machines docs, including the new OutOfMemoryError

* chore: Update version for release (#1695)

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* Release 3.3.15

* Create new partitioned TaskEvent table, and switch to it gradually as new runs are created (#1696)

* Create new partitioned TaskEvent table, and switch to it gradually as new runs are created

* Add env var for partition window in seconds

* Make startCreatedAt required in task event store

* Don't create an attempt if the run is final, batchTriggerAndWait bad continue fix (#1698)

* WIP fix for ResumeAttemptService selecting the wrong attempt (which has no error or output)

* Don’t create an attempt if the run is already in a final status

* Don’t get all the columns for the query. Improved the logging.

* Added a log to the batch example

* Filter out the undefined values

* Fix missing logs on child runs by using the root task run createdAt if it exists (#1697)

* Provider changes to support image cache (#1700)

* add env var for additional pull secrets

* make static images configurable

* optional image prefixes

* optional labels with sample rates

* add missing core paths

* remove excessive logs

* Fix run container exits after OOM retries (#1701)

* remove unused imports

* tell run to exit before force requeue

* handle exit for case where we already retried after oom

* improve retry span and add machine props

* don't try to exit run in dev

* Upgrade local dev to use electric beta.15 (#1699)

* Text fixes

* Removed pnpm files

---------

Co-authored-by: Matt Aitken <matt@mattaitken.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Eric Allam <eric@trigger.dev>
Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
2025-02-13 14:51:43 +00:00
Matt Aitken 440d413ce8 Alert Webhook improvements (#1703)
* WIP with webhook SDK function and types

* JSDocs added to the schema

* Webhooks are working

* Expanded the alert docs

* Remove duplicate export of waitUntil.js

* Use uncrypto

* Don’t rate limit webhooks

* Create slow-olives-fix.md
2025-02-13 13:44:36 +00:00
Eric Allam a2c70b450a Upgrade local dev to use electric beta.15 (#1699) 2025-02-13 11:02:01 +00:00
nicktrn a38f713e32 Fix run container exits after OOM retries (#1701)
* remove unused imports

* tell run to exit before force requeue

* handle exit for case where we already retried after oom

* improve retry span and add machine props

* don't try to exit run in dev
2025-02-12 18:32:58 +00:00
nicktrn 0924b37283 Provider changes to support image cache (#1700)
* add env var for additional pull secrets

* make static images configurable

* optional image prefixes

* optional labels with sample rates

* add missing core paths

* remove excessive logs
2025-02-12 16:57:55 +00:00
Eric Allam d622aba86d Fix missing logs on child runs by using the root task run createdAt if it exists (#1697) 2025-02-12 14:03:48 +00:00
Matt Aitken d214aa9093 Don't create an attempt if the run is final, batchTriggerAndWait bad continue fix (#1698)
* WIP fix for ResumeAttemptService selecting the wrong attempt (which has no error or output)

* Don’t create an attempt if the run is already in a final status

* Don’t get all the columns for the query. Improved the logging.

* Added a log to the batch example

* Filter out the undefined values
2025-02-12 14:02:29 +00:00
Eric Allam bd0cc541c5 Create new partitioned TaskEvent table, and switch to it gradually as new runs are created (#1696)
* Create new partitioned TaskEvent table, and switch to it gradually as new runs are created

* Add env var for partition window in seconds

* Make startCreatedAt required in task event store
2025-02-12 09:12:08 +00:00
Matt Aitken f0726affa2 Release 3.3.15 2025-02-12 00:42:27 +00:00
github-actions[bot] 0ceb37f0e8 chore: Update version for release (#1695)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2025-02-12 00:37:38 +00:00
Matt Aitken 252bf8d806 Improved the machines docs, including the new OutOfMemoryError 2025-02-12 00:25:03 +00:00
Matt Aitken efd2d21762 Detect ffmpeg OOM errors, added manual OutOfMemoryError (#1694)
* Detect ffmpeg OOM errors, added manual OutOfMemoryError

* Create eighty-spies-knock.md
2025-02-12 00:02:01 +00:00
Matt Aitken 6017c522e1 Batch queue runs that are waiting for deploy (#1693) 2025-02-11 19:06:38 +00:00
Matt Aitken a50b93faa4 Set machine when triggering docs 2025-02-11 12:43:45 +00:00
Matt Aitken d4043fc258 Release 3.3.14 2025-02-11 12:29:54 +00:00
github-actions[bot] 6ec786c7b2 chore: Update version for release (#1666)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2025-02-11 12:28:35 +00:00
Matt Aitken d2ee232fae Revert "Complete the original attempt span if retrying due to an OOM"
This reverts commit 5f652c6212.
2025-02-11 11:33:07 +00:00
Matt Aitken 5f652c6212 Complete the original attempt span if retrying due to an OOM 2025-02-11 11:01:21 +00:00
Matt Aitken 036a5061e1 Kubernetes OOMs appear as non-zero sigkills, adding support for treating these as OOMs 2025-02-11 10:32:01 +00:00
Matt Aitken bb65b2614a OOM retrying on larger machines (#1691)
* OOM retrying on larger machines

* Create forty-windows-shop.md

* Update forty-windows-shop.md

* Only retry again if the machine is different from the original
2025-02-10 17:43:59 +00:00
Matt Aitken 4dd42cd6d4 Retry heartbeat timeouts by putting back in the queue (#1689)
* If there’s a heartbeat error and no attempts we put it back in the queue to try again

* When nacking, return whether it was put back in the queue or not

* Try and nack, if it fails then fail the run

* Consolidated switch statement

* Fail executing/retrying runs
2025-02-10 13:41:32 +00:00
Matt Aitken bc7d44582d If there’s no queue concurrency limit, default to the env concurrency limit (not 1m) (#1687) 2025-02-09 21:34:40 +00:00
Victor Duarte 2771fd7408 Missing construction option in AudioWaveformExtension (#1684)
* missing construction option in `AudioWaveformExtension`

* Create lemon-fireants-repair.md

---------

Co-authored-by: Matt Aitken <matt@mattaitken.com>
2025-02-09 15:52:46 +00:00
James Ritchie 1f1a6b0405 Fixed runs page filter bug (#1681) 2025-02-07 17:32:30 +00:00
Eric Allam 9f843d0a0f Remove v2 event list (#1680) 2025-02-07 13:51:34 +00:00
Eric Allam 9dfd6a5a0a Rate limit alerts by channel for task run alerts using generic cell rate algo (#1679) 2025-02-07 13:36:37 +00:00
Eric Allam 5210d3a8bb Add internal spans to event repo (#1678) 2025-02-07 13:36:15 +00:00
Eric Allam b5867627be Use idempotency key index when finding existing runs by idempotency key in batch (#1677) 2025-02-07 13:36:03 +00:00
Eric Allam 0b555faf01 Force upgrade v1 batches to v3 (#1676)
* Force upgrade v1 batches to v3

* Actually lets process v1 batches async
2025-02-06 22:37:26 +00:00
Eric Allam dcf1ab6f38 Remove schedule constraints from TaskRun to prevent db load issues when a schedule or instance is deleted (#1674) 2025-02-06 19:16:48 +00:00
Matt Aitken c49af774ba Retry batch item completion (#1675)
* Added isPrismaRetriableError()

* Retry completeBatchTaskRunItem if they fail because of a retriable Prisma error

* Retry using Redis worker

* Handle more retriable errors. Add special condition in for race condition error

* Added Postgres connection_timeout with default 20s

* Added a simple batchTriggerAndWait example
2025-02-06 19:16:30 +00:00
Eric Allam f00ed9b13e Add the sha256 digest to the deployment image reference (#1673) 2025-02-06 19:14:34 +00:00
Eric Allam 3ec6983e45 Move run ttl and delays from graphile to redis worker (#1672) 2025-02-06 14:25:37 +00:00
Eric Allam a2c2d920b7 Use redis worker for run heartbeats and alerts (#1669)
* Move the task run heartbeats to RedisWorker

* Move alerts to redis worker, improving redis worker

* Fix typecheck errors

* Use single threaded tests for redis worker

* Enable/disable the redis workers independently

* Remove preview release from PR checks
2025-02-06 10:21:57 +00:00
nicktrn b946b9f38e Add optional env var and label to disable image pre-pulls (#1667)
* add env var to disable pre-pulls

* add label to disable pre-pulls on specific nodes
2025-02-05 16:49:57 +00:00
Eric Allam 99d38151f6 Auto-fix config.dirs of /trigger and /src/trigger (#1665) 2025-02-05 13:00:18 +00:00
Eric Allam e35b6f539a Release 3.3.13 (#1664)
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 0s
🚀 Publish Trigger.dev Docker / units (push) Failing after 0s
🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker (push) Has been skipped
2025-02-04 16:32:26 +00:00
github-actions[bot] 20d983e08a chore: Update version for release (#1654)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2025-02-04 16:31:00 +00:00
Eric Allam 3887c4a621 Fix "no tasks defined" issue (fix #1681) (#1663) 2025-02-04 16:08:57 +00:00
Eric Allam e61573600b Reduce contention on batchTaskRun when setting expected count (#1662) 2025-02-04 13:43:50 +00:00
Eric Allam 53e6d478ed Fix deploy timeout issues (#1661)
* deploy v2 streaming WIP

* finalize deployment now SSE and won't timeout

* handle zodfetchSSE connection errors

* Create tender-cycles-melt.md
2025-02-04 12:44:14 +00:00
nicktrn 7fa45ad337 Fix pre-pulling of deployment images onto worker nodes (#1660)
* lint

* get pre-pull data from finalized deployment
2025-02-04 10:07:07 +00:00
Eric Allam 42c5f4d9fa Add text_pattern_ops index on SecretKey.key to improve getSecrets perf (#1659) 2025-02-03 22:52:16 +00:00
Eric Allam 6f52b00560 Set taskRunAttemptId on batch task run items on completion (#1658) 2025-02-03 15:25:12 +00:00
D-K-P d285f7930e Added guides and examples links and snippet 2025-02-03 14:30:03 +00:00
Eric Allam b377b577ca db efficiency improvements (#1657)
- remove calls to findUnique, especially when including/selecting relations
- add some missing indexes
- add spans to $transaction calls to help track down long running txs
2025-02-03 13:59:58 +00:00
Eric Allam b67ea9a2f5 improve batch completion system for run engine v1 (#1656)
* Automatically retry TriggerTaskService when hitting a unique constraint error on idempotency key

* improve batch completion system for run engine v1

* Rename batch stuff to v3 so it's not confusing

* Handle unique constraint error on BatchTaskRunItem creation and allow different limits for batchTrigger and batchTriggerAndWait
2025-02-03 11:29:51 +00:00
D-K-P de31220820 Docs correction 2025-01-31 15:15:23 +00:00
Eric Allam 3641647add Add the keyPrefix option when redis is not in cluster mode (#1655) 2025-01-31 10:50:52 +00:00
D-K-P 664bce561d Added vercel ai sdk image generator example to the docs 2025-01-30 15:39:28 +00:00
Eric Allam 73471589ef Fixed issue with asResponse and withResponse not working on runs.retrieve (#1648) 2025-01-30 14:01:38 +00:00
Eric Allam 813d73d1a3 Fix issue with heavy contention on TaskQueue updating concurrency limit (#1653)
* Fix issue with heavy contention on TaskQueue updating concurrency limit

* Remove concurrency limit when creating background worker tasks
2025-01-30 14:01:11 +00:00
nicktrn 9ad396bdb1 Fix custom queue docs 2025-01-30 13:53:48 +00:00
James Ritchie 4e7c55920d Docs maxDuration page now states it’s a required param 2025-01-29 19:30:25 +00:00
Eric Allam 3eede3c339 Better handle redis error events (#1652) 2025-01-29 17:27:52 +00:00
Eric Allam b3afbcf558 Correctly configure redis cluster for aws elasticache (#1651) 2025-01-29 16:56:51 +00:00
Eric Allam f43de6ab74 Support redis/valkey cluster mode (#1650) 2025-01-29 16:28:51 +00:00
mintlify[bot] 3127df5b67 Add new Bulk Actions section to documentation (#1649)
* Documentation edits made through Mintlify web editor

* Moves files to correct path and improves copy

* Corrects video props to camel case and removes deprecated prop

* Link to the new bulk actions page from the replaying page

---------

Co-authored-by: mintlify[bot] <109931778+mintlify[bot]@users.noreply.github.com>
Co-authored-by: James Ritchie <james@trigger.dev>
2025-01-29 16:01:01 +00:00
Eric Allam 9fd175e860 Prevent findUnique from bringing down our database (#1633) 2025-01-29 13:57:16 +00:00
Eric Allam ef0fc6b0b4 Support for logically separated redis instances (#1647)
* Remove unnecessary disabled org check

* Add support for separate redis servers for caching, rate limiter, marqs, and the pub/sub stuff (they all default to the previously used env vars for redis)
2025-01-29 13:53:58 +00:00
Eric Allam a24b2a7732 More v1 run engine fixes (#1644)
* Remove the socket count from shared queue consumer cause it can take up to 5s to complete

* Add support for re-using queue snapshots across dequeues

* Only consider the top N orgs when dequeuing, to help mitigate large spikes in queues (like around the hour and half hour marks)
2025-01-28 16:16:37 +00:00
Eric Allam 6da5e7a4cf Various fixes for run engine v1 (#1643)
* Various fixes for run engine v1

- Make sure there are connected providers before sending a scheduled attempt message, nack and retry if there are not
- Fail runs that fail task heartbeats when pending and locked
- More and better logging around shared queue consumer
- Fix bug when failing a task run with no attempt

* Prevent findUnique from bringing down our database
2025-01-27 17:15:17 +00:00
Eric Allam 75753ec5ad Release 3.3.12 2025-01-24 19:19:55 +00:00
github-actions[bot] 40e0d2f7e9 chore: Update version for release (#1626)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2025-01-24 19:18:39 +00:00
Eric Allam 246eb4361e If a deployment is already deployed, just return the deployment so the request succeeds (#1639) 2025-01-24 18:47:26 +00:00
nicktrn ab6e6afbf5 switch to github builders 2025-01-24 18:15:40 +00:00
Eric Allam 7445d9bf35 install ca-certificates in docker image (#1638) 2025-01-24 18:01:42 +00:00
Eric Allam f6eeaa1001 Fix broken cloud deploys by using depot ephemeral registry, skip the registry proxy (#1637) 2025-01-24 17:22:35 +00:00
Eric Allam eaf46bac58 engine v1 improved dequeue selection algorithm (#1632)
* engine v1 fairness improvements (env based queue selection)

* Add ability to bias env selection by concurrency limit and capacity

* Add some randomization to the list of queues within each evn

* Configure biases using env vars

* Typo

* revert v3 catalog project ref change

* Fix v2 marqs
2025-01-23 15:01:42 +00:00
James Ritchie 16cf357380 Focus visible style refinements (#1577)
* reduces the thickness of the focus-visible

* More subtle Input style

* Updates focus-visible styles for the SimpleSelect

* Tables tab through rows and action cells only

* Refined focus visible style

* Adds the custom focus visible to the TextLink

* Moves the onClick handling to the table row rather than the cell

* Makes table row heights consistent

* Adds gap between task search bar and button

* Prevents long tag values from wrapping

* Removes unnecessary rows from table header

* Added gap between search and filters

* Fixes the schedules table pagination staying fixed to the bottom of the page

* Removed more unnecessary header table rows to prevent them being selectable

* Removed isSelected styles (not working)

* Added <tr> back to the main Table compontent

* Table row handles modifier keys

* Adds to={path} to the TableRow only

* Revert "Adds to={path} to the TableRow only"

This reverts commit 8a814d434ff85b8ab9c2305d336074f659d86094.

* Revert "Table row handles modifier keys"

This reverts commit b2394741fa2ed6b908e00673b1a395ae89ca7146.

* Table reverted to use linked cells rather than rows

* Set the tab index of a cell and style the table row when tabbed

* Tabbed row style applied to the sticky cells

* Adds isTabbableCell to each table

* Improves the spcificity of the row highlighting

* Reduces the height of the task rows to match the other tables

* Adds tab styles to fill in row dividers top and bottom

* Removed old row onClick and to props

* Removed duplicate table header row

* Creates table style variants so tables look good in the inspector panels

* Full width tables on the schedules inspector panel

* Reduce padding on Alerts page table rows

* Updates the Deploy page inspector table to the new style

* Removes the duplicate Table Row from the table headers in v2

* Adds TableRow to the EventsTable

* Adds TableRow to more tables

* Adds TableRow to more tables

* Adds TableRow to more tables

* Adds TableRow to more tables

* Fix for showing correct cursor for linked table rows

* Removed variants from some of the table components
2025-01-21 21:11:59 +00:00
Eric Allam f96bf7210b Engine v1 improvements (#1627)
* Efficiency improvements to SharedQueueConsumer and MarQS for run engine v1

* Reliabily resolve task queue for a run and ack runs where we can't find the queue

* engine v1: stop nacking after reaching the max nack count, ack to remove the message

* Handle division by 0 possibility in distributeQueues
2025-01-21 14:24:02 +00:00
James Ritchie c855e4a96c v3 pages have page titles (#1582)
* Adds tasks page title

* Adds page titles to the <head>
2025-01-21 12:07:32 +00:00
Eric Allam e1497c802d Update config docs for new telemetry property (#1610) 2025-01-20 10:29:57 +00:00
Eric Allam e2612224b4 Fix "crypto is not defined error" on Node.js 18 in dev (#1623)
* Add --experimental-global-webcrypto node option fix "crypto is not defined error" on Node.js 18 in dev

* Only add flag when on node 18

* Missed the function
2025-01-20 10:29:26 +00:00
Matt Aitken 0d38ea0feb Updated the batch REST API docs to the new endpoint 2025-01-17 13:28:02 +00:00
Eric Allam 0687ce722c Release 3.3.11 (#1621)
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 4s
🚀 Publish Trigger.dev Docker / units (push) Failing after 4s
🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker (push) Has been skipped
2025-01-17 12:06:41 +00:00
github-actions[bot] d627fa718e chore: Update version for release (#1611)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2025-01-17 12:05:15 +00:00
Matt Aitken 5798024a5b Require max duration in trigger.config. Validate TRIGGER_ACCESS_TOKEN is a PAT. (#1620)
* Show a good error if TRIGGER_ACCESS_TOKEN isn’t a PAT

* Require maxDuration in trigger.config from now on

* Changeset

* Update the trigger.config templates copy, and set the default to 1hr

* Set the maxDuration in the trigger.config files inside the repo

* Fixed the e2e tests by setting the package to 0.0.0-prerelease-20250116195421
2025-01-17 11:32:55 +00:00
Eric Allam 5750f35add Shared queue consumer telemetry improvements and perf changes (#1619)
* Shared queue consumer telemetry improvements

* findUnique -> findFirst to stop the prisma dataloader issue on api key auth

* Added action stats, audited all actions and improved reasons

* Better debug logging filtering
2025-01-17 09:15:21 +00:00
Matt Aitken 11f2832ae5 Fix for detecting internal errors 2025-01-16 17:03:39 +00:00
Matt Aitken fd04ed50ef Prevent multiple waits callback delayed by one tick
To ensure the first wait doesn't checkpoint before the second is called
2025-01-16 16:23:56 +00:00
Matt Aitken 6d17443e16 Changeset: Detect parallel waits and show a useful error message 2025-01-16 15:48:55 +00:00
Matt Aitken dd321e3616 Throw an error for concurrent waits (with nice docs link) (#1618)
* WIP preventing concurrent waits, throw an error

* Added ConcurrentWaitError (not retryable)

* Move preventMultipleWaits out of the RuntimeAPI

* Added preventMultipleWaits to the devRuntimeManager

* Added throwable InternalError. Plus new TASK_DID_CONCURRENT_WAIT code

* Docs link for troubleshooting concurrent waits

* Docs for troubleshooting concurrent waits

* preventMultipleWaits function

* Added TASK_DID_CONCURRENT_WAIT code

* Deal with InternalErrors that skipRetrying

* Added preventMultipleWaits to prod
2025-01-16 14:33:14 +00:00
Gabe O'Leary 2ad664d17d Add note about prisma-json-types-generator (#1617) 2025-01-15 23:18:42 +00:00
Matt Aitken 5eb30d026d Improve worker deployment queries (#1616)
* workerDeployment change findUnique to findFirst

* Changed the worker deployment queries to only select what they need, some of the columns are huge

* Only get the bare minimum data for tasks too
2025-01-15 20:34:28 +00:00
nicktrn 719ae83057 Add support for specifying machine preset at trigger time (#1608)
* add machine preset trigger option

* add changeset

* add examples to changeset

* machinePreset -> machine

* task options now also accept machine as simple string

* update openapi spec
2025-01-15 13:47:13 +00:00
Eric Allam 7eda22851e Prevent v2 batches from being resumed before all items have been created (#1614) 2025-01-14 14:37:04 +00:00
Eric Allam b3aa87ee26 Add custom telemetry exporter support (#1602)
* WIP langsmith & AI SDK integration

* Add changeset

* Add exporter support to deployed tasks

* Better support for external exporters and group exporters and instrumentations under the telemetry config property

* Missing changes
2025-01-13 22:43:34 +00:00
Eric Allam 14ce599218 Add reason to MarQS acknowledgeMessage to track why it's called (#1607) 2025-01-13 17:29:09 +00:00
Eric Allam 0a18e1d919 Additional dev queue consumer logging (#1606) 2025-01-13 16:37:19 +00:00
Eric Allam ca51f65394 Truncate ingested log messages to 4096, and truncate message to 256 in the summary tree view (#1604) 2025-01-13 11:35:39 +00:00
Eric Allam 46a37fdc49 Release 3.3.10 (#1601)
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 0s
🚀 Publish Trigger.dev Docker / units (push) Failing after 4s
🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker (push) Has been skipped
2025-01-10 19:52:30 +00:00
github-actions[bot] 24383dc65d chore: Update version for release (#1600)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2025-01-10 19:50:41 +00:00
Eric Allam e13edd3a20 Handle errors thrown by requests in Realtime react hooks (#1599) 2025-01-10 19:40:34 +00:00
Eric Allam 9e0f03623d Support ignoring test/spec files inside trigger dirs (fixes #1593) (#1596)
* Support ignoring test/spec files inside trigger dirs (fixes #1593)

* Add changeset
2025-01-10 19:40:08 +00:00
Eric Allam 914ceaf3c5 dev CLI: cleanup deprecated background worker files (fixes #1572) (#1595)
* dev CLI: cleanup deprecated background worker files (fixes #1572)

* Add changeset
2025-01-10 19:39:34 +00:00
nicktrn 472e5f93e1 Fix: Prevent failed resumes for very large batches (#1598)
* findUnique -> findFirst

* remove tx

* make query more legible

* cut resume payload queries in half

* remove the additional status checks

* tweak the catalog batch parent task

* fix error message
2025-01-10 18:33:25 +00:00
D-K-P 51a054f534 Added Realtime CSV importer to the docs 2025-01-09 17:13:55 +00:00
Eric Allam efd61f1374 Release 3.3.9 (#1591)
🚀 Publish Trigger.dev Docker / units (push) Failing after 3s
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 10s
🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker (push) Has been skipped
2025-01-09 11:35:16 +00:00
github-actions[bot] 2c7f7b1d5e chore: Update version for release (#1586)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2025-01-09 11:34:03 +00:00
Eric Allam c92edd61af New metadata system doc updates (#1590) 2025-01-09 11:13:48 +00:00
Eric Allam ab69b60348 New metadata deployed task fix (#1588)
* Add more logging around new metadata system

* Make the authenticated env optional when updating run metadata
2025-01-08 22:42:34 +00:00
Dan d52e3d7d57 Added LLM evaluator to example projects (#1589)
* Added repo links

* Added links to remix / nextjs webhooks examples

* Added a Realtime learn more snippet

* Added the batch llm evaluator docs

* Improved snippet

* Updated intro
2025-01-08 21:16:56 +00:00
Eric Allam 3ca0c15161 Add ability to disable metadata flushing and logging (#1587) 2025-01-08 12:37:43 +00:00
James Ritchie 0aa597104c Shortcut improvements (#1573)
* Added shortcuts (with tooltip) to the pagination

* WIP fixing the disabled hover state

* Styled the tooltip

* Added a shortcuts panel to the help menu

* Adds new shortcut to list

* Changes “meta” for “mod”

* Adds more shortcuts to the list

* Adding shortcut to open the shortcuts panel

* tweak gap between shortcut letters

* button component now has icon spacing adjustment (for lucide icons)

* Fixed some ilegal markup

* Pagination uses disabled prop rather than a disabled wrapper

* Improved the Switch styles

* Makes the shortcut modifier optional

* Added new icon based shortcut keys for mac and win

* Updated PC modifier shortcuts

* Adds a new windows key icon

* Allows variants and react nodes to be used as the modifier key

* Adds more shortcuts to the storybook

* Adds missing focus-visible styles to the pagination

* Removed test modifier keys

* number style is tabular

* Update apps/webapp/app/components/primitives/ShortcutKey.tsx

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>

* Tooltip now just 1 prop on the button component

---------

Co-authored-by: Matt Aitken <matt@mattaitken.com>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
2025-01-08 11:58:18 +00:00
Eric Allam 34f8bd588e Add ability to update parent and root run metadata from children (#1563)
* Increase the number of active streams from 2 to 5

* WIP parent metadata updates

* Fix noop metadata manager

* Implement run metadata updates from ancestor tasks

* Add changeset

* Add ability to stream into parent and root task runs

* Don't duplicate exporting run metadata types

* Add ability to fetch streams through metadata

* Couple of fixes from CodeRabbit

* Fix metadata tests

* Make sure streams are subscribed in the "background"

* Move the stream subscription stuff to the API client, expose it through `runs.fetchStream`

* Fixed run stream tests
2025-01-08 11:29:24 +00:00
Eric Allam 4243cb2451 Release 3.3.8
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 0s
🚀 Publish Trigger.dev Docker / units (push) Failing after 0s
🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker (push) Has been skipped
2025-01-08 10:29:06 +00:00
github-actions[bot] 7e8a0d96d2 chore: Update version for release (#1570)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2025-01-08 10:26:37 +00:00
Willow (GHOST) 4d19dec71a fix: add user-agent to the websocket request (#1584) 2025-01-07 22:21:32 +00:00
Eric Allam 76a5ac293e Realtime fixes: safari, timezones, and TIMED_OUT status (#1585)
* Fix realtime safari bug because of missing ReadableStream async iterable support
Limit to only safari

* Fix missing TIMED_OUT run status

* When coercing realtime date strings, make sure they are set to UTC
2025-01-07 20:47:49 +00:00
Dan 668b34d25e Adding example projects (#1564)
* Added new side menu section and updated intro

* Updated the fal ai realtime project

* Copy update

* Added example project

* Swapped walkthrough and repo

* Restore original fal task page

* Added links back to the intro
2025-01-03 20:00:40 +00:00
James Ritchie 6a2d033f32 Project setup improvements (#1579)
* Adds ‘an existing project’ to step 1

* When you click pnpm or yarn, the other tabs change too
2024-12-30 18:07:07 +00:00
James Ritchie e71df640a9 Adds more org names to the alert emails (#1561)
Co-authored-by: Matt Aitken <matt@mattaitken.com>
2024-12-30 17:54:38 +00:00
James Ritchie 330f6c548e Softened the language in the Pro upgrade modal (#1574) 2024-12-30 17:49:00 +00:00
Erin Allison 4d2412a0fe Add Support for Alternate Email Transports (#1580)
* Add support for AWS SES and SMTP email transports

Signed-off-by: Erin Allison <eallison@andrettikarting.com>

* Correct environment variable names for new email settings

Signed-off-by: Erin Allison <eallison@andrettikarting.com>

* Correct option names being passed to nodemailer for SMTP

Signed-off-by: Erin Allison <eallison@andrettikarting.com>

* Remove use of AWS SDK synthetic default export

It apparently causes issues when transpiled/bundled

Signed-off-by: Erin Allison <eallison@andrettikarting.com>

* Add documentation for new email settings

Signed-off-by: Erin Allison <eallison@andrettikarting.com>

* Move nodemailer types to devDependencies

Signed-off-by: Erin Allison <eallison@andrettikarting.com>

* Adjust mail transport error handling

Gotta keep the linter happy :)

Signed-off-by: Erin Allison <eallison@andrettikarting.com>

* Fix typecheck error on MailTransportOptions

Signed-off-by: Erin Allison <eallison@andrettikarting.com>

* Correct environment variable usage for SMTP email transport

Signed-off-by: Erin Allison <eallison@andrettikarting.com>

---------

Signed-off-by: Erin Allison <eallison@andrettikarting.com>
2024-12-30 17:38:17 +00:00
Matt Aitken 21a4fabc28 Docs update batch limits to 5MB 2024-12-18 11:47:50 +00:00
Eric Allam 3418aa614b Don't expire enqueued runs if they have lockedAt set (#1571) 2024-12-16 19:00:18 +00:00
Eric Allam 88a0754ae0 Support scopes packages in externals when found in a transitive dep #1552 (#1569)
* Support scopes packages in externals when found in a transitive dep

* fixed typescript errro

* Add changeset
2024-12-16 17:55:44 +00:00
Eric Allam f3e4597ef4 Update scrape-hacker-news.mdx 2024-12-14 21:11:37 +00:00
Eric Allam 2bf04ece54 Release 3.3.7 (#1562)
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 0s
🚀 Publish Trigger.dev Docker / units (push) Failing after 0s
🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker (push) Has been skipped
2024-12-13 15:49:52 +00:00
github-actions[bot] 295b2eb3b6 chore: Update version for release (#1558)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2024-12-13 15:48:04 +00:00
James Ritchie f0182abc83 Alert emails now contain the Org name (#1555)
* Alert emails now contain the Org name

* Adds org title to the alert parameters
2024-12-13 13:32:34 +00:00
James Ritchie 1105b9b71a 3 small improvements (#1560)
* Updated login page logos

* Adds an isSelected state to the Table

* Toast style now matches the design

* Adds a space between the upgrade panel and the list of users
2024-12-13 13:32:19 +00:00
Eric Allam 7077dbf65e @trigger.dev/react-hooks now works with React 19 (#1559) 2024-12-13 13:31:47 +00:00
Eric Allam 6b355ab9ad Upgrades and fixes to Realtime and Realtime streams (#1549)
* Fix streaming splits in realtime streams v2

* Add changeset

* Skip all flaky tests 😡

* Improve the way we stream from tasks to the server

* Improve the v1 realtime streams (Redis)

* Turn on the relay realtime stream service

* Improved the relay realtime cleanup

* Fixed consuming realtime runs w/streams after the run is already finished

* Remove some logs

* Update changeset

* Fixed runStream tests
2024-12-13 11:42:50 +00:00
Niloy 68d7139ebe Update deploy-environment-variables.mdx (#1509)
base64 with -i for input file
2024-12-13 11:16:47 +00:00
James Ritchie c58f6f7a68 Upgrading from Hobby to Pro displays a dialog to confirm (#1545)
* Upgrading from Hobby to Pro displays a dialog to confirm

* Improved the logic for upgrading to pro

* Added the spinner component to storybook

* Improved the dark style spinner
2024-12-13 11:14:47 +00:00
James Ritchie 6516e15bd8 Changes frozen to waiting (#1551)
* Offset the tooltip in more

* The Select component’s shortcut has a better hover state

* “Frozen” changed to “Waiting”
2024-12-13 11:13:34 +00:00
James Ritchie 7f2365f4e0 Adds a tooltip to the Status header showing deployment status descriptions (#1554) 2024-12-13 11:11:14 +00:00
James Ritchie f560594610 Tabs horizontally overflow scroll in the side panel inspectors (#1556) 2024-12-13 11:08:04 +00:00
Matt Aitken ebd9767926 When replaying, using the concurrencyLimit from the original run’s queue 2024-12-12 16:48:22 +00:00
Matt Aitken 1c631a1b6f Skip the RealtimeClient tests for now in CI… flaky 2024-12-12 14:39:35 +00:00
Matt Aitken 110a8ea48c Added queue updating logging. For Triggering changed the concurrency limit calculation to match deploying 2024-12-12 13:49:20 +00:00
James Ritchie 10f0ef3301 Adds v2 countdown eol text in side menu (#1546) 2024-12-11 12:24:23 +00:00
nicktrn 2e4a6307b4 Fix: Prevent some attempt creation retries from causing system failures (#1550) 2024-12-11 12:01:18 +00:00
Eric Allam 30ea5eb13a Release 3.3.6 (#1544)
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 4s
🚀 Publish Trigger.dev Docker / units (push) Failing after 19s
🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker (push) Has been skipped
2024-12-10 15:53:45 +00:00
github-actions[bot] 5846f30228 chore: Update version for release (#1537)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2024-12-10 15:52:05 +00:00
Eric Allam 3afa42c209 Make TRIGGER_REALTIME_STREAM_VERSION overridable by the project env vars (#1543) 2024-12-10 14:54:50 +00:00
Eric Allam 86b1628953 Add run metadata to runs.list endpoint (#1542) 2024-12-10 12:52:42 +00:00
Matt Aitken ea23dbd297 Throw an error if Resend responds with an error (#1540) 2024-12-10 12:37:36 +00:00
Matt Aitken 9065e64be8 Added logging when we remove a queue’s concurrency limit 2024-12-10 09:57:41 +00:00
Eric Allam 9970b9b68e Realtime streams now powered by electric (#1541)
* Realtime streams now powered by electric, and fix the streaming update duplicate issues by converting the electric Shape materialized view into a ReadableStream of changes

* Ensure realtime subscription stops when runs are finished, and add an onComplete handle to use realtime hooks

* Fix tests
2024-12-09 22:09:30 +00:00
Matt Aitken b4113134ad Fix: test form: update the callback when the task changes 2024-12-08 22:07:42 +00:00
Eric Allam 2a07ea42f1 Optionally trigger batched items sequentially to preserve order (#1536)
* Optionally trigger batched items sequentially to preserve order

* Fix infinite v3.processBatchTaskRun enqueuings by checking the attemptCount
2024-12-05 15:16:06 +00:00
Eric Allam 91afa5ebbf Release 3.3.5 (#1535)
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 3s
🚀 Publish Trigger.dev Docker / units (push) Failing after 5s
🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker (push) Has been skipped
2024-12-03 18:13:54 +00:00
github-actions[bot] 65262dc3d7 chore: Update version for release (#1534)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2024-12-03 18:12:28 +00:00
Eric Allam 9105701ae0 Fix cancelled runs breaking realtime subscriptions (#1533) 2024-12-03 16:55:27 +00:00
2284 changed files with 189725 additions and 92683 deletions
+5
View File
@@ -0,0 +1,5 @@
---
"trigger.dev": patch
---
Added experimental_devProcessCwdInBuildDir config option to opt-in to new process.cwd behavior when executing tasks in the dev CLI. Currently process.cwd maps to the "root" of your trigger.dev project (the directory that contains your trigger.config.ts file). Setting experimental_devProcessCwdInBuildDir to true changes process.cwd to instead be the temporary build directory inside of the .trigger directory.
+5
View File
@@ -0,0 +1,5 @@
---
"trigger.dev": patch
---
Fix dev runs
+6
View File
@@ -0,0 +1,6 @@
---
"trigger.dev": patch
"@trigger.dev/core": patch
---
The dev command will now use the platform-provided engine URL
+10
View File
@@ -0,0 +1,10 @@
---
"@trigger.dev/react-hooks": patch
"@trigger.dev/sdk": patch
"trigger.dev": patch
"@trigger.dev/build": patch
"@trigger.dev/core": patch
"@trigger.dev/rsc": patch
---
Run Engine 2.0 (alpha)
+5
View File
@@ -0,0 +1,5 @@
---
"@trigger.dev/sdk": patch
---
fix: Logging large objects is now much more performant and uses less memory
+5
View File
@@ -0,0 +1,5 @@
---
"trigger.dev": patch
---
Fix update command version mismatch detection
+5
View File
@@ -0,0 +1,5 @@
---
"@trigger.dev/sdk": patch
---
New internal idempotency implementation for trigger and batch trigger to prevent request retries from duplicating work
+7 -1
View File
@@ -12,7 +12,13 @@
"access": "public",
"baseBranch": "main",
"updateInternalDependencies": "patch",
"ignore": ["webapp", "proxy", "coordinator", "docker-provider", "kubernetes-provider"],
"ignore": [
"webapp",
"coordinator",
"docker-provider",
"kubernetes-provider",
"supervisor"
],
"___experimentalUnsafeOptions_WILL_CHANGE_IN_PATCH": {
"onlyUpdatePeerDependentsWhenOutOfRange": true
}
+5
View File
@@ -0,0 +1,5 @@
---
"trigger.dev": patch
---
Add external log exporters and fix missing external trace exporters in deployed tasks
+5
View File
@@ -0,0 +1,5 @@
---
"@trigger.dev/sdk": patch
---
When you create a Waitpoint token using `wait.createToken()` you get a URL back that can be used to complete it by making an HTTP POST request.
+5
View File
@@ -0,0 +1,5 @@
---
"trigger.dev": patch
---
Enhance deploy command output to better distinguish between local and remote builds
+5
View File
@@ -0,0 +1,5 @@
---
"@trigger.dev/core": patch
---
Configurable queue consumer count in supervisor session
+5
View File
@@ -0,0 +1,5 @@
---
"trigger.dev": patch
---
Fixes a bug that would allow processes that had OOM errors to be incorrectly reused when experimental_processKeepAlive was enabled
+6
View File
@@ -0,0 +1,6 @@
---
"trigger.dev": patch
"@trigger.dev/core": patch
---
Runtime agnostic SDK config via env vars
+12
View File
@@ -0,0 +1,12 @@
---
"@trigger.dev/redis-worker": major
"@trigger.dev/react-hooks": major
"@trigger.dev/sdk": major
"trigger.dev": major
"@trigger.dev/python": major
"@trigger.dev/build": major
"@trigger.dev/core": major
"@trigger.dev/rsc": major
---
Trigger.dev v4 release. Please see our upgrade to v4 docs to view the full changelog: https://trigger.dev/docs/upgrade-to-v4
+5
View File
@@ -0,0 +1,5 @@
---
"@trigger.dev/core": patch
---
Add supervisor http client option to disable debug logs
+5
View File
@@ -0,0 +1,5 @@
---
"@trigger.dev/sdk": patch
---
Fixed an issue with realtime streams that timeout and resume streaming dropping chunks
+6
View File
@@ -0,0 +1,6 @@
---
"trigger.dev": patch
"@trigger.dev/core": patch
---
Expose esbuild `keepNames` option (experimental)
+5
View File
@@ -0,0 +1,5 @@
---
"@trigger.dev/sdk": patch
---
The envvars.list() and retrieve() functions receive isSecret for each value. Secret values are always redacted.
+6
View File
@@ -0,0 +1,6 @@
---
"trigger.dev": patch
"@trigger.dev/core": patch
---
Add `experimental_autoDetectExternal` trigger config option
+6
View File
@@ -0,0 +1,6 @@
---
"trigger.dev": patch
"@trigger.dev/core": patch
---
Add project details to the whoami command
+5
View File
@@ -0,0 +1,5 @@
---
"@trigger.dev/sdk": patch
---
Fix issue where realtime streams would cut off after 5 minutes
+5
View File
@@ -0,0 +1,5 @@
---
"@trigger.dev/sdk": patch
---
Deprecate toolTask and replace with `ai.tool(mySchemaTask)`
+5
View File
@@ -0,0 +1,5 @@
---
"trigger.dev": patch
---
Log images sizes for self-hosted deploys
+6
View File
@@ -0,0 +1,6 @@
---
"@trigger.dev/sdk": patch
"trigger.dev": patch
---
Display clickable links in Cursor terminal
+5
View File
@@ -0,0 +1,5 @@
---
"trigger.dev": patch
---
Fix init.ts in custom trigger dirs
+6
View File
@@ -0,0 +1,6 @@
---
"trigger.dev": patch
"@trigger.dev/core": patch
---
Add import timings and bundle size analysis, the dev command will now warn about slow imports
+5
View File
@@ -0,0 +1,5 @@
---
"@trigger.dev/core": patch
---
Improve structured logs
+5
View File
@@ -0,0 +1,5 @@
---
"@trigger.dev/build": patch
---
syncVercelEnvVars() fix for syncing the wrong preview branch env vars
+5
View File
@@ -0,0 +1,5 @@
---
"trigger.dev": patch
---
Fix update command version range handling
+45
View File
@@ -0,0 +1,45 @@
---
"@trigger.dev/sdk": patch
---
Removes the `releaseConcurrencyOnWaitpoint` option on queues and the `releaseConcurrency` option on various wait functions. Replaced with the following default behavior:
- Concurrency is never released when a run is first blocked via a waitpoint, at either the env or queue level.
- Concurrency is always released when a run is checkpointed and shutdown, at both the env and queue level.
Additionally, environment concurrency limits now have a new "Burst Factor", defaulting to 2.0x. The "Burst Factor" allows the environment-wide concurrency limit to be higher than any individual queue's concurrency limit. For example, if you have an environment concurrency limit of 100, and a Burst Factor of 2.0x, then you can execute up to 200 runs concurrently, but any one task/queue can still only execute 100 runs concurrently.
We've done some work cleaning up the run statuses. The new statuses are:
- `PENDING_VERSION`: Task is waiting for a version update because it cannot execute without additional information (task, queue, etc.)
- `QUEUED`: Task is waiting to be executed by a worker
- `DEQUEUED`: Task has been dequeued and is being sent to a worker to start executing.
- `EXECUTING`: Task is currently being executed by a worker
- `WAITING`: Task has been paused by the system, and will be resumed by the system
- `COMPLETED`: Task has been completed successfully
- `CANCELED`: Task has been canceled by the user
- `FAILED`: Task has failed to complete, due to an error in the system
- `CRASHED`: Task has crashed and won't be retried, most likely the worker ran out of resources, e.g. memory or storage
- `SYSTEM_FAILURE`: Task has failed to complete, due to an error in the system
- `DELAYED`: Task has been scheduled to run at a specific time
- `EXPIRED`: Task has expired and won't be executed
- `TIMED_OUT`: Task has reached it's maxDuration and has been stopped
We've removed the following statuses:
- `WAITING_FOR_DEPLOY`: This is no longer used, and is replaced by `PENDING_VERSION`
- `FROZEN`: This is no longer used, and is replaced by `WAITING`
- `INTERRUPTED`: This is no longer used
- `REATTEMPTING`: This is no longer used, and is replaced by `EXECUTING`
We've also added "boolean" helpers to runs returned via the API and from Realtime:
- `isQueued`: Returns true when the status is `QUEUED`, `PENDING_VERSION`, or `DELAYED`
- `isExecuting`: Returns true when the status is `EXECUTING`, `DEQUEUED`. These count against your concurrency limits.
- `isWaiting`: Returns true when the status is `WAITING`. These do not count against your concurrency limits.
- `isCompleted`: Returns true when the status is any of the completed statuses.
- `isCanceled`: Returns true when the status is `CANCELED`
- `isFailed`: Returns true when the status is any of the failed statuses.
- `isSuccess`: Returns true when the status is `COMPLETED`
This change adds the ability to easily detect which runs are being counted against your concurrency limit by filtering for both `EXECUTING` or `DEQUEUED`.
+5
View File
@@ -0,0 +1,5 @@
---
"trigger.dev": patch
---
Init command will now correctly install v4-beta packages
+5
View File
@@ -0,0 +1,5 @@
---
"trigger.dev": patch
---
Fix metadata collapsing correctness
+5
View File
@@ -0,0 +1,5 @@
---
"trigger.dev": patch
---
Improve warm start times by eagerly creating the child TaskRunProcess when a previous run as completed
+11
View File
@@ -0,0 +1,11 @@
---
"trigger.dev": patch
"@trigger.dev/core": patch
---
- Resolve issue where CLI could get stuck during deploy finalization
- Unify local and remote build logic, with multi-platform build support
- Improve switch command; now accepts profile name as an argument
- Registry configuration is now fully managed by the webapp
- The deploy `--self-hosted` flag is no longer required
- Enhance deployment error reporting and image digest retrieval
+5
View File
@@ -0,0 +1,5 @@
---
"@trigger.dev/redis-worker": patch
---
Now each worker gets it's own pLimit concurrency limiter, and we will only ever dequeue items where there is concurrency capacity, preventing incorrectly retried jobs due to visibility timeout expiry
+5
View File
@@ -0,0 +1,5 @@
---
"trigger.dev": patch
---
Fix init.ts detection when using the sentry esbuild plugin
+8
View File
@@ -0,0 +1,8 @@
---
"trigger.dev": patch
"@trigger.dev/core": patch
---
- Correctly resolve waitpoints that come in early
- Ensure correct state before requesting suspension
- Fix race conditions in snapshot processing
+5
View File
@@ -0,0 +1,5 @@
---
"trigger.dev": patch
---
experimental processKeepAlive
+5
View File
@@ -0,0 +1,5 @@
---
"trigger.dev": patch
---
Fixes runLimiter check on #dequeueRuns
+5
View File
@@ -0,0 +1,5 @@
---
"trigger.dev": patch
---
Update nypm package to support test-based bun.lock files
+98
View File
@@ -0,0 +1,98 @@
{
"mode": "pre",
"tag": "v4-beta",
"initialVersions": {
"coordinator": "0.0.1",
"docker-provider": "0.0.1",
"kubernetes-provider": "0.0.1",
"supervisor": "0.0.1",
"webapp": "1.0.0",
"@trigger.dev/build": "3.3.17",
"trigger.dev": "3.3.17",
"@trigger.dev/core": "3.3.17",
"@trigger.dev/python": "3.3.17",
"@trigger.dev/react-hooks": "3.3.17",
"@trigger.dev/redis-worker": "3.3.17",
"@trigger.dev/rsc": "3.3.17",
"@trigger.dev/sdk": "3.3.17"
},
"changesets": [
"beige-horses-juggle",
"big-carrots-fail",
"blue-eyes-tickle",
"breezy-turtles-talk",
"chatty-snakes-hope",
"chilled-weeks-switch",
"clean-beans-compete",
"cuddly-boats-press",
"curvy-dogs-share",
"eight-ligers-help",
"eighty-rings-divide",
"fifty-beers-bake",
"flat-pianos-live",
"four-needles-add",
"fuzzy-snakes-beg",
"gentle-waves-suffer",
"gold-insects-invite",
"green-lions-relate",
"grumpy-wasps-fold",
"healthy-apricots-drop",
"hip-cups-wave",
"honest-files-decide",
"itchy-frogs-care",
"itchy-games-sort",
"late-chairs-ring",
"late-dancers-smile",
"lazy-panthers-shop",
"lazy-plums-fetch",
"light-peas-melt",
"little-birds-appear",
"moody-squids-count",
"nasty-cobras-wonder",
"nice-colts-boil",
"ninety-games-grow",
"orange-pens-smile",
"orange-rocks-grow",
"plenty-dolphins-act",
"polite-badgers-suffer",
"polite-impalas-care",
"polite-lies-fix",
"rare-beds-accept",
"real-rats-drop",
"red-chairs-begin",
"red-rings-marry",
"red-wasps-cover",
"shiny-kiwis-beam",
"silly-cows-serve",
"silly-timers-repair",
"sixty-beers-share",
"small-birds-arrive",
"small-dancers-smell",
"smart-coins-hammer",
"smooth-planets-flow",
"soft-candles-do",
"sour-mirrors-accept",
"spotty-ducks-punch",
"spotty-pants-wink",
"strong-pianos-provide",
"sweet-dolphins-invent",
"tender-jobs-collect",
"thick-bikes-laugh",
"tidy-books-smell",
"tiny-buckets-teach",
"tricky-houses-invite",
"twelve-actors-hide",
"two-eagles-report",
"two-tigers-dream",
"weak-jobs-hide",
"weak-parents-sip",
"wet-deers-think",
"wet-dragons-boil",
"wet-steaks-reflect",
"wicked-ads-walk",
"wild-mirrors-return",
"witty-cherries-tan",
"witty-donkeys-unite",
"yellow-cars-tell"
]
}
+5
View File
@@ -0,0 +1,5 @@
---
"@trigger.dev/core": patch
---
Add verbose structured log level
+5
View File
@@ -0,0 +1,5 @@
---
"@trigger.dev/sdk": patch
---
Add onCancel lifecycle hook
+5
View File
@@ -0,0 +1,5 @@
---
"trigger.dev": patch
---
Added AI assistance link when you have build errors
+5
View File
@@ -0,0 +1,5 @@
---
"@trigger.dev/sdk": patch
---
Provide realtime skipColumns option via untamperable public access tokens
+5
View File
@@ -0,0 +1,5 @@
---
"@trigger.dev/core": patch
---
Suppress external instrumentation for fetch calls from ApiClient
+5
View File
@@ -0,0 +1,5 @@
---
"trigger.dev": patch
---
Handle flush errors gracefully in dev
+5
View File
@@ -0,0 +1,5 @@
---
"trigger.dev": patch
---
Added support for Preview branches in v4 projects
+5
View File
@@ -0,0 +1,5 @@
---
"trigger.dev": patch
---
Can now set project ref using the TRIGGER_PROJECT_REF env var
+6
View File
@@ -0,0 +1,6 @@
---
"trigger.dev": patch
"@trigger.dev/core": patch
---
Add runtime version detection for display in the dashboard
+16
View File
@@ -0,0 +1,16 @@
---
"@trigger.dev/react-hooks": patch
---
Added the ability to specify a "createdAt" filter when subscribing to tags in our useRealtime hooks:
```tsx
// Only subscribe to runs created in the last 10 hours
useRealtimeRunWithTags("my-tag", { createdAt: "10h" })
```
You can also now choose to skip subscribing to specific columns by specifying the `skipColumns` option:
```tsx
useRealtimeRun(run.id, { skipColumns: ["usageDurationMs"] });
```
+6
View File
@@ -0,0 +1,6 @@
---
"@trigger.dev/sdk": patch
"@trigger.dev/core": patch
---
Improve metadata flushing efficiency by collapsing operations
+5
View File
@@ -0,0 +1,5 @@
---
"@trigger.dev/core": patch
---
fix: Realtime streams: prevent enqueuing into closed ReadableStream
+5
View File
@@ -0,0 +1,5 @@
---
"trigger.dev": patch
---
Update profile switcher
+5
View File
@@ -0,0 +1,5 @@
---
"trigger.dev": patch
---
Update base images to latest compatible versions. The `node-22` runtime now uses v22.16.0 and `bun` uses the latest v1.2.18 release. The default `node` runtime is unchanged and points at v21.7.3.
+6
View File
@@ -0,0 +1,6 @@
---
"trigger.dev": patch
"@trigger.dev/core": patch
---
Improve usage flushing
+5
View File
@@ -0,0 +1,5 @@
---
"trigger.dev": patch
---
fix: default machine config indexing now works
+6
View File
@@ -0,0 +1,6 @@
---
"trigger.dev": patch
"@trigger.dev/core": patch
---
Prevent large outputs from overwriting each other
+6
View File
@@ -0,0 +1,6 @@
---
"trigger.dev": patch
"@trigger.dev/core": patch
---
Fail fast in CI when running deploy with missing `TRIGGER_ACCESS_TOKEN` and add useful error message with link to docs
+5
View File
@@ -0,0 +1,5 @@
---
"trigger.dev": patch
---
Always print full deploy logs in CI
+5
View File
@@ -0,0 +1,5 @@
---
"trigger.dev": patch
---
TriggerApiError 4xx errors will no longer cause tasks to be retried
+5
View File
@@ -0,0 +1,5 @@
---
"@trigger.dev/build": patch
---
Add ffmpeg v7 support to existing extension: `ffmpeg({ version: "7" })`
+8
View File
@@ -0,0 +1,8 @@
---
"trigger.dev": patch
"@trigger.dev/core": patch
---
- Fix polling interval reset bug that could create duplicate intervals
- Protect against unexpected attempt number changes
- Prevent run execution zombies after warm starts
+5
View File
@@ -0,0 +1,5 @@
---
"trigger.dev": patch
---
Fix stalled run detection
+6
View File
@@ -0,0 +1,6 @@
---
"trigger.dev": patch
"@trigger.dev/core": patch
---
Managed run controller performance and reliability improvements
+5
View File
@@ -0,0 +1,5 @@
---
"trigger.dev": patch
---
Fix init.ts auto-import for deployed workers
+5
View File
@@ -0,0 +1,5 @@
---
"@trigger.dev/sdk": patch
---
Added runs.list filtering for queue and machine
+5
View File
@@ -0,0 +1,5 @@
---
"@trigger.dev/sdk": patch
---
maintain proper context in metadata.root and parent getters
+7
View File
@@ -0,0 +1,7 @@
---
"@trigger.dev/sdk": patch
"trigger.dev": patch
"@trigger.dev/core": patch
---
v4: New lifecycle hooks
+5
View File
@@ -0,0 +1,5 @@
---
"trigger.dev": patch
---
Output esbuild metafile, can be inspected after `deploy --dry run`
+6
View File
@@ -0,0 +1,6 @@
---
"trigger.dev": patch
"@trigger.dev/core": patch
---
Fix QUEUED status snapshot handler
+5
View File
@@ -0,0 +1,5 @@
---
"trigger.dev": patch
---
Serialize metadata to prevent invalid metadata from breaking run completions
+5
View File
@@ -0,0 +1,5 @@
---
"trigger.dev": patch
---
If you pass a directory when calling deploy we validate it exists and give helpful hints
+6
View File
@@ -0,0 +1,6 @@
---
"@trigger.dev/react-hooks": patch
"@trigger.dev/core": patch
---
Fixes an issue with realtime when re-subscribing to a run, that would temporarily display stale data and the changes. Now when re-subscribing to a run only the latest changes will be vended
+6
View File
@@ -0,0 +1,6 @@
---
"trigger.dev": patch
"@trigger.dev/core": patch
---
Expose esbuild `minify` option (experimental)
+5
View File
@@ -0,0 +1,5 @@
---
"trigger.dev": patch
---
Fix `syncEnvVars` for non-preview deployments
+5
View File
@@ -0,0 +1,5 @@
---
"@trigger.dev/build": patch
---
Add playwright extension
+5
View File
@@ -0,0 +1,5 @@
---
"@trigger.dev/core": patch
---
Fix Bun runtime path resolution fallback
+2 -2
View File
@@ -1,7 +1,7 @@
{
"compilerOptions": {
"target": "es2022",
"lib": ["ES2022", "DOM", "DOM.Iterable"],
"lib": ["ES2022", "DOM", "DOM.Iterable", "DOM.AsyncIterable"],
"module": "NodeNext",
"moduleResolution": "NodeNext",
"moduleDetection": "force",
@@ -10,7 +10,7 @@
"strict": true,
"alwaysStrict": true,
"strictPropertyInitialization": false,
"strictPropertyInitialization": true,
"skipLibCheck": true,
"forceConsistentCasingInFileNames": true,
"noUnusedLocals": false,
+7
View File
@@ -0,0 +1,7 @@
{
"mcpServers": {
"trigger.dev": {
"url": "http://localhost:3333/sse"
}
}
}
+24
View File
@@ -0,0 +1,24 @@
---
description: how to run commands in the monorepo
globs:
alwaysApply: true
---
Almost all commands in the monorepo should be executed when `pnpm run ...` from the root of the monorepo. For example, running tests for the `@internal/run-engine` internal package:
```
pnpm run dev --filter webapp
```
But often, when running tests, it's better to `cd` into the directory and then run tests:
```
cd apps/webapp
pnpm run test --run
```
This way you can run for a single file easily:
```
cd internal-packages/run-engine
pnpm run test ./src/engine/tests/ttl.test.ts --run
```
+6
View File
@@ -0,0 +1,6 @@
---
description: understanding the structure of the monorepo
globs:
alwaysApply: true
---
We've documented the structure of our monorepo here: [repo.md](mdc:ai/references/repo.md)
+37
View File
@@ -0,0 +1,37 @@
---
description: Making updates to the main trigger.dev remix webapp
globs: apps/webapp/**/*.tsx,apps/webapp/**/*.ts
alwaysApply: false
---
The main trigger.dev webapp, which powers it's API and dashboard and makes up the docker image that is produced as an OSS image, is a Remix 2.1.0 app that uses an express server, written in TypeScript. The following subsystems are either included in the webapp or are used by the webapp in another part of the monorepo:
- `@trigger.dev/database` exports a Prisma 5.4.1 client that is used extensively in the webapp to access a PostgreSQL instance. The schema file is [schema.prisma](mdc:internal-packages/database/prisma/schema.prisma)
- `@trigger.dev/core` is a published package and is used to share code between the `@trigger.dev/sdk` and the webapp. It includes functionality but also a load of Zod schemas for data validation. When importing from `@trigger.dev/core` in the webapp, we never import the root `@trigger.dev/core` path, instead we favor one of the subpath exports that you can find in [package.json](mdc:packages/core/package.json)
- `@internal/run-engine` has all the code needed to trigger a run and take it through it's lifecycle to completion.
- `@trigger.dev/redis-worker` is a custom redis based background job/worker system that's used in the webapp and also used inside the run engine.
## Environment variables and testing
In the webapp, all environment variables are accessed through the `env` export of [env.server.ts](mdc:apps/webapp/app/env.server.ts), instead of directly accessing `process.env`.
Ideally, the `env.server.ts` file would never be imported into a test file, either directly or indirectly. Tests should only imported classes and functions from a file matching `app/**/*.ts` of the webapp, and that file should not use environment variables, everything should be passed through as options instead. This "service/configuration" separation is important, and can be seen in a few places in the code for examples:
- [realtimeClient.server.ts](mdc:apps/webapp/app/services/realtimeClient.server.ts) is the testable service, and [realtimeClientGlobal.server.ts](mdc:apps/webapp/app/services/realtimeClientGlobal.server.ts) is the configuration
Also for writing tests in the webapp, checkout our [tests.md](mdc:ai/references/tests.md) guide
## Legacy run engine vs Run Engine 2.0
We originally the Trigger.dev "Run Engine" not as a single system, but just spread out all over the codebase, with no real separate or encapsulation. And we didn't even call it a "Run Engine". With Run Engine 2.0, we've completely rewritten big parts of the way the system works, and moved it over to an internal package called `@internal/run-engine`. So we've retroactively named the previous run engine "Legacy run engine". We're focused almost exclusively now on moving to Run Engine 2.0 and will be deprecating and removing the legacy run engine code eventually.
## Where to look for code
- The trigger API endpoint is [api.v1.tasks.$taskId.trigger.ts](mdc:apps/webapp/app/routes/api.v1.tasks.$taskId.trigger.ts)
- The batch trigger API endpoint is [api.v1.tasks.batch.ts](mdc:apps/webapp/app/routes/api.v1.tasks.batch.ts)
- Setup code for the prisma client is in [db.server.ts](mdc:apps/webapp/app/db.server.ts)
- The run engine is configured in [runEngine.server.ts](mdc:apps/webapp/app/v3/runEngine.server.ts)
- All the "services" that are found in app/v3/services/**/*.server.ts
- The code for the TaskEvent data, which is the otel data sent from tasks to our servers, is in both the [eventRepository.server.ts](mdc:apps/webapp/app/v3/eventRepository.server.ts) and also the [otlpExporter.server.ts](mdc:apps/webapp/app/v3/otlpExporter.server.ts). The otel endpoints which are hit from production and development otel exporters is [otel.v1.logs.ts](mdc:apps/webapp/app/routes/otel.v1.logs.ts) and [otel.v1.traces.ts](mdc:apps/webapp/app/routes/otel.v1.traces.ts)
- We use "presenters" to move more complex loader code into a class, and you can find those are app/v3/presenters/**/*.server.ts
File diff suppressed because it is too large Load Diff
+6
View File
@@ -0,0 +1,6 @@
---
description: How to write tests in the monorepo
globs:
alwaysApply: true
---
Follow our [tests.md](mdc:ai/references/tests.md) guide for how to write tests in the monorepo.
+9
View File
@@ -0,0 +1,9 @@
apps/docker-provider/
apps/kubernetes-provider/
apps/proxy/
apps/coordinator/
packages/rsc/
.changeset
.zed
.env
!.env.example
+31 -39
View File
@@ -1,44 +1,23 @@
\*.log
.git
.github
**/*.log
**/*.pem
**/*.tsbuildinfo
# editor
**/.cache
**/.env
**/.next
**/.output
**/.trigger
**/.tshy
**/.tshy-build
**/.turbo
**/.vercel
**/.wrangler
.idea
.vscode
**/dist
**/node_modules
# dependencies
node_modules
.pnp
.pnp.js
# testing
coverage
# next.js
.next/
build
# packages
build
dist
packages/\*\*/dist
# misc
.DS_Store
\*.pem
.turbo
.vercel
.cache
.output
.trigger
apps/\*\*/public/build
apps/webapp/build
apps/webapp/public/build
cypress/screenshots
cypress/videos
@@ -47,8 +26,21 @@ apps/**/styles/tailwind.css
packages/**/styles/tailwind.css
.changeset
references
.DS_Store
.git
.github
.idea
.pnp
.pnp.js
.vscode
coverage
build
docs
examples
out
references
CHANGESETS.md
CONTRIBUTING.md
README.md
+33 -32
View File
@@ -12,7 +12,9 @@ APP_ENV=development
APP_ORIGIN=http://localhost:3030
ELECTRIC_ORIGIN=http://localhost:3060
NODE_ENV=development
V3_ENABLED=true
# Set this to UTC because Node.js uses the system timezone
TZ="UTC"
# Redis is used for the v3 queuing and v2 concurrency control
REDIS_HOST="localhost"
@@ -22,6 +24,9 @@ REDIS_TLS_DISABLED="true"
DEV_OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:3030/otel"
DEV_OTEL_BATCH_PROCESSING_ENABLED="0"
# When the domain is set to `localhost` the CLI deploy command will only --load the image by default and not --push it
DEPLOY_REGISTRY_HOST=localhost:5000
# OPTIONAL VARIABLES
# This is used for validating emails that are allowed to log in. Every email that do not match this regex will be rejected.
# WHITELISTED_EMAILS="authorized@yahoo\.com|authorized@gmail\.com"
@@ -31,41 +36,40 @@ DEV_OTEL_BATCH_PROCESSING_ENABLED="0"
# AUTH_GITHUB_CLIENT_ID=
# AUTH_GITHUB_CLIENT_SECRET=
# Resend is an email service used for signing in to Trigger.dev via a Magic Link.
# Emails will print to the console if you leave these commented out
# Configure an email transport to allow users to sign in to Trigger.dev via a Magic Link.
# If none are configured, emails will print to the console instead.
# Uncomment one of the following blocks to allow delivery of
# Resend
### Visit https://resend.com, create an account and get your API key. Then insert it below along with your From and Reply To email addresses. Visit https://resend.com/docs for more information.
# RESEND_API_KEY=<api_key>
# EMAIL_TRANSPORT=resend
# FROM_EMAIL=
# REPLY_TO_EMAIL=
# RESEND_API_KEY=
# Generic SMTP
### Enter the configuration provided by your mail provider. Visit https://nodemailer.com/smtp/ for more information
### SMTP_SECURE = false will use STARTTLS when connecting to a server that supports it (usually port 587)
# EMAIL_TRANSPORT=smtp
# FROM_EMAIL=
# REPLY_TO_EMAIL=
# SMTP_HOST=
# SMTP_PORT=587
# SMTP_SECURE=false
# SMTP_USER=
# SMTP_PASSWORD=
# AWS Simple Email Service
### Authentication is configured using the default Node.JS credentials provider chain (https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/Package/-aws-sdk-credential-providers/#fromnodeproviderchain)
# EMAIL_TRANSPORT=aws-ses
# FROM_EMAIL=
# REPLY_TO_EMAIL=
# Remove the following line to enable logging telemetry traces to the console
LOG_TELEMETRY="false"
# CLOUD VARIABLES
POSTHOG_PROJECT_KEY=
PLAIN_API_KEY=
CLOUD_AIRTABLE_CLIENT_ID=
CLOUD_AIRTABLE_CLIENT_SECRET=
CLOUD_GITHUB_CLIENT_ID=
CLOUD_GITHUB_CLIENT_SECRET=
CLOUD_LINEAR_CLIENT_ID=
CLOUD_LINEAR_CLIENT_SECRET=
CLOUD_SLACK_APP_HOST=
CLOUD_SLACK_CLIENT_ID=
CLOUD_SLACK_CLIENT_SECRET=
# v3 variables
PROVIDER_SECRET=provider-secret # generate the actual secret with `openssl rand -hex 32`
COORDINATOR_SECRET=coordinator-secret # generate the actual secret with `openssl rand -hex 32`
# Uncomment the following line to enable the registry proxy
# ENABLE_REGISTRY_PROXY=true
# DEPOT_ORG_ID=<Depot org id>
# DEPOT_TOKEN=<Depot org token>
# DEPOT_PROJECT_ID=<Depot project id>
# DEPLOY_REGISTRY_HOST=${APP_ORIGIN} # This is the host that the deploy CLI will use to push images to the registry
# CONTAINER_REGISTRY_ORIGIN=<Container registry origin e.g. https://registry.digitalocean.com>
# CONTAINER_REGISTRY_USERNAME=<Container registry username e.g. Digital ocean email address>
# CONTAINER_REGISTRY_PASSWORD=<Container registry password e.g. Digital ocean PAT>
# DEV_OTEL_EXPORTER_OTLP_ENDPOINT="http://0.0.0.0:4318"
# These are needed for the object store (for handling large payloads/outputs)
# OBJECT_STORE_BASE_URL="https://{bucket}.{accountId}.r2.cloudflarestorage.com"
@@ -75,8 +79,5 @@ COORDINATOR_SECRET=coordinator-secret # generate the actual secret with `openssl
# These control the server-side internal telemetry
# INTERNAL_OTEL_TRACE_EXPORTER_URL=<URL to send traces to>
# INTERNAL_OTEL_TRACE_EXPORTER_AUTH_HEADER_NAME=<Header name for the auth token>
# INTERNAL_OTEL_TRACE_EXPORTER_AUTH_HEADER_VALUE=<Auth token value>
# INTERNAL_OTEL_TRACE_LOGGING_ENABLED=1
# INTERNAL_OTEL_TRACE_SAMPING_RATE=20 # this means 1/20 traces or 5% of traces will be sampled (sampled = recorded)
# INTERNAL_OTEL_TRACE_INSTRUMENT_PRISMA_ENABLED=0,
# INTERNAL_OTEL_TRACE_INSTRUMENT_PRISMA_ENABLED=0,
-14
View File
@@ -1,14 +0,0 @@
module.exports = {
root: true,
// This tells ESLint to load the config from the package `eslint-config-custom`
extends: ["custom"],
settings: {
next: {
rootDir: ["apps/*/"],
},
},
parserOptions: {
sourceType: "module",
ecmaVersion: 2020,
},
};
+5
View File
@@ -31,6 +31,11 @@ runs:
sha=$(echo ${{ github.sha }} | head -c7)
ts=$(date +%s)
tag=${env}-${sha}-${ts}
elif [[ "${{ github.ref_name }}" == re2-*-* ]]; then
env=$(echo ${{ github.ref_name }} | cut -d- -f2)
sha=$(echo ${{ github.sha }} | head -c7)
ts=$(date +%s)
tag=${env}-${sha}-${ts}
elif [[ "${{ github.ref_name }}" == v.docker.* ]]; then
version="${GITHUB_REF_NAME#v.docker.}"
tag="v${version}"
+1 -1
View File
@@ -39,4 +39,4 @@ jobs:
${{ runner.os }}-mintlify
- name: 🔗 Check for broken links
run: npx mintlify@4.0.222 broken-links
run: npx mintlify@4.0.393 broken-links
+7 -1
View File
@@ -1,5 +1,8 @@
name: "E2E"
permissions:
contents: read
on:
workflow_call:
inputs:
@@ -17,7 +20,7 @@ jobs:
strategy:
fail-fast: false
matrix:
os: [buildjet-8vcpu-ubuntu-2204, windows-latest]
os: [ubuntu-latest, windows-latest]
package-manager: ["npm", "pnpm"]
steps:
- name: ⬇️ Checkout repo
@@ -38,6 +41,9 @@ jobs:
- name: 📥 Download deps
run: pnpm install --frozen-lockfile --filter trigger.dev...
- name: 📀 Generate Prisma Client
run: pnpm run generate
- name: 🔧 Build v3 cli monorepo dependencies
run: pnpm run build --filter trigger.dev^...
+1 -31
View File
@@ -5,6 +5,7 @@ on:
types: [opened, synchronize, reopened]
paths-ignore:
- "docs/**"
- ".changeset/**"
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
@@ -28,34 +29,3 @@ jobs:
with:
package: cli-v3
secrets: inherit
preview-release:
name: Preview Release
needs: [typecheck, units, e2e]
if: github.repository == 'triggerdotdev/trigger.dev'
runs-on: buildjet-8vcpu-ubuntu-2204
steps:
- name: ⬇️ Checkout repo
uses: actions/checkout@v4
with:
fetch-depth: 0
- name: ⎔ Setup pnpm
uses: pnpm/action-setup@v4
with:
version: 8.15.5
- name: ⎔ Setup node
uses: buildjet/setup-node@v4
with:
node-version: 20.11.1
cache: "pnpm"
- name: 📥 Download deps
run: pnpm install --frozen-lockfile
- name: 🏗️ Build
run: pnpm run build --filter "@trigger.dev/*" --filter "trigger.dev"
- name: ⚡ Publish preview release
run: npx pkg-pr-new publish --no-template $(ls -d ./packages/*)
+24 -2
View File
@@ -1,5 +1,10 @@
name: "🐳 Publish Webapp"
permissions:
contents: read
packages: write
id-token: write
on:
workflow_call:
inputs:
@@ -43,13 +48,25 @@ jobs:
ref_without_tag=ghcr.io/triggerdotdev/trigger.dev
image_tags=$ref_without_tag:${{ steps.get_tag.outputs.tag }}
# if tag is a semver, also tag it as v3
# if tag is a semver, also tag it as v4
if [[ "${{ steps.get_tag.outputs.is_semver }}" == true ]]; then
image_tags=$image_tags,$ref_without_tag:v3
# TODO: switch to v4 tag on GA
image_tags=$image_tags,$ref_without_tag:v4-beta
fi
echo "image_tags=${image_tags}" >> "$GITHUB_OUTPUT"
- name: 📝 Set the build info
id: set_build_info
run: |
tag=${{ steps.get_tag.outputs.tag }}
if [[ "${{ steps.get_tag.outputs.is_semver }}" == true ]]; then
echo "BUILD_APP_VERSION=${tag}" >> "$GITHUB_OUTPUT"
fi
echo "BUILD_GIT_SHA=${{ github.sha }}" >> "$GITHUB_OUTPUT"
echo "BUILD_GIT_REF_NAME=${{ github.ref_name }}" >> "$GITHUB_OUTPUT"
echo "BUILD_TIMESTAMP_SECONDS=$(date +%s)" >> "$GITHUB_OUTPUT"
- name: 🐙 Login to GitHub Container Registry
uses: docker/login-action@v3
with:
@@ -64,3 +81,8 @@ jobs:
platforms: linux/amd64,linux/arm64
tags: ${{ steps.set_tags.outputs.image_tags }}
push: true
build-args: |
BUILD_APP_VERSION=${{ steps.set_build_info.outputs.BUILD_APP_VERSION }}
BUILD_GIT_SHA=${{ steps.set_build_info.outputs.BUILD_GIT_SHA }}
BUILD_GIT_REF_NAME=${{ steps.set_build_info.outputs.BUILD_GIT_REF_NAME }}
BUILD_TIMESTAMP_SECONDS=${{ steps.set_build_info.outputs.BUILD_TIMESTAMP_SECONDS }}
+89
View File
@@ -0,0 +1,89 @@
name: "⚒️ Publish Worker (v4)"
on:
workflow_call:
inputs:
image_tag:
description: The image tag to publish
type: string
required: false
default: ""
push:
tags:
- "re2-test-*"
- "re2-prod-*"
permissions:
id-token: write
packages: write
contents: read
jobs:
# check-branch:
# runs-on: ubuntu-latest
# steps:
# - name: Fail if re2-prod-* is pushed from a non-main branch
# if: startsWith(github.ref_name, 're2-prod-') && github.base_ref != 'main'
# run: |
# echo "🚫 re2-prod-* tags can only be pushed from the main branch."
# exit 1
build:
# needs: check-branch
strategy:
matrix:
package: [supervisor]
runs-on: ubuntu-latest
env:
DOCKER_BUILDKIT: "1"
steps:
- name: 🏭 Setup Depot CLI
uses: depot/setup-action@v1
- name: ⬇️ Checkout git repo
uses: actions/checkout@v4
- name: 📦 Get image repo
id: get_repository
run: |
if [[ "${{ matrix.package }}" == *-provider ]]; then
provider_type=$(echo "${{ matrix.package }}" | cut -d- -f1)
repo=provider/${provider_type}
else
repo="${{ matrix.package }}"
fi
echo "repo=${repo}" >> "$GITHUB_OUTPUT"
- name: "#️⃣ Get image tag"
id: get_tag
uses: ./.github/actions/get-image-tag
with:
tag: ${{ inputs.image_tag }}
- name: 📛 Set tags to push
id: set_tags
run: |
ref_without_tag=ghcr.io/triggerdotdev/${{ steps.get_repository.outputs.repo }}
image_tags=$ref_without_tag:${{ steps.get_tag.outputs.tag }}
# if tag is a semver, also tag it as v4
if [[ "${{ steps.get_tag.outputs.is_semver }}" == true ]]; then
# TODO: switch to v4 tag on GA
image_tags=$image_tags,$ref_without_tag:v4-beta
fi
echo "image_tags=${image_tags}" >> "$GITHUB_OUTPUT"
- name: 🐙 Login to GitHub Container Registry
uses: docker/login-action@v3
with:
registry: ghcr.io
username: ${{ github.repository_owner }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: 🐳 Build image and push to GitHub Container Registry
uses: depot/build-push-action@v1
with:
file: ./apps/${{ matrix.package }}/Containerfile
platforms: linux/amd64,linux/arm64
tags: ${{ steps.set_tags.outputs.image_tags }}
push: true
+9 -9
View File
@@ -23,7 +23,7 @@ jobs:
strategy:
matrix:
package: [coordinator, docker-provider, kubernetes-provider]
runs-on: buildjet-16vcpu-ubuntu-2204
runs-on: ubuntu-latest
env:
DOCKER_BUILDKIT: "1"
steps:
@@ -77,11 +77,11 @@ jobs:
REPOSITORY: ${{ steps.get_repository.outputs.repo }}
IMAGE_TAG: ${{ steps.get_tag.outputs.tag }}
- name: 🐙 Push 'v3' tag to GitHub Container Registry
if: steps.get_tag.outputs.is_semver == 'true'
run: |
docker tag infra_image "$REGISTRY/$REPOSITORY:v3"
docker push "$REGISTRY/$REPOSITORY:v3"
env:
REGISTRY: ghcr.io/triggerdotdev
REPOSITORY: ${{ steps.get_repository.outputs.repo }}
# - name: 🐙 Push 'v3' tag to GitHub Container Registry
# if: steps.get_tag.outputs.is_semver == 'true'
# run: |
# docker tag infra_image "$REGISTRY/$REPOSITORY:v3"
# docker push "$REGISTRY/$REPOSITORY:v3"
# env:
# REGISTRY: ghcr.io/triggerdotdev
# REPOSITORY: ${{ steps.get_repository.outputs.repo }}
+10 -5
View File
@@ -24,12 +24,10 @@ on:
- "packages/**"
- "!packages/**/*.md"
- "!packages/**/*.eslintrc"
- "internal-packages/**"
- "apps/**"
- "!apps/**/*.md"
- "!apps/**/*.eslintrc"
- "integrations/**"
- "!integrations/**/*.md"
- "!integrations/**/*.eslintrc"
- "pnpm-lock.yaml"
- "pnpm-workspace.yaml"
- "turbo.json"
@@ -58,15 +56,22 @@ jobs:
secrets: inherit
publish-webapp:
needs: [typecheck, units]
needs: [typecheck]
uses: ./.github/workflows/publish-webapp.yml
secrets: inherit
with:
image_tag: ${{ inputs.image_tag }}
publish-worker:
needs: [typecheck, units]
needs: [typecheck]
uses: ./.github/workflows/publish-worker.yml
secrets: inherit
with:
image_tag: ${{ inputs.image_tag }}
publish-worker-v4:
needs: [typecheck]
uses: ./.github/workflows/publish-worker-v4.yml
secrets: inherit
with:
image_tag: ${{ inputs.image_tag }}
+143
View File
@@ -0,0 +1,143 @@
name: 🧭 Helm Chart Release
on:
push:
tags:
- 'helm-v*'
workflow_dispatch:
inputs:
chart_version:
description: 'Chart version to release'
required: true
type: string
env:
REGISTRY: ghcr.io
CHART_NAME: trigger
jobs:
lint-and-test:
runs-on: ubuntu-latest
permissions:
contents: read
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Set up Helm
uses: azure/setup-helm@v4
with:
version: "3.18.3"
- name: Build dependencies
run: helm dependency build ./hosting/k8s/helm/
- name: Extract dependency charts
run: |
cd ./hosting/k8s/helm/
for file in ./charts/*.tgz; do echo "Extracting $file"; tar -xzf "$file" -C ./charts; done
- name: Lint Helm Chart
run: |
helm lint ./hosting/k8s/helm/
- name: Render templates
run: |
helm template test-release ./hosting/k8s/helm/ \
--values ./hosting/k8s/helm/values.yaml \
--output-dir ./helm-output
- name: Validate manifests
uses: docker://ghcr.io/yannh/kubeconform:v0.7.0
with:
entrypoint: '/kubeconform'
args: "-summary -output json ./helm-output"
release:
needs: lint-and-test
runs-on: ubuntu-latest
permissions:
contents: write # for gh-release
packages: write
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Set up Helm
uses: azure/setup-helm@v4
with:
version: "3.18.3"
- name: Build dependencies
run: helm dependency build ./hosting/k8s/helm/
- name: Extract dependency charts
run: |
cd ./hosting/k8s/helm/
for file in ./charts/*.tgz; do echo "Extracting $file"; tar -xzf "$file" -C ./charts; done
- name: Log in to Container Registry
uses: docker/login-action@v3
with:
registry: ${{ env.REGISTRY }}
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Extract version from tag or input
id: version
run: |
if [ "${{ github.event_name }}" = "workflow_dispatch" ]; then
VERSION="${{ github.event.inputs.chart_version }}"
else
VERSION="${{ github.ref_name }}"
VERSION="${VERSION#helm-v}"
fi
echo "version=$VERSION" >> $GITHUB_OUTPUT
echo "Releasing version: $VERSION"
- name: Check Chart.yaml version matches release version
run: |
VERSION="${{ steps.version.outputs.version }}"
CHART_VERSION=$(grep '^version:' ./hosting/k8s/helm/Chart.yaml | awk '{print $2}')
echo "Chart.yaml version: $CHART_VERSION"
echo "Release version: $VERSION"
if [ "$CHART_VERSION" != "$VERSION" ]; then
echo "❌ Chart.yaml version does not match release version!"
exit 1
fi
echo "✅ Chart.yaml version matches release version."
- name: Package Helm Chart
run: |
helm package ./hosting/k8s/helm/ --destination /tmp/
- name: Push Helm Chart to GHCR
run: |
VERSION="${{ steps.version.outputs.version }}"
CHART_PACKAGE="/tmp/${{ env.CHART_NAME }}-${VERSION}.tgz"
# Push to GHCR OCI registry
helm push "$CHART_PACKAGE" "oci://${{ env.REGISTRY }}/${{ github.repository_owner }}/charts"
- name: Create GitHub Release
id: release
uses: softprops/action-gh-release@v1
if: github.event_name == 'push'
with:
tag_name: ${{ github.ref_name }}
name: "Helm Chart ${{ steps.version.outputs.version }}"
body: |
### Installation
```bash
helm upgrade --install trigger \
oci://${{ env.REGISTRY }}/${{ github.repository_owner }}/charts/${{ env.CHART_NAME }} \
--version "${{ steps.version.outputs.version }}"
```
### Changes
See commit history for detailed changes in this release.
files: |
/tmp/${{ env.CHART_NAME }}-${{ steps.version.outputs.version }}.tgz
token: ${{ secrets.GITHUB_TOKEN }}
draft: true
prerelease: true
+7 -10
View File
@@ -1,5 +1,8 @@
name: 🦋 Changesets Release
permissions:
contents: write
on:
push:
branches:
@@ -18,6 +21,10 @@ jobs:
release:
name: 🦋 Changesets Release
runs-on: ubuntu-latest
permissions:
contents: write
packages: write
pull-requests: write
if: github.repository == 'triggerdotdev/trigger.dev'
outputs:
published: ${{ steps.changesets.outputs.published }}
@@ -89,13 +96,3 @@ jobs:
run: |
package_version=$(echo '${{ steps.changesets.outputs.publishedPackages }}' | jq -r '.[0].version')
echo "package_version=${package_version}" >> "$GITHUB_OUTPUT"
publish:
needs: release
uses: ./.github/workflows/publish.yml
secrets: inherit
# if: needs.release.outputs.published == 'true'
# disable automatic publishing for now
if: false
with:
image_tag: v${{ needs.release.outputs.published_package_version }}

Some files were not shown because too many files have changed in this diff Show More