Commit Graph

3906 Commits

Author SHA1 Message Date
Eric Allam 684e02c8c8 fix: prevent large root/parent metadata updates from endlessly retrying (#2290)
* fixing metadata WIP

* WIP

* fix: prevent large root/parent metadata updates from endlessly retrying

* Fixed other calls to handleMetadataPacket
2025-07-21 23:04:44 +01:00
Matt Aitken 4b9f33082e AI run filtering (#2285)
* Queue in run table and filtering

* Debounce the filter changes

* Remove console log

* Added machine filtering

* Added version filtering

* Filter by version in the db

* Removed duplicate classes

* Version filtering hasFilters consistency

* Added queues and machines to the bulk action summary

* runs.list filtering for queue and machine

* Fix for machine errors

* Input field now has accessory instead of shortcut

* First experiments with the UI

* Got the fake filtering working

* AI filtering is working pretty well 

* Started working on tool calling

* Tool calling is working

* Styling progress

* Working on the error

* Errors work, improved the styling

* Nice glow effect

* Tweak the darkness of the text field

* Re-ordered the UI, set AI settings to use system prompt and telemetry

* Refactored to make it testable

* Added basic evals

* Better time inputs and evals

* Removed some code comments

* Remove unused useSearchParam change

* Tidy imports

* If no OpenAI API key send json back

* Fix for merge conflict with duplicate query filters

* Another conflict resolved

* Another merge conflict resolved

* Pass the model in, allow changing it
2025-07-21 17:53:43 +01:00
Matt Aitken a90b73c7ca Filter runs by queue, machine, version (#2277)
* Queue in run table and filtering

* Debounce the filter changes

* Remove console log

* Added machine filtering

* Added version filtering

* Filter by version in the db

* Removed duplicate classes

* Version filtering hasFilters consistency

* Added queues and machines to the bulk action summary

* runs.list filtering for queue and machine

* Fix for machine errors
2025-07-21 16:21:46 +01:00
Eric Allam 07980f8ebd fix: allow updating run metadata up to 1 hour after completion (#2288) 2025-07-21 16:05:29 +01:00
Eric Allam 8d5c86fea0 v4: simplified release concurrency system and status changes (#2284)
* WIP

* Make release concurrency system extremely simple, everything just releases all the time

* update the deadlock detection to use the new lockedQueueReleaseConcurrencyOnWaitpoint column

* WIP new release concurrency system

* Remove releaseConcurrency and releaseConcurrencyOnWaitpoint

Also removed deadlock detection, and added environment burst concurrency

* Added new DEQUEUED status

Cleaned up the API run statuses, including now detecting new clients and not breaking older clients by adding an API version header to all requests

* Introduce the new "current dequeued concurrency set"

* Remove QUEUED_EXECUTING because we no longer "eagerly" release before checkpointing

* Remove waitpoint test for QUEUED_EXECUTING

* Add isWaiting

* Add changeset

* Use createdAt for ordering realtime runs instead of number

* Clarify the envCurrentDequeuedKey usage

* mock the db.server file to fix the tests

* Updated changset "EXECUTED" -> "EXECUTING"

---------

Co-authored-by: Matt Aitken <matt@mattaitken.com>
2025-07-21 15:45:27 +01:00
James Ritchie 4565f19d3a Copyable admin area table fields and autofocus search (#2280) 2025-07-18 18:48:56 +01:00
James Ritchie 3ad4b8b32d Add machine to run list (#2275)
* Access machinePreset from the run list presenter

* New icons for machine presets

* New icon + name combo label for the machine preset

* Adds new “Machine” column to the runs list

* Make a separate component for the machine tooltip info

* add machinePreset to the span presenter

* Show the Machine in the Details tab in the Run inspector

* Show an admin only separator

* Fix docs icon in the button

* Small padding tweak

* Move the Machine nearer the costs

* Don't cast the machine preset

* Remove typecast, better to have a bad label if we add a new machine and don't update thos

---------

Co-authored-by: Matt Aitken <matt@mattaitken.com>
2025-07-17 14:26:52 +01:00
Matt Aitken 6f6852621d Bundle p-map (#2274) 2025-07-16 16:36:36 +01:00
Matt Aitken 5e999c8992 Use p-map to speed up bulk actions (#2272) 2025-07-16 15:52:42 +01:00
Matt Aitken 569cecb675 The email counts should come from the completed bulk action… not this iteration (#2271) 2025-07-16 13:46:48 +01:00
Saadi Myftija c0b6c41afc Disable payload editing for run replays with large payloads (#2268)
* Use sexy scrollbars for templates and recent runs popover content

@Nick this is for you!!1

* Fix TabButton component disabled state

* Disable payload tab when replaying runs with large payloads
2025-07-15 21:54:43 +01:00
Matt Aitken 35c7a1ecb0 Bulk action 2 fixes (#2267)
* Fix for filtering by run id

Simplified to have a single filter for runId which accepts an array. This unifies the filtering by multiple runs (used by the waitpoint page) and the individual run filtering. It fixes a bug with bulk actions

* Truncate long bulk action titles
2025-07-15 14:43:19 +01:00
Saadi Myftija a64875a297 fix: regression affecting replayed runs with superjson payloads (#2266)
* Fix regression that broke superjson behavior for the replay flow

* Ignore undefined type metadata when overriding superjson payloads

Addresses https://github.com/triggerdotdev/trigger.dev/issues/1968

* Add a few unit tests for replaceSuperJsonPayload
2025-07-15 12:30:33 +01:00
Saadi Myftija b78b3dceb2 feat: link the original run from replayed runs (#2262)
* Add ID of the replayedFrom run to the TaskRun schema

* Propagate the replayedFrom run ID in the replay flow

* Link the replayed run in the run details pane
2025-07-14 15:57:54 +01:00
Matt Aitken 630e9556b0 Bulk actions 2.0 (and switch all run listing to ClickHouse) (#2264)
* useSearchParams has

* useSearchParams has

* useSearchParams has

* Consistent way to get the run filters

* Consistent way to get the run filters

* Consistent way to get the run filters

* Initial work on the new bulk actions

* Initial work on the new bulk actions

* Initial work on the new bulk actions

* WIP actions and filtering

* WIP actions and filtering

* WIP actions and filtering

* Empty filter arrays are set to undefined

* Empty filter arrays are set to undefined

* Empty filter arrays are set to undefined

* WIP prisma schema

Removed extra runtimeEnvironmentId

* WIP prisma schema

Removed extra runtimeEnvironmentId

* WIP prisma schema

Removed extra runtimeEnvironmentId

* Migrations

* Migrations

* Migrations

* BulkActionGroup changed some columns around

* BulkActionGroup changed some columns around

* BulkActionGroup changed some columns around

* New badge variant, removed unused ones

* New badge variant, removed unused ones

* New badge variant, removed unused ones

* Bulk action button

* Bulk action button

* Bulk action button

* Make the next runs page the default now

* Make the next runs page the default now

* Make the next runs page the default now

* Improved the RadioButton style

* Improved the RadioButton style

* Improved the RadioButton style

* Remove the old bulk action bar

* Remove the old bulk action bar

* Remove the old bulk action bar

* More UI progress

* More UI progress

* More UI progress

* Lots of UI changes to the Runs page

* Lots of UI changes to the Runs page

* Lots of UI changes to the Runs page

* Fixed period filter resetting everything

* Fixed period filter resetting everything

* Fixed period filter resetting everything

* Improved the Switch secondary style

* Improved the Switch secondary style

* Improved the Switch secondary style

* Buggy filter fixes

* Buggy filter fixes

* Buggy filter fixes

* Improved the filter display and fixed a bug with search param from object

* Improved the filter display and fixed a bug with search param from object

* Improved the filter display and fixed a bug with search param from object

* Clear button is minimal

* Clear button is minimal

* Clear button is minimal

* Using a presenter now

* Using a presenter now

* Using a presenter now

* Bulk actions are created, but not actually processed (yet)

* Bulk actions are created, but not actually processed (yet)

* Bulk actions are created, but not actually processed (yet)

* Bulk replay/cancel is working

* Bulk replay/cancel is working

* Bulk replay/cancel is working

* Multiple fixes, added bulk column to PG

* Multiple fixes, added bulk column to PG

* Multiple fixes, added bulk column to PG

* Bulk action run filtering working using CH

* Bulk action run filtering working using CH

* Bulk action run filtering working using CH

* Replay setting the bulk id on the runs

* Replay setting the bulk id on the runs

* Replay setting the bulk id on the runs

* Properly cap the time when doing a bulk action

* Properly cap the time when doing a bulk action

* Properly cap the time when doing a bulk action

* If the bulk action isn't recent, add it to the dropdown anyway

* If the bulk action isn't recent, add it to the dropdown anyway

* If the bulk action isn't recent, add it to the dropdown anyway

* Blank version of the bulk actions page

* Blank version of the bulk actions page

* Blank version of the bulk actions page

* Individually selected runs working

* Individually selected runs working

* Individually selected runs working

* Use selected mode if runs are checked

* Use selected mode if runs are checked

* Use selected mode if runs are checked

* Added the modal

* Added the modal

* Added the modal

* Marked the old bulk actions stuff as deprecated

* Marked the old bulk actions stuff as deprecated

* Marked the old bulk actions stuff as deprecated

* Renamed bulk action file

* Renamed bulk action file

* Renamed bulk action file

* Bulk run filter with the name and a default

* Bulk run filter with the name and a default

* Bulk run filter with the name and a default

* WIP on bulk actions page

* WIP on bulk actions page

* WIP on bulk actions page

* Updated panel, added new truncated id component

* Updated panel, added new truncated id component

* Updated panel, added new truncated id component

* Style improvements to the radio buttons

* Style improvements to the radio buttons

* Style improvements to the radio buttons

* Added an option action completion email

* Added an option action completion email

* Added an option action completion email

* Adds a blank state for the bulk actions page

* Adds a blank state for the bulk actions page

* Adds a blank state for the bulk actions page

* Nicer completed email

* Nicer completed email

* Nicer completed email

* Don't open the bulk action panel if there are no runs

* Don't open the bulk action panel if there are no runs

* Don't open the bulk action panel if there are no runs

* Runs blank state and bulk action accordion

* Runs blank state and bulk action accordion

* Runs blank state and bulk action accordion

* Updates secondary/small switch style

* Updates secondary/small switch style

* Updates secondary/small switch style

* Pagination buttons no longer split in twain (WIP)

* Pagination buttons no longer split in twain (WIP)

* Pagination buttons no longer split in twain (WIP)

* Aborting working

* Aborting working

* Aborting working

* Bulk action live reloading

* Bulk action live reloading

* Bulk action live reloading

* ListPagination works correctly in all states

* ListPagination works correctly in all states

* ListPagination works correctly in all states

* Run page, show friendlyId instead of number

* Run page, show friendlyId instead of number

* Run page, show friendlyId instead of number

* Bulk action help open by default if you have none

* Bulk action help open by default if you have none

* Bulk action help open by default if you have none

* Extra status filtering step because of replication delay

* Extra status filtering step because of replication delay

* Extra status filtering step because of replication delay

* Wider bulk action onboarding

* Wider bulk action onboarding

* Wider bulk action onboarding

* More sensible widths on the bulk action side panel

* More sensible widths on the bulk action side panel

* More sensible widths on the bulk action side panel

* Border color tweak to the RadioButton

* Border color tweak to the RadioButton

* Border color tweak to the RadioButton

* Improved the accordion component hover states

* Improved the accordion component hover states

* Improved the accordion component hover states

* Updates the bulk action blank state images to the latest UI

* Updates the bulk action blank state images to the latest UI

* Updates the bulk action blank state images to the latest UI

* Added R and C shortcuts back in

* Added R and C shortcuts back in

* Added R and C shortcuts back in

* Fix for selecting a single run

* Fix for selecting a single run

* Fix for selecting a single run

* Improved exit icon, added shortcut to modal

* Improved exit icon, added shortcut to modal

* Improved exit icon, added shortcut to modal

* Tidy imports

* Tidy imports

* Tidy imports

* Tidy imports

* Tidy imports

* Tidy imports

* Tidy imports

* Tidy imports

* Tidy imports

* Tidy imports

* Fix for grid layout when 1 page of bulk actions visible

* Fix for grid layout when 1 page of bulk actions visible

* Fix for grid layout when 1 page of bulk actions visible

* Removed the ... on the abort button

* Removed the ... on the abort button

* Removed the ... on the abort button

* Removed the ... on the abort button

* Animate the progress bar

* Set TZ="UTC" in the env example

* Filter summary in the bulk inspector

* Improves the pagination styling

* Improves the pagination styling

* Delete old bulk action routes

* Removed old Postgres RunListPresenter

* Retry any replication error where the message contains "timeout"

* Increase wait to make test less flaky

* The test was using run id instead of friendly id

* Safer array access

* Remove error log if there's a bad status

* Nicer frontend type safety with the bulk action and mode

* Switched a log to a debug log

* Retry replication unless the error is a known non-retry error

Flip the strategy to retry by default

* Make ClickHouse required

* Backfill run replication admin API endpoint

* Set a CLICKHOUSE_URL for unit tests

---------

Co-authored-by: James Ritchie <james@trigger.dev>
2025-07-14 15:50:01 +01:00
Eric Allam 70b4b1257e fix: Logging large objects is now much more performant and uses less memory (#2263)
* Logging large objects performance fixes

* Add changeset

* docs: new otel attribute count defaults

* Scope the switch/case statement with braces

* Add tests for nested map and sets

---------

Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
2025-07-14 13:16:04 +01:00
Saadi Myftija 87c21ab50a feat: enable overriding run options when replaying (#2260)
* Rearrange the layout of the replay modal

* Add run options to the replay modal

* Handle payload and metadata correctly in the shared json editor

* Apply run options in replays

* Fix clear button issue in json editor

Memoization of the clear function caused some problems. Removing it should not cause performance issues.

* Update replay modal hint

* Move machine and version fields to the top for visibility

* Use the same field ordering in the test page

* Reload queues and versions on env override

* Adapt json editor to fill full height

* Clean up a few excessive ternaries

* Avoid ui jump on env selection

* Switch to sexy scrollbars for scheduled tasks in the test page
2025-07-11 16:54:20 +02:00
Matt Aitken 4c635f72a2 Fixed queues SSE live reloading (#2259) 2025-07-11 12:53:28 +01:00
nicktrn 0c71dc74df feat: add node 22 and bun runtime support with version display (#2254)
* update node-22 image

* update bun image

* disable io_uring

* fix fallback bun path

* prevent duplicate warnings

* add runtime and version to deployments

* runtime icons

* fallback to nodejs

* prevent empty table cell menu

* log if local build on deploy

* pass io_uring env var to child

* denormalize runtime and version, display on run details

* add changesets

* disable pr checks for changeset commits..

* add runtime data to deployed bg workers
2025-07-10 16:43:10 +01:00
Eric Allam 6ff0b923e1 New internal idempotency implementation for trigger and batch trigger (#2256)
* Introduce request idempotency to prevent duplicate triggers

* Implement request idempotency on trigger

* Use x-trigger-request-idempotency-key header instead

Plus cleanup shared logic in the routes

* Add changeset

* Oops, lets not hardcode a 408

* A couple of improvements
2025-07-10 16:42:32 +01:00
Saadi Myftija 28b6be2491 feat: introduce run templates for reusing test run configs (#2253)
* Add new prisma model for task run templates

* Create run templates in a new service

* Add modal to create run templates in the test page

* Show templates list and apply values when selected

* Hide template creation time in the dropdown list, only show date

* Enable deleting run templates

* Show success toast on template creation

* Validate template label length

* Improve the template creation success indicator

* Use formAction consistently to differentiate submissions

* Type formAction for better editor support

* Prettify run template payload and metadata

* Add triggerSource, concurrencyKey and ttl to run templates
2025-07-09 19:09:25 +02:00
Saadi Myftija dbcf632b4e fix: add missing field descriptions in the test page (#2251)
* Switch to sexy scrollbars in the options pane

* Add missing field descriptions

* Add a run options link to the docs

* Make the copy for the field descriptions more concise

* Fix tooltip issue in the Select component
2025-07-09 16:25:06 +02:00
Saadi Myftija 3cfde48bfd feat: expose all run options in the test run page (#2227)
* Implement a new primitive UI component for picking durations

* Implement a new component to input run tags

* Expose all run options in the test run page

* Add subtle animations when adding/removing run tags in the test page

* Add a new resource endpoint for fetching queues

* Fetch usable queues for the selected task

* Fix width display issue in the select component

* Enable locking a run to a version from the test page

* Disable entering max attemps <0

* Validate tags

* Add recent runs popover

* Only show latest version for development environments

* Update run options when selecting a recent run

* Rearrange the test page layout

* Add subtle animation to the duration picker segments on focus

* Improve queue selection dropdown styling

* Fix disabled state issue for the SelectTrigger component

* Disable version selection field for dev envs

* Add usage hints next to the run option fields

* Add machine preset to the run options list

* Allow arbitrary queue inputs for v1 engine runs

* Show truncated run ID instead of run numbers for recent runs

Run numbers will soon get deprecated due to contention issues

* Fix duplicate queue issue

* Extract common elements across the standard and scheduled test task forms

* Apply values from recent runs to scheduled tasks too

* Add additional run options for scheduled tasks

* Use a slightly smaller font size for run option labels

* Disallow commas in the run tag input field

* Switch to a custom icon for recent runs button

* Flatten the load function test task result object

* Avoid redefining machine presets, use zod schema instead

* Fix ClockRotateLeftIcon jsx issues

* Remove recent runs button tooltip as it causes nesting errors

* Adjust the page layout to make it clear which task is currently selected

* Inline the tab group with the copy/clear buttons
2025-07-09 10:40:10 +01:00
Eric Allam b113e5a99c Fix non-MFA login (#2246)
* Fix non-MFA login

* fixed typecheck errors
2025-07-08 14:59:31 +01:00
Eric Allam b119a52e08 Implement MFA (#2244)
* Adds a new route for logging in with mfa

* New path for security page

* Adds “Security” link to account side menu

* Update the Switch component to allow label positions left and right

* Optionally hide the Close button in the Dialog title bar

* Installs `qrcode` react package for generating QR codes.

* CopyButton component now takes children

* New Security route for setting up MFA

* Adds new OTP package for the chadcn InputOTP component

* Adds new InputOTP chadcn component

* Adds InputOTP chadcn component to the MFA login screen

* InputOTP component supports variant styles

* Improvements to form handling

* Show a confirmation modal before you can disable MFA

* Revert redirect back to the dashboard for now

* Implement MFA enabling and disabling

* Refactor and cleanup mfa management code

* More cleanup

* Handle errors in the management action

* Implement mfa login flow

* recovery code input should be password

* Implement rate limiting on the mfa validation endpoint

* Better error ux

* Implement mfa emails and apply James' updates

* Use latest @better-auth/utils

* Improvements via CodeRabbit review

---------

Co-authored-by: James Ritchie <james@trigger.dev>
2025-07-08 13:44:47 +01:00
Eric Allam 493ca11c43 Add ability to disable the prisma query in the healthcheck endpoint (#2242) 2025-07-07 16:41:55 +01:00
Eric Allam 47e583c19d Properly scope the batch list to the selected environment (#2236)
Prevents showing batch runs from environments that are not in the selected environment
2025-07-04 14:41:10 +01:00
Eric Allam 469a8df0cf Prevent redis worker from enqueuing pending jobs (#2235) 2025-07-04 13:53:32 +01:00
Eric Allam 922c7cb3a6 Increase batch trigger processing job visibility timeout (#2234) 2025-07-04 13:43:26 +01:00
nicktrn 74fdfa93c5 Show project details when running the whoami command (#2231)
* add project details to whoami

* findFirst

* skip project check if no orgs found

* add changeset
2025-07-04 13:40:41 +01:00
Eric Allam bdaa2ed15a Move batch trigger processing jobs to a dedicated redis worker (#2233) 2025-07-04 12:16:05 +01:00
nicktrn afc9a6ade1 Add ECR support for remote builds (#2224)
* create repo if doesn't exist

* fresh auth token for each deploy

* optional assume role

* log when machine overrides enabled

* make test repo namespace configurable

* assume role fix and env var changes

* improve ecr check

* improve tag parsing

* tag parsing tests

* track if repo created and fix test

* missing tryCatch for sts call
2025-07-03 14:25:30 +01:00
Eric Allam 7bb7e7aedc When sharding, use the where clause in the shard key to distribute requests more evenly (#2229) 2025-07-03 14:21:58 +01:00
Eric Allam 0c2af6d87c Increase otel attribute limits and make them configurable 2025-07-02 11:30:13 +01:00
Eric Allam 57f3a89a86 Fix timeline span events tests 2025-07-02 11:30:13 +01:00
Eric Allam ceb737e9f1 Fix broken webapp test 2025-07-02 11:30:13 +01:00
Eric Allam 632d6b91e9 Fix taskSlug extraction 2025-07-02 11:30:13 +01:00
Eric Allam b548561220 Remvoe the sometimes incorrect import file name 2025-07-02 11:30:13 +01:00
Eric Allam 58d56f8381 Use hostname instead of host 2025-07-02 11:30:13 +01:00
Eric Allam 8798352be3 Super-warm starts in deployed tasks proof 2025-07-02 11:30:13 +01:00
Eric Allam 9374aea401 Split resource and span attributes 2025-07-02 11:30:13 +01:00
Eric Allam e2049fcfbf Initial dev process keep alive. moved resource attributes to span attributes because resource attributes are global and immutable 2025-07-02 11:30:13 +01:00
Matt Aitken 4a79b3b35b Fixed retrying on run replication inserts on connection errors (#2216) 2025-07-01 13:35:13 +01:00
Matt Aitken f38d35e9f6 ClickHouse replication improvements (retrying, strip bad unicode chars) (#2205)
* Add retry logic for insert operations

Add a generic retry mechanism for task run and payload inserts to handle
transient connection errors. The new #insertWithRetry method retries up to
three times with exponential backoff and jitter on retryable connection
errors such as connection resets or timeouts. Errors are logged and
recorded in tracing spans to improve observability and robustness of the
replication service.

* Replication settings are configurable

* Log out the runIds for failed batches

* Detecting bad JSON in run replication and ignoring it

* Reproduced split unicode error

* Move output file

* Massively improved the performance

* Minor performance improvements

* Unskip tests

* Remove unused test in CH package

* Fix for the ClickHouse UI explorer

* RunReplication keepAlive defaults to false

* Add concurrency_key and bulk_action_group_ids to ClickHouse task runs

* ClickHouse package doesn't need to be built anymore for the webapp

* Set the concurrency_key from the run replication service
2025-06-30 16:32:02 +01:00
Eric Allam 1b4dabb8ba v4: current concurrency sweeper (#2206)
* v4: current concurrency sweeper

* Fix webapp tests

* Ensure only a single instance performs concurrency sweeping by using redis-worker cron jobs

* Improved the mark phase

* Ensure cron jobs get rescheduled even if the handler throws an error

* Better property names
2025-06-30 16:22:19 +01:00
nicktrn fa6675eb00 Helm chart improvements (#2207)
* add helm deps

* configurablecli tag for setup commands

* enabled -> deploy

* extraEnv -> extraEnvVars

* update docs

* minio -> s3

* improve persistence config

* set cli tag to v4-beta

* chart and image bump

* fix tests

* lock chart

* workflow hotfix

* quote for niels

* make probes configurable

* fix webapp and supervisor templates

* bump chart
2025-06-28 12:59:29 +01:00
Eric Allam 8a86e7b41d Remove deprecated schedule trigger jobs when registering with the schedule engine (#2204)
* Remove deprecated schedule trigger jobs when registering with the schedule engine

* Use a tryCatch just to be absolutely sure
2025-06-27 12:32:02 +01:00
Eric Allam 0f3179f493 Provide realtime skipColumns option via untamperable public access tokens (#2201)
* Provide realtime skipColumns option via untamperable public access tokens

* Fixed realtime client tests

* Create red-rings-marry.md
2025-06-26 15:13:26 +01:00
Eric Allam b87e1c4630 Improve clickhouse client debug logging (#2197)
* Improve clickhouse client debug logging

* Stop hardcoding ScheduleEngineWorker log level

* Add DLQ debug log message
2025-06-26 15:12:54 +01:00
Eric Allam 14058d557f fix: queues no longer unpaused on deploy (#2198) 2025-06-26 15:08:52 +01:00