Commit Graph

4084 Commits

Author SHA1 Message Date
Saadi Myftija 3cfde48bfd feat: expose all run options in the test run page (#2227)
* Implement a new primitive UI component for picking durations

* Implement a new component to input run tags

* Expose all run options in the test run page

* Add subtle animations when adding/removing run tags in the test page

* Add a new resource endpoint for fetching queues

* Fetch usable queues for the selected task

* Fix width display issue in the select component

* Enable locking a run to a version from the test page

* Disable entering max attemps <0

* Validate tags

* Add recent runs popover

* Only show latest version for development environments

* Update run options when selecting a recent run

* Rearrange the test page layout

* Add subtle animation to the duration picker segments on focus

* Improve queue selection dropdown styling

* Fix disabled state issue for the SelectTrigger component

* Disable version selection field for dev envs

* Add usage hints next to the run option fields

* Add machine preset to the run options list

* Allow arbitrary queue inputs for v1 engine runs

* Show truncated run ID instead of run numbers for recent runs

Run numbers will soon get deprecated due to contention issues

* Fix duplicate queue issue

* Extract common elements across the standard and scheduled test task forms

* Apply values from recent runs to scheduled tasks too

* Add additional run options for scheduled tasks

* Use a slightly smaller font size for run option labels

* Disallow commas in the run tag input field

* Switch to a custom icon for recent runs button

* Flatten the load function test task result object

* Avoid redefining machine presets, use zod schema instead

* Fix ClockRotateLeftIcon jsx issues

* Remove recent runs button tooltip as it causes nesting errors

* Adjust the page layout to make it clear which task is currently selected

* Inline the tab group with the copy/clear buttons
2025-07-09 10:40:10 +01:00
Eric Allam b113e5a99c Fix non-MFA login (#2246)
* Fix non-MFA login

* fixed typecheck errors
2025-07-08 14:59:31 +01:00
Eric Allam b119a52e08 Implement MFA (#2244)
* Adds a new route for logging in with mfa

* New path for security page

* Adds “Security” link to account side menu

* Update the Switch component to allow label positions left and right

* Optionally hide the Close button in the Dialog title bar

* Installs `qrcode` react package for generating QR codes.

* CopyButton component now takes children

* New Security route for setting up MFA

* Adds new OTP package for the chadcn InputOTP component

* Adds new InputOTP chadcn component

* Adds InputOTP chadcn component to the MFA login screen

* InputOTP component supports variant styles

* Improvements to form handling

* Show a confirmation modal before you can disable MFA

* Revert redirect back to the dashboard for now

* Implement MFA enabling and disabling

* Refactor and cleanup mfa management code

* More cleanup

* Handle errors in the management action

* Implement mfa login flow

* recovery code input should be password

* Implement rate limiting on the mfa validation endpoint

* Better error ux

* Implement mfa emails and apply James' updates

* Use latest @better-auth/utils

* Improvements via CodeRabbit review

---------

Co-authored-by: James Ritchie <james@trigger.dev>
2025-07-08 13:44:47 +01:00
Eric Allam 493ca11c43 Add ability to disable the prisma query in the healthcheck endpoint (#2242) 2025-07-07 16:41:55 +01:00
Eric Allam 47e583c19d Properly scope the batch list to the selected environment (#2236)
Prevents showing batch runs from environments that are not in the selected environment
2025-07-04 14:41:10 +01:00
Eric Allam 469a8df0cf Prevent redis worker from enqueuing pending jobs (#2235) 2025-07-04 13:53:32 +01:00
Eric Allam 922c7cb3a6 Increase batch trigger processing job visibility timeout (#2234) 2025-07-04 13:43:26 +01:00
nicktrn 74fdfa93c5 Show project details when running the whoami command (#2231)
* add project details to whoami

* findFirst

* skip project check if no orgs found

* add changeset
2025-07-04 13:40:41 +01:00
Eric Allam bdaa2ed15a Move batch trigger processing jobs to a dedicated redis worker (#2233) 2025-07-04 12:16:05 +01:00
nicktrn afc9a6ade1 Add ECR support for remote builds (#2224)
* create repo if doesn't exist

* fresh auth token for each deploy

* optional assume role

* log when machine overrides enabled

* make test repo namespace configurable

* assume role fix and env var changes

* improve ecr check

* improve tag parsing

* tag parsing tests

* track if repo created and fix test

* missing tryCatch for sts call
2025-07-03 14:25:30 +01:00
Eric Allam 7bb7e7aedc When sharding, use the where clause in the shard key to distribute requests more evenly (#2229) 2025-07-03 14:21:58 +01:00
Eric Allam 0c2af6d87c Increase otel attribute limits and make them configurable 2025-07-02 11:30:13 +01:00
Eric Allam 57f3a89a86 Fix timeline span events tests 2025-07-02 11:30:13 +01:00
Eric Allam ceb737e9f1 Fix broken webapp test 2025-07-02 11:30:13 +01:00
Eric Allam 632d6b91e9 Fix taskSlug extraction 2025-07-02 11:30:13 +01:00
Eric Allam b548561220 Remvoe the sometimes incorrect import file name 2025-07-02 11:30:13 +01:00
Eric Allam 58d56f8381 Use hostname instead of host 2025-07-02 11:30:13 +01:00
Eric Allam 8798352be3 Super-warm starts in deployed tasks proof 2025-07-02 11:30:13 +01:00
Eric Allam 9374aea401 Split resource and span attributes 2025-07-02 11:30:13 +01:00
Eric Allam e2049fcfbf Initial dev process keep alive. moved resource attributes to span attributes because resource attributes are global and immutable 2025-07-02 11:30:13 +01:00
Matt Aitken 4a79b3b35b Fixed retrying on run replication inserts on connection errors (#2216) 2025-07-01 13:35:13 +01:00
Matt Aitken f38d35e9f6 ClickHouse replication improvements (retrying, strip bad unicode chars) (#2205)
* Add retry logic for insert operations

Add a generic retry mechanism for task run and payload inserts to handle
transient connection errors. The new #insertWithRetry method retries up to
three times with exponential backoff and jitter on retryable connection
errors such as connection resets or timeouts. Errors are logged and
recorded in tracing spans to improve observability and robustness of the
replication service.

* Replication settings are configurable

* Log out the runIds for failed batches

* Detecting bad JSON in run replication and ignoring it

* Reproduced split unicode error

* Move output file

* Massively improved the performance

* Minor performance improvements

* Unskip tests

* Remove unused test in CH package

* Fix for the ClickHouse UI explorer

* RunReplication keepAlive defaults to false

* Add concurrency_key and bulk_action_group_ids to ClickHouse task runs

* ClickHouse package doesn't need to be built anymore for the webapp

* Set the concurrency_key from the run replication service
2025-06-30 16:32:02 +01:00
Eric Allam 1b4dabb8ba v4: current concurrency sweeper (#2206)
* v4: current concurrency sweeper

* Fix webapp tests

* Ensure only a single instance performs concurrency sweeping by using redis-worker cron jobs

* Improved the mark phase

* Ensure cron jobs get rescheduled even if the handler throws an error

* Better property names
2025-06-30 16:22:19 +01:00
nicktrn fa6675eb00 Helm chart improvements (#2207)
* add helm deps

* configurablecli tag for setup commands

* enabled -> deploy

* extraEnv -> extraEnvVars

* update docs

* minio -> s3

* improve persistence config

* set cli tag to v4-beta

* chart and image bump

* fix tests

* lock chart

* workflow hotfix

* quote for niels

* make probes configurable

* fix webapp and supervisor templates

* bump chart
2025-06-28 12:59:29 +01:00
Eric Allam 8a86e7b41d Remove deprecated schedule trigger jobs when registering with the schedule engine (#2204)
* Remove deprecated schedule trigger jobs when registering with the schedule engine

* Use a tryCatch just to be absolutely sure
2025-06-27 12:32:02 +01:00
Eric Allam 0f3179f493 Provide realtime skipColumns option via untamperable public access tokens (#2201)
* Provide realtime skipColumns option via untamperable public access tokens

* Fixed realtime client tests

* Create red-rings-marry.md
2025-06-26 15:13:26 +01:00
Eric Allam b87e1c4630 Improve clickhouse client debug logging (#2197)
* Improve clickhouse client debug logging

* Stop hardcoding ScheduleEngineWorker log level

* Add DLQ debug log message
2025-06-26 15:12:54 +01:00
Eric Allam 14058d557f fix: queues no longer unpaused on deploy (#2198) 2025-06-26 15:08:52 +01:00
Eric Allam 485f71c8d4 Improve schedule engine performance (#2191)
* Improve schedule engine performance

* Convert an info log to a debug log
2025-06-23 16:55:18 +01:00
Eric Allam fb9abe5289 fix: Incorrect/missing logs on new schedule engine runs (#2185)
* Fix incorrect logs on new schedule engine triggered taskss

Also added the ability to recover schedules in the schedule engine via an Admin API endpoint in the new schedule engine

* Fixed schedule recovery failing test
2025-06-19 15:37:50 +01:00
Eric Allam f2db1b8aa5 Create dedicated alerts worker (#2184)
* Add a new alerts worker that just handle alerts

* fallback to graphile worker when a tx is required
2025-06-19 11:54:49 +01:00
nicktrn 3c93783e8b remove nodetype label requirement (#2182) 2025-06-18 11:34:35 +01:00
Matt Aitken 4ca6506655 Delay reschedule fix (v3 & v4) (#2180)
* Fix for v3 rescheduling bug…

* v4 rescheduling working
2025-06-17 18:22:15 +01:00
Matt Aitken 9f9e74b0b6 Fix hydration error (#2179)
* Remove unused hook

* The Impersonation button was causing DOM hydration issues
2025-06-17 11:17:17 +01:00
Eric Allam ea6f66bdfc Improved schedule engine (#2174) 2025-06-17 06:48:05 +01:00
Eric Allam a42c9acf44 v4: Improved run locking (#2173) 2025-06-17 06:45:59 +01:00
Matt Aitken 5f6fd8e235 Tasks page didn't include Delayed runs (#2176)
* Delayed runs weren’t shown in the Tasks page mini graph

* The Tasks page Queued count didn’t include Delayed runs (since ClickHouse switchover)
2025-06-16 14:14:09 +01:00
Matt Aitken a0815c8e3e Clickhouse performance improvements (#2175)
* Runs filter by org id and add created at to ordering

* CopyableText can accept an alternative value for copying

* The runs table now shows the ID instead of run number

* Paginating back/forwards fix

* The task stats need org id and project id too
2025-06-14 11:10:28 +01:00
Matt Aitken c3dca5835d Fix for the Kapa widget not working with SSR (#2170)
* Move all the logic into a single component, preparing for client only

* Use ClientOnly

* Fix for search param not working
2025-06-12 22:17:22 +01:00
James Ritchie 20a8e7c6c1 Updated kapa sdk to 0.1.3 (#2169) 2025-06-12 16:21:34 +01:00
James Ritchie e12c82b444 Use the Kapa AI SDK instead of the Kapa AI widget (#2113)
* Install the kapa sdk

* WIP using the SDK for the Kapa Ask AI widget

* Removes old kapa from root

* Now rendering everything inside the dialog component

* Fixes min-height of dialog content

* Remove kapa from root

* prevents kapa using reCaptcha

* Adds more functionailty with temporary UI placement for now

* Reset conversation button

* Adds a new sparkle list icon

* Adds some example questions as a blank state

* Animate in the example questions

* use “marked” package to render markdown

* Improve some animations

* Submit a question from the URL param

* adds custom scroll bar styling

* fixes modal to correct height after re-opening it

* Add button to stop generating answer mid-stream

* Adds buttons states to show submitting, generating, submittable

* Adds a helpfull sentence in the blank state

* Show a message if the chat returns an error

* Adds reset chat and feedback buttons to the bottom of an answer

* Makes sure you can give feedback in the different states of chat

* Adds a suble background to the dialog

* Fix a button inside button error

* Improve the shortcut esc key on dialog and sheet component

* Fix classname error

* organize imports

* Use our custom focus-visible

* Move the Tooltip for the button into the AskAI component

* Improved error message

* Organize imports

* Animated the modal gradient

* Small layout improvements

* Adds most asked questions from Kapa

* border glow tweak

* AskAI component is now a hook that can take a question

* remove kapa script

* Add a delay before the modal opens when usign the URL params

* Remove old component

* Update to the latest Kapa version

* Rephrased error message

* Use correct types for conversation

* Fixed types for addFeedback

* Adds DOMPurify package

* removed unused const

* Removed unnecessary platform specification

* Reset the timeout when the ai panel pops up

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>

* Fix for coderabbit bad commit

* Clean up imports

---------

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
2025-06-12 14:09:03 +01:00
Eric Allam 48e1e7250b v4: fix stuck batch issue when processing happens before transaction commits (#2163)
* v4: fix stuck batch issue when processing happens before transaction commits

* Fix flaky test
2025-06-10 17:02:34 +01:00
Eric Allam 3f2b748a80 Delete the secure param before passing it to the ClickHouse client (#2161) 2025-06-10 13:55:19 +01:00
Eric Allam e6fb32194e Add environment filter to declarative schedules (#2148) 2025-06-10 12:11:11 +01:00
Eric Allam b38405cb88 Realtime and task run performance improvements (#2158)
* Add createdAt filter to realtime subscribing with tags

* Filter realtime colums and expose ability to skip some columns

* Add sharding support for electric

* Use unkey cache for the created at filter caching

* Remove 2 unused indexes on TaskRun

* Run list now filters by a single runtime environment

* Remove project ID indexes

* Use clickhouse in task list aggregation queries instead of pg (keep pg for self-hosters)

* WIP clickhouse powered runs list
stuff

* Improve the query to get the latest tasks for the task list presenter

* Update the usage task list to use clickhouse

* Implement next runs list powered by clickhouse

* Add new index for TaskRun for the runs list, by environment ID

* Add runTags gin index

* Handle possibly malicious inputs

* Ignore claude settings

* Better handling not finding an environment on the schedule page

* Use ms since epoch in test, not seconds

* Remove unused function

* Fix test

* Use an env var for the realtime maximum createdAt filter duration (defaults to 1 day)

* Fixed the query builder to correct the group by / order by order

* Make sure runs.list still works

* Create small-birds-arrive.md
2025-06-10 12:11:01 +01:00
nicktrn 47b19a49d4 A few small v4 fixes (#2153)
* remove unused env vars

* actually use ALERT_REPLY_TO_EMAIL for alerts (non-breaking)

* increase fallback branch limit to 100M

* more unused env vars and example

* hide usage page when self-hosted

* fix for init flow

* set default or concurrency to 300 to match 100 on env

* fail fast when registry env vars are empty strings
2025-06-06 10:15:07 +01:00
Eric Allam 5301239e09 chore: remove v2 models from schema (#2107)
* Round 1 of v2 model removals

* Round 2

* Remove trigger http endpoint env

* Round 3

* Round 4

* Round 5

* Round 6

* Round 7

* Round 8

* Round 9

* Round 10

* Removed the remainder v2 code from the webapp

* GitButler WIP Commit

---------

Co-authored-by: GitButler <gitbutler@gitbutler.com>
2025-06-05 22:57:29 +01:00
Eric Allam 261dce102a v4: fix batchTriggerAndWait completion issues by processing batch chunks sequentially (#2152)
* v4: fix batchTriggerAndWait completion issues by processing batch chunks sequentially

* Remove unnecessary test
2025-06-05 17:56:56 +01:00
nicktrn e7795a06ad Fix: fixes and prerequisites for v4 self-hosting (#2150)
* remove pgadmin

* remove V3_ENABLED

* v3 is always enabled

* enfore docker machine presets by default

* rename autoremove env var

* prefix more k8s-specific env vars

* same prefix for all docker settings

* improve profile switcher copy

* supervisor can load token from file

* optional webapp worker group bootstrap

* fix error message

* fix app origin fallback for otlp endpoint

* use pnpm cache for webapp docker builds

* increase default org and env concurrency limit to 100

* optional machine preset overrides

* improve s3 pre-signing errors

* fix DOCKER_ENFORCE_MACHINE_PRESETS bool coercion

* shard unit tests

* fix for s3-compatible services

* optional object store region

* Update apps/supervisor/src/workerToken.ts

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>

* fix DEPLOY_REGISTRY_HOST example

* fix platform mock

* remove remaining v3Enabled refs

* fix error type.. bad bot

---------

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
2025-06-04 18:02:38 +01:00
Eric Allam 2b3ea692fe v4: dequeue performance improvements (split concurrency from dequeue) (#2127)
* WIP

* Run queue now works with the worker queue / master queue split

* Acking should also cause the master queue to be processed

* Convert run engine tests and run engine to use runQueue changes

* Include the util files in the test tsconfig

* coordinator target should be es2020 as well

* providers target 2020

* Fix the triggerTask tests in the webapp

* v4 now working with the new worker queues, and added the legacy master queue migration stuff

* report worker queue lengths via opentelemetry metrics

* Adding lock metrics

* Release concurrency bucket metrics

* • Updated RunQueue.removeEnvironmentQueuesFromMasterQueue() method signature to take runtimeEnvironmentId instead of masterQueue parameter
• Added automatic master queue shard calculation using this.keys.masterQueueKeyForEnvironment(runtimeEnvironmentId, this.shardCount) 
• Updated RunEngine wrapper method to use new runtimeEnvironmentId parameter
• Updated DeleteProjectService to call the method once per environment instead of once per master queue
• Simplified API by encapsulating master queue sharding logic within RunQueue class

* metrics now working, configure the run queue settings, additional metrics for run engine and redis-worker

* Fix CodeRabbit suggestions

* return undefined from dequeueFromWorkerQueue, not null

* Remove message from worker queue in certain circumstances when acking

* Update log

* Ensure master queue consumers cannot stop from a processing error, and make the consumer interval configurable via an env var

* Change how the run queue master queue consumers are disabled internally

* Fixed tests

* process the queue on nack

* Fix more tests

* Fix priority tests

* Fixed dequeueing test
2025-06-04 17:18:46 +01:00