Commit Graph

1084 Commits

Author SHA1 Message Date
github-actions[bot] 3aca603a33 chore: Update version for release (beta) (#1153)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2024-06-12 17:34:47 +01:00
nicktrn 01633c9c03 Print error logs on dev worker failure 2024-06-12 13:08:50 +01:00
nicktrn b2ba403dd3 Improve error logs cli message 2024-06-12 12:04:59 +01:00
nicktrn 68d32429b6 v3: checkpoint failover and misc fixes (#1157)
* configurable checkpoint registry namespace

* add missing task create await

* remove unused messages

* changeset

* update self-hosting docs

* capture and display stderr for failed deploys

* add missing lockfile changes

* stderr changeset

* fix cli stderr message

* update error logs label
2024-06-11 14:06:50 +01:00
Eric Allam ca94f0cac3 cli v3: fix deploys by using sub-path imports (fixes execa require ESM error) 2024-06-11 11:08:15 +01:00
Eric Allam c332519e72 v3: Add a much needed index on TaskEvent.spanId 2024-06-10 20:54:13 +01:00
nicktrn 35dbaedf69 v3: self-hosting (#1147)
* add amin email regex env var

* fix displayed init command for self-hosted setups

* shared env var to disable telemetry in cli and webapp

* pin sdk version during init

* if specified, add api url to dev command shown after init

* improve checkpoint support detection

* control forced checkpoint simulation via env var

* add public init to providers

* better checkpoint support check for coordinator

* add docker to coordinator image

* update docker provider containerfile

* bump remaining containers to node 20

* add infra image build to default publish workflow

* lockfile

* remove concurrency group from infra workflow

* add docker provider to build matrix

* fix var subst

* checkpoint test is docker specific

* enable v3 projects by default on self-hosted instances

* fix v3 setup command again

* add default posthog key

* self-hosting docs

* add latest tags to versioned infra and webapp builds

* some checkpoint errors should skip retrying

* add changeset

* shorten paragraph

* some docs updates

* update tunnelling section

* add registry setup section

* use correct cli push flag

* add checkout to v3 branch

* update the worker machine setup steps

* fix infra build

* small docs update

* remove unused feature function

* Revert "remove unused feature function"

This reverts commit cfe07887a12b6893dca8ce499964481a9b3dc9db.

* fix self-hosted v3 feature gate

* add note about missing arm support

* simplify helper script syntax
2024-06-10 14:13:04 +01:00
Eric Allam c11a77f50b cli v3: increase otel force flush timeout to 30s from 500ms 2024-06-07 20:46:03 +01:00
github-actions[bot] df7d1de16d chore: Update version for release (beta) (#1142)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2024-06-06 15:59:21 +01:00
Eric Allam 8fc8f57b39 v2: MarQS powered job executions (#1149)
* WIP

* Allow marqsv2 and v2 graphile to run in parallel

* Fix missing GraphileLogger import

* Fixed heartbeat after rebase

* Replace postgres based run counters with redis ones with a backfill

* Add back in the graphile logger

* Remove duplicate visibility timeout calls

* Clamp simple weighted strategy to max of 5
2024-06-06 13:56:27 +01:00
nicktrn 8e5ef176a4 v3: registry maintenance (#1146)
* retry checkpoints with backoff and optional failover registry for deploys

* never abort checkpoint cleanup

* simulate checkpoint failure for 5 minutes

* add flag to simulate checkpoint push failure

* add flag to control push failure simulation duration

* backoff with helper

* handle all coordinator errors

* improve stop retrying

* increase cleanup ipc timeout

* improve webapp socket.io handler error logging

* remove unused backoff function
2024-06-05 11:06:18 +01:00
Eric Allam b4f9b70ae2 v3: Support triggering tasks with non-URL friendly characters in the ID 2024-05-31 11:39:17 +01:00
Matt Aitken 51bb4c887a Support custom queue when triggering a task (#1138)
* Created a v3-catalog test script for queues

* SDK: Fix for calling trigger and passing a custom queue

* Support custom queue in TriggerTaskService

* Improved the script in the catalog so it’s clearer what’s going on

* Remove the concurrencyLimit from a queue if the limit is null

* Fix for the test code… stupid
2024-05-31 11:24:13 +01:00
Eric Allam ba71f959e2 v3: Runs List Management SDK and SDK improvements (#1133)
* Improved the existing runs API

* WIP next runs API

* Improve the returned ApiPromise to add ability to return response

* More WIP

* WI{

* Added offset/limit pagination stuff like the cursor one, and converted all API methods to use ApiPromise

* More run API stuff

- Adding schedule  output from the retrieveRun endpoint
- Ability to filter by schedule and isTest

* Remove env from retrieve run in openAPI

* prefer duplication over merge

* WIP docs

* Use spread to DRY up some run API schemas

* Finish the overview docs

* Adding changeset

* Fixed typecheck errors

* Typo fix

* Re-export zodfetch from core so the v3 CLI can use it

* Fixed type errors

---------

Co-authored-by: Matt Aitken <matt@mattaitken.com>
2024-05-31 11:02:28 +01:00
github-actions[bot] ca9e827bd3 chore: Update version for release (beta) (#1130)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2024-05-30 12:15:10 +01:00
Matt Aitken 98ef170299 Set the CLI deploy timeout to 3mins from 1min 2024-05-30 12:08:14 +01:00
Eric Allam e69ffd314a v3: Refactor attempt creation to be worker requested (#1077)
* WIP worker TaskRunAttempt creation

* Handling failing task runs that cannot create an attempt for whatever reason

* Move the visibility queue stuff into a graphile job

* Fixed task runs with unsanitized queue names

* “Borrow” the code from alerts PR to get self hosted deployments working

* Add an admin API endpoint to get info about the shared marqs queue

* Allow admins to view any project metrics

* start adding lazy attempts to prod

* lazy attempt creation for prod workers

* resurrect prod stack traces

* add exception event to failed run spans

* simplify dependency resumes

* fix typecheck

* fix merge

* fresh process for all attempts

* always try sigterm first

* stop heartbeat timeout on non-inplace replace message

* add missing ack on checkpoint creation service failure

* bypass dequeue for retries with running worker

* respect retry delays

* crash runs with invalid run status for execution

* remove debug logs

* fix nack message

* fix version locking

* fresh attempt processes in dev and prod

* improve handling of ipc timeouts

* consider checkpoint failures on cancellation

* add basic chaos monkey to checkpointer

* changeset

* control forced checkpoint simulation via env var

* fix merge

* kill old attempt processes before checkpointing

* detailed perf logging for checkpointing

* add coordinator otlp endpoint example

* improve prod run cancellation

* rename supports lazy attempts migration

* fix graceful exit

* fix retry mechanics

* clear paused state before retry

* remove checkpoint image after push

* crash worker on unrecoverable errors

* refactor unrecoverable error emit

* switch to do hosted busybox image

* increase wait for duration ipc timeout

* add changeset for misc fixes

* fix merge

* fix retry delay span runId

* fix dev retries

* improve prod worker logging

* log checkpoint sizes

* add lazy attempts catalog entries

* Fixed merge issue: use zodFetch, not wrapZodFetch

* Revert "Fixed merge issue: use zodFetch, not wrapZodFetch"

This reverts commit d137e4e1fe.

* importEnvVars uses wrapZodFetch now

* add backwards compat for retries without checkpoints

* handle more cases of unrecoverable runs

* don't kill the child process if it shouldn't be killed

---------

Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
Co-authored-by: Matt Aitken <matt@mattaitken.com>
2024-05-30 11:05:52 +01:00
Eric Allam ece6ca678a Fix issue when using SDK in non-node environments by scoping the stream import with node: 2024-05-25 21:43:16 +01:00
github-actions[bot] ea4d15536c chore: Update version for release (beta) (#1115)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2024-05-24 14:55:09 +01:00
Eric Allam 3a1b0c486a v3: env var management API (#1116)
* WIP env var management API

* Add import env var API endpoint

* Adding docs and support for using both API keys and PATs when interacting with the env var endpoints

* WIP envvar SDK

* Uploading env vars in a variety of formats now works

* Finish env var endpoints and add resolveEnvVars hook

* Add changeset
2024-05-23 16:11:25 +01:00
Matt Aitken 974bcc49e8 Speed up the Run log queries (#1126)
* Added traceId index to TaskEvent

* Get less data and get the trace from the run

* Use the read replica in the EventRepository

* Use the regular client to get the run because the redirect from test can be very fast
2024-05-23 15:32:43 +01:00
Matt Aitken 60cbb87b08 New TaskRunNumberCounter that uses taskIdentifier + environmentId (#1124)
* Switch to new TaskRunNumberCounter that uses taskIdentifier + environmentId

* Drop old TaskRunCounter table (not used anymore)

* Revert "Drop old TaskRunCounter table (not used anymore)"

This reverts commit d7311a3d03cd8020df02105da155ceec4fa24b1b.
2024-05-23 13:28:23 +01:00
Matt Aitken 1281d40e4b v2: When a run hits the rate limit reschedule the re-execution (#1125)
* Fix: API rate limit error has the correct seconds until reset

* When a v2 run hits the rate limit, reschedule using the reset timestamp

* Still throw AutoYieldRateLimitErrors

* Reschedule runs from the rate limit

* The stress test timeout should be inside the task

* If the rate limit error is thrown, don’t retry the API request
2024-05-23 13:26:03 +01:00
Matt Aitken 116766f398 Request v3 access (from the app) and disable v2 projects by default (#1123)
* Added v2Enabled and hasRequestedV3 columns to Organization

* Don’t create a project when you create an org

* Form for requesting v3 access

* Reworked the new project form with the different version states. Refined copy on early access

* If the project isn’t in the org then redirect to the new project page

* Better message for existing users

* Tidy imports

* If it’s not the managed cloud then allow them to create v2 projects
2024-05-22 19:38:42 +01:00
Eric Allam c092c0f9db v3: Prevent legacy-peer-deps=true from breaking deploys 2024-05-22 15:46:13 +01:00
Alexandre Costa a86f36cefa Fix TypeScript inclusion in tsconfig.json for cli-v3 init (#1105)
* Fix TypeScript inclusion in tsconfig.json for cli-v3 init

Fixed an issue where TypeScript files were included in the project directory when no include directive was present in tsconfig.json. Previously, the CLI added trigger.config.ts to the inclusion list by default, causing TypeScript compilation errors for other files. The fix ensures that trigger.config.ts is only added to the inclusion list if there's an existing include directive present in tsconfig.json

* Create hot-fishes-retire.md

---------

Co-authored-by: Eric Allam <eric@trigger.dev>
2024-05-21 14:40:49 +01:00
Eric Allam 5a6e79e0c0 Fix missing logs when client includes their own @opentelemetry/api package (#1106) 2024-05-21 14:38:06 +01:00
Eric Allam 3f8b6d8fce v2: Better handle recovering from platform communication errors by auto-yielding back to the platform in case of temporary API failures 2024-05-21 14:07:16 +01:00
Matt Aitken a5cba375ae Bulk replaying and canceling from the runs list (#1109)
* WIP on multi-select

* WIP on simple checkbox

* CheckboxWIthLabel and Checkbox

* Multi-selection of runs across pages is working

* Fix for selection on seconds page

* Focus the run filter on page load

* Don’t focus the checkbox

* BulkActionBar now shows/hides and has buttons

* Some state to stop escape clearing the selection when the modals are open

* Delete unused formData util

* Improvements to the page

* Created the replay resource action. It doesn’t do anything useful yet.

* Database schema created for BulkActionGroup/BulkActionItem

* The BulkActionService is creating the right data, now we need to process it

* WIP on bulk processing

* Added failed state and made the sourceRun required

* Bulk replaying is working

* WIP on bulk action filtering

* Fixed bulk filters displaying

* Filtering by batch is working

* Some fixes for the bulk id filtering

* Style tweaks

* Load the extra info in parallel

* Bulk canceling working

* Get the most recent 20 bulk actions to display in the filter menu

* Even if the run isn’t cancelable add it to the final list

* Maximum of 250 runs can be bulk actioned

* Don’t let them select more than the maximum (250 currently)

* Separate each bulk item action into it’s own separate graphile job to increase resiliency

---------

Co-authored-by: Eric Allam <eallam@icloud.com>
2024-05-20 17:17:33 +01:00
Eric Allam f4f87ad573 Defer the enqueuing of events.deliverScheduled events, to keep the graphile_jobs table smaller 2024-05-20 14:27:20 +01:00
Eric Allam 68107565a4 v3: Fix issues with posting alerts to public slack channels (#1108)
* Fix issues with posting alerts to public slack channels

* Use the actual values in the new environmentTypes column to display the environment type labels in the alerts list

* Implement environment alert options
2024-05-17 15:24:47 +01:00
github-actions[bot] a7ca1222d7 chore: Update version for release (beta) (#1104)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2024-05-16 19:00:06 +01:00
Eric Allam 6a379e4e91 Fix 3rd party otel propagation from breaking our Task Events data from being properly correlated to the correct trace 2024-05-16 12:07:21 +01:00
Matt Aitken 598906fc4c Fix for typo in v3 CLI login command 2024-05-16 09:23:45 +01:00
github-actions[bot] 72748603ab chore: Update version for release (beta) (#1102)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2024-05-15 16:21:58 +01:00
Eric Allam 21c8b847e5 Better handle resolving dependencies when there are conflicts 2024-05-15 13:26:03 +01:00
github-actions[bot] 107f2e7bd5 chore: Update version for release (beta) (#1099)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2024-05-13 16:14:14 -04:00
Eric Allam b8477ea2b0 Fixes an issue with scoped packages in additionalPackages option 2024-05-13 15:31:19 -04:00
github-actions[bot] 2e21bc1563 chore: Update version for release (beta) (#1094)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2024-05-13 17:38:50 +01:00
Matt Aitken 1477a2e309 Set the timeout for canceling checkpoints 2024-05-13 16:50:53 +01:00
James Ritchie cbe9317a86 Added padding to the email body 2024-05-09 17:39:39 +01:00
James Ritchie a823421324 Small style improvements to the emails 2024-05-09 16:49:45 +01:00
Eric Allam 0e919f56f2 Alerts v1 (#1065)
* Alerts v1

* Encrypt alert webhook secrets and allow them to be generated by the server

* Alert v1 UI

* Remove unnecessary emails

* Move to using `@react-email/components`

* WIP slack alerts

* More slack alerts WIP

* Update pnpm lock after rebase

* Finish implementing Slack alerts

* Use a more error like emoji

* New secondary variant for the segmented control

* Added a simple checkbox style variant to storybook

* Style tweak to the segmented control

* UI improvements to the alert modal

* Use searchable Select for alerts. Changed default variant for SegmentedControl

* Secondary button now using secondary colour

* segmented control style tweak

* Improved the channel column in the alerts table

* Updated logo-mono.png

* Updated email styles

* Email templates updated to new styles

* Don’t log the decrypted secret

* await enqueing the deployment alert when an index fails

* await enqueing the timeout alert

---------

Co-authored-by: James Ritchie <james@jamesritchie.co.uk>
Co-authored-by: Matt Aitken <matt@mattaitken.com>
2024-05-09 12:12:00 +01:00
github-actions[bot] 223768a025 chore: Update version for release (beta) (#1091)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2024-05-08 17:25:22 +01:00
github-actions[bot] 56a063e586 chore: Update version for release (beta) (#1087)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2024-05-06 12:34:43 +01:00
Eric Allam d490bc5cbf Add the "log" level back in as an alias to "info" 2024-05-06 12:28:17 +01:00
Eric Allam 6d9dfbc75d Add configure function to be able to configure the SDK manually
And move the ApiClient configuration away from AsyncLocalStorage and use our globals system instead
2024-05-06 12:12:30 +01:00
Eric Allam 6406924b02 Ensure @trigger.dev/sdk and @trigger.dev/core are always in the list of deployed dependencies 2024-05-06 11:01:49 +01:00
Eric Allam 2156e15266 Adding some additional telemetry during deploy to help debug issues 2024-05-06 10:34:54 +01:00
github-actions[bot] 2e4f5d893c chore: Update version for release (beta) (#1086)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2024-05-05 08:19:13 +01:00