Commit Graph

5537 Commits

Author SHA1 Message Date
Eric Allam 92d98e5499 Build before running typecheck now 2024-08-23 13:46:18 +01:00
Matt Aitken a9810b0fd8 Stop heartbeats when creating andWait checkpoints, fix for concurrency tracker (#1268)
* RESUME messages were missing some data that is needed for the concurrency tracker

* Cancel heartbeats when checkpoints are created for triggerAndWait and batchTriggerAndWait

* Added some logs when canceling heartbeats

* Improved the nested dependencies test task
2024-08-23 13:10:39 +01:00
Eric Allam f9ec66c562 v3: new build system (#1265)
* upgrade @opentelemetry packages to the latest versions

* remove v2 only packages, will be moved to a dedicated repo

* remove more v2 code and run pnpm install

* use the npm yalt package in the webapp

* convert @trigger.dev/core to tshy

* Switch from jest to vitest in @trigger.dev/core

* Fixed core test

* move core-backend code into core subpath export

* convert @trigger.dev/sdk to tshy

* Removed hono

* move core-apps to core/v3/apps, remove core-apps, start converting cli-v3

* Fix up some of the commands

* cli now building and loadable

* using package-json-from-dist to get package version now in core and cli

* dev command WIP

* cleaned up some repetition and structure of the entry point stuff

* bringing back the background worker stuff

* Indexing of the v3 catalog

* getting closer to executing dev runs...

* centralize dev logging using event emitter

* Move indexing to it’s own entry point, simplify code

* dev runs working

* Get instrumentation to work with openai

* debugging achieved internally

* provide worker files as part of the worker creation on the server

* support for cjs and esm javascript

* Fixed timeout

* worker manifest now has the config path

* auto-upgrade config to non-deprecated alternatives

* Adding package preview release

* deployment WIP

* improve the syncEnvVars output and adapt resolveEnvVars

* WIP bun runtime

* WIP bun support

* seed tasks with the machine preset if listed in the config

* deploy run executions WIP, extracted TaskRunProcess into 1 place

* deployed tasks running and executing 🎉

* support for waits and better flushing & process cleanup

* Fixed the heartbeating

* Better warning messages

* Improve and unify the indexing between dev and deploy

* Support for external deps that need node-gyp to build

* build extensions can now install custom packages and run instructions in the image. Also prisma extension now works and also works with multiple schema files

* Add back in the main/types/module to sdk

* dev no longer is Ink/React, grace period for disconnections in dev

* Fix the changeset config

* More changeset fixes

* Remove config packages

* More changeset fixes

* Fixed typescript issues (needed to revert back to zod 3.22.3

* Fix pr_checks workflow

* Remove the prepare script

* Fixed tests and package versions

* Remove cli test script

* Remove packages from tailwind watch paths

* Add repo to public packages

* Just commit the generated files and do the building at dev time

* Try and get pkg.pr.new working

* Try again

* Fix emitDecoratorMetadata importing named export from typescript

* config file backwards compat with export const config

* Fixed issue where import errors weren’t coming through

* p-retry is a prod dep

* typescript needs to be a prod dependency for emitDecoratorMetadata

* Add better debug logging to help track down import-in-the-middle bug

* An external is only considered resolvable if it resolves to the same path as the collected external

* Fix runtime checks to allow >=18.20

* Move extensions to a new build package

* Fixed building packages in dockerfile

* Remove the e2e test from publish workflow for now

* Don’t treat pkg.pr.new versions has needing upgrading

* making sure config handleError works, and discovered path aliases don’t work in config files

* Strip empty string env vars so they accidentally override real values

* Couple of things

* Update version to use preview instead of beta

* Hopefully fix re-attempts with >30s delay

* Match socket emit messages to current latest in main

* Initial guide

* Go back to beta

* Go back to the preview, and update guide to use pr preview tags

* Go back to beta

---------

Co-authored-by: Matt Aitken <matt@mattaitken.com>
2024-08-23 13:10:15 +01:00
James Ritchie 2cd8658190 If an org has been deleted, show a ☠️ icon on the admin page 2024-08-22 13:15:00 +01:00
James Ritchie e5895424db Added a link to a Next.js example in the docs 2024-08-21 15:58:18 +01:00
Matt Aitken a724a6fa01 Added Task index parentId 2024-08-21 15:54:23 +01:00
Matt Aitken ba7bbdbbdb Dashboard query improvements (#1266)
* Remove run count from JobPresenter

* Remove job/integration counts from org/projects query

* Removed v2 counts from ProjectPresenter query

* JobRun index idx_jobrun_organizationId_createdAt

* JobRun index idx_jobrun_versionId

* BatchTaskRunItem index taskRunAttemptId

* BatchTaskRunItem index taskRunId
2024-08-21 15:49:18 +01:00
Matt Aitken 7746302b51 Run failed alerts (#1262)
* Add new ProjectAlertType ‘TASK_RUN’ and then migrate channels to it

* WIP new alerts for run failures

* Consolidate failed run status into taskStatus.ts

* Added the read replica into BaseService, could be useful

* Send task run alerts code

* Revert "Added the read replica into BaseService, could be useful"

This reverts commit cc2348a40254ea3f9e11c0feaf303ecd8794433a.

* Allow adding task run alerts in the UI

* Use task run alert not attempt alert…

* Use the primary
2024-08-21 14:53:48 +01:00
Matt Aitken 6558e5f8eb Fix for permanently frozen tasks when at max concurrency
Heartbeats now free up concurrency for paused/frozen runs, then they don’t get crashed but get acked by the sharedQueueConsumer
2024-08-21 11:37:02 +01:00
Matt Aitken 9f07df8abc Don’t nack in requeueTaskRun if it’s WAITING_TO_RESUME or PAUSED 2024-08-20 19:00:29 +01:00
Matt Aitken 8ad599f90e Try set the checkpointEventId when resuming batches and dependency 2024-08-20 16:09:43 +01:00
Matt Aitken 78dac99d72 Release 3.0.0-beta.55
🚀 Publish Trigger.dev Docker / e2e (push) Failing after 4s
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 2s
🚀 Publish Trigger.dev Docker / units (push) Failing after 19s
🚀 Publish Trigger.dev Docker / publish (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-infra (push) Has been skipped
@trigger.dev/airtable@3.0.0-beta.55 @trigger.dev/astro@3.0.0-beta.55 @trigger.dev/cli@3.0.0-beta.55 @trigger.dev/core@3.0.0-beta.55 @trigger.dev/hono@3.0.0-beta.55 @trigger.dev/linear@3.0.0-beta.55 @trigger.dev/nestjs@3.0.0-beta.55 @trigger.dev/nextjs@3.0.0-beta.55 @trigger.dev/otlp-importer@3.0.0-beta.55 @trigger.dev/react@3.0.0-beta.55 @trigger.dev/replicate@3.0.0-beta.55 @trigger.dev/sdk@3.0.0-beta.55 @trigger.dev/shopify@3.0.0-beta.55 @trigger.dev/slack@3.0.0-beta.55 @trigger.dev/stripe@3.0.0-beta.55 @trigger.dev/supabase@3.0.0-beta.55 @trigger.dev/sveltekit@3.0.0-beta.55 @trigger.dev/testing@3.0.0-beta.55 @trigger.dev/typeform@3.0.0-beta.55 @trigger.dev/yalt@3.0.0-beta.55 v.docker.3.0.0-beta.55 v.docker.3.0.0-beta.56 @trigger.dev/github@3.0.0-beta.55 @trigger.dev/plain@3.0.0-beta.55 @trigger.dev/eslint-plugin@3.0.0-beta.55 @trigger.dev/resend@3.0.0-beta.55 @trigger.dev/openai@3.0.0-beta.55 @trigger.dev/integration-kit@3.0.0-beta.55 @trigger.dev/sendgrid@3.0.0-beta.55 @trigger.dev/remix@3.0.0-beta.55 @trigger.dev/core-backend@3.0.0-beta.55 trigger.dev@3.0.0-beta.55 @trigger.dev/express@3.0.0-beta.55
2024-08-20 15:50:14 +01:00
James Ritchie d6786002b2 Full Next.js guide (#1259)
* tasks no longer inside a group in the side menu (and added “cron”)

* Delay using a timezone

* Added React Not Defined error to the troubleshooting page

* Improved the React common problem

* Link to v2 docs

* New Development section and entry in Common Problems

* Concurrently running the terminal

* Fixed the .env weirdness

* Added section on creating PATs for Github actions

* Improved the Machine spec and limits page

* Quick start steps now have nice images

* Added a diagram for the lifecycle functions

* Added note about onFailure

* CRON -> cron/Cron

* WIP adding more steps to the next.js guide

* WIP next.js

* WIP adding tabbed steps for pages/app router

* References to Infisical links to their homepage so it’s clearer

* WIP updating the nextjs guide

* WIP nextjs guide

* More nextjs guide steps

* More copy

* Added rate limit trouble shooting

* Removed old prisma error title

* Added secret key step

* Added a note for logging in using a specified domain if self hosting

* typo

* App router docs copy

* Deploy copy update

* Added a favicon.png to fix a docs build error

* Removed unused snippet

* Server actions now inside a tab

* Server actions + restructured the triggering section

* Added troubleshooting snippet for react event handlers

* Added a new troubleshooting snippet for ESM

* Updated old replaying image to reflect the new UI

* Removed references to reattempting

* Updated replaying from the run page

* Added a bulk replay section

* Updated the writing tasks intro page

* Removed edge runtime code for now

* Added edge runtime – it seems to just work!

* import type
2024-08-20 14:50:39 +01:00
github-actions[bot] 9d529e9f17 chore: Update version for release (beta) (#1264)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2024-08-20 14:46:19 +01:00
Matt Aitken d48eb9bac8 Fix: was showing deleted team member dev environments (#1261)
* Env vars: don’t show deleted dev environments

* Concurrency limits: don’t show deleted dev environments

* Create/edit schedule: don’t show deleted dev environments
2024-08-20 14:03:24 +01:00
Matt Aitken 0591db5f2b Fix for short runs inside (batch)triggerAndWait (#1263)
* Test for checkpoints

* Make sourceTaskAttemptId optional on resumeBatchRun

* Removed all completions/executions logic from the shared queue consumer

* Removed the sourceTaskAttemptId from ResumeBatchRunService

* Revert "Removed all completions/executions logic from the shared queue consumer"

This reverts commit d35398d50463c81a1975bb5d5bcfca66a24b8ede.

* WIP on triggerAndWait…

* Fixed triggerAndWait continuing when a checkpoint completes

* Remove the ResumeAttempt code that fails attempts (was protecting against infinite restores)

* Removed messageBody.data.completedAttemptIds.length === 0 commented out code

* Don’t ack if there’s no batchRun

* Added the marqs?.replaceMessage back in but NOT when there’s no checkpoint. More logging

This is a fix for when some attempts fail

* Improvement to the test task that now randomly fails attempts

* When a checkpoint happens, only continue the attempt if it’s in the correct state

* Changeset for rollback in branch

* Set keepRunAlive to false when the dependent task isn’t finished

* Changeset manual version (to get inline with the hotfix branch)

* Changeset: Fixes for continuing after waits

* Latest lockfile (after manual changeset version)
2024-08-20 13:57:08 +01:00
nicktrn a64064965d Prevent runs that fail fast from never sending a heartbeat 2024-08-19 13:17:33 +01:00
Matt Aitken 41ec450bee Revert "Remove the RequeueTaskRunService"
This reverts commit 3421f5ec0b.
2024-08-15 20:43:47 +01:00
Matt Aitken 9a9bdc20c9 Revert "Disable upserting task queues when triggering a task"
This reverts commit 6652edf7a7.
2024-08-15 20:42:50 +01:00
Matt Aitken 6652edf7a7 Disable upserting task queues when triggering a task 2024-08-15 19:34:22 +01:00
Matt Aitken 369bc38560 Instead of upserting taskQueues, do a findFirst then create/update 2024-08-15 18:48:50 +01:00
Matt Aitken 27f7132509 Don’t do the auto-increment inside a transaction, this means triggerTask isn’t inside one 2024-08-15 17:44:10 +01:00
Matt Aitken b6fc68aaae Don’t use Graphile for invocation usage reporting 2024-08-15 17:22:52 +01:00
Matt Aitken 3421f5ec0b Remove the RequeueTaskRunService 2024-08-15 17:20:40 +01:00
Matt Aitken 15097a3a67 Use the replica for the v2 run and task presenters 2024-08-15 15:34:57 +01:00
Matt Aitken 1b7bb90852 Only show the first 1000 task on the v2 run page 2024-08-15 15:26:12 +01:00
Matt Aitken 7e68c02636 Don’t reload v2 run page if more than 100 tasks 2024-08-15 15:25:59 +01:00
nicktrn 01e01376f8 Fix attempt retry mechanics for new workers 2024-08-15 12:13:13 +01:00
James Ritchie db9163e775 Added a note to use batchTrigger() instead of trigger() 2024-08-14 13:25:08 +01:00
nicktrn e9bdd0c7e5 Release 3.0.0-beta.53 @trigger.dev/airtable@3.0.0-beta.53 @trigger.dev/astro@3.0.0-beta.53 @trigger.dev/core-backend@3.0.0-beta.53 @trigger.dev/core@3.0.0-beta.53 @trigger.dev/eslint-plugin@3.0.0-beta.53 @trigger.dev/express@3.0.0-beta.53 @trigger.dev/github@3.0.0-beta.53 @trigger.dev/hono@3.0.0-beta.53 @trigger.dev/integration-kit@3.0.0-beta.53 @trigger.dev/linear@3.0.0-beta.53 @trigger.dev/nestjs@3.0.0-beta.53 @trigger.dev/nextjs@3.0.0-beta.53 @trigger.dev/openai@3.0.0-beta.53 @trigger.dev/otlp-importer@3.0.0-beta.53 @trigger.dev/plain@3.0.0-beta.53 @trigger.dev/react@3.0.0-beta.53 @trigger.dev/remix@3.0.0-beta.53 @trigger.dev/sendgrid@3.0.0-beta.53 @trigger.dev/slack@3.0.0-beta.53 @trigger.dev/stripe@3.0.0-beta.53 @trigger.dev/supabase@3.0.0-beta.53 @trigger.dev/sveltekit@3.0.0-beta.53 @trigger.dev/typeform@3.0.0-beta.53 @trigger.dev/yalt@3.0.0-beta.53 trigger.dev@3.0.0-beta.53 @trigger.dev/replicate@3.0.0-beta.53 @trigger.dev/cli@3.0.0-beta.53 @trigger.dev/shopify@3.0.0-beta.53 @trigger.dev/resend@3.0.0-beta.53 @trigger.dev/sdk@3.0.0-beta.53 @trigger.dev/testing@3.0.0-beta.53 2024-08-14 11:40:38 +01:00
github-actions[bot] 7cb977ff2c chore: Update version for release (beta) (#1256)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2024-08-14 11:39:44 +01:00
James Ritchie 2940b07430 Added a description panel to the Concurrency admin page 2024-08-14 11:33:15 +01:00
James Ritchie c46e3c97b5 Fix for schedule page not scrolling 2024-08-14 11:23:16 +01:00
nicktrn 16cb1c46ac Fix restored container names 2024-08-13 18:10:09 +01:00
Matt Aitken 2b0d68a0b6 Include the old message data when replacing, so we get the projectId etc. 2024-08-13 17:09:29 +01:00
James Ritchie 2088fa392d Added depenenciesToBundle guide to bundle all packages 2024-08-13 17:06:42 +01:00
Matt Aitken f1ffc2e6c8 Added messageReplaced to concurrency tracking (when freezing) 2024-08-13 16:36:39 +01:00
Matt Aitken c0b9394099 Call subscriber.messageDequeued in dequeueMessageInSharedQueue 2024-08-13 15:31:25 +01:00
Matt Aitken 937b61913d Added more logging to TaskRunConcurrencyTracker and some more try/catches 2024-08-13 14:54:50 +01:00
nicktrn 5cf90da722 v3: fix unfreezable state crashes for runs with multiple waits (#1253)
* support named capture groups

* write crash errors to attempt.error

* make restored pod names unique per checkpoint

* use last eight characters of checkpoint id instead

* add more chaos monkey env vars

* Ignore unfreezable states

* prevent excessive queue config parsing errors

* handle dependency resume edge case

* better entry point logging

* ignore checkpoint cancellation timeouts

* add missing idempotency keys to wait for dep replays

* remove checkpoints between attempts

* fix retry container names on kubernetes

* add changeset

* fix types

* bring back internal duration timers
2024-08-13 11:44:40 +01:00
Matt Aitken da6ce3c8d5 Concurrency page and more accurate tracking (#1252)
* Initial TaskRunConcurrencyTracker implementation

* MARQS calls a subscriber to events

* When enqueuing add the extra required metadata

* Track concurrency per environment for tasks too

* Admin page for global concurrency

* Use the new concurrency tracker on the tasks page

* Useful performance test task

* getAllTaskIdentifiers()

* New page for concurrency

* BackgroundWorkerTask index for quick lookup of task identifiers

* Added a way to get concurrency for environments

* Added upgrade/request more concurrency button

* Queued task column working

* Use defer and suspense

* Added queue column to the concurrency environments table

* Some comments added for clarity

* Fixed bad log message

* Sidemenu: move lower and rename to “Concurrency limits”

* Only show the environments, not tasks. Renamed to “Concurrency limits”
2024-08-13 11:43:46 +01:00
James Ritchie 07054f13d1 Fixed v2 usage meter displaying when on paid plan (#1255)
* Fixed v2 usage meter displaying when on paid plan

* Show the free usage panel only for v3 projects
2024-08-13 11:14:28 +01:00
nicktrn c50a94b962 Increase indexing memory limit 2024-08-12 17:05:25 +01:00
Matt Aitken bb38261947 FinalizeRunService (#1250)
* WIP notes on each location where we’ll use finalize

* Initial FinalizeTaskRunService

* ExpireEnqueuedRunService uses FinalizeTaskRunService

* FailedTaskRunService uses FinalizeTaskRunService

* Allow passing in an include when finalizing the run

* CrashTaskRunService using FinalizeTaskRunService

* Remove comments

* Status is optional

* CancelAttemptService using FinalizeTaskRunService

* Import tidy

* CancelTaskRunService using FinalizeTaskRunService

* Import tidying

* CompleteAttemptService system failure switched to FinalizeTaskRunService

* Added more logging to Finalizing

* CompleteAttemptStatus COMPLETED_SUCCESSFULLY

* CompletedAttempt “SYSTEM_FAILURE”

* CompletedService final pair

* Use satisfies so we can derive types from the groups

* Only allow final states to be used with this service

* BaseService tx support, minor improvements
2024-08-08 15:32:06 +01:00
James Ritchie 0d4e3e70c1 Lots of small docs improvements (#1251)
* tasks no longer inside a group in the side menu (and added “cron”)

* Delay using a timezone

* Added React Not Defined error to the troubleshooting page

* Improved the React common problem

* Link to v2 docs

* New Development section and entry in Common Problems

* Concurrently running the terminal

* Fixed the .env weirdness

* Added section on creating PATs for Github actions

* Improved the Machine spec and limits page

* Quick start steps now have nice images

* Added a diagram for the lifecycle functions

* Added note about onFailure

* CRON -> cron/Cron

* Updated the cli-dev steps for the concurrently package

* Fixed capital letter

* Updated diagram text

* Removed dead page
2024-08-08 15:03:05 +01:00
nicktrn bf1482fdb9 Save crash errors on the attempt 2024-08-08 11:42:46 +01:00
Matt Aitken e2d3358271 Added error from the run attempt with nice display (#1249) 2024-08-07 15:50:45 +01:00
James Ritchie 6270abaab8 Remove v2 docs pages (#1246)
* Make the tooltip text color grey so it’s readable again the primary color

* Setup and styling for the Guides section

* Better Guides icon

* Improved the prerequisites

* WIP Nextjs guide and new folder structures

* Copy updates

* Useful next steps is now a snippet

* Better icons for the next steps section

* Renamed the “prerequisites” snippet

* Node.js guide

* Added a “Creating a project” guide

* New snippet for prerequisites

* New Remix guide

* Added Remix to the side bar

* Tweaked icons for creating a project page

* Added the hello world step to the onboarding steps

* Better “useful next steps” snippet card links

* Moved prerequisites

* Fixing links to images

* Moved v2 migration page to guides

* Removed dead link

* Attempt fix for redirect

* Fixed redirects

* Getting started section includes link to roadmap

* Removed icon from side menu

* Deleted all v2 pages (excluding updating mint.json)

* Removed v2 pages, redirects and versions from mint.json

* Deleted v2 snippets

* Deleted un-used pages

* Set of more useful coming soon snippets

* All snippets use the updated format

* updated folder “v3/“ with “/pages”

* Moved all main docs files to the route and updated the redirect

* Fixed URLs in the mdx pages to the new route path

* URL goes to the proper pricing page

* Better delayed runs image

* Attempt fix for self hosting page not redirecting
2024-08-07 11:13:45 +01:00
Matt Aitken 6c71ddbe67 TaskRun completedAt index 2024-08-06 17:31:12 +01:00
James Ritchie fafda21e3b V2 end of life messaging, paid plan badge, and project menu improvements (#1245)
* Updated the v2 project message to include end of life

* Removed the v3 badge. Showing a v2 badge instead

* Removed the v3 badge variant

* Show a badge in the side menu if you’re paying

* Popover menu section header and button state colours now reflects the design UI

* Added a link to the blog in the end-of-life message

* Removed missing variant from badge
2024-08-06 10:50:23 +01:00