Commit Graph

289 Commits

Author SHA1 Message Date
Eric Allam 23b43be952 feat/realtime-streams (#1470)
* WIP realtime streams

* Handle realtime with large payloads or outputs #1451

* feat: optimize Redis stream handling with batching

Add STREAM_ORIGIN to environment schema. Improve performance in
RealtimeStreams by using TextDecoderStream for simpler text
decoding and implementing batching of XADD commands for Redis
streams. Limit stream size using MAXLEN option. Update
environment variable repository with new variable type. Adjust
import statements for Redis key and value types.

* 🔧 chore: add dev dependencies for bundle analysis

* add metadata tests and a few more utilties

* Add stream tests and improve streaming

* Added AI tool tasks, descriptions to tasks

* Use the config file path to determine the workingDir, then the package.json path

* Remove stream test files

* useTaskTrigger react hook that allows triggering a task from the client

* Add streaming support for the realtime react hooks

* Add ability to stream results after useTaskTrigger

* Improve the stream throttling

* Use the runId as the ID key to bust the cache after triggering

* Upgrade to to the latest electric sql client and server

* Make realtime server backwards compat with 3.1.2 release

* Pass the runId into useRealtimeRun

* Fix scopes when specifiying reading all runs

* WIP @trigger.dev/rsc package

* Various fixes and accepted recommendations by CodeRabbit

* Regenerate pnpm lock file

* A couple tweaks to rsc and give up on rendering react in tasks for now

* Add changeset

* Remove triggerRequest from the useEffect deps

* Improve realtime & frontend authentication errors

* Fixed authorization tests

* Remove unnecessary log

* Add metadata.stream limits and improve the metadata streams structure

* Streams can now have up to 2500 entries

* Various coderabbit fixes

* additional react-hooks jsdocs
2024-11-19 13:14:25 +00:00
Eric Allam 2d8a41b18b feat: realtime (#1402)
* Denormalize run tags, increase character limit to 128

* WIP realtime subscribing to runs

* extracted the stream stuff into core, made it more reusable

* WIP tags

* Remove tags for now because it’s not support in electric

* Support async iterables, readable stream, and callback style subscription styles

* Remove tags streaming endpoint

* Add realtime rate limits and scope them to the /realtime path

* WIP rate limt per org

* Introduce per org rate limits

* WIP JWT auth

* Move migrations into new internal db package

* Resolve pnpm lock file

* Authenticating to the realtime API with JWTs are working

* realtime in the client

* Created react-hooks package and starting to move stuff in there

* Improve types for hooks

* schema tasks

* Added useBatch hook

* build uploadthing/fal demo and change how run metadata is synced to the server

* tweaks

* WIL realtime concurrency tracking

* Implement test for realtime client using testcontainers

also updated electric to latest version

* Allow customizing the expiration time of the automatic JWT created after triggering a task

* Add support for subscribing to run tags

* Improve auth types and API

* finalize the realtime API

* Fixed some example stuff

* Allow up to 10 run tags

* Remove core from docker-provider tsconfig paths to prevent it from being typechecked

* do the same for the kubernetes provider

* Fixing some typecheck errors

* Fix webapp type errors

* Update @trigger.dev/platform to 1.0.13

* Fix attw error

* Remove from/to in subscribeToRuns query params

* Add tests for the rate limit middleware and add custom JWT rate limits

* turn off webapp test parallelism

* Finish renaming jwt -> publicAccessToken and automatically give the JWT read access to the tags when using trigger

* Add changeset

* Attempt to fix unit tests in CI

* Skip running the auth rate limit middleware tests for now

* Try a beefier machine

* Try and run webapp tests separately

* Setup env vars

* Make sliding window test more reliabile
2024-10-21 15:07:08 +01:00
Eric Allam 6d0884254e feat: Add maxDuration to tasks (#1377)
* WIP

* Get max duration working on deployed runs

* Actually set the timed out runs to status = TIMED_OUT

* The client status for TIMED_OUT is now MAX_DURATION_EXCEEDED

* New TimedOutIcon

* Added new timedout icon

* Add ability to opt-out of maxDuration with timeout.None

* MAX_DURATION_EXCEEDED -> TIMED_OUT

* changeset

* Improved styling for the status tooltip content

---------

Co-authored-by: James Ritchie <james@trigger.dev>
2024-10-03 12:43:26 -07:00
James Ritchie 4906fe9cf0 Run page links to tasks & replay run modal improvements (#1364)
* Removed the inline-code accessory from the logs when calling trigger or batchTrigger from a run

* Removed re2 from the v3 catalog

* Added a Root badge to the runs list

* Keep the side panel open when switching tasks & remove links from detail panel

* Root and parent task styling inspector

* Hide the root badge if the task isn’t the root

* margin between the dev not running message

* improved spacing of items

* Improved Root badge style

* Show a table of triggered runs in the inspector

* Add parentSpanId index to the TaskRun table

* Fix for the run inspector now opening when linked from another run/replay

* Triggered runs table has a max height

* Added a description to the replay run modal and improved the styling slightly

* Only include a bottom border when the triggered run table is more than 4 items

* Improved the triggered runs table borders

* Improved the tables so they can have an optional sticky header

* Added table types to storybook

* Fix for hover states on different backgrounds & runs table select cell
2024-09-27 12:14:44 +01:00
Eric Allam 3c492bc202 feat: Run metadata (#1357)
* Run metadata

* Remove metadata from context, move it to it’s own tab

* More run metadata stuff

- Add metadata to testing
- Make using metadata outside of runs a no-op
- Add docs

* Replaying should copy over the metadata

* transfer final attempt output to the task run

* A couple of minor fixes

* Use the new clientOrThrow() method everywhere

* Cleaned up the update metadata endpoint and added an API doc page for it

* Mirror task run attempt errors and output
2024-09-26 09:23:38 +01:00
Eric Allam ba3c5bdf33 Adding missing task run hierarchy to TaskRun table (#1332)
* Add task run hierarchical relationships to the database

* Add depth and related runs to the retrieve run API response

* Remove prisma optimize

* restructure the migrations to create the index concurrently

* Delete these tsbuildinfo files

* Fix type error by adding depth to the run list presenter

* Cleanup the task hierarchy, share more code

* Remove some fields from the list run response
2024-09-20 11:09:00 +01:00
Eric Allam 4adc773c77 Auto-resolve payload/output presigned urls when retrieving a run with runs.retrieve (#1317) 2024-09-18 15:20:02 +01:00
Eric Allam cf13fbdf32 Updating docs for the move from beta -> latest (with new build system) (#1299)
* Add triggerAndWait().unwrap() to more easily get at the output or throw the subtask error

* Add taskId and runId to SubtaskUnwrapError

* WIP docs update beta -> latest

* trigger.dev init now adds @trigger.dev/build to devDependencies

* How it works doc

* Restructure some docs and update the cli commands

* Config file docs, plus aptGet and ffmpeg extensions

* Update to latest from beta docs

* Add runtime to templates

* Add --runtime option to the init CLI command

* A bunch more doc updates after feedback

* Document triggerAndWait with unwrap and result types

* beta -> latest in the webapp

* CLI update check no longer references beta

* Add major release

* Leave changeset beta, back to normal package release

* Fixed default dirs option in init command

* exclude windows-yarn variation of cli e2e tests because it’s buggy

* Remove cache to try and fix yarn e2e test workflow errors
2024-09-16 15:30:32 +01:00
Eric Allam 8578c9b281 v3: new build system fixes round 2 (#1283)
* Fixed empty env vars overriding in dev runs

* Don’t import package.json anymore

* fix node10 moduleResolution in @trigger.dev/core

* Support self-hosters pushing to a custom registry when running deploy

* dev: Fixed stuck runs when a child run fails with a process exit

* Make some doc notes about known issues and docker hub private repos

* Fix --project-ref when running deploy

* Fix —config option when deploying

* Fixing the flushing/killing process with the new build system

* Add monorepo-react-email e2e test fixture

* Fix issue with emitDecoratorMetadata and tsconfigs with extends

* Got the emit decorator metadata fixture working

* Fixed typechecking yarn e2e CLI tests in monorepos

* Add remote forced externals system, in case we come across another package that cannot be bundled (spurred on by header-generator)

* Remote externals now powered by JSON Hero to be easier to update

* resolve config source files

* Add a —javascript option to init, defaults to typescript

* Add support for prisma typed sql

* Remove msw and retry.interceptFetch

* Add missing code to the openai retries example

* Don’t generate the v3 catalog prisma client during CI

* Fixed v3-catalog task imports

* Remove interceptor usage in task file

* Only import import-in-the-middle hook if there are instrumented packages

* Fix yarn.lock file
2024-09-11 15:48:20 +01:00
Eric Allam f9ec66c562 v3: new build system (#1265)
* upgrade @opentelemetry packages to the latest versions

* remove v2 only packages, will be moved to a dedicated repo

* remove more v2 code and run pnpm install

* use the npm yalt package in the webapp

* convert @trigger.dev/core to tshy

* Switch from jest to vitest in @trigger.dev/core

* Fixed core test

* move core-backend code into core subpath export

* convert @trigger.dev/sdk to tshy

* Removed hono

* move core-apps to core/v3/apps, remove core-apps, start converting cli-v3

* Fix up some of the commands

* cli now building and loadable

* using package-json-from-dist to get package version now in core and cli

* dev command WIP

* cleaned up some repetition and structure of the entry point stuff

* bringing back the background worker stuff

* Indexing of the v3 catalog

* getting closer to executing dev runs...

* centralize dev logging using event emitter

* Move indexing to it’s own entry point, simplify code

* dev runs working

* Get instrumentation to work with openai

* debugging achieved internally

* provide worker files as part of the worker creation on the server

* support for cjs and esm javascript

* Fixed timeout

* worker manifest now has the config path

* auto-upgrade config to non-deprecated alternatives

* Adding package preview release

* deployment WIP

* improve the syncEnvVars output and adapt resolveEnvVars

* WIP bun runtime

* WIP bun support

* seed tasks with the machine preset if listed in the config

* deploy run executions WIP, extracted TaskRunProcess into 1 place

* deployed tasks running and executing 🎉

* support for waits and better flushing & process cleanup

* Fixed the heartbeating

* Better warning messages

* Improve and unify the indexing between dev and deploy

* Support for external deps that need node-gyp to build

* build extensions can now install custom packages and run instructions in the image. Also prisma extension now works and also works with multiple schema files

* Add back in the main/types/module to sdk

* dev no longer is Ink/React, grace period for disconnections in dev

* Fix the changeset config

* More changeset fixes

* Remove config packages

* More changeset fixes

* Fixed typescript issues (needed to revert back to zod 3.22.3

* Fix pr_checks workflow

* Remove the prepare script

* Fixed tests and package versions

* Remove cli test script

* Remove packages from tailwind watch paths

* Add repo to public packages

* Just commit the generated files and do the building at dev time

* Try and get pkg.pr.new working

* Try again

* Fix emitDecoratorMetadata importing named export from typescript

* config file backwards compat with export const config

* Fixed issue where import errors weren’t coming through

* p-retry is a prod dep

* typescript needs to be a prod dependency for emitDecoratorMetadata

* Add better debug logging to help track down import-in-the-middle bug

* An external is only considered resolvable if it resolves to the same path as the collected external

* Fix runtime checks to allow >=18.20

* Move extensions to a new build package

* Fixed building packages in dockerfile

* Remove the e2e test from publish workflow for now

* Don’t treat pkg.pr.new versions has needing upgrading

* making sure config handleError works, and discovered path aliases don’t work in config files

* Strip empty string env vars so they accidentally override real values

* Couple of things

* Update version to use preview instead of beta

* Hopefully fix re-attempts with >30s delay

* Match socket emit messages to current latest in main

* Initial guide

* Go back to beta

* Go back to the preview, and update guide to use pr preview tags

* Go back to beta

---------

Co-authored-by: Matt Aitken <matt@mattaitken.com>
2024-08-23 13:10:15 +01:00
nicktrn 8cae1d0875 v3: fix triggering tasks with custom queues (#1242)
* add missing dotenv requires in catalog scripts

* pass task queue options to all task trigger functions

* remove unnecessary include

* fallback to background worker task queue options

* add changeset
2024-08-01 11:26:34 +01:00
Matt Aitken 994ea7c3c8 Update the docs to reflect the new tags limit of 5 (up from 3) 2024-07-31 17:56:45 +01:00
Eric Allam 979bee50d0 Fix return type of runs.retrieve, and allow passing the type of the task to runs.retrieve 2024-07-23 23:28:14 +01:00
Eric Allam 086a0f95c5 Extract common trigger code into internal functions and add a tasks.batchTriggerAndWait function 2024-07-23 22:48:59 +01:00
Matt Aitken 55264657d7 Run tags (#1232)
* TaskRunTag migration. Removed unused TaskTag

* Show tags for a run in the run list

* API endpoint for searching tags

* Let’s not expose an API endpoint for tags at the moment

* Filter by tags working from the URL

* Tag async filter working

* We don’t need a “None” option because it’s multi-select

* When triggering a run you can add tags

* SDK runs.retrieve and runs.list with tag support

* Tidied imports

* Changed the run list query so we show all the tags even if a run only matches one of them

* Fix for dealing with weird characters in tags

* Run tags changeset

* Creating a project has a proper loading state (and blocks multiple)

* Improved the error message for tag length

* Added a tags icon for display on the run screen

* Convenient functions for creating and getting run tags

* tags.set() from inside the run function

* Replay a run passes tags through

* Less ridiculous tags for the catalog example

* Allow passing just a string for the tags

* Order by id because there’s an index on the primary key already

* Trim the tags earlier so we don’t accidentally error if a blank string is passed

* Renamed some thing from setTags to addTags

* Use findFirst for the tags project lookup

* Added an index for "TaskRunTag"("name", "id")

It massively improves the performance of run filtering based on tags

* Use `array_agg` for the run list tags so pagination works and we get a single result for each run

* Tidied imports

* More comprehensive test of tags with all triggering functions

* Support tags with tasks.trigger, tasks.batchTrigger and tasks poll variants

* Added tasks.batchTrigger tags

* Sort the tags in the UI so they’re always in the same order

* Added tooltips in the run table for delay, ttl and tags

* Added costInCents, baseCostInCents and durationMs to runs.retrieve and runs.list

* Added support for displaying a split tag if you use key_value or key:value format

* Fix code comment

* Tweaked the JSDoc to make the prefixing clearer

* Added tasks.triggerAndWait to the tags test task
2024-07-23 16:48:09 +01:00
Matt Aitken 8ba9987944 Declarative schedules (#1226)
* Added type (STATIC or DYNAMIC) to TaskSchedule. Defaults to dynamic

* WIP with dev indexing of static schedules

* Added a code comment

* First stab at deleting unused static schedules

* Dashboard changes for the static schedules

* Generate the description. Upsert the instances when editing. Fix for the friendlyId

* Don’t allow deleting of static schedules

* Don’t allow enabling/disabling of static schedules

* Added filtering for schedule types

* Syncing of schedule for deployed tasks

* Static schedules are now created for each environment

* Added a second static schedule for testing

* Add the type to the schedule task run payload and the object you get back from the SDK

* Changed static/dynamic to declarative/imperative

* Timezone example

* Changeset

* Updated scheduled docs to include declarative

* When you test a schedule it set the type to “IMPERATIVE”

* Improved the tooltip

* Fix for queue time continuing to rise when a run is canceled/expired etc

* Update the info panel on a selected declarative schedule

* Check if there are no instances. This should never happen but log an error if it does

* Throw errors and push them through to the CLI dev command

* Fail deployments if creating the background tasks or schedules fails

* Format the deployment error so it gets displayed

* Changed the maxed out schedules error message to remove bit about support
2024-07-18 20:24:54 +01:00
Eric Allam ecf1110ab7 v3: Export AbortTaskRunError from @trigger.dev/sdk/v3 2024-07-09 14:00:44 +01:00
Eric Allam 46789292b5 Add back in the run link when calling trigger 2024-07-08 17:59:48 +01:00
Eric Allam 305a6cebd4 Add tasks.triggerAndWait SDK function 2024-07-08 11:48:33 +01:00
Eric Allam 4f95c9de4e v3: Cancel awaited subtasks and reliable rate-limit recovery (#1200)
* v3: cancel subtasks when parent task runs are cancelled

* v3: recover from server rate limiting errors in a more reliable way

- Changing from sliding window to token bucket in the API rate limiter, to help smooth out traffic
- Adding spans to the API Client core & SDK functions
- Added waiting spans when retrying in the API Client
- Retrying in the API Client now respects the x-ratelimit-reset
- Retrying ApiError’s in tasks now respects the x-ratelimit-reset
- Added AbortTaskRunError that when thrown will stop retries
- Added idempotency keys SDK functions and automatically injecting the run ID when inside a task
- Added the ability to configure ApiRequestOptions (retries only for now) globally and on specific calls
- Implement the maxAttempts TaskRunOption (it wasn’t doing anything before)

* Adding some docs about the request options

* Fix type error

* Remove context propagation through graphile jobs

* Remove logger

* only select a subset of task run columns

* limit columns selected in batchTrigger as well

* added idempotency doc

* allow scoped idempotency keys, and fixed an issue with the unique index on BatchTaskRun and TaskRun

* Removed old cancel task run children code
2024-07-05 10:30:52 +01:00
Eric Allam 0e77e7ef7d v3: Delayed runs and run ttl expiration (#1193)
* v3: Trigger delayed runs and reschedule them

* Create a `@trigger.dev/core/v3/schemas` export

* fixed the `@trigger.dev/core/v3/schemas` export

* Small docs tweak

* Add ttl option when triggering tasks, expire runs after ttl

Dev runs expire in 10m by default
2024-07-01 12:01:35 +01:00
Eric Allam ecef199660 v3: Use global setTimeout to ensure cross-runtime support 2024-06-24 19:49:22 +01:00
Eric Allam 7c36a1a4b0 v3: Adding SDK functions for triggering tasks in a typesafe way (#1177)
* v3: Adding SDK functions for triggering tasks in a typesafe way, without importing task file

* Add type usages
2024-06-24 12:37:34 +01:00
Eric Allam 85a543d8ec v3: usage tracking (#1163)
* Starting to measure wall time and cpu time in the workers, and reporting that via otel and to completed task run attempts

* Move usage tracking outside of the executor

* WIP prod usage tracking

* WIP

* WIP custom fetch to openmeter

* Create a usage client

* WIP

* WIP

* Implement new machine preset stuff and send usage reports to OpenMeter from webapp

* WIP

* Expose usage info to the client

* Add usage and cost to TaskEvent

* Add ability to globally configure the task machine preset

* Report start run usage

* Change the machine docs to use presets

* setExpirationTime to 24h

* Removed logs

* Update machines.mdx

* Removed console.logs

* Handle revalidating JWT tokens

* Couple tweaks

---------

Co-authored-by: Matt Aitken <matt@mattaitken.com>
2024-06-18 09:40:23 +01:00
Matt Aitken c405ae7117 Schedule limits and timezone support (#1165)
* Added maximumScheduleInstancesLimit column to Org, default to 20

* Docs on the schedule limits and improved soft-limit communication

* Added limit info to the schedules list page

* Created a task that creates schedules, useful for testing

* Make deduplicationKey required when creating/updating a schedule using the SDK

* New schedule button shows an alert if you’re over the limit

* Added timezone to the form and db

* WIP on the timezone dropdown for the create/edit schedule form

* Use the new filter search for timezones

* Made the timezone dropdown faster by fixing the virtualization

* The preview table is working and added a nice message about daylight savings

* Created a page where you can view the full list of timezones

The URL is included in the error message if you send an invalid time using the SDK

* Creating tasks with the timezone

* Added timezone support the the scheduler and the schedules list

* Added timezone support to more of the schedules UI

* The timezone comes through to scheduled runs with nice JSDocs

* Allow setting the timezone from the SDK

* Always have a timezone on a schedule

* Updated jsdocs

* Updated catalog example

* Changed the column to be a string, not null. Added the timezone across the SDK

* API endpoint for getting the timezones

* Added an SDK function to get the list of timezones

* Added timezones to the docs

* Changeset: Added timezone support to schedules

* Added support for testing timezone

* Tidied up imports

* Imports

* Imports

* Update limits.mdx

* Fixed a couple type issues and use the already exported zodfetch

---------

Co-authored-by: Eric Allam <eallam@icloud.com>
2024-06-14 13:31:27 +01:00
Matt Aitken 51bb4c887a Support custom queue when triggering a task (#1138)
* Created a v3-catalog test script for queues

* SDK: Fix for calling trigger and passing a custom queue

* Support custom queue in TriggerTaskService

* Improved the script in the catalog so it’s clearer what’s going on

* Remove the concurrencyLimit from a queue if the limit is null

* Fix for the test code… stupid
2024-05-31 11:24:13 +01:00
Eric Allam ba71f959e2 v3: Runs List Management SDK and SDK improvements (#1133)
* Improved the existing runs API

* WIP next runs API

* Improve the returned ApiPromise to add ability to return response

* More WIP

* WI{

* Added offset/limit pagination stuff like the cursor one, and converted all API methods to use ApiPromise

* More run API stuff

- Adding schedule  output from the retrieveRun endpoint
- Ability to filter by schedule and isTest

* Remove env from retrieve run in openAPI

* prefer duplication over merge

* WIP docs

* Use spread to DRY up some run API schemas

* Finish the overview docs

* Adding changeset

* Fixed typecheck errors

* Typo fix

* Re-export zodfetch from core so the v3 CLI can use it

* Fixed type errors

---------

Co-authored-by: Matt Aitken <matt@mattaitken.com>
2024-05-31 11:02:28 +01:00
Eric Allam 3a1b0c486a v3: env var management API (#1116)
* WIP env var management API

* Add import env var API endpoint

* Adding docs and support for using both API keys and PATs when interacting with the env var endpoints

* WIP envvar SDK

* Uploading env vars in a variety of formats now works

* Finish env var endpoints and add resolveEnvVars hook

* Add changeset
2024-05-23 16:11:25 +01:00
Matt Aitken 1281d40e4b v2: When a run hits the rate limit reschedule the re-execution (#1125)
* Fix: API rate limit error has the correct seconds until reset

* When a v2 run hits the rate limit, reschedule using the reset timestamp

* Still throw AutoYieldRateLimitErrors

* Reschedule runs from the rate limit

* The stress test timeout should be inside the task

* If the rate limit error is thrown, don’t retry the API request
2024-05-23 13:26:03 +01:00
Eric Allam 3f8b6d8fce v2: Better handle recovering from platform communication errors by auto-yielding back to the platform in case of temporary API failures 2024-05-21 14:07:16 +01:00
Eric Allam 6d9dfbc75d Add configure function to be able to configure the SDK manually
And move the ApiClient configuration away from AsyncLocalStorage and use our globals system instead
2024-05-06 12:12:30 +01:00
Eric Allam 203e002087 Add runs.retrieve management API method to get info about a run by run ID 2024-05-04 22:24:42 +01:00
Eric Allam 9491a1649c v3: Support decorators (#1060)
* Adds support for `emitDecoratorMetadata: true` and `experimentalDecorators: true` in your tsconfig

* Implement task.onSuccess/onFailure and config.onSuccess/onFailure

* Added onStart and more docs for lifecycle functions

* Use onStart instead of init for TypeORM
2024-04-25 11:17:13 +01:00
Eric Allam b68012f81c v3: Various fixes for Next.js projects and projects that use v2 and v3 together (#1051)
* Fixes an issue that was treating v2 trigger directories as v3

* Make msw a normal dependency (for now) to fix Module Not Found error in Next.js.

* Extract out all the zod* stuff from core so the SDK does not import it

* Add a changeset

* Fixing typecheck errors in the webapp

* Export the Task and TaskOptions types

* Extract additional exports from core/v3 that aren’t used in the SDK

* Move to our global system from AsyncLocalStorage for the current task context storage

* Update the esbuild core bundling plugin for the new core v3 exports

* Fix v3 CLI telemetry

* Add support for tasks located in subdirectories inside trigger dirs

* Remove the env var check during deploy (too many false negatives)
2024-04-24 10:09:36 +01:00
Eric Allam 374edef020 Updates the trigger, batchTrigger and their *AndWait variants to use the first parameter for the payload/items, and the second parameter for options (#1045)
Also always returns a `TaskRunResult` object from `triggerAndWait` instead of rethrowing subtask errors in the parent
2024-04-19 14:51:51 +01:00
Eric Allam 26093896d2 v3: Fixes for using (batch)triggerAndWait with idempotency keys (#1043)
* Fixes various issues with triggerAndWait and batchTriggerAndWait

When using idempotency keys, triggerAndWait and batchTriggerAndWait will still work even if the existing runs have already been completed (or even partially completed, in the case of batchTriggerAndWait)

- TaskRunExecutionResult.id is now the run friendlyId, not the attempt friendlyId
- A single TaskRun can now have many batchItems, in the case of batchTriggerAndWait while using idempotency keys
- A run’s idempotencyKey is now added to the ctx as well as the TaskEvent and displayed in the span view
- When resolving batchTriggerAndWait, the runtimes no longer reject promises, leading to an error in the parent task

* Remove the default queue concurrency limit as we now have env and org concurrency limits

* Use the run friendlyId in the completion result id

* Added some error logging
2024-04-19 10:54:43 +01:00
nicktrn c9e1a3e9c5 v3: fix dependency trigger and wait (#1030)
* fix dev dependecy resumes

* add catalog entry with multiple trigger waits

* update docker provider lifecycle command

* switch to x for clearing run filters

* Revert "fix dev dependecy resumes"

This reverts commit b4061f2ae71f21889adcf061928e2e77c436f0ed.

* fix dependency resumes, properly this time

* add catalog entry for dependency waits in loops

* advice in docs re parallel dependency waits

* fix link from v3 to v2 docs

* move lifecycle command logging to debug only

* Removed batchOptions from the trigger options and the docs

---------

Co-authored-by: Matt Aitken <matt@mattaitken.com>
2024-04-18 14:48:43 +01:00
Eric Allam 44e1b87547 v3: various schedule fixes (#1040)
* Improve the SDK function types and expose a new APIError instead of the APIResult type

* Skip triggering scheduled tasks if the task isn’t found in the current deployment

Also fixes an issue when editing the environments of a schedule
2024-04-17 15:10:23 +01:00
Matt Aitken 4986bfda2e Scheduled tasks (#1036)
* Database schema and migrations for schedules

* Added schedules to the side menu

* The pagination can optionally hide the page numbers for a compactive mode

* Filters for the schedule page

* Added triggerSource (“STANDARD”, “SCHEDULED”) to BackgroundWorkerTask

* Added the ability to disabled a LinkButton

* Started work on the schedule page

* Environment buttons

* The new schedule form styles

* Added a cxouple of extra fields

* Allow a checkbox to have a rich label

* Added cronstrue package to the webapp

* WIP creating tasks using the form

* Improved the form styling

* Creating schedules is working in the UI

* Minor improvements

* Basic schedule table is displaying

* Creating tasks with triggerSource = scheduled. Refactored how task metadata is stored and accessed to be cleaner

* Resource route

* Fixes in the form

* Added a gap between the environments

* WIP on OpenAI generating CRON expressions

* AI generated CRON expressions is working

* Fix for the CRON field being uneditable after an AI generation

* Improvements

* Table padding

* useThrottle now behaves correctly

* Added filtering to the schedules list

* Improved the layout and fixed CRON search

* Fixed pagination for the schedules list. Just use a regular Prisma query

* Page size of 20

* Added links to the schedule rows

* Get rid of the Last run column for now

* Implement triggered scheduled tasks

Also implemented superjson payloads and dev runtime environment “presence” with RuntimeEnvironmentSession

* Move CronPattern and CreateSchedule into a common client-accessible file

* Latest UI changes

* Fix for creating a task schedule with a blank dedup key

* Refactor the human to cron stuff into a separate file and use json_object OpenAI response format

* Fix for trying to use a hook on the server-side…

* A couple of fixes to the new schedule form

* WIP on viewing a scheduled run

* Make the filters all optional

* Use the RunListPresenter from the schedule presenter

* Display a table of runs… the wrong runs but still

* Runs from the schedule

* Deleting schedules from the UI

* Tidied imports and fixed name of options object

* Disabling a schedule

* Editing schedules

* Tidied imports

* Added icons to the task list, needs some design love

* Added a tooltip for CRON pattersn

* Show the last run in the schedules table

* Some tweaks

* Added a placeholder to the CRON AI field

* Improved the trigger source icon

* Dim out disabled schedules

* Scheduled tasks have the correct icon in a run

* Added the task source icon to the test task list

* Added the date field component to storybook

* Style improvements to the date field

* Implement Task Schedule API

* added a medium sized variant to the date field

* Fixed replay run for superjson payload types

Also now linking from the replaying run to the original run (using span links)

Also added a project metrics prometheus endpoint to detect the state of the queues

* WIP on allowing different forms for testing

* If you pass a string to prettyPrintPacket which is json or superjson, safeParse it first

* Test page, deal json and superjson. Started splitting UI for schedules

* WIP on schedule form

* Removed the Label from the DateField

* WIP on test schedule form

* Fix for the runs page showing the wrong message when there are no runs from time filtering

* Removed labels from the DateField

* Fixes for the form

* Test runs are working for schedules

* Fix for nextScheduledTimestamps in triggerScheduledTask

* Implement idempotency key support and fix issue with cancelled runs causing concurrency usage

* Added API documentation for the schedule API

* Remove log

* Removed console log from runs page

* Transform the recent runs test data on the server

* Fix for hydration mismatch

* Current date as the default for the test form

* Recent payloads working

* Delete schedule modal

* Deal with empty strings from the form

* Set the initial value for the scheduled test form

* Add option to print console logs in the dev CLI locally (issue #1014)

* Export queue from the SDK

* Fix for schedules list when you have no schedule tasks

* Blank states improved

* Make task schedules more generic, to support additional schedule generators in the future

* Removed log from maqrs

* Removed “v3/schedules” export from the SDK

---------

Co-authored-by: Eric Allam <eallam@icloud.com>
Co-authored-by: James Ritchie <james@jamesritchie.co.uk>
2024-04-17 11:14:23 +01:00
Matt Aitken f854cb90eb Replay\Cancel a run (v3) (#1006)
* WIP on replaying a task from the run page

* Don’t pass the existing runs idempotency key, it will cause the replay to always return the original run

* Replay from the run list

* Don’t use fetchers in the replay/cancel dialogs

* API endpoint for replaying a run

* REST API docs (mostly coming soon) but added replay run

* replayRun function added to the SDK

* Cancel run added to the SDK

* v3-catalog file to test canceling and replaying

* Changed the SDK to be runs.replay and runs.cancel

* Removed comment

* Latest lockfile
2024-04-08 19:08:44 +01:00
Eric Allam b271742dca v3: implement configurable log levels via config file and env var (#985)
* v3: implement configurable log levels via config file and TRIGGER_LOG_LEVEL

Also, test runs automatically set the TRIGGER_LOG_LEVEL to debug

* Fix type error and changeset

* Added a Node.js runtime check for the CLI dev command
2024-03-28 22:00:38 +00:00
Eric Allam f93eae300e v3: superjson dynamic import and deploy fixes (#982)
* v3: Dynamically import superjson and fix some bundling issues

* Added changeset

* Better error handling in the registry proxy and catch uncaught exceptions and unhandled promise rejections instead of crashing the server

* Await the prettyPrintPackage
2024-03-28 16:22:00 +00:00
nicktrn b66d5525ef v3: machine config (#978)
* add and use machine config

* assign tasks to worker nodes only

* add secure flag to zod connection

* changeset

* add pre stop hook

* don't use secure connection by default

* pass more identifiers to provider and apply labels
2024-03-28 14:21:49 +00:00
Eric Allam 719c0a0b94 v3: better clock management for spans/logs in CRIU envs (#980)
* durable clock WIP

* Fixed incorrect span timings around checkpoints by implementing a precise wall clock that resets after restores

* Add changeset
2024-03-28 14:20:44 +00:00
Matt Aitken fc473105e3 Better error messages if you haven’t set the API URL or SECRET for the v3 client 2024-03-27 14:08:13 +00:00
Eric Allam 215b60c742 v3: superjson output support (#971)
* superjson output support

* Better support for superjson in the task events, limiting output attributes

* Offload large outputs to object store (r2)

* Finishing up the offloading of large outputs/payloads to an object store
2024-03-27 11:57:22 +00:00
Eric Allam 9e02b45808 Fixed export 2024-03-25 17:24:23 +00:00
Eric Allam 395abe1b92 v3: Various fixes to support papermark use-case (#970)
* Adding a changeset for v3

* Add a version field to @trigger.dev/core-apps package.json

* Build trigger.dev when doing a prerelease

* bundle @trigger.dev/core-apps with trigger.dev cli

* Fix the init command config template

* Don’t use * for the @trigger.dev/core dep version specifier

* strip workspace: from the package version before installing it

* Added dependenciesToBundle config option to bundle ESM only packages

* Added logging around resolving dependency paths

* Try again

* Resolve dependencies based on the project dir first

* flip the bundled default

* Adding some logs around dev task completion notifications

* Adding some additional logs

* Add more logs

* Write out the log using process.stdout

* Store pending completion notifications and resume them when awaited (fixes race condition)

* Cleanup some of the logs

* Copy over the postinstall step from the projects package.json

* Add support for including additional files when deploying (e.g. prisma schema)

* Don’t run scripts when resolving deps

* copy all the files just in case anything is needed in postinstall

* Remove duplicate option

* Use the tag when outputting the dev command

* Remove the postinstall script

* add the trigger dir to the config if the default is not chosen

* Remove the “hud” display in the dev command

* trigger file names with dashes now work

* Better file watching in dev

* Much better duplicate ID experience now

* Much better “Project not found” error

* Export the handleError function types from sdk

* Add support for configuring instrumentation

* Upgrade and unify @opentelemetry/* packages (and remove storybook from the webapp)

* Fix typescript error in react package

* Upgrade react types in webapp

* Allow span icons to be determined based on the span name (e.g. prisma:)

* Ignore built-in env vars when checking for env vars, and allow continuing the deployment even if missing env vars were detected

* Improve the retry.fetch default behavior and option structure

* Fixed typescript errors with packages/email react types

* Update the retry.fetch docs
2024-03-25 17:02:49 +00:00
nicktrn 9ecf07731a v3: small features and fixes for prod infra (#968)
* add checkpoint restore events

* fix retries.enabledInDev

* fix tsconfig paths

* fix provider build and dev

* update kubernetes provider and fix builds again

* upgrade prod base to node 20

* update infra publish workflow

* rethrow k8s errors after logging

* shorten index container names

* fix error type assertion

* rename type assertion

* remove resource limits for now

* add missing run id on create

* add push to deploy command for self-hosting

* checkpointing fixes

* update coordinator image

* ensure valid registry login

* delete checkpoint archive after successful push

* log options on error

* structured logs for socket connections

* fix structured log merge

* exit process after checkpointing

* update restore pull secret name

* append shortcode to restore names

* log handler payload

* disable post start lifecycle hook

* pass in coordinator host via volume

* replace dapi with taskinfo

* add missing restore label

* don't restart restored containers

* remove init container from create

* atomic post-completion checkpoints

* switch to run id for container names

* improve wait accuracy

* measure basic checkpoint perf

* always log disconnect reason

* use system clock to end wait spans

* checkpoint readiness and cancel signals

* restore from checkpoint events and fix statuses

* remove attempt id env var

* restore dependencies from events

* reconnect wip

* lifecycle hooks are back

* fix hooks and improve reconnect

* make docker send postStart hook

* only checkpoint for retry if large delay

* fix a few more resume issues

* lifecycle hook fixes

* skip connection handler when waiting for post start hook

---------

Co-authored-by: Eric Allam <eric@trigger.dev>
2024-03-25 16:28:56 +00:00
Matt Aitken cf9d466b9e Basic JSDocs for the v3 SDK (#963)
* task(), retry and queue

* Added machine JSDocs

* Add JSdocs for the run function params
2024-03-22 14:09:06 +00:00