* magic links on span event errors
* prevent task monitor from processing errors handled elsewhere
* exclusively use internal error code enum for completion data
* add complete attempt service opts
* reattempts need to go via the queue for task controllers that may have exited
* only infer retry config if completed via crash or system failure
* enhance error before deciding if retriable
* retry on SIGTERM
* enable retry config helper for latest sdk
* don't retry heartbeat timeouts for now
* enable task monitor to update fatal errors
* add missing service
* update retry config since package version
* don't alter completion time when updating existing error
* refactor finalize run service
* refactor complete attempt service
* remove separate graceful exit handling
* refactor task status helpers
* clearly separate statuses in prisma schema
* all non-final statuses should be failable
* new import payload error code
* store default retry config if none set on task
* failed run service now respects retries
* fix merged task retry config indexing
* some errors should never be retried
* finalize run service takes care of acks now
* execution payload helper now with single object arg
* internal error code enum export
* unify failed and crashed run retries
* Prevent uncaught socket ack exceptions (#1415)
* catch all the remaining socket acks that could possibly throw
* wrap the remaining handlers in try catch
* New onboarding question (#1404)
* Updated “Twitter” to be “X (Twitter)”
* added Textarea to storybook
* Updated textarea styling to match input field
* WIP adding new text field to org creation page
* Added description to field
* Submit feedback to Plain when an org signs up
* Formatting improvement
* type improvement
* removed userId
* Moved submitting to Plain into its own file
* Change orgName with name
* use sendToPlain function for the help & feedback email form
* use name not orgName
* import cleanup
* Downgrading plan form uses sendToPlain
* Get the userId from requireUser only
* Added whitespace-pre-wrap to the message property on the run page
* use requireUserId
* Removed old Plain submit code
* Added a new Context page for the docs (#1416)
* Added a new context page with task context properties
* Removed code comments
* Added more crosslinks
* Fix updating many environment variables at once (#1413)
* Move code example to the side menu
* New docs example for creating a HN email summary
* doc: add instructions to create new reference project and run it locally (#1417)
* doc: add instructions to create new reference project and run it locally
* doc: Add instruction for running tunnel
* minor language improvement
* Fix several restore and resume bugs (#1418)
* try to correct resume messages with missing checkpoint
* prevent creating checkpoints for outdated task waits
* prevent creating checkpoints for outdated batch waits
* use heartbeats to check for and clean up any leftover containers
* lint
* improve exec logging
* improve resume attempt logs
* fix for resuming parents of canceled child runs
* separate SIGTERM from maybe OOM errors
* pretty errors can have magic dashboard links
* prevent uncancellable checkpoints
* simplify task run error code enum export
* grab the last, not the first child run
* Revert "prevent creating checkpoints for outdated batch waits"
This reverts commit f2b5c2ac42.
* Revert "grab the last, not the first child run"
This reverts commit 89ec5c8bfd.
* Revert "prevent creating checkpoints for outdated task waits"
This reverts commit 11066b4e74.
* more logs for resume message handling
* add magic error link comment
* add changeset
* chore: Update version for release (#1410)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
* Release 3.0.13
* capture ffmpeg oom errors
* respect maxAttempts=1 when failing before first attempt creation
* request worker exit on fatal errors
* fix error code merge
* add new error code to should retry
* pretty segfault errors
* pretty internal errors for attempt spans
* decrease oom false positives
* fix timeline event color for failed runs
* auto-retry packet import and export
* add sdk version check and complete event while completing attempt
* all internal errors become crashes by default
* use pretty error helpers exclusively
* error to debug log
* zodfetch fixes
* rename import payload to task input error
* fix true non-zero exit error display
* fix retry config parsing
* correctly mark crashes as crashed
* add changeset
* remove non-zero exit comment
* pretend we don't support default default retry configs yet
---------
Co-authored-by: James Ritchie <james@trigger.dev>
Co-authored-by: shubham yadav <126192924+yadavshubham01@users.noreply.github.com>
Co-authored-by: Tarun Pratap Singh <101409098+Wackyator@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
* Denormalize run tags, increase character limit to 128
* WIP realtime subscribing to runs
* extracted the stream stuff into core, made it more reusable
* WIP tags
* Remove tags for now because it’s not support in electric
* Support async iterables, readable stream, and callback style subscription styles
* Remove tags streaming endpoint
* Add realtime rate limits and scope them to the /realtime path
* WIP rate limt per org
* Introduce per org rate limits
* WIP JWT auth
* Move migrations into new internal db package
* Resolve pnpm lock file
* Authenticating to the realtime API with JWTs are working
* realtime in the client
* Created react-hooks package and starting to move stuff in there
* Improve types for hooks
* schema tasks
* Added useBatch hook
* build uploadthing/fal demo and change how run metadata is synced to the server
* tweaks
* WIL realtime concurrency tracking
* Implement test for realtime client using testcontainers
also updated electric to latest version
* Allow customizing the expiration time of the automatic JWT created after triggering a task
* Add support for subscribing to run tags
* Improve auth types and API
* finalize the realtime API
* Fixed some example stuff
* Allow up to 10 run tags
* Remove core from docker-provider tsconfig paths to prevent it from being typechecked
* do the same for the kubernetes provider
* Fixing some typecheck errors
* Fix webapp type errors
* Update @trigger.dev/platform to 1.0.13
* Fix attw error
* Remove from/to in subscribeToRuns query params
* Add tests for the rate limit middleware and add custom JWT rate limits
* turn off webapp test parallelism
* Finish renaming jwt -> publicAccessToken and automatically give the JWT read access to the tags when using trigger
* Add changeset
* Attempt to fix unit tests in CI
* Skip running the auth rate limit middleware tests for now
* Try a beefier machine
* Try and run webapp tests separately
* Setup env vars
* Make sliding window test more reliabile
* try to correct resume messages with missing checkpoint
* prevent creating checkpoints for outdated task waits
* prevent creating checkpoints for outdated batch waits
* use heartbeats to check for and clean up any leftover containers
* lint
* improve exec logging
* improve resume attempt logs
* fix for resuming parents of canceled child runs
* separate SIGTERM from maybe OOM errors
* pretty errors can have magic dashboard links
* prevent uncancellable checkpoints
* simplify task run error code enum export
* grab the last, not the first child run
* Revert "prevent creating checkpoints for outdated batch waits"
This reverts commit f2b5c2ac42.
* Revert "grab the last, not the first child run"
This reverts commit 89ec5c8bfd.
* Revert "prevent creating checkpoints for outdated task waits"
This reverts commit 11066b4e74.
* more logs for resume message handling
* add magic error link comment
* add changeset
* Updated “Twitter” to be “X (Twitter)”
* added Textarea to storybook
* Updated textarea styling to match input field
* WIP adding new text field to org creation page
* Added description to field
* Submit feedback to Plain when an org signs up
* Formatting improvement
* type improvement
* removed userId
* Moved submitting to Plain into its own file
* Change orgName with name
* use sendToPlain function for the help & feedback email form
* use name not orgName
* import cleanup
* Downgrading plan form uses sendToPlain
* Get the userId from requireUser only
* Added whitespace-pre-wrap to the message property on the run page
* use requireUserId
* Removed old Plain submit code
* Added a new dropdown help and feedback menu to the side menu
* Added a shortcut to the popover menu
* Removed dev cli connected button for now
* Contact us form uses original Feedback component to prevent broken links
* Improved the messaging when selecting different options in the email form
* buttons style tweak
* SideMenuItem supports the trailingIconClassName
* Adding a consistent focus-visible states
* Removing tooltips for now
* Squashed commit of the following:
commit 7d11123c0a
Author: Eric Goldman <eric@sequin.io>
Date: Mon Sep 30 17:54:06 2024 -0700
Add sequin guide (#1368)
Co-authored-by: James Ritchie <james@trigger.dev>
commit 8da495ac00
Author: nicktrn <55853254+nicktrn@users.noreply.github.com>
Date: Mon Sep 30 13:42:22 2024 +0100
Improve checkpoint reliability and cleanup of temp files (#1367)
* improve cleanup reliability
* improve logging
* bye-bye execa
* fix for trailing newlines
* prettier errors
* trim args and log output by default
* fix archive cleanup
* prevent potential memleak
* more cleanup debug logs
* ignore abort during cleanup
* rename checkpoint dir env var and move to helper
* add global never throw override
* add tmp cleaner
* also clean up checkpoint dir by default
* split by any whitespace, not just tabs
* only create tmp cleaner if paths to clean
commit 69ec68ee31
Author: Eric Allam <eallam@icloud.com>
Date: Sun Sep 29 19:18:39 2024 -0700
Release 3.0.9
commit a6ea8444c9
Author: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Date: Sun Sep 29 19:17:26 2024 -0700
chore: Update version for release (#1366)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
commit 4c1ee3d6ea
Author: Eric Allam <eallam@icloud.com>
Date: Sun Sep 29 19:09:38 2024 -0700
fix: run metadata not working when using npx/pnpm dlx
* More support for custom-focus
* More custom focus styles added
* Support for focus-visible style for the Segmented control
* Fixed table triple dot menu z-index issue
* Improved help menu wording
* When you submit the help form, close the modal
* focus-visible style for radio buttons
* button prop is now optional in the SideMenu component
* focus styling for a text link
* Deleted unused sequin files
* Checkbox component can have its label styles
* Improved the dialog footer
* Handle sending feedback to Slack using Plain
* WIP making the modal conditional
* Moved the Plain form action into the select plan file
* removed comment
* Show a confirmation diaglog if you’re downgrading from Pro to Hobby
* Downgrading to Hobby works
* Use redirectWithErrorMessage instead of throw error
* The cancel form now submits the data correctly
* Modals don’t trigger when you upgrade
* Copy improvements
* Added a tooltip to explain the link to the pricing page
* Squashed commit of the following:
commit 886429b0ed
Author: Matt Aitken <matt@mattaitken.com>
Date: Tue Oct 8 18:36:46 2024 +0100
Removed emails, @trigger.dev/database and @trigger.dev/otlp-importer from changesets config
commit f65157a14f
Author: Matt Aitken <matt@mattaitken.com>
Date: Tue Oct 8 18:31:27 2024 +0100
Lockfile with run-engine removed
commit 3d67bb81e9
Author: Matt Aitken <matt@mattaitken.com>
Date: Tue Oct 8 18:24:31 2024 +0100
Removed run-engine from the webapp package.json/tsconfig
commit d30e9713f4
Author: Matt Aitken <matt@mattaitken.com>
Date: Tue Oct 8 18:06:04 2024 +0100
Dockerfile fix because the database package has been moved
commit f2babbf637
Author: Matt Aitken <matt@mattaitken.com>
Date: Tue Oct 8 09:41:22 2024 -0700
Internal packages (testcontainers, redis-worker and zod-worker) (#1392)
* Some notes on the new run engine
* lockfile with setup for the run engine
* Documenting where TaskRun is currently mutated, to try figure out the shape of the new system
* Added notes about how triggering currently works
* Details about when triggering happens
* Lots of notes about waitpoints
* Started scaffolding the RunEngine
* Sketch of Prisma waitpoint schema while it’s fresh in my mind
* Got Prisma working with testcontainers
* Use beforeEach/afterEach
* Simple Prisma and Redis test
* Return Redis options instead of a client
* Simplified things
* A very simple FIFO pull-based queue to check the tests working properly
* Use vitest extend
* Separate redis, postgres and combined tests for faster testing
* Some fixes and test improvements
* Pass a logger into the queue
* A queue processor that processes items from the given queue as fast as it can
* Test for retrying an item that wasn’t processed
* First draft of waitpoints in the Prisma schema
* Remove the custom logger from the test
* Added a completedAt to Waitpoint
* Notes on the flow for an execution starting
* Added redlock, moved some files around
* Starting point for the TaskRunExecutionSnapshot table
* Added relationships to TaskRunExecutionSnapshot
* Change some tsconfig
* Moved some things around
* Added some packages
* WIP on the RunQueue
* Fix for some imports
* Key producer with some tests
* Removed the nv type from the keys… it’s not useful to do global queries
* Passing unit tests for all the public key producer functions
* Some basic tests passing for the RunQueue
* Simple enqueue test working
* Enqueue and dequeue for dev is working
* Don’t log everything during the tests
* Enqueuing/dequeuing from the shared queue is working
* Tests for getting a shared queue
* The key producer sharedQueue can now be named, to allow multiple separate queues
* The key producer uses the name of the queue as the input
* Extra info in the Prisma schema
* Dequeuing a message gets the payload and sets the task concurrency all in one Lua script
* Adding more keys so we can read the concurrency from the queue
* Setting the concurrency with dequeue and enquque is working
* Improved the tests and fixed some bugs
* Acking is resetting the concurrencies
* Check the key has been removed after acking
* Nacking is working
* Changed the package to CommonJS + Node10 so it works with Redlock
* Moved the database, otel and emails packages to be in internal-packages
* Moved some Prisma code to the database package
* Started using the RunEngine for triggering
* Progress on run engine triggering, first waitpoint code
* Create a delay waitpoint
* Moved ZodWorker to an internal package so it can be used in the run engine as well as the webapp
* Web app now uses the zod worker package
* Added parseNaturalLanguageDuration to core/apps
* internal-packages/zod-worker in the lockfile
* Pass in the master queue, remove old rebalance workers code
* Add masterQueue to TaskRun
* Fixed the tests
* Moved waitpoint code into the run engine, also the zod worker
* Completing waitpoints
* An experiment to create a new test container with environment
* More changes to triggering
* Started testing triggering
* Test for a run getting triggered and being enqueued
* Removed dequeueMessageInEnv
* Update dev queue tests to use the shared queue function
* Schema changes for TaskRunExecutionSnapshot
* First execution snapshot when the run is created. Dequeue run function added to the engine
* Separate internal package for testcontainers so they can be used elsewhere
* Remove the simple queue and testcontainers from the run-engine. They’re going to be separate
* Fix for the wrong path to the Prisma schem,a
* Added the testcontainers package to the run-engine
* redis-worker package, just a copy of the simple queue for now
* The queue now uses Lua to enqueue dequeue
* The queue now has a catalog and an invisible period after dequeuing
* Added a visibility timeout and acking, with tests
* Added more Redis connection logging, deleted todos
* Visibility timeouts are now defined on the catalog and can be overridden when enqueuing
* Dequeue multiple items at once
* Test for dequeuing multiple items
* Export some types to be used elsewhere
* Partial refactor of the processor
* First stab at a worker with concurrency and NodeWorkers
* Don’t have a default visibility timeout in the queue
* Worker setup and processing items in a simple test
* Process jobs in parallel with retrying
* Get the attempt when dequeuing
* Workers do exponential backoff
* Moved todos
* DLQ functionality
* DLQ tests
* Same cluster for all keys in the same queue
* Added DLQ tests
* Whitespace
* Redis pubsub to redrive from the worker
* Fixed database paths
* Fix for path to zod-worker
* Fixes for typecheck errors, mostly with TS versions and module resolution
* Redlock required a patch
* Moved the new DB migrations to the new database package folder
* Remove the run-engine package
* Remove the RunEngine prisma schema changes
* Delete triggerTaskV2
* Remove zodworker test script (no tests)
* Update test-containers readme
* Generate the client first
* Use a specific version of the prisma package
* Generate the prisma client before running the unit tests
commit fc609470c4
Author: Dan <8297864+D-K-P@users.noreply.github.com>
Date: Tue Oct 8 14:36:03 2024 +0100
Supabase database webhook example upgrade (#1386)
* Added overview for guides and examples section and split them all out
* New supabase guide wip
* Updated images and improved docs
* Trimmed the supabase prereqs
* Supabase guide wip
* more updates
* Replaced old database webhook guide
* Created one intro page and removed snippets
* Updated guide sidebar titles
* Code updates
* More improvements
* Updates and added images
* Compressed image
* Updated guides descriptions and edge function basic
* Removed bold
* Updated redirects
* Fixed broken links
* Updated intro
commit 07f82eae54
Author: nicktrn <55853254+nicktrn@users.noreply.github.com>
Date: Tue Oct 8 13:28:54 2024 +0100
Release 3.0.11
commit 13ebfcc5b9
Author: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Date: Tue Oct 8 13:24:38 2024 +0100
chore: Update version for release (#1381)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
commit 2a04d17e1f
Author: nicktrn <55853254+nicktrn@users.noreply.github.com>
Date: Tue Oct 8 09:24:23 2024 +0100
Simplify showLogs expression
commit 002ae4bc0d
Author: nicktrn <55853254+nicktrn@users.noreply.github.com>
Date: Tue Oct 8 09:22:29 2024 +0100
Fix dotenv overrides for dev runs (#1388)
* override dashboard dev env vars with local .env
* add changeset
* add simple task for testing env vars
commit 047cb00417
Author: nicktrn <55853254+nicktrn@users.noreply.github.com>
Date: Tue Oct 8 09:22:05 2024 +0100
Disable schedules for deleted orgs on next tick (#1383)
* disable schedules for deleted orgs
* add debug logs
commit 2c014f725a
Author: James Ritchie <james@trigger.dev>
Date: Sun Oct 6 13:02:00 2024 -0700
Override log retention (#1385)
* set full log retention as admin
* If run.logsDeletedAt is set, don’t bother getting the trace
commit a69e04f03a
Author: nicktrn <55853254+nicktrn@users.noreply.github.com>
Date: Sat Oct 5 14:18:58 2024 +0100
Include push output in logs for self-hosted deploys (#1382)
* include push output in logs
* changeset
commit c5488df8ca
Author: nicktrn <55853254+nicktrn@users.noreply.github.com>
Date: Sat Oct 5 13:12:47 2024 +0100
Fix CLI downgrade check (#1380)
* fix downgrade detection
* remove unused semver package from webapp
* add changeset
commit 1caec27f72
Author: Eric Allam <eric@trigger.dev>
Date: Fri Oct 4 15:33:35 2024 -0700
docs: Max duration (#1379)
* maxDuration docs
* Update the init command to set the maxDuration and include a commented out maxDuration in the config file
commit e14c954e12
Author: Eric Allam <eallam@icloud.com>
Date: Fri Oct 4 15:02:05 2024 -0700
Release 3.0.10
commit 8e61f5d631
Author: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Date: Fri Oct 4 14:59:07 2024 -0700
chore: Update version for release (#1378)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
commit 08db565a72
Author: Eric Allam <eallam@icloud.com>
Date: Thu Oct 3 12:38:25 2024 -0700
improve the timed out description
commit 6d0884254e
Author: Eric Allam <eric@trigger.dev>
Date: Thu Oct 3 12:43:26 2024 -0700
feat: Add maxDuration to tasks (#1377)
* WIP
* Get max duration working on deployed runs
* Actually set the timed out runs to status = TIMED_OUT
* The client status for TIMED_OUT is now MAX_DURATION_EXCEEDED
* New TimedOutIcon
* Added new timedout icon
* Add ability to opt-out of maxDuration with timeout.None
* MAX_DURATION_EXCEEDED -> TIMED_OUT
* changeset
* Improved styling for the status tooltip content
---------
Co-authored-by: James Ritchie <james@trigger.dev>
commit 665ccf88b7
Author: nicktrn <55853254+nicktrn@users.noreply.github.com>
Date: Thu Oct 3 12:33:18 2024 +0100
Update github actions and self-hosting docs
commit 1ff7b861ca
Author: Eric Allam <eric@trigger.dev>
Date: Wed Oct 2 18:26:36 2024 -0700
Add max queue depth limits (#1376)
* Add runs to an env queue, as well as the actual queue
* Add queue size limit guard on triggering tasks
commit c531a9d986
Author: Eric Allam <eric@trigger.dev>
Date: Wed Oct 2 15:30:39 2024 -0700
fix: cleanup ttl expire run graphile jobs (#1373)
* fix: remove ttl expire run graphile jobs when a run is started or completed
* Update expireEnqueuedRun.server.ts
commit 0bf500fa95
Author: Matt Aitken <matt@mattaitken.com>
Date: Wed Oct 2 15:30:16 2024 -0700
Prioritize finishing waited runs (#1375)
* If a tree node is missing, estimate the size as zero
* Task to test prioritizing finishing existing runs after triggerAndWaits
* When requeuing a run with a checkpoint, put it in the queue with the parent run time so it’s correctly prioritized
* The same change but if there’s no checkpoint
* Squashed commit of the following:
commit 886429b0ed
Author: Matt Aitken <matt@mattaitken.com>
Date: Tue Oct 8 18:36:46 2024 +0100
Removed emails, @trigger.dev/database and @trigger.dev/otlp-importer from changesets config
commit f65157a14f
Author: Matt Aitken <matt@mattaitken.com>
Date: Tue Oct 8 18:31:27 2024 +0100
Lockfile with run-engine removed
commit 3d67bb81e9
Author: Matt Aitken <matt@mattaitken.com>
Date: Tue Oct 8 18:24:31 2024 +0100
Removed run-engine from the webapp package.json/tsconfig
commit d30e9713f4
Author: Matt Aitken <matt@mattaitken.com>
Date: Tue Oct 8 18:06:04 2024 +0100
Dockerfile fix because the database package has been moved
commit f2babbf637
Author: Matt Aitken <matt@mattaitken.com>
Date: Tue Oct 8 09:41:22 2024 -0700
Internal packages (testcontainers, redis-worker and zod-worker) (#1392)
* Some notes on the new run engine
* lockfile with setup for the run engine
* Documenting where TaskRun is currently mutated, to try figure out the shape of the new system
* Added notes about how triggering currently works
* Details about when triggering happens
* Lots of notes about waitpoints
* Started scaffolding the RunEngine
* Sketch of Prisma waitpoint schema while it’s fresh in my mind
* Got Prisma working with testcontainers
* Use beforeEach/afterEach
* Simple Prisma and Redis test
* Return Redis options instead of a client
* Simplified things
* A very simple FIFO pull-based queue to check the tests working properly
* Use vitest extend
* Separate redis, postgres and combined tests for faster testing
* Some fixes and test improvements
* Pass a logger into the queue
* A queue processor that processes items from the given queue as fast as it can
* Test for retrying an item that wasn’t processed
* First draft of waitpoints in the Prisma schema
* Remove the custom logger from the test
* Added a completedAt to Waitpoint
* Notes on the flow for an execution starting
* Added redlock, moved some files around
* Starting point for the TaskRunExecutionSnapshot table
* Added relationships to TaskRunExecutionSnapshot
* Change some tsconfig
* Moved some things around
* Added some packages
* WIP on the RunQueue
* Fix for some imports
* Key producer with some tests
* Removed the nv type from the keys… it’s not useful to do global queries
* Passing unit tests for all the public key producer functions
* Some basic tests passing for the RunQueue
* Simple enqueue test working
* Enqueue and dequeue for dev is working
* Don’t log everything during the tests
* Enqueuing/dequeuing from the shared queue is working
* Tests for getting a shared queue
* The key producer sharedQueue can now be named, to allow multiple separate queues
* The key producer uses the name of the queue as the input
* Extra info in the Prisma schema
* Dequeuing a message gets the payload and sets the task concurrency all in one Lua script
* Adding more keys so we can read the concurrency from the queue
* Setting the concurrency with dequeue and enquque is working
* Improved the tests and fixed some bugs
* Acking is resetting the concurrencies
* Check the key has been removed after acking
* Nacking is working
* Changed the package to CommonJS + Node10 so it works with Redlock
* Moved the database, otel and emails packages to be in internal-packages
* Moved some Prisma code to the database package
* Started using the RunEngine for triggering
* Progress on run engine triggering, first waitpoint code
* Create a delay waitpoint
* Moved ZodWorker to an internal package so it can be used in the run engine as well as the webapp
* Web app now uses the zod worker package
* Added parseNaturalLanguageDuration to core/apps
* internal-packages/zod-worker in the lockfile
* Pass in the master queue, remove old rebalance workers code
* Add masterQueue to TaskRun
* Fixed the tests
* Moved waitpoint code into the run engine, also the zod worker
* Completing waitpoints
* An experiment to create a new test container with environment
* More changes to triggering
* Started testing triggering
* Test for a run getting triggered and being enqueued
* Removed dequeueMessageInEnv
* Update dev queue tests to use the shared queue function
* Schema changes for TaskRunExecutionSnapshot
* First execution snapshot when the run is created. Dequeue run function added to the engine
* Separate internal package for testcontainers so they can be used elsewhere
* Remove the simple queue and testcontainers from the run-engine. They’re going to be separate
* Fix for the wrong path to the Prisma schem,a
* Added the testcontainers package to the run-engine
* redis-worker package, just a copy of the simple queue for now
* The queue now uses Lua to enqueue dequeue
* The queue now has a catalog and an invisible period after dequeuing
* Added a visibility timeout and acking, with tests
* Added more Redis connection logging, deleted todos
* Visibility timeouts are now defined on the catalog and can be overridden when enqueuing
* Dequeue multiple items at once
* Test for dequeuing multiple items
* Export some types to be used elsewhere
* Partial refactor of the processor
* First stab at a worker with concurrency and NodeWorkers
* Don’t have a default visibility timeout in the queue
* Worker setup and processing items in a simple test
* Process jobs in parallel with retrying
* Get the attempt when dequeuing
* Workers do exponential backoff
* Moved todos
* DLQ functionality
* DLQ tests
* Same cluster for all keys in the same queue
* Added DLQ tests
* Whitespace
* Redis pubsub to redrive from the worker
* Fixed database paths
* Fix for path to zod-worker
* Fixes for typecheck errors, mostly with TS versions and module resolution
* Redlock required a patch
* Moved the new DB migrations to the new database package folder
* Remove the run-engine package
* Remove the RunEngine prisma schema changes
* Delete triggerTaskV2
* Remove zodworker test script (no tests)
* Update test-containers readme
* Generate the client first
* Use a specific version of the prisma package
* Generate the prisma client before running the unit tests
commit fc609470c4
Author: Dan <8297864+D-K-P@users.noreply.github.com>
Date: Tue Oct 8 14:36:03 2024 +0100
Supabase database webhook example upgrade (#1386)
* Added overview for guides and examples section and split them all out
* New supabase guide wip
* Updated images and improved docs
* Trimmed the supabase prereqs
* Supabase guide wip
* more updates
* Replaced old database webhook guide
* Created one intro page and removed snippets
* Updated guide sidebar titles
* Code updates
* More improvements
* Updates and added images
* Compressed image
* Updated guides descriptions and edge function basic
* Removed bold
* Updated redirects
* Fixed broken links
* Updated intro
commit 07f82eae54
Author: nicktrn <55853254+nicktrn@users.noreply.github.com>
Date: Tue Oct 8 13:28:54 2024 +0100
Release 3.0.11
commit 13ebfcc5b9
Author: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Date: Tue Oct 8 13:24:38 2024 +0100
chore: Update version for release (#1381)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
commit 2a04d17e1f
Author: nicktrn <55853254+nicktrn@users.noreply.github.com>
Date: Tue Oct 8 09:24:23 2024 +0100
Simplify showLogs expression
commit 002ae4bc0d
Author: nicktrn <55853254+nicktrn@users.noreply.github.com>
Date: Tue Oct 8 09:22:29 2024 +0100
Fix dotenv overrides for dev runs (#1388)
* override dashboard dev env vars with local .env
* add changeset
* add simple task for testing env vars
commit 047cb00417
Author: nicktrn <55853254+nicktrn@users.noreply.github.com>
Date: Tue Oct 8 09:22:05 2024 +0100
Disable schedules for deleted orgs on next tick (#1383)
* disable schedules for deleted orgs
* add debug logs
commit 2c014f725a
Author: James Ritchie <james@trigger.dev>
Date: Sun Oct 6 13:02:00 2024 -0700
Override log retention (#1385)
* set full log retention as admin
* If run.logsDeletedAt is set, don’t bother getting the trace
commit a69e04f03a
Author: nicktrn <55853254+nicktrn@users.noreply.github.com>
Date: Sat Oct 5 14:18:58 2024 +0100
Include push output in logs for self-hosted deploys (#1382)
* include push output in logs
* changeset
commit c5488df8ca
Author: nicktrn <55853254+nicktrn@users.noreply.github.com>
Date: Sat Oct 5 13:12:47 2024 +0100
Fix CLI downgrade check (#1380)
* fix downgrade detection
* remove unused semver package from webapp
* add changeset
commit 1caec27f72
Author: Eric Allam <eric@trigger.dev>
Date: Fri Oct 4 15:33:35 2024 -0700
docs: Max duration (#1379)
* maxDuration docs
* Update the init command to set the maxDuration and include a commented out maxDuration in the config file
commit e14c954e12
Author: Eric Allam <eallam@icloud.com>
Date: Fri Oct 4 15:02:05 2024 -0700
Release 3.0.10
commit 8e61f5d631
Author: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Date: Fri Oct 4 14:59:07 2024 -0700
chore: Update version for release (#1378)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
commit 08db565a72
Author: Eric Allam <eallam@icloud.com>
Date: Thu Oct 3 12:38:25 2024 -0700
improve the timed out description
commit 6d0884254e
Author: Eric Allam <eric@trigger.dev>
Date: Thu Oct 3 12:43:26 2024 -0700
feat: Add maxDuration to tasks (#1377)
* WIP
* Get max duration working on deployed runs
* Actually set the timed out runs to status = TIMED_OUT
* The client status for TIMED_OUT is now MAX_DURATION_EXCEEDED
* New TimedOutIcon
* Added new timedout icon
* Add ability to opt-out of maxDuration with timeout.None
* MAX_DURATION_EXCEEDED -> TIMED_OUT
* changeset
* Improved styling for the status tooltip content
---------
Co-authored-by: James Ritchie <james@trigger.dev>
commit 665ccf88b7
Author: nicktrn <55853254+nicktrn@users.noreply.github.com>
Date: Thu Oct 3 12:33:18 2024 +0100
Update github actions and self-hosting docs
commit 1ff7b861ca
Author: Eric Allam <eric@trigger.dev>
Date: Wed Oct 2 18:26:36 2024 -0700
Add max queue depth limits (#1376)
* Add runs to an env queue, as well as the actual queue
* Add queue size limit guard on triggering tasks
commit c531a9d986
Author: Eric Allam <eric@trigger.dev>
Date: Wed Oct 2 15:30:39 2024 -0700
fix: cleanup ttl expire run graphile jobs (#1373)
* fix: remove ttl expire run graphile jobs when a run is started or completed
* Update expireEnqueuedRun.server.ts
commit 0bf500fa95
Author: Matt Aitken <matt@mattaitken.com>
Date: Wed Oct 2 15:30:16 2024 -0700
Prioritize finishing waited runs (#1375)
* If a tree node is missing, estimate the size as zero
* Task to test prioritizing finishing existing runs after triggerAndWaits
* When requeuing a run with a checkpoint, put it in the queue with the parent run time so it’s correctly prioritized
* The same change but if there’s no checkpoint
* Revert "Squashed commit of the following:"
This reverts commit b837b5a558dc5187d89b09135276d16f6856f455.
* Removed console logs
* cleaned up conditionals
* Unlock free plan state
* Fixed subscribe button if you’re already github verified
* Simplified the downgrade reasons logic
* made periodEnd required
---------
Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
* Some notes on the new run engine
* lockfile with setup for the run engine
* Documenting where TaskRun is currently mutated, to try figure out the shape of the new system
* Added notes about how triggering currently works
* Details about when triggering happens
* Lots of notes about waitpoints
* Started scaffolding the RunEngine
* Sketch of Prisma waitpoint schema while it’s fresh in my mind
* Got Prisma working with testcontainers
* Use beforeEach/afterEach
* Simple Prisma and Redis test
* Return Redis options instead of a client
* Simplified things
* A very simple FIFO pull-based queue to check the tests working properly
* Use vitest extend
* Separate redis, postgres and combined tests for faster testing
* Some fixes and test improvements
* Pass a logger into the queue
* A queue processor that processes items from the given queue as fast as it can
* Test for retrying an item that wasn’t processed
* First draft of waitpoints in the Prisma schema
* Remove the custom logger from the test
* Added a completedAt to Waitpoint
* Notes on the flow for an execution starting
* Added redlock, moved some files around
* Starting point for the TaskRunExecutionSnapshot table
* Added relationships to TaskRunExecutionSnapshot
* Change some tsconfig
* Moved some things around
* Added some packages
* WIP on the RunQueue
* Fix for some imports
* Key producer with some tests
* Removed the nv type from the keys… it’s not useful to do global queries
* Passing unit tests for all the public key producer functions
* Some basic tests passing for the RunQueue
* Simple enqueue test working
* Enqueue and dequeue for dev is working
* Don’t log everything during the tests
* Enqueuing/dequeuing from the shared queue is working
* Tests for getting a shared queue
* The key producer sharedQueue can now be named, to allow multiple separate queues
* The key producer uses the name of the queue as the input
* Extra info in the Prisma schema
* Dequeuing a message gets the payload and sets the task concurrency all in one Lua script
* Adding more keys so we can read the concurrency from the queue
* Setting the concurrency with dequeue and enquque is working
* Improved the tests and fixed some bugs
* Acking is resetting the concurrencies
* Check the key has been removed after acking
* Nacking is working
* Changed the package to CommonJS + Node10 so it works with Redlock
* Moved the database, otel and emails packages to be in internal-packages
* Moved some Prisma code to the database package
* Started using the RunEngine for triggering
* Progress on run engine triggering, first waitpoint code
* Create a delay waitpoint
* Moved ZodWorker to an internal package so it can be used in the run engine as well as the webapp
* Web app now uses the zod worker package
* Added parseNaturalLanguageDuration to core/apps
* internal-packages/zod-worker in the lockfile
* Pass in the master queue, remove old rebalance workers code
* Add masterQueue to TaskRun
* Fixed the tests
* Moved waitpoint code into the run engine, also the zod worker
* Completing waitpoints
* An experiment to create a new test container with environment
* More changes to triggering
* Started testing triggering
* Test for a run getting triggered and being enqueued
* Removed dequeueMessageInEnv
* Update dev queue tests to use the shared queue function
* Schema changes for TaskRunExecutionSnapshot
* First execution snapshot when the run is created. Dequeue run function added to the engine
* Separate internal package for testcontainers so they can be used elsewhere
* Remove the simple queue and testcontainers from the run-engine. They’re going to be separate
* Fix for the wrong path to the Prisma schem,a
* Added the testcontainers package to the run-engine
* redis-worker package, just a copy of the simple queue for now
* The queue now uses Lua to enqueue dequeue
* The queue now has a catalog and an invisible period after dequeuing
* Added a visibility timeout and acking, with tests
* Added more Redis connection logging, deleted todos
* Visibility timeouts are now defined on the catalog and can be overridden when enqueuing
* Dequeue multiple items at once
* Test for dequeuing multiple items
* Export some types to be used elsewhere
* Partial refactor of the processor
* First stab at a worker with concurrency and NodeWorkers
* Don’t have a default visibility timeout in the queue
* Worker setup and processing items in a simple test
* Process jobs in parallel with retrying
* Get the attempt when dequeuing
* Workers do exponential backoff
* Moved todos
* DLQ functionality
* DLQ tests
* Same cluster for all keys in the same queue
* Added DLQ tests
* Whitespace
* Redis pubsub to redrive from the worker
* Fixed database paths
* Fix for path to zod-worker
* Fixes for typecheck errors, mostly with TS versions and module resolution
* Redlock required a patch
* Moved the new DB migrations to the new database package folder
* Remove the run-engine package
* Remove the RunEngine prisma schema changes
* Delete triggerTaskV2
* Remove zodworker test script (no tests)
* Update test-containers readme
* Generate the client first
* Use a specific version of the prisma package
* Generate the prisma client before running the unit tests
* WIP
* Get max duration working on deployed runs
* Actually set the timed out runs to status = TIMED_OUT
* The client status for TIMED_OUT is now MAX_DURATION_EXCEEDED
* New TimedOutIcon
* Added new timedout icon
* Add ability to opt-out of maxDuration with timeout.None
* MAX_DURATION_EXCEEDED -> TIMED_OUT
* changeset
* Improved styling for the status tooltip content
---------
Co-authored-by: James Ritchie <james@trigger.dev>
* If a tree node is missing, estimate the size as zero
* Task to test prioritizing finishing existing runs after triggerAndWaits
* When requeuing a run with a checkpoint, put it in the queue with the parent run time so it’s correctly prioritized
* The same change but if there’s no checkpoint
* Removed the inline-code accessory from the logs when calling trigger or batchTrigger from a run
* Removed re2 from the v3 catalog
* Added a Root badge to the runs list
* Keep the side panel open when switching tasks & remove links from detail panel
* Root and parent task styling inspector
* Hide the root badge if the task isn’t the root
* margin between the dev not running message
* improved spacing of items
* Improved Root badge style
* Show a table of triggered runs in the inspector
* Add parentSpanId index to the TaskRun table
* Fix for the run inspector now opening when linked from another run/replay
* Triggered runs table has a max height
* Added a description to the replay run modal and improved the styling slightly
* Only include a bottom border when the triggered run table is more than 4 items
* Improved the triggered runs table borders
* Improved the tables so they can have an optional sticky header
* Added table types to storybook
* Fix for hover states on different backgrounds & runs table select cell
* Run metadata
* Remove metadata from context, move it to it’s own tab
* More run metadata stuff
- Add metadata to testing
- Make using metadata outside of runs a no-op
- Add docs
* Replaying should copy over the metadata
* transfer final attempt output to the task run
* A couple of minor fixes
* Use the new clientOrThrow() method everywhere
* Cleaned up the update metadata endpoint and added an API doc page for it
* Mirror task run attempt errors and output
* Add safe rootTaskRunId index and a README to @trigger.dev/database
* Associate child runs with the span ID of the span in the parent run that triggered the child run
* Update deprecation notice doc links
* Add task run hierarchical relationships to the database
* Add depth and related runs to the retrieve run API response
* Remove prisma optimize
* restructure the migrations to create the index concurrently
* Delete these tsbuildinfo files
* Fix type error by adding depth to the run list presenter
* Cleanup the task hierarchy, share more code
* Remove some fields from the list run response
* Fix for run filtering not working with some special characters (double encoded)
* Add the full dependentTaskAttempt to a ResumeBatchRunService log
* Added RESUME_AFTER_DEPENDENCY_WITH_ACK
* Set the delay to 5s
* If a checkpoint has been created, the coordinator won’t continue the run with RESUME_AFTER_DEPENDENCY_WITH_ACK
* If we’re keeping the run alive then set socket.data.requiresCheckpointResumeWithMessage to undefined
* Log out the data before and after setting socket.data.requiresCheckpointResumeWithMessage
* Add triggerAndWait().unwrap() to more easily get at the output or throw the subtask error
* Add taskId and runId to SubtaskUnwrapError
* WIP docs update beta -> latest
* trigger.dev init now adds @trigger.dev/build to devDependencies
* How it works doc
* Restructure some docs and update the cli commands
* Config file docs, plus aptGet and ffmpeg extensions
* Update to latest from beta docs
* Add runtime to templates
* Add --runtime option to the init CLI command
* A bunch more doc updates after feedback
* Document triggerAndWait with unwrap and result types
* beta -> latest in the webapp
* CLI update check no longer references beta
* Add major release
* Leave changeset beta, back to normal package release
* Fixed default dirs option in init command
* exclude windows-yarn variation of cli e2e tests because it’s buggy
* Remove cache to try and fix yarn e2e test workflow errors
* WIP on using react-window-splitter
* WIP with new resizable panels and SSR
* Use the cookie package
* Resizable storybook page
* Increase indexing memory limit
* Fixed v2 usage meter displaying when on paid plan (#1255)
* Fixed v2 usage meter displaying when on paid plan
* Show the free usage panel only for v3 projects
* Concurrency page and more accurate tracking (#1252)
* Initial TaskRunConcurrencyTracker implementation
* MARQS calls a subscriber to events
* When enqueuing add the extra required metadata
* Track concurrency per environment for tasks too
* Admin page for global concurrency
* Use the new concurrency tracker on the tasks page
* Useful performance test task
* getAllTaskIdentifiers()
* New page for concurrency
* BackgroundWorkerTask index for quick lookup of task identifiers
* Added a way to get concurrency for environments
* Added upgrade/request more concurrency button
* Queued task column working
* Use defer and suspense
* Added queue column to the concurrency environments table
* Some comments added for clarity
* Fixed bad log message
* Sidemenu: move lower and rename to “Concurrency limits”
* Only show the environments, not tasks. Renamed to “Concurrency limits”
* v3: fix unfreezable state crashes for runs with multiple waits (#1253)
* support named capture groups
* write crash errors to attempt.error
* make restored pod names unique per checkpoint
* use last eight characters of checkpoint id instead
* add more chaos monkey env vars
* Ignore unfreezable states
* prevent excessive queue config parsing errors
* handle dependency resume edge case
* better entry point logging
* ignore checkpoint cancellation timeouts
* add missing idempotency keys to wait for dep replays
* remove checkpoints between attempts
* fix retry container names on kubernetes
* add changeset
* fix types
* bring back internal duration timers
* Added more logging to TaskRunConcurrencyTracker and some more try/catches
* Call subscriber.messageDequeued in dequeueMessageInSharedQueue
* Added messageReplaced to concurrency tracking (when freezing)
* Added depenenciesToBundle guide to bundle all packages
* Include the old message data when replacing, so we get the projectId etc.
* Fix restored container names
* Fix for schedule page not scrolling
* Added a description panel to the Concurrency admin page
* chore: Update version for release (beta) (#1256)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
* Release 3.0.0-beta.53
* Added a note to use batchTrigger() instead of trigger()
* The latest react-window-splitter fixes the ESM issues
* Set sensible defaults for the run page
* Deployments page
* Test page
* Schedules page
* Latest version of react-window-splitter (0.2.5)
* Updated to the latest version: react-window-splitter
* Callout if runs don’t start right away now has some top margin
* Small padding fix
* styled the handle focus state
* Added isStaticAtRest prop to resizable panel
* Updated resizable storybook
* Inline code blocks behave nicer when text wraps
* Added ElectricSQL to docker-compose, available on 3060
* Extracted some logic out of the eventRepository for getting a trace. This will be used on the frontend
* Use the new util
* More restructuring ready to use the trace summary from the frontend
* Using ElectricSQL for the run page data
* Min size for resizable panel on test page
* Don’t load the trace in the RunPresenter anymore
* Fix for the resizable panels on the run page
* Added overflow hidden to the panel group
* min size for the test page left hand panel
* Updated to latest window-splitter version
* Removed unused const
* One fix for client-server mismatch
* Slight improvement in the loading state
* Restructured the page so the loading is better
* Improvement to the loading states
* Improved the loading behaviour with the inspector
* WIP on auth, having problems with it
* Upgrade Remix to 2.9.1 (same as PR #1096)
* Switched structure around again so we only call the useTrace hook from the client
* Added auth to the sync
* Overscan more rows in the tree view
* Fix for TS error
* Remove duplicate import
* Revert "Upgrade Remix to 2.9.1 (same as PR #1096)"
This reverts commit e63ee9e924.
* save cookie only when id is used
* Deployment table now scrolls
* removed imports
* A lot of changes to make the inspector live too… WIP
* More major overhauls to get the synced version of the run page working…
* If a span is completed show that
* Set the debounce much lower for selecting the span view
* Load the details run inspector data on demand
* Delete the SpanPresenter
* Use the async payload because it deals with superjson
* Fixed weird merge conflict
* Share some inspector timeline components
* A couple of layout tweaks
* Improved the run inspector loading states
* Fix for paragaph errors
* Fix for focusing on a span
* Undefined typre for useSyncedShape
* ELECTRIC_ORIGIN env var doesn’t have a default, added to the examples
* Updated @electric-sql/react package to the latest
* Fix the timeline duration stretching
* Added some better error handling for the electric sync
* More logging
* Better error when there are bad responses
* Turn off resizable snapshots, there’s a bug
* Added getSpan back
* Added SpanPresenter back
* Updated to the new Electric hooks package
* Made a copy so we have the old run page and the new electric one
* Put the main eventRepository back for now
---------
Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
Co-authored-by: James Ritchie <james@trigger.dev>
Co-authored-by: James Ritchie <james@jamesritchie.co.uk>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
nicktrn