* chore(docker): use bitnami legacy repo
* chore(helm): use bitnami legacy repo
* Make Helm webapp chart images configurable
Adds configurability for init and token syncer container images through
new values in the Helm chart configuration
* chore(helm): refactor utility image config
* chore(helm): bump chart version to 4.0.3
---------
Co-authored-by: LeoKaynan <leokaynan@hotmail.com>
* Initial work on upgrading to 6.14.0
Set the output to node_modules still to make it easier
* Use ./generated Prisma folder, update types to fix issues
* Docker compose restart Clickhouse
* Prisma instrumentation update
* Docker
* Removed database dockerignore file, add generated prisma client to the top-level one
* Delete v3-catalog package.json
* Resolved pnpm lock file
* Log errors for very slow queries
* Create schema and migration for organization access tokens
* Add helpers for creating and authenticating OATs
* Adapt the auth service to also accept OATs
* Accept OATs in the whoami v2 endpoint
* Enable deployments with the CLI using OATs
* Avoid reading env variables directly in the token utils
* Remove duplicate cli token utils
* Validate ENCRYPTION_KEY length when parsing env vars
* Make token utils a server-only module
* Disallow revoking already revoked OATs
* Simplify generics in authenticateRequest
* Use 32 bytes mock encryption key in the test setup
* Update dummy encryption key values in tests and templates
* Add a column in the OATs table to differentiate between user and system generated
* Simplify args for v3ProjectPath
Co-authored-by: Matt Aitken <matt@mattaitken.com>
* Add index on org id and createdAt
* Avoid storing the encrypted oat token and its obfuscated version in the DB at all
It is a safer approach. Also we do not need to ever read the decrypted token value after creation.
* Fix prisma update condition
* Add token type to the OAT table index
* Accept OATs in the mcp auth flow
* Simplify env auth flow around the /projects endpoints
---------
Co-authored-by: Matt Aitken <matt@mattaitken.com>
* Sentry WIP
* Configure sentry for uploading and releasing during the publish webapp step
* Delete source maps after uploading
* Forward logger.error calls to sentry through Logger.onError
* Couple tweaks to the dockerfile
* Add retry logic for insert operations
Add a generic retry mechanism for task run and payload inserts to handle
transient connection errors. The new #insertWithRetry method retries up to
three times with exponential backoff and jitter on retryable connection
errors such as connection resets or timeouts. Errors are logged and
recorded in tracing spans to improve observability and robustness of the
replication service.
* Replication settings are configurable
* Log out the runIds for failed batches
* Detecting bad JSON in run replication and ignoring it
* Reproduced split unicode error
* Move output file
* Massively improved the performance
* Minor performance improvements
* Unskip tests
* Remove unused test in CH package
* Fix for the ClickHouse UI explorer
* RunReplication keepAlive defaults to false
* Add concurrency_key and bulk_action_group_ids to ClickHouse task runs
* ClickHouse package doesn't need to be built anymore for the webapp
* Set the concurrency_key from the run replication service
* remove unused env vars
* actually use ALERT_REPLY_TO_EMAIL for alerts (non-breaking)
* increase fallback branch limit to 100M
* more unused env vars and example
* hide usage page when self-hosted
* fix for init flow
* set default or concurrency to 300 to match 100 on env
* fail fast when registry env vars are empty strings
* runs replication leader lock expiration fix
* Allow configuring the container image --max-old-space-size using NODE_MAX_OLD_SPACE_SIZE
* Ability to configure the clickhouse keep alive settings
* Add some logging because we might not be able to do telemetry
* WIP clickhouse package with test containers setup
* More clickhouse client setup now with otel and real tests, and the v1 of raw run events
* Add some additional columns to raw_run_events_v1
* WIP runs dashboard service
* Create a new run engine event bus event for the runs dashboard to hook into
* Track run events in the run engine
* make sure engine v1 runs get synced to CH
* Update the attemptNumber of v3 task runs
* Restructure the run events to be more sparse
* emit more stuff
* Setup replication package
* scaffold the replication package
* replication wip
* resolve conflicts
* more replication stuff
* Add ability to drop the replication slot completely on teardown
* Use the new single replacingmergetree task events table for replication
* get it working
* insert payloads into their own table only on insert and then join
* prepare for using clickhouse cloud and now running ch migrations during boot in the entrypoint.sh
* Handover WIP and tests
* Testing the replication service
* Remove the runs dashboard stuff that we aren't using anymore
* Added a test for large payloads
* hacky typecheck fix
* Fix new internal package typecheck issues and start adding telemetry to the replication service
* tracing over spans, some other improvements
* Improvements to the runs replication service, now ready for testing
* Some fixes and cleanups
* Don't need this code anymore
* move transaction types into the runs replication service
* only send spans where there are transaction events
* A couple of suggested tweaks
* refactor: docker compose migration
* fix compose download link
* set static name for electric container
---------
Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
* fix and restructure dockerignore
* switch to using pnpm deploy
* pass webapp node image as build arg
* ensure pnpm is downloaded at build, not runtime
* Fix streaming splits in realtime streams v2
* Add changeset
* Skip all flaky tests 😡
* Improve the way we stream from tasks to the server
* Improve the v1 realtime streams (Redis)
* Turn on the relay realtime stream service
* Improved the relay realtime cleanup
* Fixed consuming realtime runs w/streams after the run is already finished
* Remove some logs
* Update changeset
* Fixed runStream tests
* Realtime streams now powered by electric, and fix the streaming update duplicate issues by converting the electric Shape materialized view into a ReadableStream of changes
* Ensure realtime subscription stops when runs are finished, and add an onComplete handle to use realtime hooks
* Fix tests
* WIP realtime streams
* Handle realtime with large payloads or outputs #1451
* feat: optimize Redis stream handling with batching
Add STREAM_ORIGIN to environment schema. Improve performance in
RealtimeStreams by using TextDecoderStream for simpler text
decoding and implementing batching of XADD commands for Redis
streams. Limit stream size using MAXLEN option. Update
environment variable repository with new variable type. Adjust
import statements for Redis key and value types.
* 🔧 chore: add dev dependencies for bundle analysis
* add metadata tests and a few more utilties
* Add stream tests and improve streaming
* Added AI tool tasks, descriptions to tasks
* Use the config file path to determine the workingDir, then the package.json path
* Remove stream test files
* useTaskTrigger react hook that allows triggering a task from the client
* Add streaming support for the realtime react hooks
* Add ability to stream results after useTaskTrigger
* Improve the stream throttling
* Use the runId as the ID key to bust the cache after triggering
* Upgrade to to the latest electric sql client and server
* Make realtime server backwards compat with 3.1.2 release
* Pass the runId into useRealtimeRun
* Fix scopes when specifiying reading all runs
* WIP @trigger.dev/rsc package
* Various fixes and accepted recommendations by CodeRabbit
* Regenerate pnpm lock file
* A couple tweaks to rsc and give up on rendering react in tasks for now
* Add changeset
* Remove triggerRequest from the useEffect deps
* Improve realtime & frontend authentication errors
* Fixed authorization tests
* Remove unnecessary log
* Add metadata.stream limits and improve the metadata streams structure
* Streams can now have up to 2500 entries
* Various coderabbit fixes
* additional react-hooks jsdocs
* Denormalize run tags, increase character limit to 128
* WIP realtime subscribing to runs
* extracted the stream stuff into core, made it more reusable
* WIP tags
* Remove tags for now because it’s not support in electric
* Support async iterables, readable stream, and callback style subscription styles
* Remove tags streaming endpoint
* Add realtime rate limits and scope them to the /realtime path
* WIP rate limt per org
* Introduce per org rate limits
* WIP JWT auth
* Move migrations into new internal db package
* Resolve pnpm lock file
* Authenticating to the realtime API with JWTs are working
* realtime in the client
* Created react-hooks package and starting to move stuff in there
* Improve types for hooks
* schema tasks
* Added useBatch hook
* build uploadthing/fal demo and change how run metadata is synced to the server
* tweaks
* WIL realtime concurrency tracking
* Implement test for realtime client using testcontainers
also updated electric to latest version
* Allow customizing the expiration time of the automatic JWT created after triggering a task
* Add support for subscribing to run tags
* Improve auth types and API
* finalize the realtime API
* Fixed some example stuff
* Allow up to 10 run tags
* Remove core from docker-provider tsconfig paths to prevent it from being typechecked
* do the same for the kubernetes provider
* Fixing some typecheck errors
* Fix webapp type errors
* Update @trigger.dev/platform to 1.0.13
* Fix attw error
* Remove from/to in subscribeToRuns query params
* Add tests for the rate limit middleware and add custom JWT rate limits
* turn off webapp test parallelism
* Finish renaming jwt -> publicAccessToken and automatically give the JWT read access to the tags when using trigger
* Add changeset
* Attempt to fix unit tests in CI
* Skip running the auth rate limit middleware tests for now
* Try a beefier machine
* Try and run webapp tests separately
* Setup env vars
* Make sliding window test more reliabile
* WIP on using react-window-splitter
* WIP with new resizable panels and SSR
* Use the cookie package
* Resizable storybook page
* Increase indexing memory limit
* Fixed v2 usage meter displaying when on paid plan (#1255)
* Fixed v2 usage meter displaying when on paid plan
* Show the free usage panel only for v3 projects
* Concurrency page and more accurate tracking (#1252)
* Initial TaskRunConcurrencyTracker implementation
* MARQS calls a subscriber to events
* When enqueuing add the extra required metadata
* Track concurrency per environment for tasks too
* Admin page for global concurrency
* Use the new concurrency tracker on the tasks page
* Useful performance test task
* getAllTaskIdentifiers()
* New page for concurrency
* BackgroundWorkerTask index for quick lookup of task identifiers
* Added a way to get concurrency for environments
* Added upgrade/request more concurrency button
* Queued task column working
* Use defer and suspense
* Added queue column to the concurrency environments table
* Some comments added for clarity
* Fixed bad log message
* Sidemenu: move lower and rename to “Concurrency limits”
* Only show the environments, not tasks. Renamed to “Concurrency limits”
* v3: fix unfreezable state crashes for runs with multiple waits (#1253)
* support named capture groups
* write crash errors to attempt.error
* make restored pod names unique per checkpoint
* use last eight characters of checkpoint id instead
* add more chaos monkey env vars
* Ignore unfreezable states
* prevent excessive queue config parsing errors
* handle dependency resume edge case
* better entry point logging
* ignore checkpoint cancellation timeouts
* add missing idempotency keys to wait for dep replays
* remove checkpoints between attempts
* fix retry container names on kubernetes
* add changeset
* fix types
* bring back internal duration timers
* Added more logging to TaskRunConcurrencyTracker and some more try/catches
* Call subscriber.messageDequeued in dequeueMessageInSharedQueue
* Added messageReplaced to concurrency tracking (when freezing)
* Added depenenciesToBundle guide to bundle all packages
* Include the old message data when replacing, so we get the projectId etc.
* Fix restored container names
* Fix for schedule page not scrolling
* Added a description panel to the Concurrency admin page
* chore: Update version for release (beta) (#1256)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
* Release 3.0.0-beta.53
* Added a note to use batchTrigger() instead of trigger()
* The latest react-window-splitter fixes the ESM issues
* Set sensible defaults for the run page
* Deployments page
* Test page
* Schedules page
* Latest version of react-window-splitter (0.2.5)
* Updated to the latest version: react-window-splitter
* Callout if runs don’t start right away now has some top margin
* Small padding fix
* styled the handle focus state
* Added isStaticAtRest prop to resizable panel
* Updated resizable storybook
* Inline code blocks behave nicer when text wraps
* Added ElectricSQL to docker-compose, available on 3060
* Extracted some logic out of the eventRepository for getting a trace. This will be used on the frontend
* Use the new util
* More restructuring ready to use the trace summary from the frontend
* Using ElectricSQL for the run page data
* Min size for resizable panel on test page
* Don’t load the trace in the RunPresenter anymore
* Fix for the resizable panels on the run page
* Added overflow hidden to the panel group
* min size for the test page left hand panel
* Updated to latest window-splitter version
* Removed unused const
* One fix for client-server mismatch
* Slight improvement in the loading state
* Restructured the page so the loading is better
* Improvement to the loading states
* Improved the loading behaviour with the inspector
* WIP on auth, having problems with it
* Upgrade Remix to 2.9.1 (same as PR #1096)
* Switched structure around again so we only call the useTrace hook from the client
* Added auth to the sync
* Overscan more rows in the tree view
* Fix for TS error
* Remove duplicate import
* Revert "Upgrade Remix to 2.9.1 (same as PR #1096)"
This reverts commit e63ee9e924.
* save cookie only when id is used
* Deployment table now scrolls
* removed imports
* A lot of changes to make the inspector live too… WIP
* More major overhauls to get the synced version of the run page working…
* If a span is completed show that
* Set the debounce much lower for selecting the span view
* Load the details run inspector data on demand
* Delete the SpanPresenter
* Use the async payload because it deals with superjson
* Fixed weird merge conflict
* Share some inspector timeline components
* A couple of layout tweaks
* Improved the run inspector loading states
* Fix for paragaph errors
* Fix for focusing on a span
* Undefined typre for useSyncedShape
* ELECTRIC_ORIGIN env var doesn’t have a default, added to the examples
* Updated @electric-sql/react package to the latest
* Fix the timeline duration stretching
* Added some better error handling for the electric sync
* More logging
* Better error when there are bad responses
* Turn off resizable snapshots, there’s a bug
* Added getSpan back
* Added SpanPresenter back
* Updated to the new Electric hooks package
* Made a copy so we have the old run page and the new electric one
* Put the main eventRepository back for now
---------
Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
Co-authored-by: James Ritchie <james@trigger.dev>
Co-authored-by: James Ritchie <james@jamesritchie.co.uk>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
* WIP on using react-window-splitter
* WIP with new resizable panels and SSR
* Use the cookie package
* Resizable storybook page
* Increase indexing memory limit
* Fixed v2 usage meter displaying when on paid plan (#1255)
* Fixed v2 usage meter displaying when on paid plan
* Show the free usage panel only for v3 projects
* Concurrency page and more accurate tracking (#1252)
* Initial TaskRunConcurrencyTracker implementation
* MARQS calls a subscriber to events
* When enqueuing add the extra required metadata
* Track concurrency per environment for tasks too
* Admin page for global concurrency
* Use the new concurrency tracker on the tasks page
* Useful performance test task
* getAllTaskIdentifiers()
* New page for concurrency
* BackgroundWorkerTask index for quick lookup of task identifiers
* Added a way to get concurrency for environments
* Added upgrade/request more concurrency button
* Queued task column working
* Use defer and suspense
* Added queue column to the concurrency environments table
* Some comments added for clarity
* Fixed bad log message
* Sidemenu: move lower and rename to “Concurrency limits”
* Only show the environments, not tasks. Renamed to “Concurrency limits”
* v3: fix unfreezable state crashes for runs with multiple waits (#1253)
* support named capture groups
* write crash errors to attempt.error
* make restored pod names unique per checkpoint
* use last eight characters of checkpoint id instead
* add more chaos monkey env vars
* Ignore unfreezable states
* prevent excessive queue config parsing errors
* handle dependency resume edge case
* better entry point logging
* ignore checkpoint cancellation timeouts
* add missing idempotency keys to wait for dep replays
* remove checkpoints between attempts
* fix retry container names on kubernetes
* add changeset
* fix types
* bring back internal duration timers
* Added more logging to TaskRunConcurrencyTracker and some more try/catches
* Call subscriber.messageDequeued in dequeueMessageInSharedQueue
* Added messageReplaced to concurrency tracking (when freezing)
* Added depenenciesToBundle guide to bundle all packages
* Include the old message data when replacing, so we get the projectId etc.
* Fix restored container names
* Fix for schedule page not scrolling
* Added a description panel to the Concurrency admin page
* chore: Update version for release (beta) (#1256)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
* Release 3.0.0-beta.53
* Added a note to use batchTrigger() instead of trigger()
* The latest react-window-splitter fixes the ESM issues
* Set sensible defaults for the run page
* Deployments page
* Test page
* Schedules page
* Latest version of react-window-splitter (0.2.5)
* Updated to the latest version: react-window-splitter
* Callout if runs don’t start right away now has some top margin
* Small padding fix
* styled the handle focus state
* Added isStaticAtRest prop to resizable panel
* Updated resizable storybook
* Inline code blocks behave nicer when text wraps
* Added ElectricSQL to docker-compose, available on 3060
* Extracted some logic out of the eventRepository for getting a trace. This will be used on the frontend
* Use the new util
* More restructuring ready to use the trace summary from the frontend
* Using ElectricSQL for the run page data
* Min size for resizable panel on test page
* Don’t load the trace in the RunPresenter anymore
* Fix for the resizable panels on the run page
* Added overflow hidden to the panel group
* min size for the test page left hand panel
* Updated to latest window-splitter version
* Removed unused const
* One fix for client-server mismatch
* Slight improvement in the loading state
* Restructured the page so the loading is better
* Improvement to the loading states
* Improved the loading behaviour with the inspector
* WIP on auth, having problems with it
* Upgrade Remix to 2.9.1 (same as PR #1096)
* Switched structure around again so we only call the useTrace hook from the client
* Added auth to the sync
* Overscan more rows in the tree view
* Fix for TS error
* Remove duplicate import
* Revert "Upgrade Remix to 2.9.1 (same as PR #1096)"
This reverts commit e63ee9e924.
* save cookie only when id is used
* Deployment table now scrolls
* removed imports
* A lot of changes to make the inspector live too… WIP
* More major overhauls to get the synced version of the run page working…
* If a span is completed show that
* Set the debounce much lower for selecting the span view
* Load the details run inspector data on demand
* Delete the SpanPresenter
* Use the async payload because it deals with superjson
* Fixed weird merge conflict
* Share some inspector timeline components
* A couple of layout tweaks
* Improved the run inspector loading states
* Fix for paragaph errors
* Fix for focusing on a span
* Undefined typre for useSyncedShape
* ELECTRIC_ORIGIN env var doesn’t have a default, added to the examples
* Updated @electric-sql/react package to the latest
* Fix the timeline duration stretching
* Added some better error handling for the electric sync
* More logging
* Better error when there are bad responses
* Turn off resizable snapshots, there’s a bug
---------
Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
Co-authored-by: James Ritchie <james@trigger.dev>
Co-authored-by: James Ritchie <james@jamesritchie.co.uk>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
* upgrade @opentelemetry packages to the latest versions
* remove v2 only packages, will be moved to a dedicated repo
* remove more v2 code and run pnpm install
* use the npm yalt package in the webapp
* convert @trigger.dev/core to tshy
* Switch from jest to vitest in @trigger.dev/core
* Fixed core test
* move core-backend code into core subpath export
* convert @trigger.dev/sdk to tshy
* Removed hono
* move core-apps to core/v3/apps, remove core-apps, start converting cli-v3
* Fix up some of the commands
* cli now building and loadable
* using package-json-from-dist to get package version now in core and cli
* dev command WIP
* cleaned up some repetition and structure of the entry point stuff
* bringing back the background worker stuff
* Indexing of the v3 catalog
* getting closer to executing dev runs...
* centralize dev logging using event emitter
* Move indexing to it’s own entry point, simplify code
* dev runs working
* Get instrumentation to work with openai
* debugging achieved internally
* provide worker files as part of the worker creation on the server
* support for cjs and esm javascript
* Fixed timeout
* worker manifest now has the config path
* auto-upgrade config to non-deprecated alternatives
* Adding package preview release
* deployment WIP
* improve the syncEnvVars output and adapt resolveEnvVars
* WIP bun runtime
* WIP bun support
* seed tasks with the machine preset if listed in the config
* deploy run executions WIP, extracted TaskRunProcess into 1 place
* deployed tasks running and executing 🎉
* support for waits and better flushing & process cleanup
* Fixed the heartbeating
* Better warning messages
* Improve and unify the indexing between dev and deploy
* Support for external deps that need node-gyp to build
* build extensions can now install custom packages and run instructions in the image. Also prisma extension now works and also works with multiple schema files
* Add back in the main/types/module to sdk
* dev no longer is Ink/React, grace period for disconnections in dev
* Fix the changeset config
* More changeset fixes
* Remove config packages
* More changeset fixes
* Fixed typescript issues (needed to revert back to zod 3.22.3
* Fix pr_checks workflow
* Remove the prepare script
* Fixed tests and package versions
* Remove cli test script
* Remove packages from tailwind watch paths
* Add repo to public packages
* Just commit the generated files and do the building at dev time
* Try and get pkg.pr.new working
* Try again
* Fix emitDecoratorMetadata importing named export from typescript
* config file backwards compat with export const config
* Fixed issue where import errors weren’t coming through
* p-retry is a prod dep
* typescript needs to be a prod dependency for emitDecoratorMetadata
* Add better debug logging to help track down import-in-the-middle bug
* An external is only considered resolvable if it resolves to the same path as the collected external
* Fix runtime checks to allow >=18.20
* Move extensions to a new build package
* Fixed building packages in dockerfile
* Remove the e2e test from publish workflow for now
* Don’t treat pkg.pr.new versions has needing upgrading
* making sure config handleError works, and discovered path aliases don’t work in config files
* Strip empty string env vars so they accidentally override real values
* Couple of things
* Update version to use preview instead of beta
* Hopefully fix re-attempts with >30s delay
* Match socket emit messages to current latest in main
* Initial guide
* Go back to beta
* Go back to the preview, and update guide to use pr preview tags
* Go back to beta
---------
Co-authored-by: Matt Aitken <matt@mattaitken.com>
* superjson output support
* Better support for superjson in the task events, limiting output attributes
* Offload large outputs to object store (r2)
* Finishing up the offloading of large outputs/payloads to an object store
* Introducing Modular Asynchronous Reliable Queueing System (MarQS). Works in dev
* Convert MarQS to using lua and dealing with concurrency
* Simplified the timeout queue and current concurrency is now a set instead of a flat value (to support idempotency)
* Implement task heartbeating and reconnect the background workers CLI when the websocket connection reconnects
* Start adding internal telemetry support for the server
* Get env vars to work in dev and implement prisma tracing in webapp
* Cleanup telemetry and implement it in the consumer
* Implement dequeuing a message from a parent shared queue
* Implement a custom logger exporter instead of using console log exporter
* Use node instead of shell for generating protocol buffer code
* Propogate trace context into debug logs, and allow turning off logger exporter through env vars
* Switch to using baselime for internal otel data
* Make orgMember optional to fix type issues
* Provide the CLI dev env vars through the CLI, don’t build dotenv into facade
* Removed the logger import
* Address Matt’s comments
* Addressing more of Matt’s comments
* Handle sending an execution after a websocket connection closes
* Remove auth from the env attributes to prevent obfuscation
* WIP execution concurrency controls implemented via Redis
- Split up resuming a run and executing a run
- Added some new statuses to better show what is going on in a run
- Removed preprocessing runs
* WIP
* Convert to using ZSETs and adding env vars
* Removed unused import
* Improve run number generation using advistory locks, and only on start
* More execution concurrency stuff
* Add support for job concurrency limits and concurrency limit groups
* Create wild-swans-battle.md
* Increase slots refresh timeout to 10s
* Try to fix Redis connection issues
* Don’t be so strict about the APP_ENV
* Add the blank tls option to the normal redis client as well
* Add docs
Eric Allam