* WIP realtime streams
* Handle realtime with large payloads or outputs #1451
* feat: optimize Redis stream handling with batching
Add STREAM_ORIGIN to environment schema. Improve performance in
RealtimeStreams by using TextDecoderStream for simpler text
decoding and implementing batching of XADD commands for Redis
streams. Limit stream size using MAXLEN option. Update
environment variable repository with new variable type. Adjust
import statements for Redis key and value types.
* 🔧 chore: add dev dependencies for bundle analysis
* add metadata tests and a few more utilties
* Add stream tests and improve streaming
* Added AI tool tasks, descriptions to tasks
* Use the config file path to determine the workingDir, then the package.json path
* Remove stream test files
* useTaskTrigger react hook that allows triggering a task from the client
* Add streaming support for the realtime react hooks
* Add ability to stream results after useTaskTrigger
* Improve the stream throttling
* Use the runId as the ID key to bust the cache after triggering
* Upgrade to to the latest electric sql client and server
* Make realtime server backwards compat with 3.1.2 release
* Pass the runId into useRealtimeRun
* Fix scopes when specifiying reading all runs
* WIP @trigger.dev/rsc package
* Various fixes and accepted recommendations by CodeRabbit
* Regenerate pnpm lock file
* A couple tweaks to rsc and give up on rendering react in tasks for now
* Add changeset
* Remove triggerRequest from the useEffect deps
* Improve realtime & frontend authentication errors
* Fixed authorization tests
* Remove unnecessary log
* Add metadata.stream limits and improve the metadata streams structure
* Streams can now have up to 2500 entries
* Various coderabbit fixes
* additional react-hooks jsdocs
* Denormalize run tags, increase character limit to 128
* WIP realtime subscribing to runs
* extracted the stream stuff into core, made it more reusable
* WIP tags
* Remove tags for now because it’s not support in electric
* Support async iterables, readable stream, and callback style subscription styles
* Remove tags streaming endpoint
* Add realtime rate limits and scope them to the /realtime path
* WIP rate limt per org
* Introduce per org rate limits
* WIP JWT auth
* Move migrations into new internal db package
* Resolve pnpm lock file
* Authenticating to the realtime API with JWTs are working
* realtime in the client
* Created react-hooks package and starting to move stuff in there
* Improve types for hooks
* schema tasks
* Added useBatch hook
* build uploadthing/fal demo and change how run metadata is synced to the server
* tweaks
* WIL realtime concurrency tracking
* Implement test for realtime client using testcontainers
also updated electric to latest version
* Allow customizing the expiration time of the automatic JWT created after triggering a task
* Add support for subscribing to run tags
* Improve auth types and API
* finalize the realtime API
* Fixed some example stuff
* Allow up to 10 run tags
* Remove core from docker-provider tsconfig paths to prevent it from being typechecked
* do the same for the kubernetes provider
* Fixing some typecheck errors
* Fix webapp type errors
* Update @trigger.dev/platform to 1.0.13
* Fix attw error
* Remove from/to in subscribeToRuns query params
* Add tests for the rate limit middleware and add custom JWT rate limits
* turn off webapp test parallelism
* Finish renaming jwt -> publicAccessToken and automatically give the JWT read access to the tags when using trigger
* Add changeset
* Attempt to fix unit tests in CI
* Skip running the auth rate limit middleware tests for now
* Try a beefier machine
* Try and run webapp tests separately
* Setup env vars
* Make sliding window test more reliabile
* WIP
* Get max duration working on deployed runs
* Actually set the timed out runs to status = TIMED_OUT
* The client status for TIMED_OUT is now MAX_DURATION_EXCEEDED
* New TimedOutIcon
* Added new timedout icon
* Add ability to opt-out of maxDuration with timeout.None
* MAX_DURATION_EXCEEDED -> TIMED_OUT
* changeset
* Improved styling for the status tooltip content
---------
Co-authored-by: James Ritchie <james@trigger.dev>
* If a tree node is missing, estimate the size as zero
* Task to test prioritizing finishing existing runs after triggerAndWaits
* When requeuing a run with a checkpoint, put it in the queue with the parent run time so it’s correctly prioritized
* The same change but if there’s no checkpoint
* Removed the inline-code accessory from the logs when calling trigger or batchTrigger from a run
* Removed re2 from the v3 catalog
* Added a Root badge to the runs list
* Keep the side panel open when switching tasks & remove links from detail panel
* Root and parent task styling inspector
* Hide the root badge if the task isn’t the root
* margin between the dev not running message
* improved spacing of items
* Improved Root badge style
* Show a table of triggered runs in the inspector
* Add parentSpanId index to the TaskRun table
* Fix for the run inspector now opening when linked from another run/replay
* Triggered runs table has a max height
* Added a description to the replay run modal and improved the styling slightly
* Only include a bottom border when the triggered run table is more than 4 items
* Improved the triggered runs table borders
* Improved the tables so they can have an optional sticky header
* Added table types to storybook
* Fix for hover states on different backgrounds & runs table select cell
* Run metadata
* Remove metadata from context, move it to it’s own tab
* More run metadata stuff
- Add metadata to testing
- Make using metadata outside of runs a no-op
- Add docs
* Replaying should copy over the metadata
* transfer final attempt output to the task run
* A couple of minor fixes
* Use the new clientOrThrow() method everywhere
* Cleaned up the update metadata endpoint and added an API doc page for it
* Mirror task run attempt errors and output
* Add task run hierarchical relationships to the database
* Add depth and related runs to the retrieve run API response
* Remove prisma optimize
* restructure the migrations to create the index concurrently
* Delete these tsbuildinfo files
* Fix type error by adding depth to the run list presenter
* Cleanup the task hierarchy, share more code
* Remove some fields from the list run response
* Add triggerAndWait().unwrap() to more easily get at the output or throw the subtask error
* Add taskId and runId to SubtaskUnwrapError
* WIP docs update beta -> latest
* trigger.dev init now adds @trigger.dev/build to devDependencies
* How it works doc
* Restructure some docs and update the cli commands
* Config file docs, plus aptGet and ffmpeg extensions
* Update to latest from beta docs
* Add runtime to templates
* Add --runtime option to the init CLI command
* A bunch more doc updates after feedback
* Document triggerAndWait with unwrap and result types
* beta -> latest in the webapp
* CLI update check no longer references beta
* Add major release
* Leave changeset beta, back to normal package release
* Fixed default dirs option in init command
* exclude windows-yarn variation of cli e2e tests because it’s buggy
* Remove cache to try and fix yarn e2e test workflow errors
* Ignore /packages/cli-v3/src/package.json
* Added more logs when resuming a dependency, added the runId
* A task for reproducing a race condition with checkpoints
* Fix for doing remote image build when not self-hosting
* Set team members, alerts and schedule limits to 100m for self-hosting
* Import fix
* Set the checkpointEventId in marqs when the checkpoint is created for batchTriggerAndWait
This should fix a horrible race condition when at max concurrency
* Fixed empty env vars overriding in dev runs
* Don’t import package.json anymore
* fix node10 moduleResolution in @trigger.dev/core
* Support self-hosters pushing to a custom registry when running deploy
* dev: Fixed stuck runs when a child run fails with a process exit
* Make some doc notes about known issues and docker hub private repos
* Fix --project-ref when running deploy
* Fix —config option when deploying
* Fixing the flushing/killing process with the new build system
* Add monorepo-react-email e2e test fixture
* Fix issue with emitDecoratorMetadata and tsconfigs with extends
* Got the emit decorator metadata fixture working
* Fixed typechecking yarn e2e CLI tests in monorepos
* Add remote forced externals system, in case we come across another package that cannot be bundled (spurred on by header-generator)
* Remote externals now powered by JSON Hero to be easier to update
* resolve config source files
* Add a —javascript option to init, defaults to typescript
* Add support for prisma typed sql
* Remove msw and retry.interceptFetch
* Add missing code to the openai retries example
* Don’t generate the v3 catalog prisma client during CI
* Fixed v3-catalog task imports
* Remove interceptor usage in task file
* Only import import-in-the-middle hook if there are instrumented packages
* Fix yarn.lock file
* When resuming a batch, only do marqs operations once
* Made TaskRunDependency clearer in the Prisma schema
* New ResumeDependentParentsService service, use it from checkpoints
* WIP on making resuming more robust
* Turn the declarative schedules off because they make debugging other runs painful
* Resuming batches when there’s an attempt is working
* If there’s no attempt then create one
* Added a log if there are no span events to complete
* If Graphile addJob doesn’t return a row, log and return undefined. No throw
* Pass prisma into the ResumeDependentParentsService
* Removed the todos
* Pass Prisma through to the checkpoint service
* Fix for not checking the batch item correctly
* Fix for when a log flush times out and the process is checkpointed
* Fix for when a log flush times out and the process is checkpointed
* Another test run that does batches with failed subtasks
* Don’t call ResumeTaskRunDependenciesService anymore (we have a new service)
* Only resume if the run is in a final state
* If an attempt doesn’t exist, fix for creating queue with sanitized name
* If DEV then don’t resume using marqs/batches. The CLI manages it
* We don’t need to check the run status again, it’s in the main function now
* Added TaskRunAttempt taskRunId index
* Only allow calling ResumeDependentParentsService with a run ID
* Put the flushing back to what it was
* Support custom config file names & paths
* Fix entry point paths on windows
* Support custom conditions
Add support for custom conditions (for bundling and running), to support being able to import `ai/rsc` with the “react-server” condition.
- Fixed an issue where symlinking unresolvable externals after rebuilding caused the build to hang
- Fixed an issue with external not working with subpath exports (e.g. “ai” would not match “ai/rsc”)
- Protect better against build extensions breaking builds
* Add changeset
* Fix passing CLI process.env down to the task processes
* Remove unused import
* reviving the e2e CLI tests
* Another attempt at fixing windows
* yet another windows attempt (yawa)
* Output index child stdout and stderr (yawa)
* normalize import paths for windows
* Added some logging (yawa)
* normalize the loader path as well
* Added some logging to figure out why the entry points aren’t being found on windows
* Fix for entry point detection on windows
* Normalize runner import loader path
* Normalize import paths in dev and make sure rewritten build manifest paths are correct on windows as well
* Various cleanup after windows fixes
* Remove the webapp e2e for now
* Add node10 type resolution support for subpath exports (still does not actually import in Node 10 or Browserify/Parcel)
https://github.com/andrewbranch/example-subpath-exports-ts-compat
* init using templates again but downloaded from the repo this time
* Adding init schedule example
* Support for js init
* init now working with js
* Fix issues with links in terminals that don’t support them. Also skip update check of version starts with 0.0.0
* Add an error to the final attempt if there isn’t one
* Improved the checkpointResumer test task
* Using triggerAndWait or batchTriggerAndWait frees up concurrency
Normally it frees up just env and org concurrency. If it’s a recursive task then it will free up the run concurrency too (e.g. a task calling itself).
* Removed the filepath and export name from attempt spans
* Run inspector: only show an error if the run is in a finished state
* Add support + docs for custom esbuild plugins
* Fixed the missing subpath export
* Add support for specifying the prisma client generator
* Add `—env-file` support to dev and deploy
* RESUME messages were missing some data that is needed for the concurrency tracker
* Cancel heartbeats when checkpoints are created for triggerAndWait and batchTriggerAndWait
* Added some logs when canceling heartbeats
* Improved the nested dependencies test task
* upgrade @opentelemetry packages to the latest versions
* remove v2 only packages, will be moved to a dedicated repo
* remove more v2 code and run pnpm install
* use the npm yalt package in the webapp
* convert @trigger.dev/core to tshy
* Switch from jest to vitest in @trigger.dev/core
* Fixed core test
* move core-backend code into core subpath export
* convert @trigger.dev/sdk to tshy
* Removed hono
* move core-apps to core/v3/apps, remove core-apps, start converting cli-v3
* Fix up some of the commands
* cli now building and loadable
* using package-json-from-dist to get package version now in core and cli
* dev command WIP
* cleaned up some repetition and structure of the entry point stuff
* bringing back the background worker stuff
* Indexing of the v3 catalog
* getting closer to executing dev runs...
* centralize dev logging using event emitter
* Move indexing to it’s own entry point, simplify code
* dev runs working
* Get instrumentation to work with openai
* debugging achieved internally
* provide worker files as part of the worker creation on the server
* support for cjs and esm javascript
* Fixed timeout
* worker manifest now has the config path
* auto-upgrade config to non-deprecated alternatives
* Adding package preview release
* deployment WIP
* improve the syncEnvVars output and adapt resolveEnvVars
* WIP bun runtime
* WIP bun support
* seed tasks with the machine preset if listed in the config
* deploy run executions WIP, extracted TaskRunProcess into 1 place
* deployed tasks running and executing 🎉
* support for waits and better flushing & process cleanup
* Fixed the heartbeating
* Better warning messages
* Improve and unify the indexing between dev and deploy
* Support for external deps that need node-gyp to build
* build extensions can now install custom packages and run instructions in the image. Also prisma extension now works and also works with multiple schema files
* Add back in the main/types/module to sdk
* dev no longer is Ink/React, grace period for disconnections in dev
* Fix the changeset config
* More changeset fixes
* Remove config packages
* More changeset fixes
* Fixed typescript issues (needed to revert back to zod 3.22.3
* Fix pr_checks workflow
* Remove the prepare script
* Fixed tests and package versions
* Remove cli test script
* Remove packages from tailwind watch paths
* Add repo to public packages
* Just commit the generated files and do the building at dev time
* Try and get pkg.pr.new working
* Try again
* Fix emitDecoratorMetadata importing named export from typescript
* config file backwards compat with export const config
* Fixed issue where import errors weren’t coming through
* p-retry is a prod dep
* typescript needs to be a prod dependency for emitDecoratorMetadata
* Add better debug logging to help track down import-in-the-middle bug
* An external is only considered resolvable if it resolves to the same path as the collected external
* Fix runtime checks to allow >=18.20
* Move extensions to a new build package
* Fixed building packages in dockerfile
* Remove the e2e test from publish workflow for now
* Don’t treat pkg.pr.new versions has needing upgrading
* making sure config handleError works, and discovered path aliases don’t work in config files
* Strip empty string env vars so they accidentally override real values
* Couple of things
* Update version to use preview instead of beta
* Hopefully fix re-attempts with >30s delay
* Match socket emit messages to current latest in main
* Initial guide
* Go back to beta
* Go back to the preview, and update guide to use pr preview tags
* Go back to beta
---------
Co-authored-by: Matt Aitken <matt@mattaitken.com>
* Test for checkpoints
* Make sourceTaskAttemptId optional on resumeBatchRun
* Removed all completions/executions logic from the shared queue consumer
* Removed the sourceTaskAttemptId from ResumeBatchRunService
* Revert "Removed all completions/executions logic from the shared queue consumer"
This reverts commit d35398d50463c81a1975bb5d5bcfca66a24b8ede.
* WIP on triggerAndWait…
* Fixed triggerAndWait continuing when a checkpoint completes
* Remove the ResumeAttempt code that fails attempts (was protecting against infinite restores)
* Removed messageBody.data.completedAttemptIds.length === 0 commented out code
* Don’t ack if there’s no batchRun
* Added the marqs?.replaceMessage back in but NOT when there’s no checkpoint. More logging
This is a fix for when some attempts fail
* Improvement to the test task that now randomly fails attempts
* When a checkpoint happens, only continue the attempt if it’s in the correct state
* Changeset for rollback in branch
* Set keepRunAlive to false when the dependent task isn’t finished
* Changeset manual version (to get inline with the hotfix branch)
* Changeset: Fixes for continuing after waits
* Latest lockfile (after manual changeset version)
* Initial TaskRunConcurrencyTracker implementation
* MARQS calls a subscriber to events
* When enqueuing add the extra required metadata
* Track concurrency per environment for tasks too
* Admin page for global concurrency
* Use the new concurrency tracker on the tasks page
* Useful performance test task
* getAllTaskIdentifiers()
* New page for concurrency
* BackgroundWorkerTask index for quick lookup of task identifiers
* Added a way to get concurrency for environments
* Added upgrade/request more concurrency button
* Queued task column working
* Use defer and suspense
* Added queue column to the concurrency environments table
* Some comments added for clarity
* Fixed bad log message
* Sidemenu: move lower and rename to “Concurrency limits”
* Only show the environments, not tasks. Renamed to “Concurrency limits”
* PropertyTable component changed to use sub components
* Separate run span component
* Early WIP on tabs that use search query
* Shortcut key tabs for the span panel
* The span timeline is working
* When runs get expired, update the OTEL event with an error
* Improved the expired error message
* Reveal env vars when editing
* Tightened things up a bit
* Added detail tab properties
* Progress dashed line
* Move the env label next to the Run number title
* Top level cancel/replay buttons
* Replay with a different payload and environment
* Fix for non json payloads
* Hide the clear/copy buttons
* UI improvements with large payloads
* Close the panels when you replay/cancel
* Added the new timeline to spans
* Use the u-turn left icon for replay
* Added an index for spanId on TaskRun
* Remove replay/cancel buttons the span view
* Replay shortcut works inside the code editor
* Split the log/span inspector between Overview and Detail as well
* More improvements to the inspector
* Context and output improvements
* Focus on run working
* Added version to run.ctx
* Added some padding to the detail view
* Added context tab with shortcut
* Only load the replay data when the dialog is open
* Replaying uses the tags from the original run
* Links are now text links
* Removed version links for now because we don’t have dropdown filters for them yet
* Tabs are now outside of the scrollview
* The inspector is now 30% of the width by default
* Allow replaying and editing SuperJSON payloads
* Deleted unused CodeGroup file
* Increase the tags limit to 5, do the limiting on the server
* The admin tooltip now always shows basic org, project and user info
* Fix for schedule inspector disabled state layout
* Remove new unused span metadata and context
* Removed unused import
* TaskRunTag migration. Removed unused TaskTag
* Show tags for a run in the run list
* API endpoint for searching tags
* Let’s not expose an API endpoint for tags at the moment
* Filter by tags working from the URL
* Tag async filter working
* We don’t need a “None” option because it’s multi-select
* When triggering a run you can add tags
* SDK runs.retrieve and runs.list with tag support
* Tidied imports
* Changed the run list query so we show all the tags even if a run only matches one of them
* Fix for dealing with weird characters in tags
* Run tags changeset
* Creating a project has a proper loading state (and blocks multiple)
* Improved the error message for tag length
* Added a tags icon for display on the run screen
* Convenient functions for creating and getting run tags
* tags.set() from inside the run function
* Replay a run passes tags through
* Less ridiculous tags for the catalog example
* Allow passing just a string for the tags
* Order by id because there’s an index on the primary key already
* Trim the tags earlier so we don’t accidentally error if a blank string is passed
* Renamed some thing from setTags to addTags
* Use findFirst for the tags project lookup
* Added an index for "TaskRunTag"("name", "id")
It massively improves the performance of run filtering based on tags
* Use `array_agg` for the run list tags so pagination works and we get a single result for each run
* Tidied imports
* More comprehensive test of tags with all triggering functions
* Support tags with tasks.trigger, tasks.batchTrigger and tasks poll variants
* Added tasks.batchTrigger tags
* Sort the tags in the UI so they’re always in the same order
* Added tooltips in the run table for delay, ttl and tags
* Added costInCents, baseCostInCents and durationMs to runs.retrieve and runs.list
* Added support for displaying a split tag if you use key_value or key:value format
* Fix code comment
* Tweaked the JSDoc to make the prefixing clearer
* Added tasks.triggerAndWait to the tags test task
* Added type (STATIC or DYNAMIC) to TaskSchedule. Defaults to dynamic
* WIP with dev indexing of static schedules
* Added a code comment
* First stab at deleting unused static schedules
* Dashboard changes for the static schedules
* Generate the description. Upsert the instances when editing. Fix for the friendlyId
* Don’t allow deleting of static schedules
* Don’t allow enabling/disabling of static schedules
* Added filtering for schedule types
* Syncing of schedule for deployed tasks
* Static schedules are now created for each environment
* Added a second static schedule for testing
* Add the type to the schedule task run payload and the object you get back from the SDK
* Changed static/dynamic to declarative/imperative
* Timezone example
* Changeset
* Updated scheduled docs to include declarative
* When you test a schedule it set the type to “IMPERATIVE”
* Improved the tooltip
* Fix for queue time continuing to rise when a run is canceled/expired etc
* Update the info panel on a selected declarative schedule
* Check if there are no instances. This should never happen but log an error if it does
* Throw errors and push them through to the CLI dev command
* Fail deployments if creating the background tasks or schedules fails
* Format the deployment error so it gets displayed
* Changed the maxed out schedules error message to remove bit about support
Eric Allam