* WIP realtime streams
* Handle realtime with large payloads or outputs #1451
* feat: optimize Redis stream handling with batching
Add STREAM_ORIGIN to environment schema. Improve performance in
RealtimeStreams by using TextDecoderStream for simpler text
decoding and implementing batching of XADD commands for Redis
streams. Limit stream size using MAXLEN option. Update
environment variable repository with new variable type. Adjust
import statements for Redis key and value types.
* 🔧 chore: add dev dependencies for bundle analysis
* add metadata tests and a few more utilties
* Add stream tests and improve streaming
* Added AI tool tasks, descriptions to tasks
* Use the config file path to determine the workingDir, then the package.json path
* Remove stream test files
* useTaskTrigger react hook that allows triggering a task from the client
* Add streaming support for the realtime react hooks
* Add ability to stream results after useTaskTrigger
* Improve the stream throttling
* Use the runId as the ID key to bust the cache after triggering
* Upgrade to to the latest electric sql client and server
* Make realtime server backwards compat with 3.1.2 release
* Pass the runId into useRealtimeRun
* Fix scopes when specifiying reading all runs
* WIP @trigger.dev/rsc package
* Various fixes and accepted recommendations by CodeRabbit
* Regenerate pnpm lock file
* A couple tweaks to rsc and give up on rendering react in tasks for now
* Add changeset
* Remove triggerRequest from the useEffect deps
* Improve realtime & frontend authentication errors
* Fixed authorization tests
* Remove unnecessary log
* Add metadata.stream limits and improve the metadata streams structure
* Streams can now have up to 2500 entries
* Various coderabbit fixes
* additional react-hooks jsdocs
* Fixed empty env vars overriding in dev runs
* Don’t import package.json anymore
* fix node10 moduleResolution in @trigger.dev/core
* Support self-hosters pushing to a custom registry when running deploy
* dev: Fixed stuck runs when a child run fails with a process exit
* Make some doc notes about known issues and docker hub private repos
* Fix --project-ref when running deploy
* Fix —config option when deploying
* Fixing the flushing/killing process with the new build system
* Add monorepo-react-email e2e test fixture
* Fix issue with emitDecoratorMetadata and tsconfigs with extends
* Got the emit decorator metadata fixture working
* Fixed typechecking yarn e2e CLI tests in monorepos
* Add remote forced externals system, in case we come across another package that cannot be bundled (spurred on by header-generator)
* Remote externals now powered by JSON Hero to be easier to update
* resolve config source files
* Add a —javascript option to init, defaults to typescript
* Add support for prisma typed sql
* Remove msw and retry.interceptFetch
* Add missing code to the openai retries example
* Don’t generate the v3 catalog prisma client during CI
* Fixed v3-catalog task imports
* Remove interceptor usage in task file
* Only import import-in-the-middle hook if there are instrumented packages
* Fix yarn.lock file
* Support custom config file names & paths
* Fix entry point paths on windows
* Support custom conditions
Add support for custom conditions (for bundling and running), to support being able to import `ai/rsc` with the “react-server” condition.
- Fixed an issue where symlinking unresolvable externals after rebuilding caused the build to hang
- Fixed an issue with external not working with subpath exports (e.g. “ai” would not match “ai/rsc”)
- Protect better against build extensions breaking builds
* Add changeset
* Fix passing CLI process.env down to the task processes
* Remove unused import
* reviving the e2e CLI tests
* Another attempt at fixing windows
* yet another windows attempt (yawa)
* Output index child stdout and stderr (yawa)
* normalize import paths for windows
* Added some logging (yawa)
* normalize the loader path as well
* Added some logging to figure out why the entry points aren’t being found on windows
* Fix for entry point detection on windows
* Normalize runner import loader path
* Normalize import paths in dev and make sure rewritten build manifest paths are correct on windows as well
* Various cleanup after windows fixes
* Remove the webapp e2e for now
* Add node10 type resolution support for subpath exports (still does not actually import in Node 10 or Browserify/Parcel)
https://github.com/andrewbranch/example-subpath-exports-ts-compat
* init using templates again but downloaded from the repo this time
* Adding init schedule example
* Support for js init
* init now working with js
* Fix issues with links in terminals that don’t support them. Also skip update check of version starts with 0.0.0
* upgrade @opentelemetry packages to the latest versions
* remove v2 only packages, will be moved to a dedicated repo
* remove more v2 code and run pnpm install
* use the npm yalt package in the webapp
* convert @trigger.dev/core to tshy
* Switch from jest to vitest in @trigger.dev/core
* Fixed core test
* move core-backend code into core subpath export
* convert @trigger.dev/sdk to tshy
* Removed hono
* move core-apps to core/v3/apps, remove core-apps, start converting cli-v3
* Fix up some of the commands
* cli now building and loadable
* using package-json-from-dist to get package version now in core and cli
* dev command WIP
* cleaned up some repetition and structure of the entry point stuff
* bringing back the background worker stuff
* Indexing of the v3 catalog
* getting closer to executing dev runs...
* centralize dev logging using event emitter
* Move indexing to it’s own entry point, simplify code
* dev runs working
* Get instrumentation to work with openai
* debugging achieved internally
* provide worker files as part of the worker creation on the server
* support for cjs and esm javascript
* Fixed timeout
* worker manifest now has the config path
* auto-upgrade config to non-deprecated alternatives
* Adding package preview release
* deployment WIP
* improve the syncEnvVars output and adapt resolveEnvVars
* WIP bun runtime
* WIP bun support
* seed tasks with the machine preset if listed in the config
* deploy run executions WIP, extracted TaskRunProcess into 1 place
* deployed tasks running and executing 🎉
* support for waits and better flushing & process cleanup
* Fixed the heartbeating
* Better warning messages
* Improve and unify the indexing between dev and deploy
* Support for external deps that need node-gyp to build
* build extensions can now install custom packages and run instructions in the image. Also prisma extension now works and also works with multiple schema files
* Add back in the main/types/module to sdk
* dev no longer is Ink/React, grace period for disconnections in dev
* Fix the changeset config
* More changeset fixes
* Remove config packages
* More changeset fixes
* Fixed typescript issues (needed to revert back to zod 3.22.3
* Fix pr_checks workflow
* Remove the prepare script
* Fixed tests and package versions
* Remove cli test script
* Remove packages from tailwind watch paths
* Add repo to public packages
* Just commit the generated files and do the building at dev time
* Try and get pkg.pr.new working
* Try again
* Fix emitDecoratorMetadata importing named export from typescript
* config file backwards compat with export const config
* Fixed issue where import errors weren’t coming through
* p-retry is a prod dep
* typescript needs to be a prod dependency for emitDecoratorMetadata
* Add better debug logging to help track down import-in-the-middle bug
* An external is only considered resolvable if it resolves to the same path as the collected external
* Fix runtime checks to allow >=18.20
* Move extensions to a new build package
* Fixed building packages in dockerfile
* Remove the e2e test from publish workflow for now
* Don’t treat pkg.pr.new versions has needing upgrading
* making sure config handleError works, and discovered path aliases don’t work in config files
* Strip empty string env vars so they accidentally override real values
* Couple of things
* Update version to use preview instead of beta
* Hopefully fix re-attempts with >30s delay
* Match socket emit messages to current latest in main
* Initial guide
* Go back to beta
* Go back to the preview, and update guide to use pr preview tags
* Go back to beta
---------
Co-authored-by: Matt Aitken <matt@mattaitken.com>
* WIP env var management API
* Add import env var API endpoint
* Adding docs and support for using both API keys and PATs when interacting with the env var endpoints
* WIP envvar SDK
* Uploading env vars in a variety of formats now works
* Finish env var endpoints and add resolveEnvVars hook
* Add changeset
* Better error output when an ESM only package is required
* Add better errors for npm package errors, and bring deploy errors to the dev CLI
* Handle depot build errors and write out a temporary file to view the build logs
* Add changeset
* A bunch of changes to the config file:
- No more default export, the config is exported via the `config` named export
- `handleError` is moved to a separate named export
- Added the ability to include additional packages (e.g. wrangler) that aren’t included by default (think CLI usage)
- More reliably install packages in dev
- The config is now built first instead of directly imported, meaning we can now support typescript config files (`trigger.config.ts`)
* Update to trigger.config.ts in `trigger.dev init`
* WIP proxy/deploy
* WIP, registry proxy in express working
* A couple of notes, preparing for indexing
* Move the changes to prod-worker into the new file
* Deploy command working with indexing and runs (docker provider only for now)
* Removed ts-expect-error directive
* Fixed build command
* Support for batch triggers
* Various trace fixes and improvements
* Moved trace into the logger, fixed span cardinality issues with waits and triggers
* Add support for attaching a debugger in dev with —debugger flag
* Introducing Modular Asynchronous Reliable Queueing System (MarQS). Works in dev
* Convert MarQS to using lua and dealing with concurrency
* Simplified the timeout queue and current concurrency is now a set instead of a flat value (to support idempotency)
* Implement task heartbeating and reconnect the background workers CLI when the websocket connection reconnects
* Start adding internal telemetry support for the server
* Get env vars to work in dev and implement prisma tracing in webapp
* Cleanup telemetry and implement it in the consumer
* Implement dequeuing a message from a parent shared queue
* Implement a custom logger exporter instead of using console log exporter
* Use node instead of shell for generating protocol buffer code
* Propogate trace context into debug logs, and allow turning off logger exporter through env vars
* Switch to using baselime for internal otel data
* Make orgMember optional to fix type issues
* Provide the CLI dev env vars through the CLI, don’t build dotenv into facade
* Removed the logger import
* Address Matt’s comments
* Addressing more of Matt’s comments
* Handle sending an execution after a websocket connection closes
* Remove auth from the env attributes to prevent obfuscation
* feat: BYO Auth
Define client-side auth resolvers to be able to supply custom authentication credentials for integrations before a run is performed
- Added new defineAuthResolver
- Update all integrations to support the new auth resolvers
- Strip internal symbols from .d.ts in integrations and trigger-sdk
- Added BYO Auth docs
- Update Dynamic Schedule to support associated account IDs
- Create external accounts just-in-time
- Added Account ID field to test job when there are external auth integrations
- Show Account ID on run dashboard
- Added new Run error state called “Unresolved auth”
* Added changeset
* Remove @internal from TriggerIntegration public methods
* Add void to the result union
* DynamicTriggers now work with the new BYO auth system, and added a bunch of docs and docs changes
* Add additional key material for registering dynamic trigger task
* Add new define* instance methods to the overview
* Setup project-wide prettier
* Remove old workspace file
* Remove old debugging directives
* New top-level .prettierignore
* Updated Prettier config settings
* Contrubuting guide: Fix for some bad code blocks
* Added more ignores
* Improved the format script command
* printWidth set to 100
* Formatted entire repo (pnpm run format)
Eric Allam