* WIP realtime streams
* Handle realtime with large payloads or outputs #1451
* feat: optimize Redis stream handling with batching
Add STREAM_ORIGIN to environment schema. Improve performance in
RealtimeStreams by using TextDecoderStream for simpler text
decoding and implementing batching of XADD commands for Redis
streams. Limit stream size using MAXLEN option. Update
environment variable repository with new variable type. Adjust
import statements for Redis key and value types.
* 🔧 chore: add dev dependencies for bundle analysis
* add metadata tests and a few more utilties
* Add stream tests and improve streaming
* Added AI tool tasks, descriptions to tasks
* Use the config file path to determine the workingDir, then the package.json path
* Remove stream test files
* useTaskTrigger react hook that allows triggering a task from the client
* Add streaming support for the realtime react hooks
* Add ability to stream results after useTaskTrigger
* Improve the stream throttling
* Use the runId as the ID key to bust the cache after triggering
* Upgrade to to the latest electric sql client and server
* Make realtime server backwards compat with 3.1.2 release
* Pass the runId into useRealtimeRun
* Fix scopes when specifiying reading all runs
* WIP @trigger.dev/rsc package
* Various fixes and accepted recommendations by CodeRabbit
* Regenerate pnpm lock file
* A couple tweaks to rsc and give up on rendering react in tasks for now
* Add changeset
* Remove triggerRequest from the useEffect deps
* Improve realtime & frontend authentication errors
* Fixed authorization tests
* Remove unnecessary log
* Add metadata.stream limits and improve the metadata streams structure
* Streams can now have up to 2500 entries
* Various coderabbit fixes
* additional react-hooks jsdocs
* Denormalize run tags, increase character limit to 128
* WIP realtime subscribing to runs
* extracted the stream stuff into core, made it more reusable
* WIP tags
* Remove tags for now because it’s not support in electric
* Support async iterables, readable stream, and callback style subscription styles
* Remove tags streaming endpoint
* Add realtime rate limits and scope them to the /realtime path
* WIP rate limt per org
* Introduce per org rate limits
* WIP JWT auth
* Move migrations into new internal db package
* Resolve pnpm lock file
* Authenticating to the realtime API with JWTs are working
* realtime in the client
* Created react-hooks package and starting to move stuff in there
* Improve types for hooks
* schema tasks
* Added useBatch hook
* build uploadthing/fal demo and change how run metadata is synced to the server
* tweaks
* WIL realtime concurrency tracking
* Implement test for realtime client using testcontainers
also updated electric to latest version
* Allow customizing the expiration time of the automatic JWT created after triggering a task
* Add support for subscribing to run tags
* Improve auth types and API
* finalize the realtime API
* Fixed some example stuff
* Allow up to 10 run tags
* Remove core from docker-provider tsconfig paths to prevent it from being typechecked
* do the same for the kubernetes provider
* Fixing some typecheck errors
* Fix webapp type errors
* Update @trigger.dev/platform to 1.0.13
* Fix attw error
* Remove from/to in subscribeToRuns query params
* Add tests for the rate limit middleware and add custom JWT rate limits
* turn off webapp test parallelism
* Finish renaming jwt -> publicAccessToken and automatically give the JWT read access to the tags when using trigger
* Add changeset
* Attempt to fix unit tests in CI
* Skip running the auth rate limit middleware tests for now
* Try a beefier machine
* Try and run webapp tests separately
* Setup env vars
* Make sliding window test more reliabile
* Fixed empty env vars overriding in dev runs
* Don’t import package.json anymore
* fix node10 moduleResolution in @trigger.dev/core
* Support self-hosters pushing to a custom registry when running deploy
* dev: Fixed stuck runs when a child run fails with a process exit
* Make some doc notes about known issues and docker hub private repos
* Fix --project-ref when running deploy
* Fix —config option when deploying
* Fixing the flushing/killing process with the new build system
* Add monorepo-react-email e2e test fixture
* Fix issue with emitDecoratorMetadata and tsconfigs with extends
* Got the emit decorator metadata fixture working
* Fixed typechecking yarn e2e CLI tests in monorepos
* Add remote forced externals system, in case we come across another package that cannot be bundled (spurred on by header-generator)
* Remote externals now powered by JSON Hero to be easier to update
* resolve config source files
* Add a —javascript option to init, defaults to typescript
* Add support for prisma typed sql
* Remove msw and retry.interceptFetch
* Add missing code to the openai retries example
* Don’t generate the v3 catalog prisma client during CI
* Fixed v3-catalog task imports
* Remove interceptor usage in task file
* Only import import-in-the-middle hook if there are instrumented packages
* Fix yarn.lock file
* Support custom config file names & paths
* Fix entry point paths on windows
* Support custom conditions
Add support for custom conditions (for bundling and running), to support being able to import `ai/rsc` with the “react-server” condition.
- Fixed an issue where symlinking unresolvable externals after rebuilding caused the build to hang
- Fixed an issue with external not working with subpath exports (e.g. “ai” would not match “ai/rsc”)
- Protect better against build extensions breaking builds
* Add changeset
* Fix passing CLI process.env down to the task processes
* Remove unused import
* reviving the e2e CLI tests
* Another attempt at fixing windows
* yet another windows attempt (yawa)
* Output index child stdout and stderr (yawa)
* normalize import paths for windows
* Added some logging (yawa)
* normalize the loader path as well
* Added some logging to figure out why the entry points aren’t being found on windows
* Fix for entry point detection on windows
* Normalize runner import loader path
* Normalize import paths in dev and make sure rewritten build manifest paths are correct on windows as well
* Various cleanup after windows fixes
* Remove the webapp e2e for now
* Add node10 type resolution support for subpath exports (still does not actually import in Node 10 or Browserify/Parcel)
https://github.com/andrewbranch/example-subpath-exports-ts-compat
* init using templates again but downloaded from the repo this time
* Adding init schedule example
* Support for js init
* init now working with js
* Fix issues with links in terminals that don’t support them. Also skip update check of version starts with 0.0.0
* upgrade @opentelemetry packages to the latest versions
* remove v2 only packages, will be moved to a dedicated repo
* remove more v2 code and run pnpm install
* use the npm yalt package in the webapp
* convert @trigger.dev/core to tshy
* Switch from jest to vitest in @trigger.dev/core
* Fixed core test
* move core-backend code into core subpath export
* convert @trigger.dev/sdk to tshy
* Removed hono
* move core-apps to core/v3/apps, remove core-apps, start converting cli-v3
* Fix up some of the commands
* cli now building and loadable
* using package-json-from-dist to get package version now in core and cli
* dev command WIP
* cleaned up some repetition and structure of the entry point stuff
* bringing back the background worker stuff
* Indexing of the v3 catalog
* getting closer to executing dev runs...
* centralize dev logging using event emitter
* Move indexing to it’s own entry point, simplify code
* dev runs working
* Get instrumentation to work with openai
* debugging achieved internally
* provide worker files as part of the worker creation on the server
* support for cjs and esm javascript
* Fixed timeout
* worker manifest now has the config path
* auto-upgrade config to non-deprecated alternatives
* Adding package preview release
* deployment WIP
* improve the syncEnvVars output and adapt resolveEnvVars
* WIP bun runtime
* WIP bun support
* seed tasks with the machine preset if listed in the config
* deploy run executions WIP, extracted TaskRunProcess into 1 place
* deployed tasks running and executing 🎉
* support for waits and better flushing & process cleanup
* Fixed the heartbeating
* Better warning messages
* Improve and unify the indexing between dev and deploy
* Support for external deps that need node-gyp to build
* build extensions can now install custom packages and run instructions in the image. Also prisma extension now works and also works with multiple schema files
* Add back in the main/types/module to sdk
* dev no longer is Ink/React, grace period for disconnections in dev
* Fix the changeset config
* More changeset fixes
* Remove config packages
* More changeset fixes
* Fixed typescript issues (needed to revert back to zod 3.22.3
* Fix pr_checks workflow
* Remove the prepare script
* Fixed tests and package versions
* Remove cli test script
* Remove packages from tailwind watch paths
* Add repo to public packages
* Just commit the generated files and do the building at dev time
* Try and get pkg.pr.new working
* Try again
* Fix emitDecoratorMetadata importing named export from typescript
* config file backwards compat with export const config
* Fixed issue where import errors weren’t coming through
* p-retry is a prod dep
* typescript needs to be a prod dependency for emitDecoratorMetadata
* Add better debug logging to help track down import-in-the-middle bug
* An external is only considered resolvable if it resolves to the same path as the collected external
* Fix runtime checks to allow >=18.20
* Move extensions to a new build package
* Fixed building packages in dockerfile
* Remove the e2e test from publish workflow for now
* Don’t treat pkg.pr.new versions has needing upgrading
* making sure config handleError works, and discovered path aliases don’t work in config files
* Strip empty string env vars so they accidentally override real values
* Couple of things
* Update version to use preview instead of beta
* Hopefully fix re-attempts with >30s delay
* Match socket emit messages to current latest in main
* Initial guide
* Go back to beta
* Go back to the preview, and update guide to use pr preview tags
* Go back to beta
---------
Co-authored-by: Matt Aitken <matt@mattaitken.com>
* WIP env var management API
* Add import env var API endpoint
* Adding docs and support for using both API keys and PATs when interacting with the env var endpoints
* WIP envvar SDK
* Uploading env vars in a variety of formats now works
* Finish env var endpoints and add resolveEnvVars hook
* Add changeset
* Better error output when an ESM only package is required
* Add better errors for npm package errors, and bring deploy errors to the dev CLI
* Handle depot build errors and write out a temporary file to view the build logs
* Add changeset
* A bunch of changes to the config file:
- No more default export, the config is exported via the `config` named export
- `handleError` is moved to a separate named export
- Added the ability to include additional packages (e.g. wrangler) that aren’t included by default (think CLI usage)
- More reliably install packages in dev
- The config is now built first instead of directly imported, meaning we can now support typescript config files (`trigger.config.ts`)
* Update to trigger.config.ts in `trigger.dev init`
* WIP proxy/deploy
* WIP, registry proxy in express working
* A couple of notes, preparing for indexing
* Move the changes to prod-worker into the new file
* Deploy command working with indexing and runs (docker provider only for now)
* Removed ts-expect-error directive
* Fixed build command
* Support for batch triggers
* Various trace fixes and improvements
* Moved trace into the logger, fixed span cardinality issues with waits and triggers
* Add support for attaching a debugger in dev with —debugger flag
* Introducing Modular Asynchronous Reliable Queueing System (MarQS). Works in dev
* Convert MarQS to using lua and dealing with concurrency
* Simplified the timeout queue and current concurrency is now a set instead of a flat value (to support idempotency)
* Implement task heartbeating and reconnect the background workers CLI when the websocket connection reconnects
* Start adding internal telemetry support for the server
* Get env vars to work in dev and implement prisma tracing in webapp
* Cleanup telemetry and implement it in the consumer
* Implement dequeuing a message from a parent shared queue
* Implement a custom logger exporter instead of using console log exporter
* Use node instead of shell for generating protocol buffer code
* Propogate trace context into debug logs, and allow turning off logger exporter through env vars
* Switch to using baselime for internal otel data
* Make orgMember optional to fix type issues
* Provide the CLI dev env vars through the CLI, don’t build dotenv into facade
* Removed the logger import
* Address Matt’s comments
* Addressing more of Matt’s comments
* Handle sending an execution after a websocket connection closes
* Remove auth from the env attributes to prevent obfuscation
* Setup basic storybook page
* storybook preview pages don’t have padding
* Added a storybook story for the side menu
* WIP on a new side menu component
* New variant for the side menu button style
* Side menu items can now recieve a badge ‘count’
* Storybook side menu update
* Added a new font size
* you can now set the spacing between an icon and the text
* Moving the error and count badge to the right side of the button
* Added a menu sub item button style
* added sub items to the menu and all the rest of the items
* Proper background class used
* Added a side menu header and main org header
* Removed the no-mobile blocker overlay so you can use the site on a phone
* Better section name
* Created a cusom popover trigger that can take anything as children. Also the popover titles can now take a variant
* Added a popover for the user avatar menu item
* Removed the tool tip
* Arrow Popover can no also go full width
* neater conditiona;
* Removed unnessary div. And error icon can display an error message
* menu item buttons can take a variant
* Added placeholder items for the org dropdown
* add a max height to the org menu popover and handle scrollbars
* Added a border if the menu scrolls when you have lots of project items
* warning badges display tooltips if you give them some error text
* Removed the sticky header so the scrollbar looks better
* Fixed a layout issue from removing the sticky header
* Better spacing between lists
* Popover can take any react component icon and an icon classname
* Side menu headers now have a dropdown by clicking the ellipsis icon
* Side menu items can take an icon colour
* Updated styling for the dropdown Popover menus
* Installed Tabler icons
* Updated some icons to use Tabler
* Started defining the HttpTrigger class
* RequestFilterSchema is part of core
* Use the RequestFilterSchema from core
* EventSpecificationExample now derived from schema
* Need to define the class like EventTrigger
* Catalog reference that uses the httpTrigger
* The logic for the httpTrigger is in the separate file, just call it from the client using defineHttpTrigger
* Basic HttpTrigger
* Import fixes
* Transforming httpTrigger to the format we want
* Initial httpTrigger types working
* Added HTTP generic type
* Added some notes
* Rejigged some things to support verifying payloads
* Use new security function for verifying
* Added truncating text support if side menu names get long
* Renamed some stuff
* Removed test copy
* HttpEndpoin that isn’t a trigger
* Renamed hostname to source
* Created HttpEndpoint as a top-level concept, it produces triggers using onRequest
* Started working on registration of the httpEndpoints
* Removed the new TriggerMetadata for now, http will use static
* Added PayloadType to EventRecord. Default is JSON.
* The payload from an HTTPEndpoint is now a Request
* Initial TriggerHttpEndpoint migrations
* Initial server side work
* Migration to add shortcode to RuntimeEnvironment
* Added TriggerHttpEndpointEnvironment table
* Made the httptrigger catalog example runnable
* Progress creating/updating httpEndpoints
* HTTPEndpointEnvironment updating now works
* Added a note about EventRecord HttpEndpoint relationship
* When creating environments, create a shortcode
* Fix for secret not being retrieved correctly
* Request filtering
* Add DOM types to Core
* Early work dealing with a webhook arriving
* Added some todo notes to discuss tomorrow
* Added relations to the endpoint so we can call the user’s server
* Basic handling of immediate responses
* The immediate responses are now going back to the webhook provider
* Handling is now generating events and responding where relevant. Not yet matching filters and doing verification
* Added source to TriggerHttpEndpointEnvironment
* The event filter is now optional
* The source is passed through from the httpEndpointEnv
* Added optional payloadType when ingesting an event
* Added tabler
* Updated @remix-run/testing usage for remix 2
* Fixed org menu header width and added org and project types
* Added custom icons for Runs and Endpoints
* Updated the menu icons
* storybook side menu shows static project and org data
* Updated side menu
* Plumbed in paths
* Updated eventFilter to deal with undefined/null payloads
* Started work on filtering, but it’s not working
* Strict null
* async filtering it working
* Added environment shortcodes to the seedCloud script
* Don’t pass the secret back, added a verifyPayload function that gets called before runs start
* Fix for skipping triggering runs
* The verify callback is now passed through to the trigger so it can be called
* If verifying fails, have a reason which we can show to the user
* Don’t verify if it’s a test run
* Fixed the log in the example code
* Added loader and action to storybook preview
* Fix for svg attribute name issues in EndpointIcon
* Started work on the side menu
* This logic has moved to a separate file, in a different branch
* Don’t get the environments when getting orgs
* Lots of fixes for the new SideMenu
* Refactored some org/project things to fit the new structure better
* Some progress restructuring the app for the new sidebar
* More changes to data loading
* Deleted some old pages. The redirecting is working if you’re on the root but logged in
* Basic styling for the account page. We need a back button
* Fixes for the integrations pages
* Breadcrumb implemented
* Add a border to the bottom of the breadcrumb
* Feedback is working, not in Storybook
* Removed expandSidebar from the Handle
* Org level breadcrumbs
* Getting the project in the org route now works properly
* orgslug route before using the presenter
* More UI improvements
* Removed commented out line
* Use presenters for org and root
* Removed forceActive
* Split the project selector and user menu out
* The side menu items have correct active states
* Fixed the scrolling
* Put the org name in
* Improved the admin features
* Created the HTTP Endpoints list page
* Work on the HTTP endpoint page
* WIP on the HTTP Endpoint page
* Fix for response filter not showing
* Change page URLs to “http-endpoints” (from httpendpoints)
* Slugify the HTTP endpoint id
* Added todos for EventRecord associations
* Removed the Job/Trigger tab for now
* First draft of the help panels
* Rejigged the HTTP endpoint page so it splits out the deployed and not deployed environments
* Removed irrelevant columns
* Added a link to the deployment guide
* Removed the background image from the frameworks pages
Also…
- reordered the frameworks so supported ones come first in the list
- added logos to 3 framework pages that were missing them
- removed lots of un-used imports
* Fixed an issue where the breadcrumb could be hidden on small screens so broken the pages
The breadcrumb is always visible now
* Renamed ‘Onboarding’ to ‘Framework setup’
* Added a new http-endpoint icon to the side menu
* Side menu now has a nice looking scrollbar
* Scrollbar on the usage and billing page updated to the nice style
* Icon and colour updates to the side bar menu items
* Small copy improvement
* Fixed bug with the height of the org menu dropdown
* Page breadcrumbs now don’t include duplicate path links
* renamed uiBorder to ui-border
* Remove the padding from the small menu item if it’s used as a button
* Made the paragraph text a bit smaller
* EventRecord has relationships with TriggerHttpEndpoint/TriggerHttpEndpointEnvironment
* Improved the presenter used by the side menu. Also added the HTTP Endpoint badge count to the sidebar
* The dropdown org menu has the same badge as the side menu
* Improved the Projects page
* Fixed button style miss-match in the jobs table dropdown
* Made the popover dialogs support Headers and improved the regenerate API key popover
* Made the active status of a job more subtle
* Allow the Job to have a link on its Trigger
* Simplified loader because we ditched the Jobs menu. Also showing no runs help panel now
* Fixed type errors
* Pages that are org level can scroll if the content gets too long
* Fixed a redirect bug so you now redirect to the org page
* Updated the shortcode migration, to add additional letters to the end
* Removed the nav bar from the welcome onboarding page
* Only show the countBadge if the number is > 0
* Jobs icon now has a default colour the same as the nav menu
* Capital I for integration
* Clone the request before calling verify
* Separate the checks out so they can be easily logged
* Make the failure reason optional
* Made the verification error look better, if it’s provided
* Working WhatsApp example
* Can set the immediateResponseFilter to null in an update
* Deal with a null immediateResponseFilter
* Cal.com webhook working
* Header height reduced slightly
* Fixed button padding
* Removed un-used imports
* http endpoint table displays a message if you have no entries in the table
* Removed some duplicate text and swapped primary buttons for secondary style
* All ‘HTTP Endpoints’ now ‘HTTP endpoints’
* You have to use a RequestFilter when using HTTP Trigger
* Synchronous eventDispatcher filtering again. Better content-type check
* Stripe example
* Side menu “Usage & Billing” is just “Usage” if self-hosting
* Fix for filtering not working
* Removed filter log
* Fix for jobs being optional and null
* Removed old nav bar from the invite page
* Deleted old nav bar component
* import cleanup
* NoTask message only displays if the run has status “SUCCESS”
* Added a global page loading indicator
* Added documentation buttons to the most important pages
* Added GitHub
* Use “verifyRequestSignature” helpr
* Removed todo
* Fixed the layout of the Rerun Job popover
* Jobs table has margin bottom so it sits off the bottom of the page
* Small improvement to the search bar layout on the integration page
* HTTP endpoint and Trigger concept docs
* Reference docs and JSDocs for HTTP endpoint
* SDK reference docs for HttpEndpoint
* Added an HttpTrigger SDK reference, which mainly just point you elsewhere
* Improved the copy in the help panels
* Removed unused code
* Improved HTTP endpoint comment about setting the secret
* Delete unused asyncFilter
* Fixed the delete modal not having a title
* Always try get the arrayBuffer for an HTTP request, catch and use undefined as the default
* Add extra method to invokeTrigger
* Use a transaction for the secret store
* verifyRequestSignature docs page and JSDocs
* Added verifyRequestSignature docs page to nav
* Warn if they use the same HttpEndpoint id twice
* Added another job using the same HttpEndpoint
* Renamed the api route to http-endpoints from httpendpoints
* Debug bar for VSCode is now in the commandCenter
---------
Co-authored-by: James Ritchie <james@jamesritchie.co.uk>
commit c028db229cece2bf7d7acf04e7b1c08fea61798c
Author: Matt Aitken <matt@mattaitken.com>
Date: Fri Oct 13 16:04:54 2023 +0100
Some fixes for getting the endpoint id
commit e31b7ac5f2d451f70e3d42a3fb3e5f718d93485b
Author: Matt Aitken <matt@mattaitken.com>
Date: Fri Oct 13 15:39:18 2023 +0100
Added a link in the instructions if Node isn’t detected
commit 1a59cced775fbeac3e5b94e0c151b490c838d8f8
Author: Matt Aitken <matt@mattaitken.com>
Date: Fri Oct 13 15:39:05 2023 +0100
Fixed some conflict issues
commit 77263ed26bcf7025071ac0faf15fb867fff245d0
Author: Matt Aitken <matt@mattaitken.com>
Date: Fri Oct 13 15:38:56 2023 +0100
Added a changeset
commit 1a7df3f37f0c65d68cb9a9fe7ac50328bbb4d879
Author: GuyBorderless <100129659+guy-borderless@users.noreply.github.com>
Date: Fri Oct 13 17:31:01 2023 +0300
feat: enable deno (#531)
* adding deno reference project and JSRuntime file
* accessing runtime specific functions through JsRuntime
* adding support for both deno.json and deno.jsonc
* no automatic setup for deno projects currently
---------
Co-authored-by: Guy Kedem <guy@propertiz.net>
Co-authored-by: Matt Aitken <matt@mattaitken.com>
* feat: BYO Auth
Define client-side auth resolvers to be able to supply custom authentication credentials for integrations before a run is performed
- Added new defineAuthResolver
- Update all integrations to support the new auth resolvers
- Strip internal symbols from .d.ts in integrations and trigger-sdk
- Added BYO Auth docs
- Update Dynamic Schedule to support associated account IDs
- Create external accounts just-in-time
- Added Account ID field to test job when there are external auth integrations
- Show Account ID on run dashboard
- Added new Run error state called “Unresolved auth”
* Added changeset
* Remove @internal from TriggerIntegration public methods
* Add void to the result union
* DynamicTriggers now work with the new BYO auth system, and added a bunch of docs and docs changes
* Add additional key material for registering dynamic trigger task
* Add new define* instance methods to the overview
* Setup project-wide prettier
* Remove old workspace file
* Remove old debugging directives
* New top-level .prettierignore
* Updated Prettier config settings
* Contrubuting guide: Fix for some bad code blocks
* Added more ignores
* Improved the format script command
* printWidth set to 100
* Formatted entire repo (pnpm run format)
Eric Allam