* feat: BYO Auth
Define client-side auth resolvers to be able to supply custom authentication credentials for integrations before a run is performed
- Added new defineAuthResolver
- Update all integrations to support the new auth resolvers
- Strip internal symbols from .d.ts in integrations and trigger-sdk
- Added BYO Auth docs
- Update Dynamic Schedule to support associated account IDs
- Create external accounts just-in-time
- Added Account ID field to test job when there are external auth integrations
- Show Account ID on run dashboard
- Added new Run error state called “Unresolved auth”
* Added changeset
* Remove @internal from TriggerIntegration public methods
* Add void to the result union
* DynamicTriggers now work with the new BYO auth system, and added a bunch of docs and docs changes
* Add additional key material for registering dynamic trigger task
* Add new define* instance methods to the overview
* Added stripInternal to SDK tsconfig
* Statuses can now be set from a run, and are stored in the database
* Added the key to the returned status
* Made the test job have an extra step and only pass in some of the options
* client.getRunStatuses() and the corresponding endpoint
* client.getRun() now includes status info
* Fixed circular dependency schema
* Translate null to undefined
* Added the react package to the nextjs-reference tsconfig
* Removed unused OpenAI integration from nextjs-reference project
* New hooks for getting the statuses
* Disabled most of the nextjs-reference jobs
* Updated the hooks UI
* Updated the endpoints to deal with null statuses values
* The hook is working, with an example
* Changeset: “You can create statuses in your Jobs that can then be read using React hooks”
* Changeset config is back to the old changelog style
* WIP on new React hooks guide
* Guide docs for the new hooks
* Added the status hooks to the React hooks guide
* Removed the links to the status hooks reference for now
* Re-ordered the hooks
* Fix for an error in the docs
* Set a default of a blank array for the GetRunSchema
* updated the @trigger.dev/astro package to support typescript
* set up a example astro project
* docs: manual setup of Trigger.dev in an Astro project
* doc: updated the manual setup docs for astro
* updated the webapp onboarding for astro
* cleaned up the example astro project
* removed next from dependency 😅
* updated the readme
* moved trigger.ts file and job folder to src directory
* created a .env.example file
* updated tsconfig
* updated to use aliases
* updated package.json
* updated the astro onboarding page
* updated the manual installation guide for astro
* updated to use alias
* added astro as a dev dependency
* Remove extra comma I left in
* Changed Object.create() to a Record, so we have some type safety
* Added the localhost URL to the API in the Astro example
* Added the TRIGGER_API_URL to the trigger client in the example
* Added the CLI dev dependency to the Astro example
* Update the running instructions to be closer to Remix's
* Create friendly-carpets-collect.md
* Fix bad package name for Astro
---------
Co-authored-by: Matt Aitken <matt@mattaitken.com>
* init remix project
* feat: implemented the remix adapter package
* updated the remix package to return the action variable
* added an example job to the example remix project
* fix: Ensure Side Effects Execution in Remix API Routes
* updated the readme in @trigger.dev/remix package
* doc: added the manual setup guide for remix
* updated the webapp onboarding for remix
* updated the manual setup guide
* added a license to the @trigger.dev/remix package
* added a wait function to the example job
* added a link to the remix webapp onboarding
* Updated the onboarding page
For now, we'll only support existing projects. The button that links to the manual installation was just a blank rectangle – added some text to it.
* Added the Server API key to the instructions
* Remix manual docs
Made some things a bit clearer
- file paths
- exporting the job rather than using a function wrapper
- how to run the app and CLI dev command
* Export the job
* Update api.trigger.ts
* Added tsconfig.json paths
* Remove next dependency
* Package set to 2.1.0 and set dependencies
Shouldn't rely on a specific version in main dependencies. In this case we can just have devDependency for @remix/node
* Changeset
---------
Co-authored-by: Matt Aitken <matt@mattaitken.com>
* Removed next references in the ReadMe
* FAQ improvements to update Next.js references
* Added a new Frameworks page with logos to link to the quick start guides
* Renamed the mdx file so the link works
* Added basic platform guides
* Introduction doesn’t reference next.js only
* Added a note about serverless and restructured the side menu
* Added more frameworks pages to the manual install section
* Fixed SVG error
* Coming soon guides now using snippets so they can be in 2 places while we work on them
* Removed redirects
* Added a new named icon
* increased fireworks particle count slightly
* Creating a route for the onboarding and link to access it anytime from the side menu
* Moved the onboarding into new components
* Updated the routes for the onboarding page
* Onboarding is now Setup and the frameworks overview page now links to different framework page in the onboarding side nav
* URLs and pages renamed to ‘setup’
* URLs and pages renamed to ‘setup’
* Fixed some old named component imports
* WIP on a new less hacked-in background image
* Added svg logos for all the frameworks
* New paths for the new framework pages
* Side menu collapses once you selected a framework
* Temporary Nuxt page
* New routes for the new frameworks
* New ‘framework coming soon’ component - now used in the coming soon framework pages
* Moved/removed files
* Moved all svg framework logos into a new folder and file
* balanced the logo sizes
* Framework coming soon pages take a name and url
* New express svg logo
* New component for a Framework item in the grid
* Added information and design for the coming soon frameworks
* Change the grid to 1 row if jobs === 0
* Updated the framework coming soon snippet and added it to the relevant pages
* Express now 2nd in all lists
* Added link to long running server support discussion
* New logo to include next.js + supabase quick start
* roadmap now links to homepage with page anchor to roadmap section
* changelog now links to Github releases page
* Removed reference to Next.js
* Docs pages with coming soon info all share the same framework snippets
* Fixed issue of missing props
* Moved the fireworks effect into a separate file to make it easier for adding more framework onboarding pages
* Info callouts have a more muted style
* Added a callout to the top of the next onboarding to reference Serverless-only support
* Moved the callout explaining scheduled triggers not working in dev to the bottom of the test panel
This is to resolve eric’s UX ticket: https://github.com/triggerdotdev/trigger.dev/issues/416
* Added groundwork for Nest.js framework
* Using framer motion for the menu transition instead of css - now has a slight spring and is snappier
* redirect the manual setup page to the new nextjs setup page
* Fixed broken docs link
* Added a readme.md to next.js package
* Capital I for integration
* Renamed component: NextDevCommand to RunDevCommand
* Made required props non-optional
* CLI now points at manual install guides if no Next project detected
* Explicitly made supported default to false (it was working but a bit hard to read)
* Added links to all github issues for the frameworks coming soon component
---------
Co-authored-by: Matt Aitken <matt@mattaitken.com>
* fix: use NODE_ENV when starting from docker image
When starting from `pnpm run start`, NODE_ENV is fixed to production, causing problems like #186.
So, I added `start_docker`, a process dedicated to docker, and started from it.
The process is exactly the same as `start` except that NODE_ENV is not overwritten.
* fix: align execution name with others
* Change: command name to match review
* CLI create-integration command now accepts an Open AI api key
* Create integration docs separated into multiple pages
* Initial Airtable integration commit, with OpenAI generated code
* OAuth page coming soon
* Export DisplayProperty from the SDK
* TSConfig made to match GitHub’s with paths
* getRecords
* Removed duplicate Stripe job from the catalog
* Renamed Airtable apiKey option to token
* First Airtable job
* Export Collaborator and Attachment field types
* A typesafe example that uses runTask
* WIP on new integration tasks… not working yet
* Attempt with class
* Revert "Attempt with class"
This reverts commit 93a48330019f754c3216c5b49964fa4b0218bd3f.
* WIP changing how tasks work
* Mock of async local storage
* Moved client creation from constructor
* New approach with a clone method on TriggerIntegration
* Added runTask to Airtable which is used by integration tasks
* Added the Airtable icon and connection when using runTask
* base().table() is working
* runTask options moved to the 3rd param, made optional with optional name
* Added some generic arguments
* Added generic type to table
* Removed old comment
* We don’t need to repeat the icon
* getRecords and getRecord now returning the right data and types
* Creating records
* Update records
* Delete records
* The internal properties of integrations are now hidden by the TypeScript types
* Sprinkled a Prettify in there
* Improved the types
* Added Airtable to the integration catalog
* Early work on Airtable webhook registration
* More progress with webhooks
* connectionKey needs to be cloned for webhooks to work
* connectionKey needs to be cloned for webhooks to work
* It was unclear that the ActivateSourceService was using a graphileJob id
* ActivateSourceService optionally takes a jobId, if missing it generate a unique id
* When retrying trigger registration, don’t pass an id so it is generated
* Removed Airtable webhooks tasks from the job-catalog example
* Added TriggerSourceOption, removed TriggerSourceEvent
* WIP with new ExternalSource options
* ExternalSourceTrigger setup
* DynamicTrigger changed to options, will need some more work
* filter gets options passed to it
* SourceMetadata v2 renamed to SourceMetadataV2, kept original
* Started versioning the backend
* Moved param order on io.getEvent and io.cancelEvent
* The runTask stuff that allows unknown to work is back
* Indexing for v1 and v2, with version on “activateSource” schema
* Added todos, to deal with Airtable SDK calls inside the webhook handler
* “deliverHttpSourceRequest” queueName changed to the source id so they process in order
* ActivateSource changes to deal with old and new data formats
* Update existing TriggerSources to v2
* Fix for dynamic.ts typescript errors, need to revisit this later
* UpdateSourceService v1 and v2, with new v2 API endpoint
* Removed unused imports
* More progress on v1 and v2
* Airtable webhooks are now triggering a job
* Moved webhooks to a new file
* You can do API calls in the webhook handler now, Airtable webhook data is being processed
* Airtable events coming through
* Defined the Airtable table payload type
* TriggerSource metadata is being stored and used
* Removed some logs
* Added filtering and don’t allow any webhooks that use automated sources
* Resend switched to new integration
* Moved Resend test jobs to the catalog, and tested it worked
* WIP on Slack, there are compile errors
* Created a generic type that strips out indexes
* Slack updated to use new integration
* SendGrid migrated over
* Integration runTask is now allowing regular types
* Changed io.runTask types so it only allows Json-able types
* OpenAI models tasks working
* Added Airtable changes to runTask
* Removed the index signature crap from the Slack integration
* Don’t need to cast the callback result
* Updated Resend
* Re-ordered runTask params
* WIP on openai
* onAccountUpdated is Connect only
* Removed RunTaskResult
* Handle Resend errors, the official SDK doesn’t expose them properly at the moment
* Removed OmitIndexSignature
* OpenAI converted to new integration, with backwards compatible functions
* Put the openai catalog back to what it was originally
* Export a standard retry with backoff, to be used
* Use the standard exponential backoff in the integrations
* Retry options moved earlier so they can be overriden by a task
* GitHub tasks migrated
* Added sources, fixed one bundling issue
* Added GitHub jobs to catalog
* Remove duplicate options
* Deduplicate events
* Removed duplicate Job
* Switched Plain over
* Set the Plain icon
* Converted Stripe over
* Supabase adapted
* Typeform working
* Added dynamic-schedule to catalog
* Added background-fetch job catalog
* Created dynamic-triggers catalog file
* Fixed old general file with runTask param order
* Dynamic triggers working
* SendGrid updated to use the same tsconfig as other integrations
* Removed Airtable webhook, until we have batch support
* Added OAuth airtable auth example
* Created beta changeset tag
* Beta changesets for most packages
---------
Co-authored-by: Eric Allam <eallam@icloud.com>
This commit fixes various issues with run executions, including a pretty gnarly memory bloat issue when resuming a run that had a decent number of completed tasks (e.g. anything over a few). Other issues fixed:
- Run executions no longer are bound to a queue, which will allow more parallel runs in a single job (instead of 1).
- Serverless function timeouts (504) errors are now handled better, and no longer are retried using the graphile worker failure/retry mechanism (causing massive delays).
- Fixed the job_key design of the performRunExecutionV2 task, which will ensure resumed runs are executed
- Added a mechanism to measure the amount of execution time a given run has accrued, and added a maximum execution duration on the org to be able to limit total execution time for a single run
Performance degradation came from the syntax highlighting of large code blocks and from doing that on the server and the client, so fixed this in a couple of ways:
1. Stream the task details data using defer and Suspense/Await
2. Skipped syntax highlighting code blocks with more than 1k lines
* Don’t show the Ready To Run Job prompt if you have an Integration that needs attention
* Don’t highlight the row red or green
* Improvements to the contrast of the app sections and dividers
* Removed the Jobs page title as it’s duped info
* using the new border variable
* Sticky last table cell
* Improved the sticky last table cell
* Make any last cell in a table sticky by adding isSticky to it
* Added a dropdown menu to the menu table cell
* table rows can be marked as disabled by adding disabled
* Using the jobTestPath function for the test path
* tidy up imports
* Removed un-used props
* Removed the green badge variant
* Added a new status badge to the Runs table
* Clicking the gradient clicks the row
* Delete Job triggers a modal popup
* Added a large danger button type
* Added some modal styling and started adding data
* Added more styling and data to the delete job modal
* A table can now be given a full width prop
* Large danger button added to Storybook
* Danger button disabled state looks disabled now
* New active badge component to display in the table and logic for showing the env data
* Style updates to the dialog component
* active and job status badges can now have a small size
* Added a new named icon
* The Job page shows the Job status in the PageInfoRow
* Small badge style update
* Runs table has a sticky right cell
* Created a JobStatusTable component
* Added some placeholder help panel content for disabling a Job
* WIP creating a Settings page
* Added a delete button that triggers the delete modal – just need data hooking up
* Implemented deleting jobs from the dashboard
---------
Co-authored-by: Eric Allam <eallam@icloud.com>
* WIP job run performance improvements
- Added a `perf` tool to better measure job run performance under heavy load
- Removed `runFinished` job (not really needed)
- startQueuedRuns now uses a jobKey with replace
- Fixed an issue with ZodWorker when using jobKey
* Publish improvement docker images
* fixed the improvement docker publishing
* Downgrade back to prisma 4.16.0 because 5.1.x broke docker builds
* Changes to how queued runs work
- Split the worker into two different workers, one dedicated to performRunExecution
- Schedule performRunExecution in a single place, with a queue and using a round robin manually controlled concurrency
- Remove startQueuedRuns
- All runs are queued before they are started
- Setting the worker maxPoolSize to the same as the worker concurrency
- Starting to be able to split the docker image
* Remove queue name from startRun graphile job
* Make the prisma connection pool stuff configurable through env vars
* Hardcode (for now) the max concurrent runs limit
* Rewrite performRunExecution to be more performant
PerformRunExecutionV2:
- Does not create and manage jobRunExecution records
- Does not reimplement retrying, uses graphile worker retrying instead
I’ve kept around PerformRunExecutionV1 so this works when deploying. Definitely needs LOTS of testing
* Fix issues with cached tasks
- Limit the size of the cached tasks sent when executing a run, using the knapsack problem dynamic programming approach
- Actually USE the cached tasks in IO by using the idempotencyKey instead of the task ID
- Remove output from all logs
- Added a stress test job catalog
* Forgot to commit the logger updates
* Never log connectionString
* Login to docker hub to get around rate limits
* Add additional logging to the graphile workers
* Fix the *_ENABLED env vars
* Allow adding and removing jobs to be done from the webapp
* Don’t set the job to failed if it’s being retried
* Deprecated queue options in the job and removed startPosition. Now using the job/env combo as the job queue name
* Dequeung jobs doesn’t check if the runner is initialized
* Fixed issues with retrying a run getting stuck on a cancelled task, and errors from parsing the results of dequeing a job
* Remove queued round robin thing that isn’t used anymore
* Added slack to job catalog
* Better forwards compat
* Added long delay
* Fixed lock file
Eric Allam