Commit Graph

212 Commits

Author SHA1 Message Date
Eric Allam 756024da78 Add support for Run Notifications #401 (#738)
* Add support for Run Notifications #401

* Fixed typo and added changeset
2023-11-16 14:19:42 +00:00
Eric Allam cb1825bfaf Add OpenAI support for 4.16.0 (#726)
* Add OpenAI support for 4.16.0

* Add support for background polling and use that in OpenAI integration to power assistants

* Much improved OpenAI docs

* Added backgroundPoll docs

* Implements waitForEvent and added docs for more built in tasks

* Add sendEvent API referenc

* Write the task libray

* Add changeset and warning for waitForEvent
2023-11-09 16:59:58 +00:00
nicktrn d02173442c Feature: io.sendEvents() (#728)
* Add io.sendEvents

* Add examples to built-ins catalog entry

* Add docs

* Add changeset

---------

Co-authored-by: Matt Aitken <matt@mattaitken.com>
2023-11-09 16:42:23 +00:00
Matt Aitken 690b308fea verifyRequestSignature – added an error if the passed in secret is undefined or empty 2023-11-07 10:52:55 +00:00
Eric Allam a74716a1b9 Added waitForRequest built-in tasks (#725)
* Added waitForRequest built-in tasks

* Create grumpy-buttons-thank.md
2023-11-06 18:30:49 +00:00
Eric Allam 4a0f030ebf Adding no-cache to our client fetch to fix Next.js POST caching 2023-11-06 17:38:18 +00:00
Matt Aitken 9c6838e5c8 HTTP endpoints, HTTP triggers and new sidebar (#720)
* Setup basic storybook page

* storybook preview pages don’t have padding

* Added a storybook story for the side menu

* WIP on a new side menu component

* New variant for the side menu button style

* Side menu items can now recieve a badge ‘count’

* Storybook side menu update

* Added a new font size

* you can now set the spacing between an icon and the text

* Moving the error and count badge to the right side of the button

* Added a menu sub item button style

* added sub items to the menu and all the rest of the items

* Proper background class used

* Added a side menu header and main org header

* Removed the no-mobile blocker overlay so you can use the site on a phone

* Better section name

* Created a cusom popover trigger that can take anything as children. Also the popover titles can now take a variant

* Added a popover for the user avatar menu item

* Removed the tool tip

* Arrow Popover can no also go full width

* neater conditiona;

* Removed unnessary div. And error icon can display an error message

* menu item buttons can take a variant

* Added placeholder items for the org dropdown

* add a max height to the org menu popover and handle scrollbars

* Added a border if the menu scrolls when you have lots of project items

* warning badges display tooltips if you give them some error text

* Removed the sticky header so the scrollbar looks better

* Fixed a layout issue from removing the sticky header

* Better spacing between lists

* Popover can take any react component icon and an icon classname

* Side menu headers now have a dropdown by clicking the ellipsis icon

* Side menu items can take an icon colour

* Updated styling for the dropdown Popover menus

* Installed Tabler icons

* Updated some icons to use Tabler

* Started defining the HttpTrigger class

* RequestFilterSchema is part of core

* Use the RequestFilterSchema from core

* EventSpecificationExample now derived from schema

* Need to define the class like EventTrigger

* Catalog reference that uses the httpTrigger

* The logic for the httpTrigger is in the separate file, just call it from the client using defineHttpTrigger

* Basic HttpTrigger

* Import fixes

* Transforming httpTrigger to the format we want

* Initial httpTrigger types working

* Added HTTP generic type

* Added some notes

* Rejigged some things to support verifying payloads

* Use new security function for verifying

* Added truncating text support if side menu names get long

* Renamed some stuff

* Removed test copy

* HttpEndpoin that isn’t a trigger

* Renamed hostname to source

* Created HttpEndpoint as a top-level concept, it produces triggers using onRequest

* Started working on registration of the httpEndpoints

* Removed the new TriggerMetadata for now, http will use static

* Added PayloadType to EventRecord. Default is JSON.

* The payload from an HTTPEndpoint is now a Request

* Initial TriggerHttpEndpoint migrations

* Initial server side work

* Migration to add shortcode to RuntimeEnvironment

* Added TriggerHttpEndpointEnvironment table

* Made the httptrigger catalog example runnable

* Progress creating/updating httpEndpoints

* HTTPEndpointEnvironment updating now works

* Added a note about EventRecord HttpEndpoint relationship

* When creating environments, create a shortcode

* Fix for secret not being retrieved correctly

* Request filtering

* Add DOM types to Core

* Early work dealing with a webhook arriving

* Added some todo notes to discuss tomorrow

* Added relations to the endpoint so we can call the user’s server

* Basic handling of immediate responses

* The immediate responses are now going back to the webhook provider

* Handling is now generating events and responding where relevant. Not yet matching filters and doing verification

* Added source to TriggerHttpEndpointEnvironment

* The event filter is now optional

* The source is passed through from the httpEndpointEnv

* Added optional payloadType when ingesting an event

* Added tabler

* Updated @remix-run/testing usage for remix 2

* Fixed org menu header width and added org and project types

* Added custom icons for Runs and Endpoints

* Updated the menu icons

* storybook side menu shows static project and org data

* Updated side menu

* Plumbed in paths

* Updated eventFilter to deal with undefined/null payloads

* Started work on filtering, but it’s not working

* Strict null

* async filtering it working

* Added environment shortcodes to the seedCloud script

* Don’t pass the secret back, added a verifyPayload function that gets called before runs start

* Fix for skipping triggering runs

* The verify callback is now passed through to the trigger so it can be called

* If verifying fails, have a reason which we can show to the user

* Don’t verify if it’s a test run

* Fixed the log in the example code

* Added loader and action to storybook preview

* Fix for svg attribute name issues in EndpointIcon

* Started work on the side menu

* This logic has moved to a separate file, in a different branch

* Don’t get the environments when getting orgs

* Lots of fixes for the new SideMenu

* Refactored some org/project things to fit the new structure better

* Some progress restructuring the app for the new sidebar

* More changes to data loading

* Deleted some old pages. The redirecting is working if you’re on the root but logged in

* Basic styling for the account page. We need a back button

* Fixes for the integrations pages

* Breadcrumb implemented

* Add a border to the bottom of the breadcrumb

* Feedback is working, not in Storybook

* Removed expandSidebar from the Handle

* Org level breadcrumbs

* Getting the project in the org route now works properly

* orgslug route before using the presenter

* More UI improvements

* Removed commented out line

* Use presenters for org and root

* Removed forceActive

* Split the project selector and user menu out

* The side menu items have correct active states

* Fixed the scrolling

* Put the org name in

* Improved the admin features

* Created the HTTP Endpoints list page

* Work on the HTTP endpoint page

* WIP on the HTTP Endpoint page

* Fix for response filter not showing

* Change page URLs to “http-endpoints” (from httpendpoints)

* Slugify the HTTP endpoint id

* Added todos for EventRecord associations

* Removed the Job/Trigger tab for now

* First draft of the help panels

* Rejigged the HTTP endpoint page so it splits out the deployed and not deployed environments

* Removed irrelevant columns

* Added a link to the deployment guide

* Removed the background image from the frameworks pages

Also…
- reordered the frameworks so supported ones come first in the list
- added logos to 3 framework pages that were missing them
- removed lots of un-used imports

* Fixed an issue where the breadcrumb could be hidden on small screens so broken the pages

The breadcrumb is always visible now

* Renamed ‘Onboarding’ to ‘Framework setup’

* Added a new http-endpoint icon to the side menu

* Side menu now has a nice looking scrollbar

* Scrollbar on the usage and billing page updated to the nice style

* Icon and colour updates to the side bar menu items

* Small copy improvement

* Fixed bug with the height of the org menu dropdown

* Page breadcrumbs now don’t include duplicate path links

* renamed uiBorder to ui-border

* Remove the padding from the small menu item if it’s used as a button

* Made the paragraph text a bit smaller

* EventRecord has relationships with TriggerHttpEndpoint/TriggerHttpEndpointEnvironment

* Improved the presenter used by the side menu. Also added the HTTP Endpoint badge count to the sidebar

* The dropdown org menu has the same badge as the side menu

* Improved the Projects page

* Fixed button style miss-match in the jobs table dropdown

* Made the popover dialogs support Headers and improved the regenerate API key popover

* Made the active status of a job more subtle

* Allow the Job to have a link on its Trigger

* Simplified loader because we ditched the Jobs menu. Also showing no runs help panel now

* Fixed type errors

* Pages that are org level can scroll if the content gets too long

* Fixed a redirect bug so you now redirect to the org page

* Updated the shortcode migration, to add additional letters to the end

* Removed the nav bar from the welcome onboarding page

* Only show the countBadge if the number is > 0

* Jobs icon now has a default colour the same as the nav menu

* Capital I for integration

* Clone the request before calling verify

* Separate the checks out so they can be easily logged

* Make the failure reason optional

* Made the verification error look better, if it’s provided

* Working WhatsApp example

* Can set the immediateResponseFilter to null in an update

* Deal with a null immediateResponseFilter

* Cal.com webhook working

* Header height reduced slightly

* Fixed button padding

* Removed un-used imports

* http endpoint table displays a message if you have no entries in the table

* Removed some duplicate text and swapped primary buttons for secondary style

* All ‘HTTP Endpoints’ now ‘HTTP endpoints’

* You have to use a RequestFilter when using HTTP Trigger

* Synchronous eventDispatcher filtering again. Better content-type check

* Stripe example

* Side menu “Usage & Billing” is just “Usage” if self-hosting

* Fix for filtering not working

* Removed filter log

* Fix for jobs being optional and null

* Removed old nav bar from the invite page

* Deleted old nav bar component

* import cleanup

* NoTask message only displays if the run has status “SUCCESS”

* Added a global page loading indicator

* Added documentation buttons to the most important pages

* Added GitHub

* Use “verifyRequestSignature” helpr

* Removed todo

* Fixed the layout of the Rerun Job popover

* Jobs table has margin bottom so it sits off the bottom of the page

* Small improvement to the search bar layout on the integration page

* HTTP endpoint and Trigger concept docs

* Reference docs and JSDocs for HTTP endpoint

* SDK reference docs for HttpEndpoint

* Added an HttpTrigger SDK reference, which mainly just point you elsewhere

* Improved the copy in the help panels

* Removed unused code

* Improved HTTP endpoint comment about setting the secret

* Delete unused asyncFilter

* Fixed the delete modal not having a title

* Always try get the arrayBuffer for an HTTP request, catch and use undefined as the default

* Add extra method to invokeTrigger

* Use a transaction for the secret store

* verifyRequestSignature docs page and JSDocs

* Added verifyRequestSignature docs page to nav

* Warn if they use the same HttpEndpoint id twice

* Added another job using the same HttpEndpoint

* Renamed the api route to http-endpoints from httpendpoints

* Debug bar for VSCode is now in the commandCenter

---------

Co-authored-by: James Ritchie <james@jamesritchie.co.uk>
2023-11-06 15:32:40 +00:00
Alexandre Costa 7e57f1f3dd [TRI-1449] Display warning message when duplicate job IDs are detected (#699)
* add warning message when duplicate job IDs are detected

* fix warning message for duplicated job IDs in defineJob

* Create bright-buckets-bow.md

---------

Co-authored-by: Eric Allam <eric@trigger.dev>
2023-11-03 11:10:26 +00:00
nicktrn cf8f994604 Feature: io.random() (#716)
* Add io.random

* Add changeset

---------

Co-authored-by: Eric Allam <eric@trigger.dev>
2023-11-03 11:07:12 +00:00
Eric Allam 620b83832b Implement manually invokable jobs through the invokeTrigger (#700)
* Implement manually invokable jobs through the invokeTrigger

Also implemented a job run notification system, that will POST details of a run on completion. This combines with the task callbackUrl system to implement the invokeAndWait

* Document the invoke trigger

* batch invoke and wait

* background fetch timeouts

* Use @whatwg-node/fetch instead of the polyfilled fetch

* Fix some outdated dependencies in webapp

* Improved subtask error propogation messages

* Document the OpenAI changes and the batch invoke stuff

* Fix dequeuing jobs

* Don’t retry the OpenAI completion background task

* Added OpenAI changesets

* Use the new ResumeTaskService in ProcessCallbackTimeout as well
2023-11-03 11:05:00 +00:00
Eric Allam c1710ae72d fix: Buffer import breaks @trigger.dev/react (#714) 2023-10-31 15:28:55 +00:00
Hemachandar 9c4be40adc fix: append nanoid for dynamic trigger reg event (#628)
* fix: append nanoid for dynamic trigger reg event

* use idempotency-key as event-id

* add changeset
2023-10-30 16:24:03 +00:00
Hemachandar c4533c36cb fix: set error message in runTask & executeJob (#634)
* fix: set error message in runTask & executeJob

* add changeset
2023-10-26 11:46:09 +01:00
hmacr 6e1b8a11d4 feat: allow cancelling job runs for an event-id 2023-10-26 11:18:11 +01:00
Eric Allam 044d38e390 Improvement: Auto Execution Yielding (#612)
* Auto-yield run execution to help prevent duplicate task executions

* Add auto-yield config to endpoints

* Refactor run execution with buffer and limits

Introduced constants RUN_CHUNK_EXECUTION_BUFFER and MAX_RUN_CHUNK_EXECUTION_LIMIT. Adjusted PerformRunExecutionV2Service to use the new constants to fine-tune execution timings and buffers.

* Add endpoint probing functionality

Added new `RESPONSE_TIMEOUT_STATUS_CODES` in `consts.ts` to manage timeout responses. Additional functions `detectResponseIsTimeout(response: Response)` was added in `endpoint.server.ts` to detect if a response was a timeout based on the status codes from `RESPONSE_TIMEOUT_STATUS_CODES`.

Update actions to use new endpoint probing endpoint service. This allows for the early probing of endpoints to determine if they're up and running.

A new class `ProbeEndpointService` was created in `probeEndpoint.server.ts` which makes HTTP requests to a given endpoint and updates its properties based on the result.

Finally, `detectResponseIsTimeout(response)` is used in `performRunExecutionV2.server.ts` for marking the execution as succeeded when facing a timeout.

* Refactored probe method in EndpointApi class

The probe method of the EndpointApi class has been refactored to remove the error handling part and it now takes a timeout sent from the client directly. The corresponding changes were also made in the ProbeEndpointService and TriggerClient objects to reflect the alterations in the probe method.

The error handling related to the timeout has been removed and the responsibility of handling the timeout has been shifted to the client. Thus, the probe method has been greatly simplified. The `probeEndpoint.server.ts` file was also changed to accommodate the change in behavior of the probe result.

In the `triggerClient.ts` the timeout for probe is now read from the incoming request object. For backward compatibility, if no timeout is provided in the request, the default value of 15 minutes is used.

* Remove performRunExecution v1 enqueue function
* Better document limits and add docs on increasing function timeouts
* Upgrade webapp docker container to use 18.18.2
* force clients to yield when a run is executing in a gracefully shutting down worker
* Renamed task `key` to `cacheKey` and added more task documentation
* Index the `@trigger.dev/sdk` version on Endpoints
2023-10-20 17:44:10 +01:00
Rutam Prita Mishra 975c5f1d3c [TRI-1333] : Upgrade all packages to use Node18 and use fetch instead node-fetch (#581)
* [TRI-1333] : Upgrade all packages to use Node18 and use fetch instead of node-fetch

* Changeset Added

* Update pnpm-lock to compatible pnpm v7.18.1

* Restored pnpm-lock to the last known good point compatible with pnpm v7.18.1

* Update chilled-plants-exercise.md

---------

Co-authored-by: Eric Allam <eallam@icloud.com>
2023-10-10 11:42:02 +01:00
Hemachandar 2e9452ab7c feat: allow cancelling jobs from trigger-client sdk (#562)
* feat: allow cancelling jobs from trigger-client sdk

* use presenter instead of service for non-mutating logic
2023-10-05 16:51:59 +01:00
nicktrn 6a992a1995 Replicate integration and remote callbacks (#507)
* Support tasks with remote callbacks

* Add common integration tsconfig

* Add Replicate integration

* Basic job catalog example

* Integration catalog entry

* Check for callbackUrl during executeTask

* Fix getAll

* Improve JSDoc

* Bump version

* Remove named queue

* Simplify runTask types

* Trust the types

* Fail tasks on timeout

* Callback timeout as param

* Mess with types

* performRunExecutionV1

* Update runTask docs

* Shorten callback task methods

* Fix run method return type

* Image processing jobs

* Replicate docs

* Text output example

* Changeset

* Version bump

* Roll back ugly types

* Remove missing types

* Quicker return when waiting on remote callback

* Remote callback example

* Bump version

* Remove schema parsing

* Only schedule positive callback timeout

* Decrease callback secret length

* Explicit default timeouts

* Import deployments tasks

* JSDoc

* Deployments docs

* Fix runTask examples, mention wrappers

---------

Co-authored-by: Eric Allam <eric@trigger.dev>
2023-10-04 10:46:17 +01:00
Eric Allam ab9e4a989c Improves the performance of run resuming (#522)
* Improves the perform of run resuming

When runs resume, we try and make sure that tasks that have already been completed are cached and reused. Worst case scenario the client needs to hit the API server once for a non-cached task that is indeed completed on the server, but this can get pretty expensive when there are a larger number of tasks.

This commit does 2 different things to help:

- noop tasks are no longer “cached” using the cachedTasks strategy, instead their idempotency keys are shoved into a bloom filter and the client tests for their inclusion in the bloom filter before running them (since they don’t have any concept of output, this works)
- Additional cached tasks are lazy loaded when a task is run. This allows us to progressively fetch additional tasks to be cached on the client, which will cut down on cache misses by a decent amount

* Create warm-carrots-float.md

* Make io.yield backwards compat with older platform versions

* Better support old clients connecting to server versions that support lazy loading cached tasks

* Fixed type errors when settings headers with unknown value

* Better yield not support error message

* Rename _version to _serverVersion to be more clear
2023-10-03 16:44:18 +01:00
Eric Allam 421c249e50 Add some documentation around canceling scheduled events 2023-09-30 22:18:52 +01:00
Eric Allam 50137a6f24 Decouple zod (#500)
🚀 Publish Trigger.dev Docker / ʦ TypeScript (push) Has been cancelled
🚀 Publish Trigger.dev Docker / Unit Tests (push) Has been cancelled
🚀 Publish Trigger.dev Docker / e2e Tests (push) Has been cancelled
🚀 Publish Trigger.dev Docker / publish (push) Has been cancelled
Zod Schemas is no longer required for validating/inferring event triggers. We’ve taken inspiration from how domain-functions did it: https://github.com/seasonedcc/domain-functions/pull/114
2023-09-22 14:44:03 -07:00
Matt Aitken 7b29a92f6a Removed dynamicTrigger @internal from toJSON 2023-09-22 12:24:07 -07:00
Eric Allam 24465c9a7b Add back in Job.toJSON to fix the testing package 2023-09-22 11:21:37 -07:00
Eric Allam c0dfa8048a feat: BYO Auth (#491)
* feat: BYO Auth

Define client-side auth resolvers to be able to supply custom authentication credentials for integrations before a run is performed

- Added new defineAuthResolver
- Update all integrations to support the new auth resolvers
- Strip internal symbols from .d.ts in integrations and trigger-sdk
- Added BYO Auth docs
- Update Dynamic Schedule to support associated account IDs
- Create external accounts just-in-time
- Added Account ID field to test job when there are external auth integrations
- Show Account ID on run dashboard
- Added new Run error state called “Unresolved auth”

* Added changeset

* Remove @internal from TriggerIntegration public methods

* Add void to the result union

* DynamicTriggers now work with the new BYO auth system, and added a bunch of docs and docs changes

* Add additional key material for registering dynamic trigger task

* Add new define* instance methods to the overview
2023-09-22 08:54:31 -07:00
nicktrn 15f17d27e0 Going exponential with Linear (#478)
* Unleash GPT magic

* Clean up after GPT

* All the hooks

* Provisional integration catalog entry

* Sample webhook jobs

* Attachments with alpha warnings

* Remove some verbose logs

* Fix IP restrictions

* Remove tunnel

* Revert "Remove tunnel"

This reverts commit c5b69ce6524e3b40c26b66cdc56e087b8576b8c6.

* Resolve event name clashes

* Remove circular dependency

* Use correct payload uuid

* Schema fixes

* Fix webhook event name

* Start to Linearify catalog entry

* Remove todo

* More catalog updates

* Make OAuth work

* Rename webhook helper

* Schema juggling

* More discrimination

* Add Issue SLA event

* Simplify triggers

* Handle rate limits

* Fix Project schema

* Payload examples

* Improve event props

* Remove redundant source metadata

* One type to rule them all

* Recursive WithoutFunctions type

* Linear output serializer

* Some tasks

* Update catalog entry

* Dynamic usage sample

* Bump version

* Remove tunnel

* More tasks

* Add optional skipRetrying on runTask errors

* Fail fast on user errors

* Entity getter tasks

* Another couple of tasks

* Token to apiKey

* Sort tasks

* Add filtered issue SLA triggers

* Add docs

* Type fixes

* Job catalog examples

* Serialization helper docs

* Add changeset

* Refactor webhooks

* Enhance properties

* Pagination helper and docs

* Clean up imports

* Change misc catalog job

---------

Co-authored-by: Matt Aitken <matt@mattaitken.com>
2023-09-21 15:14:54 -07:00
Matt Aitken 91fc1e80f3 Use the bell icon for the new status Tasks
🚀 Publish Trigger.dev Docker / ʦ TypeScript (push) Has been cancelled
🚀 Publish Trigger.dev Docker / Unit Tests (push) Has been cancelled
🚀 Publish Trigger.dev Docker / e2e Tests (push) Has been cancelled
🚀 Publish Trigger.dev Docker / publish (push) Has been cancelled
2023-09-21 15:01:35 -07:00
Matt Aitken 6ce87f169a Fixed dependency 2023-09-21 11:35:12 -07:00
Matt Aitken ad14983e21 React status hooks (#493)
* Added stripInternal to SDK tsconfig

* Statuses can now be set from a run, and are stored in the database

* Added the key to the returned status

* Made the test job have an extra step and only pass in some of the options

* client.getRunStatuses() and the corresponding endpoint

* client.getRun() now includes status info

* Fixed circular dependency schema

* Translate null to undefined

* Added the react package to the nextjs-reference tsconfig

* Removed unused OpenAI integration from nextjs-reference project

* New hooks for getting the statuses

* Disabled most of the nextjs-reference jobs

* Updated the hooks UI

* Updated the endpoints to deal with null statuses values

* The hook is working, with an example

* Changeset: “You can create statuses in your Jobs that can then be read using React hooks”

* Changeset config is back to the old changelog style

* WIP on new React hooks guide

* Guide docs for the new hooks

* Added the status hooks to the React hooks guide

* Removed the links to the status hooks reference for now

* Re-ordered the hooks

* Fix for an error in the docs

* Set a default of a blank array for the GetRunSchema
2023-09-21 10:24:37 -07:00
Ravan bf24593b7f fix: Added retry config and showed error from subtask in web (#468)
* fix: Added retry config and showed error from subtask in web

* Still fallback to "Task errored" if there's no output

* Create rude-carrots-enjoy.md

---------

Co-authored-by: Matt Aitken <matt@mattaitken.com>
2023-09-12 16:25:29 +01:00
Matt Aitken 8ddd151051 Re-ordered runTask params in docs, and example Jobs in the app. 2023-09-06 18:31:50 +01:00
Matt Aitken 090bc2e7d2 OverridableRunTaskOptions allows integrations to expose override task options to users 2023-09-05 17:59:46 +01:00
Matt Aitken ecd050bece Airtable integration with webhook and integration changes (#399)
* CLI create-integration command now accepts an Open AI api key

* Create integration docs separated into multiple pages

* Initial Airtable integration commit, with OpenAI generated code

* OAuth page coming soon

* Export DisplayProperty from the SDK

* TSConfig made to match GitHub’s with paths

* getRecords

* Removed duplicate Stripe job from the catalog

* Renamed Airtable apiKey option to token

* First Airtable job

* Export Collaborator and Attachment field types

* A typesafe example that uses runTask

* WIP on new integration tasks… not working yet

* Attempt with class

* Revert "Attempt with class"

This reverts commit 93a48330019f754c3216c5b49964fa4b0218bd3f.

* WIP changing how tasks work

* Mock of async local storage

* Moved client creation from constructor

* New approach with a clone method on TriggerIntegration

* Added runTask to Airtable which is used by integration tasks

* Added the Airtable icon and connection when using runTask

* base().table() is working

* runTask options moved to the 3rd param, made optional with optional name

* Added some generic arguments

* Added generic type to table

* Removed old comment

* We don’t need to repeat the icon

* getRecords and getRecord now returning the right data and types

* Creating records

* Update records

* Delete records

* The internal properties of integrations are now hidden by the TypeScript types

* Sprinkled a Prettify in there

* Improved the types

* Added Airtable to the integration catalog

* Early work on Airtable webhook registration

* More progress with webhooks

* connectionKey needs to be cloned for webhooks to work

* connectionKey needs to be cloned for webhooks to work

* It was unclear that the ActivateSourceService was using a graphileJob id

* ActivateSourceService optionally takes a jobId, if missing it generate a unique id

* When retrying trigger registration, don’t pass an id so it is generated

* Removed Airtable webhooks tasks from the job-catalog example

* Added TriggerSourceOption, removed TriggerSourceEvent

* WIP with new ExternalSource options

* ExternalSourceTrigger setup

* DynamicTrigger changed to options, will need some more work

* filter gets options passed to it

* SourceMetadata v2 renamed to SourceMetadataV2, kept original

* Started versioning the backend

* Moved param order on io.getEvent and io.cancelEvent

* The runTask stuff that allows unknown to work is back

* Indexing for v1 and v2, with version on “activateSource” schema

* Added todos, to deal with Airtable SDK calls inside the webhook handler

* “deliverHttpSourceRequest” queueName changed to the source id so they process in order

* ActivateSource changes to deal with old and new data formats

* Update existing TriggerSources to v2

* Fix for dynamic.ts typescript errors, need to revisit this later

* UpdateSourceService v1 and v2, with new v2 API endpoint

* Removed unused imports

* More progress on v1 and v2

* Airtable webhooks are now triggering a job

* Moved webhooks to a new file

* You can do API calls in the webhook handler now, Airtable webhook data is being processed

* Airtable events coming through

* Defined the Airtable table payload type

* TriggerSource metadata is being stored and used

* Removed some logs

* Added filtering and don’t allow any webhooks that use automated sources

* Resend switched to new integration

* Moved Resend test jobs to the catalog, and tested it worked

* WIP on Slack, there are compile errors

* Created a generic type that strips out indexes

* Slack updated to use new integration

* SendGrid migrated over

* Integration runTask is now allowing regular types

* Changed io.runTask types so it only allows Json-able types

* OpenAI models tasks working

* Added Airtable changes to runTask

* Removed the index signature crap from the Slack integration

* Don’t need to cast the callback result

* Updated Resend

* Re-ordered runTask params

* WIP on openai

* onAccountUpdated is Connect only

* Removed RunTaskResult

* Handle Resend errors, the official SDK doesn’t expose them properly at the moment

* Removed OmitIndexSignature

* OpenAI converted to new integration, with backwards compatible functions

* Put the openai catalog back to what it was originally

* Export a standard retry with backoff, to be used

* Use the standard exponential backoff in the integrations

* Retry options moved earlier so they can be overriden by a task

* GitHub tasks migrated

* Added sources, fixed one bundling issue

* Added GitHub jobs to catalog

* Remove duplicate options

* Deduplicate events

* Removed duplicate Job

* Switched Plain over

* Set the Plain icon

* Converted Stripe over

* Supabase adapted

* Typeform working

* Added dynamic-schedule to catalog

* Added background-fetch job catalog

* Created dynamic-triggers catalog file

* Fixed old general file with runTask param order

* Dynamic triggers working

* SendGrid updated to use the same tsconfig as other integrations

* Removed Airtable webhook, until we have batch support

* Added OAuth airtable auth example

* Created beta changeset tag

* Beta changesets for most packages

---------

Co-authored-by: Eric Allam <eallam@icloud.com>
2023-09-05 14:21:42 +01:00
Eric Allam 916a353660 bugfix: retrying tasks no longer incorrectly complete successfully 2023-08-28 10:16:47 +01:00
James Ritchie 708ebbde14 Features: Disabling and deleting jobs (#381)
* Don’t show the Ready To Run Job prompt if you have an Integration that needs attention

* Don’t highlight the row red or green

* Improvements to the contrast of the app sections and dividers

* Removed the Jobs page title as it’s duped info

* using the new border variable

* Sticky last table cell

* Improved the sticky last table cell

* Make any last cell in a table sticky by adding isSticky to it

* Added a dropdown menu to the menu table cell

* table rows can be marked as disabled by adding disabled

* Using the jobTestPath function for the test path

* tidy up imports

* Removed un-used props

* Removed the green badge variant

* Added a new status badge to the Runs table

* Clicking the gradient clicks the row

* Delete Job triggers a modal popup

* Added a large danger button type

* Added some modal styling and started adding data

* Added more styling and data to the delete job modal

* A table can now be given a full width prop

* Large danger button added to Storybook

* Danger button disabled state looks disabled now

* New active badge component to display in the table and logic for showing the env data

* Style updates to the dialog component

* active and job status badges can now have a small size

* Added a new named icon

* The Job page shows the Job status in the PageInfoRow

* Small badge style update

* Runs table has a sticky right cell

* Created a JobStatusTable component

* Added some placeholder help panel content for disabling a Job

* WIP creating a Settings page

* Added a delete button that triggers the delete modal – just need data hooking up

* Implemented deleting jobs from the dashboard

---------

Co-authored-by: Eric Allam <eallam@icloud.com>
2023-08-24 10:30:00 +01:00
Eric Allam 3ce5397072 Implement the disable job feature backend when indexing endpoints (#382) 2023-08-23 10:30:06 +01:00
Eric Allam 302bd02ff1 Issue #377: only expose the external eventId in the API (#380)
* Issue #377: only expose the external eventId in the API

* Create eighty-zebras-bow.md
2023-08-22 10:46:46 +01:00
Eric Allam c6df574689 Fixed io.cancelEvent (shouldn’t be a noop) 2023-08-21 17:48:21 +01:00
Eric Allam b9dc7ce89e Merge branch 'feat/cancelEvent-function' of https://github.com/Chigala/trigger.dev into Chigala-feat/cancelEvent-function
# Conflicts:
#	apps/webapp/app/platform/zodWorker.server.ts
2023-08-21 16:57:36 +01:00
Eric Allam b1b9321ad2 Job run performance improvements and adding "worker only" mode (#360)
* WIP job run performance improvements

- Added a `perf` tool to better measure job run performance under heavy load
- Removed `runFinished` job (not really needed)
- startQueuedRuns now uses a jobKey with replace
- Fixed an issue with ZodWorker when using jobKey

* Publish improvement docker images

* fixed the improvement docker publishing

* Downgrade back to prisma 4.16.0 because 5.1.x broke docker builds

* Changes to how queued runs work

- Split the worker into two different workers, one dedicated to performRunExecution
- Schedule performRunExecution in a single place, with a queue and using a round robin manually controlled concurrency
- Remove startQueuedRuns
- All runs are queued before they are started
- Setting the worker maxPoolSize to the same as the worker concurrency
- Starting to be able to split the docker image

* Remove queue name from startRun graphile job

* Make the prisma connection pool stuff configurable through env vars

* Hardcode (for now) the max concurrent runs limit

* Rewrite performRunExecution to be more performant

PerformRunExecutionV2:

- Does not create and manage jobRunExecution records
- Does not reimplement retrying, uses graphile worker retrying instead

I’ve kept around PerformRunExecutionV1 so this works when deploying. Definitely needs LOTS of testing

* Fix issues with cached tasks

- Limit the size of the cached tasks sent when executing a run, using the knapsack problem dynamic programming approach
- Actually USE the cached tasks in IO by using the idempotencyKey instead of the task ID
- Remove output from all logs
- Added a stress test job catalog

* Forgot to commit the logger updates

* Never log connectionString

* Login to docker hub to get around rate limits

* Add additional logging to the graphile workers

* Fix the *_ENABLED  env vars

* Allow adding and removing jobs to be done from the webapp

* Don’t set the job to failed if it’s being retried

* Deprecated queue options in the job and removed startPosition. Now using the job/env combo as the job queue name

* Dequeung jobs doesn’t check if the runner is initialized

* Fixed issues with retrying a run getting stuck on a cancelled task, and errors from parsing the results of dequeing a job

* Remove queued round robin thing that isn’t used anymore

* Added slack to job catalog

* Better forwards compat

* Added long delay

* Fixed lock file
2023-08-18 15:25:26 +01:00
Wesley 3ee7cd6ff6 feat: parse callback result with zod before return (#342) 2023-08-18 14:33:42 +01:00
Chigala ef1a1ff343 added io.cancelEvent support and fixed other requested changes 2023-08-18 06:21:57 +01:00
Chigala 24c8f08267 feat: added the ability to cancel Events based on the event ID 2023-08-14 14:56:48 +01:00
Matt Aitken 9638499163 Add your first endpoint in the dashboard (#269)
* Created the sheet

* Started work on the resource route

* Created ValidateCreateEndpointService and the EndpointValidateApi (name TBC)

* The TriggerClient responds with the id

* Tidied some stuff up

* Alternative: EndpointApi doesn’t take endpointSlug. Instead just Ping() does

* Create cool-snakes-deny.md

---------

Co-authored-by: Eric Allam <eallam@icloud.com>
2023-08-07 11:28:34 +01:00
Eric Allam 6d04f6c62a Added default retry settings for all integration tasks 2023-08-02 11:53:09 +01:00
Matt Aitken 9ff4c0dbd5 Project-wide Prettier setup (#237)
* Setup project-wide prettier

* Remove old workspace file

* Remove old debugging directives

* New top-level .prettierignore

* Updated Prettier config settings

* Contrubuting guide: Fix for some bad code blocks

* Added more ignores

* Improved the format script command

* printWidth set to 100

* Formatted entire repo (pnpm run format)
2023-08-01 10:21:22 +01:00
Matt Aitken 92233f2e90 Core package and move to @latest (#235)
* Changeset should ignore the example projects

* Removed note in Contributing instructions about not adding a changeset for internal

* Renamed @trigger.dev/internal to @trigger.dev/core. Set sdk and internal to be ES2020, so we don’t get errors about private identifiers

* Env vars in nextjs-example use square bracket syntax to avoid Turbo Repo errors

* Set the tsconfigs back for core and sdk

* @examples/nextjs compile error with undefined tasks

* Set the example projects to use ES2015 to avoid private modifier complaints

* package-tester example, which will use built packages

* package-tester package.json

* Created a readme for the package-tester

* Added all the packages to package-tester

* Create .env.local.example and readme instructions

* Added name to the package.json

* Upgraded @types/react and @types/react-dom everywhrre, so we can use server actions

* Upgraded @types/react and @types/react-dom everywhrre, so we can use server actions

* Reworked the react package build, so it generates separate files

* The SDK no longer bundles core

* package-tester setup with a server action and react hooks

* Added new SDK methods with logging

* Working tsup settings for client and server

* Explicit react hooks return types

* Added all the hooks for testing

* Added OpenAI step to the job to check types are still ok in integrations

* Get rid of rogue Changeset ignores

* Changeset: @trigger.dev/core is now a separate package

* Exited prerelease mode, added a changeset

* Changed @next to @latest

* Deleted seed.js, this shouldn’t be committed

* Ignore seed.js

* The webapp was importing @trigger.dev/core with a folder path rather package name…

* More webapp imports instead of @trigger.dev/core were a folder path

* Accidentally edited Stripe internal API url in the comment

* Added “sideEffects”: false so @trigger.dev/core is tree shaken by Remix
2023-07-31 18:47:34 +01:00
Eric Allam 7e4d72f959 Made the migration non-destructive and fixed other typechecking issues 2023-07-28 18:19:41 +01:00
Chigala 3e203ea967 Merge branch 'main' into feature/multiple-eventname-support 2023-07-28 17:53:31 +01:00
Eric Allam 64477f6b3a @trigger.dev/sdk: Adding some type helpers for getting the payload and IO types from jobs and triggers 2023-07-28 14:42:28 +01:00
Eric Allam 9351c05126 Initial @trigger.dev/stripe integration, with support for triggers and tasks (#231)
* Initial Stripe integration, with support for triggers and tasks

* Adding stripe integration to the catalog
2023-07-27 22:09:07 +01:00