Commit Graph

15 Commits

Author SHA1 Message Date
Eric Allam 74db2de1bc Use graphile strategy 0 (no named queues) and remove all named queues 2024-06-05 19:08:00 +01:00
Jacob Paris ae5ec86d0b fix: ensure enums satisfy prisma types 2024-05-21 03:44:22 -06:00
Jacob Paris 38965d8921 fix: vite cant resolve real values from prisma, only types are ok 2024-05-18 01:31:22 -06:00
Eric Allam 2496917a7a v2: Graphile auto-cleanup and auto-endpoint disabling (#1103)
* Auto-cleanup failed graphile jobs instead of keeping them around

* Disable endpoint after a period of sequential indexing failures

* Remove log
2024-05-15 21:15:06 +01:00
Matt Aitken 1ccf38e440 Return a 404 so the webhook provider doesn’t retry 2024-05-10 16:00:42 +01:00
Matt Aitken 73fb7cfeb3 Prevent HTTP source requests if the org has runs disabled 2024-05-10 15:58:51 +01:00
Matt Aitken 9c6838e5c8 HTTP endpoints, HTTP triggers and new sidebar (#720)
* Setup basic storybook page

* storybook preview pages don’t have padding

* Added a storybook story for the side menu

* WIP on a new side menu component

* New variant for the side menu button style

* Side menu items can now recieve a badge ‘count’

* Storybook side menu update

* Added a new font size

* you can now set the spacing between an icon and the text

* Moving the error and count badge to the right side of the button

* Added a menu sub item button style

* added sub items to the menu and all the rest of the items

* Proper background class used

* Added a side menu header and main org header

* Removed the no-mobile blocker overlay so you can use the site on a phone

* Better section name

* Created a cusom popover trigger that can take anything as children. Also the popover titles can now take a variant

* Added a popover for the user avatar menu item

* Removed the tool tip

* Arrow Popover can no also go full width

* neater conditiona;

* Removed unnessary div. And error icon can display an error message

* menu item buttons can take a variant

* Added placeholder items for the org dropdown

* add a max height to the org menu popover and handle scrollbars

* Added a border if the menu scrolls when you have lots of project items

* warning badges display tooltips if you give them some error text

* Removed the sticky header so the scrollbar looks better

* Fixed a layout issue from removing the sticky header

* Better spacing between lists

* Popover can take any react component icon and an icon classname

* Side menu headers now have a dropdown by clicking the ellipsis icon

* Side menu items can take an icon colour

* Updated styling for the dropdown Popover menus

* Installed Tabler icons

* Updated some icons to use Tabler

* Started defining the HttpTrigger class

* RequestFilterSchema is part of core

* Use the RequestFilterSchema from core

* EventSpecificationExample now derived from schema

* Need to define the class like EventTrigger

* Catalog reference that uses the httpTrigger

* The logic for the httpTrigger is in the separate file, just call it from the client using defineHttpTrigger

* Basic HttpTrigger

* Import fixes

* Transforming httpTrigger to the format we want

* Initial httpTrigger types working

* Added HTTP generic type

* Added some notes

* Rejigged some things to support verifying payloads

* Use new security function for verifying

* Added truncating text support if side menu names get long

* Renamed some stuff

* Removed test copy

* HttpEndpoin that isn’t a trigger

* Renamed hostname to source

* Created HttpEndpoint as a top-level concept, it produces triggers using onRequest

* Started working on registration of the httpEndpoints

* Removed the new TriggerMetadata for now, http will use static

* Added PayloadType to EventRecord. Default is JSON.

* The payload from an HTTPEndpoint is now a Request

* Initial TriggerHttpEndpoint migrations

* Initial server side work

* Migration to add shortcode to RuntimeEnvironment

* Added TriggerHttpEndpointEnvironment table

* Made the httptrigger catalog example runnable

* Progress creating/updating httpEndpoints

* HTTPEndpointEnvironment updating now works

* Added a note about EventRecord HttpEndpoint relationship

* When creating environments, create a shortcode

* Fix for secret not being retrieved correctly

* Request filtering

* Add DOM types to Core

* Early work dealing with a webhook arriving

* Added some todo notes to discuss tomorrow

* Added relations to the endpoint so we can call the user’s server

* Basic handling of immediate responses

* The immediate responses are now going back to the webhook provider

* Handling is now generating events and responding where relevant. Not yet matching filters and doing verification

* Added source to TriggerHttpEndpointEnvironment

* The event filter is now optional

* The source is passed through from the httpEndpointEnv

* Added optional payloadType when ingesting an event

* Added tabler

* Updated @remix-run/testing usage for remix 2

* Fixed org menu header width and added org and project types

* Added custom icons for Runs and Endpoints

* Updated the menu icons

* storybook side menu shows static project and org data

* Updated side menu

* Plumbed in paths

* Updated eventFilter to deal with undefined/null payloads

* Started work on filtering, but it’s not working

* Strict null

* async filtering it working

* Added environment shortcodes to the seedCloud script

* Don’t pass the secret back, added a verifyPayload function that gets called before runs start

* Fix for skipping triggering runs

* The verify callback is now passed through to the trigger so it can be called

* If verifying fails, have a reason which we can show to the user

* Don’t verify if it’s a test run

* Fixed the log in the example code

* Added loader and action to storybook preview

* Fix for svg attribute name issues in EndpointIcon

* Started work on the side menu

* This logic has moved to a separate file, in a different branch

* Don’t get the environments when getting orgs

* Lots of fixes for the new SideMenu

* Refactored some org/project things to fit the new structure better

* Some progress restructuring the app for the new sidebar

* More changes to data loading

* Deleted some old pages. The redirecting is working if you’re on the root but logged in

* Basic styling for the account page. We need a back button

* Fixes for the integrations pages

* Breadcrumb implemented

* Add a border to the bottom of the breadcrumb

* Feedback is working, not in Storybook

* Removed expandSidebar from the Handle

* Org level breadcrumbs

* Getting the project in the org route now works properly

* orgslug route before using the presenter

* More UI improvements

* Removed commented out line

* Use presenters for org and root

* Removed forceActive

* Split the project selector and user menu out

* The side menu items have correct active states

* Fixed the scrolling

* Put the org name in

* Improved the admin features

* Created the HTTP Endpoints list page

* Work on the HTTP endpoint page

* WIP on the HTTP Endpoint page

* Fix for response filter not showing

* Change page URLs to “http-endpoints” (from httpendpoints)

* Slugify the HTTP endpoint id

* Added todos for EventRecord associations

* Removed the Job/Trigger tab for now

* First draft of the help panels

* Rejigged the HTTP endpoint page so it splits out the deployed and not deployed environments

* Removed irrelevant columns

* Added a link to the deployment guide

* Removed the background image from the frameworks pages

Also…
- reordered the frameworks so supported ones come first in the list
- added logos to 3 framework pages that were missing them
- removed lots of un-used imports

* Fixed an issue where the breadcrumb could be hidden on small screens so broken the pages

The breadcrumb is always visible now

* Renamed ‘Onboarding’ to ‘Framework setup’

* Added a new http-endpoint icon to the side menu

* Side menu now has a nice looking scrollbar

* Scrollbar on the usage and billing page updated to the nice style

* Icon and colour updates to the side bar menu items

* Small copy improvement

* Fixed bug with the height of the org menu dropdown

* Page breadcrumbs now don’t include duplicate path links

* renamed uiBorder to ui-border

* Remove the padding from the small menu item if it’s used as a button

* Made the paragraph text a bit smaller

* EventRecord has relationships with TriggerHttpEndpoint/TriggerHttpEndpointEnvironment

* Improved the presenter used by the side menu. Also added the HTTP Endpoint badge count to the sidebar

* The dropdown org menu has the same badge as the side menu

* Improved the Projects page

* Fixed button style miss-match in the jobs table dropdown

* Made the popover dialogs support Headers and improved the regenerate API key popover

* Made the active status of a job more subtle

* Allow the Job to have a link on its Trigger

* Simplified loader because we ditched the Jobs menu. Also showing no runs help panel now

* Fixed type errors

* Pages that are org level can scroll if the content gets too long

* Fixed a redirect bug so you now redirect to the org page

* Updated the shortcode migration, to add additional letters to the end

* Removed the nav bar from the welcome onboarding page

* Only show the countBadge if the number is > 0

* Jobs icon now has a default colour the same as the nav menu

* Capital I for integration

* Clone the request before calling verify

* Separate the checks out so they can be easily logged

* Make the failure reason optional

* Made the verification error look better, if it’s provided

* Working WhatsApp example

* Can set the immediateResponseFilter to null in an update

* Deal with a null immediateResponseFilter

* Cal.com webhook working

* Header height reduced slightly

* Fixed button padding

* Removed un-used imports

* http endpoint table displays a message if you have no entries in the table

* Removed some duplicate text and swapped primary buttons for secondary style

* All ‘HTTP Endpoints’ now ‘HTTP endpoints’

* You have to use a RequestFilter when using HTTP Trigger

* Synchronous eventDispatcher filtering again. Better content-type check

* Stripe example

* Side menu “Usage & Billing” is just “Usage” if self-hosting

* Fix for filtering not working

* Removed filter log

* Fix for jobs being optional and null

* Removed old nav bar from the invite page

* Deleted old nav bar component

* import cleanup

* NoTask message only displays if the run has status “SUCCESS”

* Added a global page loading indicator

* Added documentation buttons to the most important pages

* Added GitHub

* Use “verifyRequestSignature” helpr

* Removed todo

* Fixed the layout of the Rerun Job popover

* Jobs table has margin bottom so it sits off the bottom of the page

* Small improvement to the search bar layout on the integration page

* HTTP endpoint and Trigger concept docs

* Reference docs and JSDocs for HTTP endpoint

* SDK reference docs for HttpEndpoint

* Added an HttpTrigger SDK reference, which mainly just point you elsewhere

* Improved the copy in the help panels

* Removed unused code

* Improved HTTP endpoint comment about setting the secret

* Delete unused asyncFilter

* Fixed the delete modal not having a title

* Always try get the arrayBuffer for an HTTP request, catch and use undefined as the default

* Add extra method to invokeTrigger

* Use a transaction for the secret store

* verifyRequestSignature docs page and JSDocs

* Added verifyRequestSignature docs page to nav

* Warn if they use the same HttpEndpoint id twice

* Added another job using the same HttpEndpoint

* Renamed the api route to http-endpoints from httpendpoints

* Debug bar for VSCode is now in the commandCenter

---------

Co-authored-by: James Ritchie <james@jamesritchie.co.uk>
2023-11-06 15:32:40 +00:00
Matt Aitken ecd050bece Airtable integration with webhook and integration changes (#399)
* CLI create-integration command now accepts an Open AI api key

* Create integration docs separated into multiple pages

* Initial Airtable integration commit, with OpenAI generated code

* OAuth page coming soon

* Export DisplayProperty from the SDK

* TSConfig made to match GitHub’s with paths

* getRecords

* Removed duplicate Stripe job from the catalog

* Renamed Airtable apiKey option to token

* First Airtable job

* Export Collaborator and Attachment field types

* A typesafe example that uses runTask

* WIP on new integration tasks… not working yet

* Attempt with class

* Revert "Attempt with class"

This reverts commit 93a48330019f754c3216c5b49964fa4b0218bd3f.

* WIP changing how tasks work

* Mock of async local storage

* Moved client creation from constructor

* New approach with a clone method on TriggerIntegration

* Added runTask to Airtable which is used by integration tasks

* Added the Airtable icon and connection when using runTask

* base().table() is working

* runTask options moved to the 3rd param, made optional with optional name

* Added some generic arguments

* Added generic type to table

* Removed old comment

* We don’t need to repeat the icon

* getRecords and getRecord now returning the right data and types

* Creating records

* Update records

* Delete records

* The internal properties of integrations are now hidden by the TypeScript types

* Sprinkled a Prettify in there

* Improved the types

* Added Airtable to the integration catalog

* Early work on Airtable webhook registration

* More progress with webhooks

* connectionKey needs to be cloned for webhooks to work

* connectionKey needs to be cloned for webhooks to work

* It was unclear that the ActivateSourceService was using a graphileJob id

* ActivateSourceService optionally takes a jobId, if missing it generate a unique id

* When retrying trigger registration, don’t pass an id so it is generated

* Removed Airtable webhooks tasks from the job-catalog example

* Added TriggerSourceOption, removed TriggerSourceEvent

* WIP with new ExternalSource options

* ExternalSourceTrigger setup

* DynamicTrigger changed to options, will need some more work

* filter gets options passed to it

* SourceMetadata v2 renamed to SourceMetadataV2, kept original

* Started versioning the backend

* Moved param order on io.getEvent and io.cancelEvent

* The runTask stuff that allows unknown to work is back

* Indexing for v1 and v2, with version on “activateSource” schema

* Added todos, to deal with Airtable SDK calls inside the webhook handler

* “deliverHttpSourceRequest” queueName changed to the source id so they process in order

* ActivateSource changes to deal with old and new data formats

* Update existing TriggerSources to v2

* Fix for dynamic.ts typescript errors, need to revisit this later

* UpdateSourceService v1 and v2, with new v2 API endpoint

* Removed unused imports

* More progress on v1 and v2

* Airtable webhooks are now triggering a job

* Moved webhooks to a new file

* You can do API calls in the webhook handler now, Airtable webhook data is being processed

* Airtable events coming through

* Defined the Airtable table payload type

* TriggerSource metadata is being stored and used

* Removed some logs

* Added filtering and don’t allow any webhooks that use automated sources

* Resend switched to new integration

* Moved Resend test jobs to the catalog, and tested it worked

* WIP on Slack, there are compile errors

* Created a generic type that strips out indexes

* Slack updated to use new integration

* SendGrid migrated over

* Integration runTask is now allowing regular types

* Changed io.runTask types so it only allows Json-able types

* OpenAI models tasks working

* Added Airtable changes to runTask

* Removed the index signature crap from the Slack integration

* Don’t need to cast the callback result

* Updated Resend

* Re-ordered runTask params

* WIP on openai

* onAccountUpdated is Connect only

* Removed RunTaskResult

* Handle Resend errors, the official SDK doesn’t expose them properly at the moment

* Removed OmitIndexSignature

* OpenAI converted to new integration, with backwards compatible functions

* Put the openai catalog back to what it was originally

* Export a standard retry with backoff, to be used

* Use the standard exponential backoff in the integrations

* Retry options moved earlier so they can be overriden by a task

* GitHub tasks migrated

* Added sources, fixed one bundling issue

* Added GitHub jobs to catalog

* Remove duplicate options

* Deduplicate events

* Removed duplicate Job

* Switched Plain over

* Set the Plain icon

* Converted Stripe over

* Supabase adapted

* Typeform working

* Added dynamic-schedule to catalog

* Added background-fetch job catalog

* Created dynamic-triggers catalog file

* Fixed old general file with runTask param order

* Dynamic triggers working

* SendGrid updated to use the same tsconfig as other integrations

* Removed Airtable webhook, until we have batch support

* Added OAuth airtable auth example

* Created beta changeset tag

* Beta changesets for most packages

---------

Co-authored-by: Eric Allam <eallam@icloud.com>
2023-09-05 14:21:42 +01:00
Chigala aa7458fe37 refactor: reduced max retries on some task to 1 on development 2023-08-24 23:50:33 +01:00
Eric Allam 4d316c5524 Fixed the OAuth2 callback urls by constructing URL using x-forwarded-proto header 2023-07-03 13:47:58 +01:00
Eric Allam fd8fd22bdc Fixed https missing from url 2023-06-22 21:35:49 +01:00
Eric Allam 786f2e16ef Static scheduled triggers are working, and now adding graphile worker jobs are done in prisma txs 2023-05-23 14:00:26 +01:00
Eric Allam f6ea37381a Re-work how triggers work to better support dynamic triggers 2023-05-16 16:34:51 +01:00
Eric Allam 1bba822e20 Added projects, moved envs to projects, and renamed executions to runs 2023-04-27 13:50:22 +01:00
Eric Allam da0290e494 Adding support for connections, integrations, webhooks 2023-04-22 09:52:45 -07:00