Files
Matt Aitken ecd050bece Airtable integration with webhook and integration changes (#399)
* CLI create-integration command now accepts an Open AI api key

* Create integration docs separated into multiple pages

* Initial Airtable integration commit, with OpenAI generated code

* OAuth page coming soon

* Export DisplayProperty from the SDK

* TSConfig made to match GitHub’s with paths

* getRecords

* Removed duplicate Stripe job from the catalog

* Renamed Airtable apiKey option to token

* First Airtable job

* Export Collaborator and Attachment field types

* A typesafe example that uses runTask

* WIP on new integration tasks… not working yet

* Attempt with class

* Revert "Attempt with class"

This reverts commit 93a48330019f754c3216c5b49964fa4b0218bd3f.

* WIP changing how tasks work

* Mock of async local storage

* Moved client creation from constructor

* New approach with a clone method on TriggerIntegration

* Added runTask to Airtable which is used by integration tasks

* Added the Airtable icon and connection when using runTask

* base().table() is working

* runTask options moved to the 3rd param, made optional with optional name

* Added some generic arguments

* Added generic type to table

* Removed old comment

* We don’t need to repeat the icon

* getRecords and getRecord now returning the right data and types

* Creating records

* Update records

* Delete records

* The internal properties of integrations are now hidden by the TypeScript types

* Sprinkled a Prettify in there

* Improved the types

* Added Airtable to the integration catalog

* Early work on Airtable webhook registration

* More progress with webhooks

* connectionKey needs to be cloned for webhooks to work

* connectionKey needs to be cloned for webhooks to work

* It was unclear that the ActivateSourceService was using a graphileJob id

* ActivateSourceService optionally takes a jobId, if missing it generate a unique id

* When retrying trigger registration, don’t pass an id so it is generated

* Removed Airtable webhooks tasks from the job-catalog example

* Added TriggerSourceOption, removed TriggerSourceEvent

* WIP with new ExternalSource options

* ExternalSourceTrigger setup

* DynamicTrigger changed to options, will need some more work

* filter gets options passed to it

* SourceMetadata v2 renamed to SourceMetadataV2, kept original

* Started versioning the backend

* Moved param order on io.getEvent and io.cancelEvent

* The runTask stuff that allows unknown to work is back

* Indexing for v1 and v2, with version on “activateSource” schema

* Added todos, to deal with Airtable SDK calls inside the webhook handler

* “deliverHttpSourceRequest” queueName changed to the source id so they process in order

* ActivateSource changes to deal with old and new data formats

* Update existing TriggerSources to v2

* Fix for dynamic.ts typescript errors, need to revisit this later

* UpdateSourceService v1 and v2, with new v2 API endpoint

* Removed unused imports

* More progress on v1 and v2

* Airtable webhooks are now triggering a job

* Moved webhooks to a new file

* You can do API calls in the webhook handler now, Airtable webhook data is being processed

* Airtable events coming through

* Defined the Airtable table payload type

* TriggerSource metadata is being stored and used

* Removed some logs

* Added filtering and don’t allow any webhooks that use automated sources

* Resend switched to new integration

* Moved Resend test jobs to the catalog, and tested it worked

* WIP on Slack, there are compile errors

* Created a generic type that strips out indexes

* Slack updated to use new integration

* SendGrid migrated over

* Integration runTask is now allowing regular types

* Changed io.runTask types so it only allows Json-able types

* OpenAI models tasks working

* Added Airtable changes to runTask

* Removed the index signature crap from the Slack integration

* Don’t need to cast the callback result

* Updated Resend

* Re-ordered runTask params

* WIP on openai

* onAccountUpdated is Connect only

* Removed RunTaskResult

* Handle Resend errors, the official SDK doesn’t expose them properly at the moment

* Removed OmitIndexSignature

* OpenAI converted to new integration, with backwards compatible functions

* Put the openai catalog back to what it was originally

* Export a standard retry with backoff, to be used

* Use the standard exponential backoff in the integrations

* Retry options moved earlier so they can be overriden by a task

* GitHub tasks migrated

* Added sources, fixed one bundling issue

* Added GitHub jobs to catalog

* Remove duplicate options

* Deduplicate events

* Removed duplicate Job

* Switched Plain over

* Set the Plain icon

* Converted Stripe over

* Supabase adapted

* Typeform working

* Added dynamic-schedule to catalog

* Added background-fetch job catalog

* Created dynamic-triggers catalog file

* Fixed old general file with runTask param order

* Dynamic triggers working

* SendGrid updated to use the same tsconfig as other integrations

* Removed Airtable webhook, until we have batch support

* Added OAuth airtable auth example

* Created beta changeset tag

* Beta changesets for most packages

---------

Co-authored-by: Eric Allam <eallam@icloud.com>
2023-09-05 14:21:42 +01:00

297 lines
7.8 KiB
TypeScript

import { Webhooks } from "@octokit/webhooks";
import { ExternalSource, TriggerIntegration, HandlerEvent } from "@trigger.dev/sdk";
import type { Logger } from "@trigger.dev/sdk";
import { safeJsonParse, omit } from "@trigger.dev/integration-kit";
import { Octokit } from "octokit";
import { z } from "zod";
import { Github } from "./index";
type WebhookData = {
id: number;
active: boolean;
events: string[];
config: {
url: string;
};
};
function webhookData(data: any): data is WebhookData {
return (
typeof data === "object" &&
data !== null &&
typeof data.id === "number" &&
typeof data.config === "object"
);
}
export function createRepoEventSource(
integration: Github
): ExternalSource<Github, { owner: string; repo: string }, "HTTP", {}> {
return new ExternalSource("HTTP", {
id: "github.repo",
version: "0.1.1",
schema: z.object({ owner: z.string(), repo: z.string() }),
integration,
key: (params) => `${params.owner}/${params.repo}`,
properties: (params) => [
{
label: "Owner",
text: params.owner,
url: `https://github.com/${params.owner}`,
},
{
label: "Repo",
text: params.repo,
url: `https://github.com/${params.owner}/${params.repo}`,
},
],
filter: (params) => ({
repository: {
full_name: [`${params.owner}/${params.repo}`],
},
}),
handler: webhookHandler,
register: async (event, io, ctx) => {
const { params, source: httpSource, options } = event;
const registeredOptions = {
event: options.event.desired,
};
if (httpSource.active && webhookData(httpSource.data)) {
const hasMissingOptions = Object.values(options).some(
(option) => option.missing.length > 0
);
if (!hasMissingOptions) return;
// We need to update the webhook to add the new events and then return
const newWebhookData = await io.integration.updateWebhook("update-webhook", {
owner: params.owner,
repo: params.repo,
hookId: httpSource.data.id,
url: httpSource.url,
secret: httpSource.secret,
addEvents: options.event.missing,
});
return {
data: newWebhookData,
options: registeredOptions,
};
}
const webhooks = await io.integration.listWebhooks("list-webhooks", {
owner: params.owner,
repo: params.repo,
});
const existingWebhook = webhooks.find((w) => w.config.url === httpSource.url);
// There is an existing webhook, but it's not synced with Trigger.dev, so we need to update it with the secret
if (existingWebhook && existingWebhook.active) {
const updatedWebhook = await io.integration.updateWebhook("update-webhook", {
owner: params.owner,
repo: params.repo,
hookId: existingWebhook.id,
url: httpSource.url,
secret: httpSource.secret,
addEvents: options.event.missing,
});
return {
data: updatedWebhook,
options: registeredOptions,
};
}
const webhook = await io.integration.createWebhook("create-webhook", {
owner: params.owner,
repo: params.repo,
events: options.event.desired,
url: httpSource.url,
secret: httpSource.secret,
});
return { data: webhook, options: registeredOptions };
},
});
}
export function createOrgEventSource(
integration: Github
): ExternalSource<Github, { org: string }, "HTTP", {}> {
return new ExternalSource("HTTP", {
id: "github.org",
version: "0.1.1",
integration,
schema: z.object({ org: z.string() }),
key: (params) => params.org,
properties: (params) => [
{
label: "Org",
text: params.org,
url: `https://github.com/${params.org}`,
},
],
filter: (params) => ({
organization: {
login: [params.org],
},
}),
handler: webhookHandler,
register: async (event, io, ctx) => {
const { params, source: httpSource, options } = event;
const registeredOptions = {
event: options.event.desired,
};
const hasMissingOptions = Object.values(options).some((option) => option.missing.length > 0);
if (
httpSource.active &&
webhookData(httpSource.data) &&
httpSource.secret &&
hasMissingOptions
) {
const existingData = httpSource.data;
// We need to update the webhook to add the new events and then return
const newWebhookData = await io.integration.updateOrgWebhook("update-webhook", {
org: params.org,
hookId: existingData.id,
url: httpSource.url,
secret: httpSource.secret,
addEvents: options.event.missing,
});
return {
secret: httpSource.secret,
data: newWebhookData,
options: registeredOptions,
};
}
const webhooks = await io.integration.listOrgWebhooks("list-webhooks", {
org: params.org,
});
const existingWebhook = webhooks.find((w) => w.config.url === httpSource.url);
const secret = Math.random().toString(36).slice(2);
if (existingWebhook && existingWebhook.active) {
const updatedWebhook = await io.integration.updateOrgWebhook("update-webhook", {
org: params.org,
hookId: existingWebhook.id,
url: httpSource.url,
secret,
});
return {
secret,
data: updatedWebhook,
options: registeredOptions,
};
}
const webhook = await io.integration.createOrgWebhook("create-webhook", {
org: params.org,
events: options.event.desired,
url: httpSource.url,
secret,
});
return { secret, data: webhook, options: registeredOptions };
},
});
}
// Parses the body of a request
// If it's a Buffer, it will be parsed as JSON
function parseBody(body: any) {
if (Buffer.isBuffer(body)) {
return safeJsonParse(body.toString());
}
if (typeof body === "string") {
return safeJsonParse(body);
}
return body;
}
async function webhookHandler(event: HandlerEvent<"HTTP">, logger: Logger) {
logger.debug("[inside github integration] Handling github repo event");
const { rawEvent: request, source } = event;
if (!request.body) {
logger.debug("[inside github integration] No body found");
return;
}
const rawBody = await request.text();
const deliveryId = request.headers.get("x-github-delivery");
const hookId = request.headers.get("x-github-hook-id");
const signature = request.headers.get("x-hub-signature-256");
if (source.secret && signature) {
const githubWebhooks = new Webhooks({
secret: source.secret,
});
if (!githubWebhooks.verify(rawBody, signature)) {
logger.debug("[inside github integration] Unable to verify the signature of the rawBody", {
signature,
secret: source.secret,
});
return;
}
}
const name = request.headers.get("x-github-event") ?? "unknown";
const allHeaders = Object.fromEntries(request.headers.entries());
const context = omit(allHeaders, [
"x-github-event",
"x-github-delivery",
"x-hub-signature-256",
"x-hub-signature",
"content-type",
"content-length",
"accept",
"accept-encoding",
"x-forwarded-proto",
]);
const payload = parseBody(rawBody);
if (!payload) {
logger.debug("[inside github integration] Unable to parse the rawBody");
return;
}
logger.debug("[inside github integration] Returning an event for the webhook!", {
name,
payload,
context,
});
return {
events: [
{
id: [hookId, deliveryId].join(":"),
source: "github.com",
payload,
name,
context,
},
],
};
}