Compare commits

...

20 Commits

Author SHA1 Message Date
Eric Allam 8bd1ca8b66 Fix pnpm lock file 2023-08-21 14:01:38 +01:00
github-actions[bot] 7f5ce165a0 chore: Update version for release (#361)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2023-08-21 13:45:35 +01:00
Eric Allam 7df1b90928 Tweak some of the task max attempts
🚀 Publish Trigger.dev Docker / ʦ TypeScript (push) Has been cancelled
🚀 Publish Trigger.dev Docker / Unit Tests (push) Has been cancelled
🚀 Publish Trigger.dev Docker / e2e Tests (push) Has been cancelled
🚀 Publish Trigger.dev Docker / publish (push) Has been cancelled
2023-08-21 09:52:55 +01:00
Atharva Deshpande 57a7e00bfa Fix spelling mistake in README.md (#371) 2023-08-19 15:23:22 +01:00
Eric Allam 7362feed71 Fix for old clients missing the cached tasks 2023-08-19 07:32:34 +01:00
Eric Allam b1b9321ad2 Job run performance improvements and adding "worker only" mode (#360)
* WIP job run performance improvements

- Added a `perf` tool to better measure job run performance under heavy load
- Removed `runFinished` job (not really needed)
- startQueuedRuns now uses a jobKey with replace
- Fixed an issue with ZodWorker when using jobKey

* Publish improvement docker images

* fixed the improvement docker publishing

* Downgrade back to prisma 4.16.0 because 5.1.x broke docker builds

* Changes to how queued runs work

- Split the worker into two different workers, one dedicated to performRunExecution
- Schedule performRunExecution in a single place, with a queue and using a round robin manually controlled concurrency
- Remove startQueuedRuns
- All runs are queued before they are started
- Setting the worker maxPoolSize to the same as the worker concurrency
- Starting to be able to split the docker image

* Remove queue name from startRun graphile job

* Make the prisma connection pool stuff configurable through env vars

* Hardcode (for now) the max concurrent runs limit

* Rewrite performRunExecution to be more performant

PerformRunExecutionV2:

- Does not create and manage jobRunExecution records
- Does not reimplement retrying, uses graphile worker retrying instead

I’ve kept around PerformRunExecutionV1 so this works when deploying. Definitely needs LOTS of testing

* Fix issues with cached tasks

- Limit the size of the cached tasks sent when executing a run, using the knapsack problem dynamic programming approach
- Actually USE the cached tasks in IO by using the idempotencyKey instead of the task ID
- Remove output from all logs
- Added a stress test job catalog

* Forgot to commit the logger updates

* Never log connectionString

* Login to docker hub to get around rate limits

* Add additional logging to the graphile workers

* Fix the *_ENABLED  env vars

* Allow adding and removing jobs to be done from the webapp

* Don’t set the job to failed if it’s being retried

* Deprecated queue options in the job and removed startPosition. Now using the job/env combo as the job queue name

* Dequeung jobs doesn’t check if the runner is initialized

* Fixed issues with retrying a run getting stuck on a cancelled task, and errors from parsing the results of dequeing a job

* Remove queued round robin thing that isn’t used anymore

* Added slack to job catalog

* Better forwards compat

* Added long delay

* Fixed lock file
2023-08-18 15:25:26 +01:00
Dan a69f756e34 Added delay example to init.ts so it works with the pages router (#363) 2023-08-18 15:22:18 +01:00
Matt Aitken 0f6e580641 Added SendGrid to the integrations table in docs 2023-08-18 15:13:16 +01:00
Chigala 796f1209f2 Docs/sendgrid doc (#359)
* added sengrid integration to the catalog

* added documention for sendgrid integration

* updated the mint.json file
2023-08-18 15:09:04 +01:00
Liran Tal 4ce96b7d28 fix: avoid code duplication by repostioning the default switch case (#356) 2023-08-18 14:48:07 +01:00
Wesley b86ffa0d3d test/add test on get user pkg manager (#349)
* chore: make possible to write tests aside with implementations

* test: write getUserPkgManager test TO-DOs

* test: add test to building the path

* test: add tests to manual file checking

* test: add tests to npm_config_user_agent

* Update packages/cli/src/utils/getUserPkgManager.spec.ts
2023-08-18 14:43:19 +01:00
Wesley 3ee7cd6ff6 feat: parse callback result with zod before return (#342) 2023-08-18 14:33:42 +01:00
Wesley 74686c00cb feature/TRI 712/create eslint plugin to ensure uniqueness on task keys (#324)
* test: add test to no-duplicated-task-keys rule

* feat: implement no-duplicated-task-keys

* chore: create testing scripts

* refactor: fill in rule meta

* feat: export no-duplicated-task-keys rule

* chore: bump package version to 0.0.1

* chore: create eslint plugin

* feat: create no-duplicated-task-keys rule

* refactor: delete no-duplicated-task-keys from config-custom

* refactor: rename eslint-plugin folder

* feat: cover case from nextjs-example

* feat: cover additional cases from examples/package-tester

* chore: add eslint-plugin on nextjs-example

* feat: add more cases from `nextjs-example`

* revert: rollback changes on eslint-config-custom

* Set version to 2.0.9, inline with other packages

* Create chilly-pianos-try.md

---------

Co-authored-by: Matt Aitken <matt@mattaitken.com>
2023-08-18 12:57:12 +01:00
Liran Tal 8f3e550d03 fix: small typo in function docblocks (#362) 2023-08-18 12:17:45 +01:00
Matt Aitken 591422b8cf CLI: added whoami command, fixed TypeScript error 2023-08-18 11:18:51 +01:00
neo773 bbaa6ba156 [TRI-1006] New @trigger.dev/cli whoami command. (#316)
* FIx: [TRI-1006] New @trigger.dev/cli whoami command.

* revert dev.ts zod schema to original

* Removed telemetry

* remove all telemetry

* Made the clientId optional again

---------

Co-authored-by: Matt Aitken <matt@mattaitken.com>
2023-08-18 10:41:06 +01:00
Liran Tal e20fa3c2ec fix: explicitly set a charset for rendered HTML documents (#355) 2023-08-18 06:49:39 +01:00
Liran Tal 57514f8771 fix: code quality nitpicks for some issues spotted (#357) 2023-08-18 06:48:48 +01:00
Wesley 83bcefe10c refactor: remove .js extension from cli package (#350)
* refactor: remove `.js` extension from cli package

* revert: rollback changes

* revert: rollback unnecessary changes

* Update createIntegration.ts

---------

Co-authored-by: Eric Allam <eallam@icloud.com>
2023-08-17 16:03:00 +01:00
Wesley 4ca9f182d7 test/TRI 887/setup jest on cli package (#326)
* chore: run `pnpm install --prefer-frozen-lockfile`

* chore: install `jest` and `ts-jest`

* test: create `jest.config.js`

* chore: install `@types/jest`

* chore: add `jest` on `tsconfig.json` `types` property

* chore: create test script

* test: create example test 🎉🎊🕺🏼

* chore: install `@gmrchk/cli-testing-library`

* chore: override `tsconfig` with `@trigger.dev/tsconfig`

* test: make poc of cli test

* chore: run build before test

* docs: improve comment

* Added a changeset

---------

Co-authored-by: Matt Aitken <matt@mattaitken.com>
2023-08-16 22:35:23 +01:00
121 changed files with 3198 additions and 887 deletions
+16
View File
@@ -4,6 +4,7 @@ on:
push:
branches:
- main
- improvements/*
tags:
- "v.docker.*"
paths:
@@ -95,6 +96,12 @@ jobs:
name: e2e Tests
runs-on: buildjet-4vcpu-ubuntu-2204
steps:
- name: 🐳 Login to Docker Hub
uses: docker/login-action@v2
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: ⬇️ Checkout repo
uses: actions/checkout@v3
with:
@@ -154,6 +161,11 @@ jobs:
version: ${{ steps.get_version.outputs.version }}
short_sha: ${{ steps.get_commit.outputs.sha_short }}
steps:
- name: 🐳 Login to Docker Hub
uses: docker/login-action@v2
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: ⬇️ Checkout repo
uses: actions/checkout@v3
@@ -167,6 +179,10 @@ jobs:
IMAGE_TAG="v${ORIGINAL_VERSION}"
fi
echo "IMAGE_TAG=${IMAGE_TAG}"
elif [[ $GITHUB_REF == refs/heads/improvements/* ]]; then
ORIGINAL_VERSION="${GITHUB_REF#refs/heads/improvements/}"
IMAGE_TAG="${ORIGINAL_VERSION}.rc"
echo "IMAGE_TAG=${IMAGE_TAG}"
elif [[ $GITHUB_REF == refs/heads/* ]]; then
IMAGE_TAG="${GITHUB_REF#refs/heads/}"
echo "IMAGE_TAG=${IMAGE_TAG}"
+1 -1
View File
@@ -62,4 +62,4 @@ We provide an official trigger.dev docker image you can use to easily self-host
## Development
To setup and develop locally or contribute to the open source project, follow our [developement guide](./CONTRIBUTING.md).
To setup and develop locally or contribute to the open source project, follow our [development guide](./CONTRIBUTING.md).
@@ -273,6 +273,7 @@ function BlankTasks({
basicStatus: RunBasicStatus;
}) {
switch (basicStatus) {
default:
case "COMPLETED":
return <Paragraph variant="small">There were no tasks for this run.</Paragraph>;
case "FAILED":
@@ -288,8 +289,6 @@ function BlankTasks({
<TaskCardSkeleton />
</div>
);
default:
return <Paragraph variant="small">There were no tasks for this run.</Paragraph>;
}
}
+2 -1
View File
@@ -1,6 +1,7 @@
export const LIVE_ENVIRONMENT = "live";
export const DEV_ENVIRONMENT = "development";
export const MAX_LIVE_PROJECTS = 1;
export const DEFAULT_MAX_CONCURRENT_RUNS = 10000;
export const DEFAULT_MAX_CONCURRENT_RUNS = 10;
export const MAX_CONCURRENT_RUNS_LIMIT = 20;
export const PREPROCESS_RETRY_LIMIT = 2;
export const EXECUTE_JOB_RETRY_LIMIT = 10;
+11 -3
View File
@@ -2,6 +2,7 @@ import { PrismaClient, Prisma } from "@trigger.dev/database";
import invariant from "tiny-invariant";
import { z } from "zod";
import { logger } from "./services/logger.server";
import { env } from "./env.server";
export type PrismaTransactionClient = Omit<
PrismaClient,
@@ -84,8 +85,16 @@ function getClient() {
const { DATABASE_URL } = process.env;
invariant(typeof DATABASE_URL === "string", "DATABASE_URL env var not set");
const databaseUrl = new URL(DATABASE_URL);
// We need to add the connection_limit and pool_timeout query params to the url, in a way that works if the DATABASE_URL already has query params
const query = databaseUrl.searchParams;
query.set("connection_limit", env.DATABASE_CONNECTION_LIMIT.toString());
query.set("pool_timeout", env.DATABASE_POOL_TIMEOUT.toString());
databaseUrl.search = query.toString();
// Remove the username:password in the url and print that to the console
const urlWithoutCredentials = new URL(DATABASE_URL);
const urlWithoutCredentials = new URL(databaseUrl.href);
urlWithoutCredentials.password = "";
console.log(`🔌 setting up prisma client to ${urlWithoutCredentials.toString()}`);
@@ -93,8 +102,7 @@ function getClient() {
const client = new PrismaClient({
datasources: {
db: {
url: DATABASE_URL,
// We can't set directUrl here, and we don't have to
url: databaseUrl.href,
},
},
log: [
+2 -2
View File
@@ -74,7 +74,7 @@ function serveTheBots(
{
// Use onAllReady to wait for the entire document to be ready
onAllReady() {
responseHeaders.set("Content-Type", "text/html");
responseHeaders.set("Content-Type", "text/html; charset=utf-8");
let body = new PassThrough();
pipe(body);
resolve(
@@ -114,7 +114,7 @@ function serveBrowsers(
// use onShellReady to wait until a suspense boundary is triggered
onShellReady() {
shellReady = true;
responseHeaders.set("Content-Type", "text/html");
responseHeaders.set("Content-Type", "text/html; charset=utf-8");
let body = new PassThrough();
pipe(body);
resolve(
+9
View File
@@ -4,6 +4,8 @@ import { SecretStoreOptionsSchema } from "./services/secrets/secretStore.server"
const EnvironmentSchema = z.object({
NODE_ENV: z.union([z.literal("development"), z.literal("production"), z.literal("test")]),
DATABASE_URL: z.string(),
DATABASE_CONNECTION_LIMIT: z.coerce.number().int().default(10),
DATABASE_POOL_TIMEOUT: z.coerce.number().int().default(60),
DIRECT_URL: z.string(),
SESSION_SECRET: z.string(),
MAGIC_LINK_SECRET: z.string(),
@@ -31,6 +33,13 @@ const EnvironmentSchema = z.object({
RESEND_API_KEY: z.string().optional(),
PLAIN_API_KEY: z.string().optional(),
RUNTIME_PLATFORM: z.enum(["docker-compose", "ecs", "local"]).default("local"),
WORKER_SCHEMA: z.string().default("graphile_worker"),
WORKER_CONCURRENCY: z.coerce.number().int().default(10),
WORKER_POLL_INTERVAL: z.coerce.number().int().default(1000),
EXECUTION_WORKER_CONCURRENCY: z.coerce.number().int().default(10),
EXECUTION_WORKER_POLL_INTERVAL: z.coerce.number().int().default(1000),
WORKER_ENABLED: z.string().default("true"),
EXECUTION_WORKER_ENABLED: z.string().default("true"),
});
export type Environment = z.infer<typeof EnvironmentSchema>;
@@ -0,0 +1,58 @@
import { JobRun, JobRunExecution } from "@trigger.dev/database";
import { PrismaClientOrTransaction } from "~/db.server";
import { executionWorker } from "~/services/worker.server";
export async function enqueueRunExecutionV1(
execution: JobRunExecution,
queueId: string,
concurrency: number,
tx: PrismaClientOrTransaction,
runAt?: Date
) {
const job = await executionWorker.enqueue(
"performRunExecution",
{
id: execution.id,
},
{
queueName: `job:queue:${queueId}`,
tx,
runAt,
jobKey: `execution:${execution.runId}`,
}
);
}
export type EnqueueRunExecutionV2Options = {
runAt?: Date;
resumeTaskId?: string;
isRetry?: boolean;
};
export async function enqueueRunExecutionV2(
run: JobRun,
tx: PrismaClientOrTransaction,
options: EnqueueRunExecutionV2Options = {}
) {
const job = await executionWorker.enqueue(
"performRunExecutionV2",
{
id: run.id,
reason: run.status === "PREPROCESSING" ? "PREPROCESS" : "EXECUTE_JOB",
resumeTaskId: options.resumeTaskId,
isRetry: typeof options.isRetry === "boolean" ? options.isRetry : false,
},
{
queueName: `job:${run.jobId}:env:${run.environmentId}`,
tx,
runAt: options.runAt,
jobKey: `job_run:${run.id}`,
}
);
}
export async function dequeueRunExecutionV2(run: JobRun, tx: PrismaClientOrTransaction) {
return await executionWorker.dequeue(`job_run:${run.id}`, {
tx,
});
}
+116 -8
View File
@@ -51,6 +51,7 @@ const AddJobResultsSchema = z.array(GraphileJobSchema);
export type ZodTasks<TConsumerSchema extends MessageCatalogSchema> = {
[K in keyof TConsumerSchema]: {
queueName?: string | ((payload: z.infer<TConsumerSchema[K]>) => string);
jobKey?: string | ((payload: z.infer<TConsumerSchema[K]>) => string | undefined);
priority?: number;
maxAttempts?: number;
jobKeyMode?: "replace" | "preserve_run_at" | "unsafe_dedupe";
@@ -76,7 +77,12 @@ export type ZodWorkerEnqueueOptions = TaskSpec & {
tx?: PrismaClientOrTransaction;
};
export type ZodWorkerDequeueOptions = {
tx?: PrismaClientOrTransaction;
};
export type ZodWorkerOptions<TMessageCatalog extends MessageCatalogSchema> = {
name: string;
runnerOptions: RunnerOptions;
prisma: PrismaClient;
schema: TMessageCatalog;
@@ -85,6 +91,7 @@ export type ZodWorkerOptions<TMessageCatalog extends MessageCatalogSchema> = {
};
export class ZodWorker<TMessageCatalog extends MessageCatalogSchema> {
#name: string;
#schema: TMessageCatalog;
#prisma: PrismaClient;
#runnerOptions: RunnerOptions;
@@ -93,6 +100,7 @@ export class ZodWorker<TMessageCatalog extends MessageCatalogSchema> {
#runner?: GraphileRunner;
constructor(options: ZodWorkerOptions<TMessageCatalog>) {
this.#name = options.name;
this.#schema = options.schema;
this.#prisma = options.prisma;
this.#runnerOptions = options.runnerOptions;
@@ -105,7 +113,7 @@ export class ZodWorker<TMessageCatalog extends MessageCatalogSchema> {
return true;
}
logger.debug("Initializing worker queue with options", {
this.#logDebug("Initializing worker queue with options", {
runnerOptions: this.#runnerOptions,
});
@@ -121,6 +129,54 @@ export class ZodWorker<TMessageCatalog extends MessageCatalogSchema> {
throw new Error("Failed to initialize worker queue");
}
this.#runner?.events.on("pool:create", ({ workerPool }) => {
this.#logDebug("pool:create");
});
this.#runner?.events.on("pool:listen:connecting", ({ workerPool, attempts }) => {
this.#logDebug("pool:create", { attempts });
});
this.#runner?.events.on("pool:listen:success", ({ workerPool, client }) => {
this.#logDebug("pool:listen:success");
});
this.#runner?.events.on("pool:listen:error", ({ error }) => {
this.#logDebug("pool:listen:error", { error });
});
this.#runner?.events.on("pool:gracefulShutdown", ({ message }) => {
this.#logDebug("pool:gracefulShutdown", { workerMessage: message });
});
this.#runner?.events.on("pool:gracefulShutdown:error", ({ error }) => {
this.#logDebug("pool:gracefulShutdown:error", { error });
});
this.#runner?.events.on("worker:create", ({ worker }) => {
this.#logDebug("worker:create", { workerId: worker.workerId });
});
this.#runner?.events.on("worker:release", ({ worker }) => {
this.#logDebug("worker:release", { workerId: worker.workerId });
});
this.#runner?.events.on("worker:stop", ({ worker, error }) => {
this.#logDebug("worker:stop", { workerId: worker.workerId, error });
});
this.#runner?.events.on("worker:fatalError", ({ worker, error, jobError }) => {
this.#logDebug("worker:fatalError", { workerId: worker.workerId, error, jobError });
});
this.#runner?.events.on("gracefulShutdown", ({ signal }) => {
this.#logDebug("gracefulShutdown", { signal });
});
this.#runner?.events.on("stop", () => {
this.#logDebug("stop");
});
return true;
}
@@ -133,23 +189,34 @@ export class ZodWorker<TMessageCatalog extends MessageCatalogSchema> {
payload: z.infer<TMessageCatalog[K]>,
options?: ZodWorkerEnqueueOptions
): Promise<GraphileJob> {
if (!this.#runner) {
throw new Error("Worker not initialized");
}
const task = this.#tasks[identifier];
const optionsWithoutTx = omit(options ?? {}, ["tx"]);
const taskWithoutJobKey = omit(task, ["jobKey"]);
const spec = {
...optionsWithoutTx,
...task,
...taskWithoutJobKey,
};
if (typeof task.queueName === "function") {
spec.queueName = task.queueName(payload);
}
if (typeof task.jobKey === "function") {
const jobKey = task.jobKey(payload);
if (jobKey) {
spec.jobKey = jobKey;
}
}
logger.debug("Enqueuing worker task", {
identifier,
payload,
spec,
});
const job = await this.#addJob(
identifier as string,
payload,
@@ -167,6 +234,17 @@ export class ZodWorker<TMessageCatalog extends MessageCatalogSchema> {
return job;
}
public async dequeue(
jobKey: string,
option?: ZodWorkerDequeueOptions
): Promise<GraphileJob | undefined> {
const results = await this.#removeJob(jobKey, option?.tx ?? this.#prisma);
logger.debug("dequeued worker task", { results, jobKey });
return results;
}
async #addJob(
identifier: string,
payload: unknown,
@@ -192,8 +270,8 @@ export class ZodWorker<TMessageCatalog extends MessageCatalogSchema> {
spec.maxAttempts || null,
spec.jobKey || null,
spec.priority || null,
spec.jobKeyMode || null,
spec.flags || null
spec.flags || null,
spec.jobKeyMode || null
);
const rows = AddJobResultsSchema.safeParse(results);
@@ -209,6 +287,32 @@ export class ZodWorker<TMessageCatalog extends MessageCatalogSchema> {
return job as GraphileJob;
}
async #removeJob(jobKey: string, tx: PrismaClientOrTransaction) {
try {
const result = await tx.$queryRawUnsafe(
`SELECT * FROM graphile_worker.remove_job(
job_key => $1::text
)`,
jobKey
);
const job = AddJobResultsSchema.safeParse(result);
if (!job.success) {
logger.debug("results returned from remove_job could not be parsed", {
error: job.error.flatten(),
result,
jobKey,
});
return;
}
return job.data[0] as GraphileJob;
} catch (e) {
throw new Error(`Failed to remove job from queue, ${e}}`);
}
}
#createTaskListFromTasks() {
const taskList: TaskList = {};
@@ -324,4 +428,8 @@ export class ZodWorker<TMessageCatalog extends MessageCatalogSchema> {
throw error;
}
}
#logDebug(message: string, args?: any) {
logger.debug(`[worker][${this.#name}] ${message}`, args);
}
}
@@ -107,6 +107,32 @@ export class RunTaskService {
});
if (existingTask) {
if (existingTask.status === "CANCELED") {
const existingTaskStatus =
(taskBody.delayUntil && taskBody.delayUntil.getTime() > Date.now()) || taskBody.trigger
? "WAITING"
: taskBody.noop
? "COMPLETED"
: "RUNNING";
const resumedExistingTask = await tx.task.update({
where: {
id: existingTask.id,
},
data: {
status: existingTaskStatus,
startedAt: new Date(),
completedAt: existingTaskStatus === "COMPLETED" ? new Date() : undefined,
},
include: {
run: true,
attempts: true,
},
});
return resumedExistingTask;
}
return existingTask;
}
@@ -2,6 +2,7 @@ import { parse } from "@conform-to/zod";
import { ActionFunction, json } from "@remix-run/node";
import { z } from "zod";
import { redirectWithSuccessMessage } from "~/models/message.server";
import { logger } from "~/services/logger.server";
import { CancelRunService } from "~/services/runs/cancelRun.server";
export const cancelSchema = z.object({
@@ -31,7 +32,19 @@ export const action: ActionFunction = async ({ request, params }) => {
request,
`Canceled run. Any pending tasks will be canceled.`
);
} catch (error: any) {
return json({ errors: { body: error.message } }, { status: 400 });
} catch (error) {
if (error instanceof Error) {
logger.error("Failed to cancel run", {
error: {
name: error.name,
message: error.message,
stack: error.stack,
},
});
return json({ errors: { body: error.message } }, { status: 400 });
} else {
logger.error("Failed to cancel run", { error });
return json({ errors: { body: "Unknown error" } }, { status: 400 });
}
}
};
@@ -162,6 +162,10 @@ export class EndpointApi {
}
async executeJobRequest(options: RunJobBody) {
logger.debug("executeJobRequest()", {
options,
});
const response = await safeFetch(this.url, {
method: "POST",
headers: {
@@ -4,6 +4,7 @@ import { plain } from "./integrations/plain";
import { resend } from "./integrations/resend";
import { slack } from "./integrations/slack";
import { stripe } from "./integrations/stripe";
import { sendgrid } from "./integrations/sendgrid";
import { supabaseManagement, supabase } from "./integrations/supabase";
import { typeform } from "./integrations/typeform";
import type { Integration } from "./types";
@@ -34,8 +35,9 @@ export const integrationCatalog = new IntegrationCatalog({
plain,
resend,
slack,
typeform,
stripe,
supabaseManagement,
supabase,
sendgrid,
typeform,
});
@@ -109,15 +109,9 @@ export class IntegrationConnectionCreatedService {
});
// We need to start the run again
await workerQueue.enqueue(
"startRun",
{
id: run.id,
},
{
queueName: `job-queue:${run.queue.id}`,
}
);
await workerQueue.enqueue("startRun", {
id: run.id,
});
}
}
}
@@ -0,0 +1,60 @@
import type { Integration } from "../types";
export const sendgrid: Integration = {
identifier: "sendgrid",
name: "SendGrid",
packageName: "@trigger.dev/sendgrid@latest",
authenticationMethods: {
apikey: {
type: "apikey",
help: {
samples: [
{
title: "Creating the client",
code: `
import { SendGrid } from "@trigger.dev/sendgrid";
const sendgrid = new SendGrid({
id: "__SLUG__",
apiKey: process.env.SENDGRID_API_KEY!,
});
`,
},
{
title: "Using the client",
code: `
client.defineJob({
id: "send-sendgrid-email",
name: "Send SendGrid Email",
version: "0.1.0",
trigger: eventTrigger({
name: "send.email",
schema: z.object({
to: z.string(),
subject: z.string(),
text: z.string(),
}),
}),
integrations: {
sendgrid,
},
run: async (payload, io, ctx) => {
await io.sendgrid.sendEmail({
to: payload.to,
from: "Trigger.dev <hello@email.trigger.dev>",
subject: payload.subject,
text: payload.text,
});
},
});
`,
highlight: [
[13, 15],
[17, 22],
],
},
],
},
},
},
};
@@ -1,18 +1,18 @@
import type { Endpoint, Integration, Job, JobIntegration, JobVersion } from "@trigger.dev/database";
import {
IntegrationConfig,
JobMetadata,
SCHEDULED_EVENT,
TriggerMetadata,
} from "@trigger.dev/core";
import type { Endpoint, Integration, Job, JobIntegration, JobVersion } from "@trigger.dev/database";
import { DEFAULT_MAX_CONCURRENT_RUNS } from "~/consts";
import type { PrismaClient } from "~/db.server";
import { prisma } from "~/db.server";
import { ExtendedEndpoint, findEndpoint } from "~/models/endpoint.server";
import type { RuntimeEnvironment } from "~/models/runtimeEnvironment.server";
import type { AuthenticatedEnvironment } from "../apiAuth.server";
import { logger } from "../logger.server";
import type { RuntimeEnvironment } from "~/models/runtimeEnvironment.server";
import { RegisterScheduleSourceService } from "../schedules/registerScheduleSource.server";
import { ExtendedEndpoint, findEndpoint } from "~/models/endpoint.server";
export class RegisterJobService {
#prismaClient: PrismaClient;
@@ -166,13 +166,9 @@ export class RegisterJobService {
});
// Upsert the JobQueue
const queueName =
typeof metadata.queue === "string"
? metadata.queue
: typeof metadata.queue === "object"
? metadata.queue.name
: "default";
const queueName = "default";
// Job Queues are going to be deprecated or used for something else, we're just doing this for now
const jobQueue = await this.#prismaClient.jobQueue.upsert({
where: {
environmentId_name: {
@@ -187,16 +183,10 @@ export class RegisterJobService {
},
},
name: queueName,
maxJobs:
typeof metadata.queue === "object"
? metadata.queue.maxConcurrent || DEFAULT_MAX_CONCURRENT_RUNS
: DEFAULT_MAX_CONCURRENT_RUNS,
maxJobs: DEFAULT_MAX_CONCURRENT_RUNS,
},
update: {
maxJobs:
typeof metadata.queue === "object"
? metadata.queue.maxConcurrent || DEFAULT_MAX_CONCURRENT_RUNS
: DEFAULT_MAX_CONCURRENT_RUNS,
maxJobs: DEFAULT_MAX_CONCURRENT_RUNS,
},
});
@@ -245,10 +235,10 @@ export class RegisterJobService {
version: metadata.version,
eventSpecification,
preprocessRuns: metadata.preprocessRuns,
startPosition: metadata.startPosition === "initial" ? "INITIAL" : "LATEST",
startPosition: "LATEST",
},
update: {
startPosition: metadata.startPosition === "initial" ? "INITIAL" : "LATEST",
startPosition: "LATEST",
eventSpecification,
preprocessRuns: metadata.preprocessRuns,
queue: {
+1 -1
View File
@@ -5,6 +5,6 @@ import { sensitiveDataReplacer } from "./sensitiveDataReplacer";
export const logger = new Logger(
"webapp",
(process.env.APP_LOG_LEVEL ?? "debug") as LogLevel,
["examples"],
["examples", "output", "connectionString", "payload"],
sensitiveDataReplacer
);
@@ -1,5 +1,6 @@
import { PrismaClient, prisma } from "~/db.server";
import { workerQueue } from "../worker.server";
import { executionWorker } from "../worker.server";
import { dequeueRunExecutionV2 } from "~/models/jobRunExecution.server";
export class CancelRunService {
#prismaClient: PrismaClient;
@@ -17,21 +18,11 @@ export class CancelRunService {
},
});
const shouldDecrementQueue = run.status === "STARTED" || run.status === "PREPROCESSING";
await tx.jobRun.update({
where: { id: runId },
data: {
status: "CANCELED",
completedAt: new Date(),
queue: shouldDecrementQueue
? {
update: {
jobCount: {
decrement: 1,
},
},
}
: undefined,
},
});
@@ -48,13 +39,7 @@ export class CancelRunService {
},
});
await workerQueue.enqueue(
"startQueuedRuns",
{
id: run.queueId,
},
{ tx }
);
await dequeueRunExecutionV2(run, tx);
});
} catch (error) {
throw error;
@@ -1,6 +1,5 @@
import { EXECUTE_JOB_RETRY_LIMIT } from "~/consts";
import { $transaction, Prisma, PrismaClient, prisma } from "~/db.server";
import { workerQueue } from "../worker.server";
import { enqueueRunExecutionV2 } from "~/models/jobRunExecution.server";
const RESUMABLE_STATUSES = ["FAILURE", "TIMED_OUT", "ABORTED", "CANCELED"];
@@ -17,86 +16,26 @@ export class ContinueRunService {
async (tx) => {
const run = await tx.jobRun.findUniqueOrThrow({
where: { id: runId },
include: {
queue: true,
},
});
if (!RESUMABLE_STATUSES.includes(run.status)) {
throw new Error("Run is not resumable");
}
if (run.queue.jobCount >= run.queue.maxJobs) {
await tx.jobRun.update({
where: { id: runId },
data: {
status: "QUEUED",
queuedAt: new Date(),
startedAt: null,
completedAt: null,
output: Prisma.DbNull,
timedOutAt: null,
timedOutReason: null,
},
});
} else {
await tx.jobRun.update({
where: { id: runId },
data: {
status: "STARTED",
queuedAt: null,
startedAt: new Date(),
completedAt: null,
output: Prisma.DbNull,
timedOutAt: null,
timedOutReason: null,
queue: {
update: {
jobCount: {
increment: 1,
},
},
},
},
});
await tx.jobRun.update({
where: { id: runId },
data: {
status: "QUEUED",
queuedAt: new Date(),
startedAt: null,
completedAt: null,
output: Prisma.DbNull,
timedOutAt: null,
timedOutReason: null,
},
});
const execution = await tx.jobRunExecution.create({
data: {
run: {
connect: {
id: runId,
},
},
status: "PENDING",
reason: "EXECUTE_JOB",
retryLimit: EXECUTE_JOB_RETRY_LIMIT,
isRetry: true,
},
});
const job = await workerQueue.enqueue(
"performRunExecution",
{
id: execution.id,
},
{ tx }
);
await tx.jobRunExecution.update({
where: { id: execution.id },
data: {
graphileJobId: job.id,
},
});
await workerQueue.enqueue(
"startQueuedRuns",
{
id: run.queueId,
},
{ tx }
);
}
await enqueueRunExecutionV2(run, tx);
},
{ timeout: 10000 }
);
@@ -75,7 +75,7 @@ export class CreateRunService {
{
id: run.id,
},
{ tx, queueName: `job-queue:${jobQueue.id}` }
{ tx }
);
return run;
@@ -1,27 +1,26 @@
import type { Task } from "@trigger.dev/database";
import {
ApiEventLogSchema,
CachedTaskSchema,
RunJobCanceledWithTask,
RunJobError,
RunJobResumeWithTask,
RunJobRetryWithTask,
RunJobSuccess,
RunSourceContextSchema,
} from "@trigger.dev/core";
import type { Task } from "@trigger.dev/database";
import { generateErrorMessage } from "zod-error";
import { EXECUTE_JOB_RETRY_LIMIT } from "~/consts";
import { $transaction, PrismaClient, PrismaClientOrTransaction, prisma } from "~/db.server";
import { enqueueRunExecutionV1 } from "~/models/jobRunExecution.server";
import { resolveRunConnections } from "~/models/runConnection.server";
import { formatError } from "~/utils/formatErrors.server";
import { safeJsonZodParse } from "~/utils/json";
import { EndpointApi } from "../endpointApi.server";
import { workerQueue } from "../worker.server";
import { formatError } from "~/utils/formatErrors.server";
import { logger } from "../logger.server";
type FoundRunExecution = NonNullable<Awaited<ReturnType<typeof findRunExecution>>>;
export class PerformRunExecutionService {
export class PerformRunExecutionV1Service {
#prismaClient: PrismaClient;
constructor(prismaClient: PrismaClient = prisma) {
@@ -162,22 +161,7 @@ export class PerformRunExecutionService {
},
});
const job = await workerQueue.enqueue(
"performRunExecution",
{
id: runExecution.id,
},
{ tx }
);
await tx.jobRunExecution.update({
where: {
id: runExecution.id,
},
data: {
graphileJobId: job.id,
},
});
await enqueueRunExecutionV1(runExecution, run.queue.id, run.queue.maxJobs, tx);
});
}
}
@@ -201,6 +185,12 @@ export class PerformRunExecutionService {
data: {
status: "STARTED",
startedAt,
run: {
update: {
status: run.status === "QUEUED" ? "STARTED" : run.status,
startedAt: run.startedAt ?? new Date(),
},
},
},
});
@@ -388,14 +378,6 @@ export class PerformRunExecutionService {
completedAt: new Date(),
},
});
await workerQueue.enqueue(
"runFinished",
{
id: run.id,
},
{ tx }
);
});
}
@@ -426,22 +408,13 @@ export class PerformRunExecutionService {
},
});
const graphileJob = await workerQueue.enqueue(
"performRunExecution",
{
id: newJobExecution.id,
},
{ tx, runAt: data.task.delayUntil ?? undefined }
await enqueueRunExecutionV1(
newJobExecution,
run.queue.id,
run.queue.maxJobs,
tx,
data.task.delayUntil ?? undefined
);
await tx.jobRunExecution.update({
where: {
id: newJobExecution.id,
},
data: {
graphileJobId: graphileJob.id,
},
});
}
});
}
@@ -522,22 +495,13 @@ export class PerformRunExecutionService {
},
});
const graphileJob = await workerQueue.enqueue(
"performRunExecution",
{
id: newJobExecution.id,
},
{ tx, runAt: data.retryAt }
await enqueueRunExecutionV1(
newJobExecution,
run.queue.id,
run.queue.maxJobs,
tx,
data.retryAt
);
await tx.jobRunExecution.update({
where: {
id: newJobExecution.id,
},
data: {
graphileJobId: graphileJob.id,
},
});
});
}
@@ -557,6 +521,13 @@ export class PerformRunExecutionService {
// So when retryCount is 1, retryDelayInMs is 500ms
// When retryCount is 2, retryDelayInMs is 750ms
// When retryCount is 3, retryDelayInMs is 1125ms
// When retryCount is 4, retryDelayInMs is 1687ms
// When retryCount is 5, retryDelayInMs is 2531ms
// When retryCount is 6, retryDelayInMs is 3796ms
// When retryCount is 7, retryDelayInMs is 5694ms
// When retryCount is 8, retryDelayInMs is 8541ms
// When retryCount is 9, retryDelayInMs is 12812ms
// When retryCount is 10, retryDelayInMs is 19218ms
const retryDelayInMs = Math.round(500 * Math.pow(1.5, retryCount - 1));
await tx.jobRunExecution.update({
@@ -572,20 +543,13 @@ export class PerformRunExecutionService {
const runAt = new Date(Date.now() + retryDelayInMs);
const job = await workerQueue.enqueue(
"performRunExecution",
{ id: execution.id },
{ runAt, tx }
await enqueueRunExecutionV1(
execution,
execution.run.queue.id,
execution.run.queue.maxJobs,
tx,
runAt
);
await tx.jobRunExecution.update({
where: {
id: execution.id,
},
data: {
graphileJobId: job.id,
},
});
});
}
@@ -617,13 +581,6 @@ export class PerformRunExecutionService {
},
});
await workerQueue.enqueue(
"runFinished",
{
id: run.id,
},
{ tx }
);
break;
}
case "PREPROCESS": {
@@ -645,14 +602,6 @@ export class PerformRunExecutionService {
},
});
await workerQueue.enqueue(
"runFinished",
{
id: run.id,
},
{ tx }
);
break;
}
@@ -675,22 +624,7 @@ export class PerformRunExecutionService {
},
});
const job = await workerQueue.enqueue(
"performRunExecution",
{
id: runExecution.id,
},
{ tx }
);
await tx.jobRunExecution.update({
where: {
id: runExecution.id,
},
data: {
graphileJobId: job.id,
},
});
await enqueueRunExecutionV1(runExecution, run.queue.id, run.queue.maxJobs, tx);
break;
}
@@ -733,6 +667,7 @@ async function findRunExecution(prisma: PrismaClientOrTransaction, id: string) {
endpoint: true,
organization: true,
externalAccount: true,
queue: true,
runConnections: {
include: {
integration: true,
@@ -0,0 +1,593 @@
import {
ApiEventLogSchema,
CachedTask,
RunJobError,
RunJobResumeWithTask,
RunJobRetryWithTask,
RunJobSuccess,
RunSourceContextSchema,
} from "@trigger.dev/core";
import type { Task } from "@trigger.dev/database";
import { generateErrorMessage } from "zod-error";
import { $transaction, PrismaClient, PrismaClientOrTransaction, prisma } from "~/db.server";
import { enqueueRunExecutionV2 } from "~/models/jobRunExecution.server";
import { resolveRunConnections } from "~/models/runConnection.server";
import { formatError } from "~/utils/formatErrors.server";
import { safeJsonZodParse } from "~/utils/json";
import { EndpointApi } from "../endpointApi.server";
import { logger } from "../logger.server";
type FoundRun = NonNullable<Awaited<ReturnType<typeof findRun>>>;
type FoundTask = FoundRun["tasks"][number];
export class PerformRunExecutionV2Service {
#prismaClient: PrismaClient;
constructor(prismaClient: PrismaClient = prisma) {
this.#prismaClient = prismaClient;
}
public async call(
id: string,
reason: "PREPROCESS" | "EXECUTE_JOB",
isRetry: boolean = false,
resumeTaskId?: string
) {
const run = await findRun(this.#prismaClient, id);
if (!run) {
return;
}
switch (reason) {
case "PREPROCESS": {
await this.#executePreprocessing(run);
break;
}
case "EXECUTE_JOB": {
await this.#executeJob(run, isRetry, resumeTaskId);
break;
}
}
}
// Execute the preprocessing step of a run, which will send the payload to the endpoint and give the job
// an opportunity to generate run properties based on the payload.
// If the endpoint is not available, or the response is not ok,
// the run execution will be marked as failed and the run will start
async #executePreprocessing(run: FoundRun) {
const client = new EndpointApi(run.environment.apiKey, run.endpoint.url);
const event = ApiEventLogSchema.parse({ ...run.event, id: run.eventId });
const { response, parser } = await client.preprocessRunRequest({
event,
job: {
id: run.version.job.slug,
version: run.version.version,
},
run: {
id: run.id,
isTest: run.isTest,
},
environment: {
id: run.environment.id,
slug: run.environment.slug,
type: run.environment.type,
},
organization: {
id: run.organization.id,
slug: run.organization.slug,
title: run.organization.title,
},
account: run.externalAccount
? {
id: run.externalAccount.identifier,
metadata: run.externalAccount.metadata,
}
: undefined,
});
if (!response) {
return await this.#failRunExecution(this.#prismaClient, "PREPROCESS", run, {
message: "Could not connect to the endpoint",
});
}
if (!response.ok) {
return await this.#failRunExecution(this.#prismaClient, "PREPROCESS", run, {
message: `Endpoint responded with ${response.status} status code`,
});
}
const rawBody = await response.text();
const safeBody = safeJsonZodParse(parser, rawBody);
if (!safeBody) {
return await this.#failRunExecution(this.#prismaClient, "PREPROCESS", run, {
message: "Endpoint responded with invalid JSON",
});
}
if (!safeBody.success) {
return await this.#failRunExecution(this.#prismaClient, "PREPROCESS", run, {
message: generateErrorMessage(safeBody.error.issues),
});
}
if (safeBody.data.abort) {
return this.#failRunExecution(
this.#prismaClient,
"PREPROCESS",
run,
{ message: "Endpoint aborted the run" },
"ABORTED"
);
} else {
await $transaction(this.#prismaClient, async (tx) => {
await tx.jobRun.update({
where: {
id: run.id,
},
data: {
status: "STARTED",
startedAt: new Date(),
properties: safeBody.data.properties,
},
});
await enqueueRunExecutionV2(run, tx);
});
}
}
async #executeJob(run: FoundRun, isRetry: boolean, resumeTaskId?: string) {
if (run.status === "CANCELED") {
await this.#cancelExecution(run);
return;
}
const client = new EndpointApi(run.environment.apiKey, run.endpoint.url);
const event = ApiEventLogSchema.parse({ ...run.event, id: run.eventId });
const startedAt = new Date();
await this.#prismaClient.jobRun.update({
where: {
id: run.id,
},
data: {
status: run.status === "QUEUED" ? "STARTED" : run.status,
startedAt: run.startedAt ?? new Date(),
},
});
const connections = await resolveRunConnections(run.runConnections);
if (!connections.success) {
return this.#failRunExecutionWithRetry({
message: `Could not resolve all connections for run ${run.id}, attempting to retry`,
});
}
let resumedTask: Task | undefined;
if (resumeTaskId) {
resumedTask =
(await this.#prismaClient.task.findUnique({
where: {
id: resumeTaskId,
},
})) ?? undefined;
if (resumedTask) {
resumedTask = await this.#prismaClient.task.update({
where: {
id: resumeTaskId,
},
data: {
status: resumedTask.noop ? "COMPLETED" : "RUNNING",
completedAt: resumedTask.noop ? new Date() : undefined,
},
});
}
}
const sourceContext = RunSourceContextSchema.safeParse(run.event.sourceContext);
const { response, parser, errorParser } = await client.executeJobRequest({
event,
job: {
id: run.version.job.slug,
version: run.version.version,
},
run: {
id: run.id,
isTest: run.isTest,
startedAt,
isRetry,
},
environment: {
id: run.environment.id,
slug: run.environment.slug,
type: run.environment.type,
},
organization: {
id: run.organization.id,
slug: run.organization.slug,
title: run.organization.title,
},
account: run.externalAccount
? {
id: run.externalAccount.identifier,
metadata: run.externalAccount.metadata,
}
: undefined,
connections: connections.auth,
source: sourceContext.success ? sourceContext.data : undefined,
tasks: prepareTasksForRun([run.tasks, resumedTask].flat().filter(Boolean)),
});
if (!response) {
return await this.#failRunExecutionWithRetry({
message: `Connection could not be established to the endpoint (${run.endpoint.url})`,
});
}
const rawBody = await response.text();
if (!response.ok) {
logger.debug("Endpoint responded with non-200 status code", {
status: response.status,
runId: run.id,
endpoint: run.endpoint.url,
});
const errorBody = safeJsonZodParse(errorParser, rawBody);
if (errorBody && errorBody.success) {
// Only retry if the error isn't a 4xx
if (response.status >= 400 && response.status <= 499) {
return await this.#failRunExecution(
this.#prismaClient,
"EXECUTE_JOB",
run,
errorBody.data
);
} else {
return await this.#failRunExecutionWithRetry(errorBody.data);
}
}
// Only retry if the error isn't a 4xx
if (response.status >= 400 && response.status <= 499 && response.status !== 408) {
return await this.#failRunExecution(this.#prismaClient, "EXECUTE_JOB", run, {
message: `Endpoint responded with ${response.status} status code`,
});
} else {
return await this.#failRunExecutionWithRetry({
message: `Endpoint responded with ${response.status} status code`,
});
}
}
const safeBody = safeJsonZodParse(parser, rawBody);
if (!safeBody) {
return await this.#failRunExecution(this.#prismaClient, "EXECUTE_JOB", run, {
message: "Endpoint responded with invalid JSON",
});
}
if (!safeBody.success) {
return await this.#failRunExecution(this.#prismaClient, "EXECUTE_JOB", run, {
message: generateErrorMessage(safeBody.error.issues),
});
}
const status = safeBody.data.status;
switch (status) {
case "SUCCESS": {
await this.#completeRunWithSuccess(run, safeBody.data);
break;
}
case "RESUME_WITH_TASK": {
await this.#resumeRunWithTask(run, safeBody.data, isRetry);
break;
}
case "ERROR": {
await this.#failRunWithError(run, safeBody.data);
break;
}
case "RETRY_WITH_TASK": {
await this.#retryRunWithTask(run, safeBody.data, isRetry);
break;
}
case "CANCELED": {
await this.#cancelExecution(run);
break;
}
default: {
const _exhaustiveCheck: never = status;
throw new Error(`Non-exhaustive match for value: ${status}`);
}
}
}
async #completeRunWithSuccess(run: FoundRun, data: RunJobSuccess) {
await this.#prismaClient.jobRun.update({
where: { id: run.id },
data: {
completedAt: new Date(),
status: "SUCCESS",
output: data.output ?? undefined,
},
});
}
async #resumeRunWithTask(run: FoundRun, data: RunJobResumeWithTask, isRetry: boolean) {
return await $transaction(this.#prismaClient, async (tx) => {
// If the task has an operation, then the next performRunExecution will occur
// when that operation has finished
if (!data.task.operation) {
await enqueueRunExecutionV2(run, tx, {
runAt: data.task.delayUntil ?? undefined,
resumeTaskId: data.task.id,
isRetry,
});
}
});
}
async #failRunWithError(execution: FoundRun, data: RunJobError) {
return await $transaction(this.#prismaClient, async (tx) => {
if (data.task) {
await tx.task.update({
where: {
id: data.task.id,
},
data: {
status: "ERRORED",
completedAt: new Date(),
output: data.error ?? undefined,
},
});
}
await this.#failRunExecution(tx, "EXECUTE_JOB", execution, data.error ?? undefined);
});
}
async #retryRunWithTask(run: FoundRun, data: RunJobRetryWithTask, isRetry: boolean) {
return await $transaction(this.#prismaClient, async (tx) => {
// We need to check for an existing task attempt
const existingAttempt = await tx.taskAttempt.findFirst({
where: {
taskId: data.task.id,
status: "PENDING",
},
orderBy: {
number: "desc",
},
});
if (existingAttempt) {
await tx.taskAttempt.update({
where: {
id: existingAttempt.id,
},
data: {
status: "ERRORED",
error: formatError(data.error),
},
});
}
// We need to create a new task attempt
await tx.taskAttempt.create({
data: {
taskId: data.task.id,
number: existingAttempt ? existingAttempt.number + 1 : 1,
status: "PENDING",
runAt: data.retryAt,
},
});
await tx.task.update({
where: {
id: data.task.id,
},
data: {
status: "WAITING",
},
});
await enqueueRunExecutionV2(run, tx, {
runAt: data.retryAt,
resumeTaskId: data.task.id,
isRetry,
});
});
}
async #failRunExecutionWithRetry(output: Record<string, any>): Promise<void> {
throw new Error(JSON.stringify(output));
}
async #failRunExecution(
prisma: PrismaClientOrTransaction,
reason: "EXECUTE_JOB" | "PREPROCESS",
run: FoundRun,
output: Record<string, any>,
status: "FAILURE" | "ABORTED" = "FAILURE"
): Promise<void> {
await $transaction(prisma, async (tx) => {
switch (reason) {
case "EXECUTE_JOB": {
// If the execution is an EXECUTE_JOB reason, we need to fail the run
await tx.jobRun.update({
where: { id: run.id },
data: {
completedAt: new Date(),
status,
output,
},
});
break;
}
case "PREPROCESS": {
// If the status is ABORTED, we need to fail the run
if (status === "ABORTED") {
await tx.jobRun.update({
where: { id: run.id },
data: {
completedAt: new Date(),
status,
output,
},
});
break;
}
await tx.jobRun.update({
where: {
id: run.id,
},
data: {
status: "STARTED",
startedAt: new Date(),
},
});
await enqueueRunExecutionV2(run, tx);
break;
}
}
});
}
async #cancelExecution(run: FoundRun) {
return;
}
}
function prepareTasksForRun(tasks: FoundTask[]): CachedTask[] {
// We need to limit the cached tasks to not be too large >3.5MB when serialized
const TOTAL_CACHED_TASK_BYTE_LIMIT = 3500000;
const cachedTasks = new Map<string, CachedTask>(); // Cache for prepared tasks
const cachedTaskSizes = new Map<string, number>(); // Cache for calculated task sizes
// Helper function to get the cached prepared task, or prepare and cache if not already cached
function getCachedTask(task: FoundTask): CachedTask {
const taskId = task.id;
if (!cachedTasks.has(taskId)) {
cachedTasks.set(taskId, prepareTaskForRun(task));
}
return cachedTasks.get(taskId)!;
}
// Helper function to get the cached task size, or calculate and cache if not already cached
function getCachedTaskSize(task: CachedTask): number {
const taskId = task.id;
if (!cachedTaskSizes.has(taskId)) {
cachedTaskSizes.set(taskId, calculateCachedTaskSize(task));
}
return cachedTaskSizes.get(taskId)!;
}
// Create a dynamic programming array to store intermediate results
const dp: number[][] = [];
for (let i = 0; i <= tasks.length; i++) {
dp[i] = [];
for (let j = 0; j <= TOTAL_CACHED_TASK_BYTE_LIMIT; j++) {
dp[i][j] = 0;
}
}
// Fill the dynamic programming array
for (let i = 1; i <= tasks.length; i++) {
const task = tasks[i - 1];
const cachedTask = getCachedTask(task);
const taskSize = getCachedTaskSize(cachedTask);
for (let j = 0; j <= TOTAL_CACHED_TASK_BYTE_LIMIT; j++) {
if (taskSize <= j) {
dp[i][j] = Math.max(dp[i - 1][j], dp[i - 1][j - taskSize] + taskSize);
} else {
dp[i][j] = dp[i - 1][j];
}
}
}
// Traverse the dynamic programming array to find the included tasks
const tasksToRun: CachedTask[] = [];
let j = TOTAL_CACHED_TASK_BYTE_LIMIT;
for (let i = tasks.length; i > 0 && j > 0; i--) {
if (dp[i][j] !== dp[i - 1][j]) {
const task = tasks[i - 1];
const cachedTask = getCachedTask(task);
tasksToRun.unshift(cachedTask);
j -= getCachedTaskSize(cachedTask);
}
}
return tasksToRun;
}
function prepareTaskForRun(task: FoundTask): CachedTask {
return {
id: task.idempotencyKey, // We should eventually move this back to task.id
status: task.status,
idempotencyKey: task.idempotencyKey,
noop: task.noop,
output: task.output as any,
parentId: task.parentId,
};
}
function calculateCachedTaskSize(task: CachedTask): number {
return JSON.stringify(task).length;
}
async function findRun(prisma: PrismaClientOrTransaction, id: string) {
return await prisma.jobRun.findUnique({
where: { id },
include: {
environment: true,
endpoint: true,
organization: true,
externalAccount: true,
queue: true,
runConnections: {
include: {
integration: true,
connection: {
include: {
dataReference: true,
},
},
},
},
tasks: {
where: {
status: {
in: ["COMPLETED"],
},
},
},
event: true,
version: {
include: {
job: true,
organization: true,
},
},
},
});
}
@@ -1,56 +0,0 @@
import type { PrismaClient } from "~/db.server";
import { prisma } from "~/db.server";
import { workerQueue } from "../worker.server";
import { z } from "zod";
import { RawEventSchema, SendEventOptionsSchema } from "@trigger.dev/core";
import { IngestSendEvent } from "../events/ingestSendEvent.server";
const SendEventOutputSchema = z.object({
events: z.array(RawEventSchema),
options: SendEventOptionsSchema.optional(),
});
export class RunFinishedService {
#prismaClient: PrismaClient;
#ingestEventService = new IngestSendEvent();
constructor(prismaClient: PrismaClient = prisma) {
this.#prismaClient = prismaClient;
}
public async call(id: string) {
const run = await this.#prismaClient.jobRun.findUniqueOrThrow({
where: { id },
include: {
environment: {
include: {
project: true,
organization: true,
},
},
},
});
// Make sure to start any queued runs once this run is finished
await workerQueue.enqueue("startQueuedRuns", {
id: run.queueId,
});
if (
run.status === "SUCCESS" &&
run.output &&
typeof run.output === "object" &&
"events" in run.output
) {
// If the run successfully completes, we will parse the output and
// if it's in the form of { events: Array<RawEvent> } then we will send the events
const parsedOutput = SendEventOutputSchema.safeParse(run.output);
if (parsedOutput.success) {
for (const newEvent of parsedOutput.data.events) {
await this.#ingestEventService.call(run.environment, newEvent, parsedOutput.data.options);
}
}
}
}
}
@@ -1,56 +0,0 @@
import type { PrismaClient } from "~/db.server";
import { prisma } from "~/db.server";
import { workerQueue } from "../worker.server";
export class StartQueuedRunsService {
#prismaClient: PrismaClient;
constructor(prismaClient: PrismaClient = prisma) {
this.#prismaClient = prismaClient;
}
public async call(id: string) {
const queue = await this.#prismaClient.jobQueue.findUnique({
where: { id },
include: {
runs: {
where: {
status: "QUEUED",
},
orderBy: {
queuedAt: "asc",
},
take: 1,
},
},
});
if (!queue) {
return;
}
if (queue.runs.length === 0) {
return;
}
if (queue.jobCount >= queue.maxJobs) {
return;
}
const run = queue.runs[0];
if (!run) {
return;
}
await workerQueue.enqueue(
"startRun",
{
id: run.id,
},
{
queueName: `job-queue:${queue.id}`,
}
);
}
}
@@ -1,7 +1,7 @@
import type { ConnectionType, Integration, IntegrationConnection } from "@trigger.dev/database";
import { EXECUTE_JOB_RETRY_LIMIT, PREPROCESS_RETRY_LIMIT } from "~/consts";
import type { PrismaClient, PrismaClientOrTransaction } from "~/db.server";
import { prisma } from "~/db.server";
import { enqueueRunExecutionV2 } from "~/models/jobRunExecution.server";
import { workerQueue } from "../worker.server";
type FoundRun = NonNullable<Awaited<ReturnType<typeof findRun>>>;
@@ -21,34 +21,20 @@ export class StartRunService {
return;
}
if (run.queue.jobCount >= run.queue.maxJobs) {
await this.#queueRun(id);
} else {
const runConnectionsByKey = await createRunConnections(this.#prismaClient, run);
const runConnectionsByKey = await createRunConnections(this.#prismaClient, run);
if (hasMissingConnections(runConnectionsByKey)) {
await this.#handleMissingConnections(id, runConnectionsByKey);
} else {
await this.#startRun(id, run, runConnectionsByKey);
}
if (hasMissingConnections(runConnectionsByKey)) {
await this.#handleMissingConnections(id, runConnectionsByKey);
} else {
await this.#startRun(id, run, runConnectionsByKey);
}
}
#runIsStartable(run: FoundRun) {
const startableStatuses = ["PENDING", "QUEUED", "WAITING_ON_CONNECTIONS"] as const;
const startableStatuses = ["PENDING", "WAITING_ON_CONNECTIONS"] as const;
return startableStatuses.includes(run.status);
}
async #queueRun(id: string) {
await this.#prismaClient.jobRun.update({
where: { id },
data: {
status: "QUEUED",
queuedAt: new Date(),
},
});
}
async #startRun(id: string, run: FoundRun, runConnectionsByKey: RunConnectionsByKey) {
const createRunConnections = Object.entries(runConnectionsByKey)
.map(([key, runConnection]) =>
@@ -69,89 +55,35 @@ export class StartRunService {
)
.filter(Boolean);
const updateRunAndCreateExecution = async () => {
const updateRun = async () => {
if (run.preprocess) {
// Start the jobRun and increment the jobCount
await this.#prismaClient.jobRun.update({
return await this.#prismaClient.jobRun.update({
where: { id },
data: {
status: "PREPROCESSING",
queue: {
update: {
jobCount: {
increment: 1,
},
},
},
runConnections: {
create: createRunConnections,
},
},
});
return await this.#prismaClient.jobRunExecution.create({
data: {
run: {
connect: {
id,
},
},
status: "PENDING",
reason: "PREPROCESS",
retryLimit: PREPROCESS_RETRY_LIMIT,
},
});
} else {
// Start the jobRun and increment the jobCount
await this.#prismaClient.jobRun.update({
return await this.#prismaClient.jobRun.update({
where: { id },
data: {
status: "STARTED",
startedAt: new Date(),
queue: {
update: {
jobCount: {
increment: 1,
},
},
},
status: "QUEUED",
queuedAt: new Date(),
runConnections: {
create: createRunConnections,
},
},
});
return await this.#prismaClient.jobRunExecution.create({
data: {
run: {
connect: {
id,
},
},
status: "PENDING",
reason: "EXECUTE_JOB",
retryLimit: EXECUTE_JOB_RETRY_LIMIT,
},
});
}
};
const execution = await updateRunAndCreateExecution();
const updatedRun = await updateRun();
const job = await workerQueue.enqueue("performRunExecution", {
id: execution.id,
});
await this.#prismaClient.jobRunExecution.update({
where: { id: execution.id },
data: {
graphileJobId: job.id,
},
});
await workerQueue.enqueue("startQueuedRuns", {
id: run.queueId,
});
await enqueueRunExecutionV2(updatedRun, this.#prismaClient);
}
async #handleMissingConnections(id: string, runConnectionsByKey: RunConnectionsByKey) {
@@ -1,7 +1,3 @@
import type { Task } from "@trigger.dev/database";
import { EXECUTE_JOB_RETRY_LIMIT } from "~/consts";
import { $transaction, PrismaClient, PrismaClientOrTransaction, prisma } from "~/db.server";
import { workerQueue } from "../worker.server";
import {
FetchOperationSchema,
FetchRequestInit,
@@ -10,9 +6,13 @@ import {
RedactString,
calculateRetryAt,
} from "@trigger.dev/core";
import type { Task } from "@trigger.dev/database";
import { $transaction, PrismaClient, PrismaClientOrTransaction, prisma } from "~/db.server";
import { enqueueRunExecutionV2 } from "~/models/jobRunExecution.server";
import { formatUnknownError } from "~/utils/formatErrors.server";
import { safeJsonFromResponse } from "~/utils/json";
import { logger } from "../logger.server";
import { formatUnknownError } from "~/utils/formatErrors.server";
import { workerQueue } from "../worker.server";
type FoundTask = Awaited<ReturnType<typeof findTask>>;
@@ -192,7 +192,7 @@ export class PerformTaskOperationService {
});
}
async #resumeTaskWithError(task: Task, output: any) {
async #resumeTaskWithError(task: NonNullable<FoundTask>, output: any) {
await $transaction(this.#prismaClient, async (tx) => {
await tx.task.update({
where: { id: task.id },
@@ -243,34 +243,8 @@ export class PerformTaskOperationService {
});
}
async #resumeRunExecution(task: Task, prisma: PrismaClientOrTransaction) {
await $transaction(prisma, async (tx) => {
const newJobExecution = await tx.jobRunExecution.create({
data: {
runId: task.runId,
reason: "EXECUTE_JOB",
status: "PENDING",
retryLimit: EXECUTE_JOB_RETRY_LIMIT,
},
});
const graphileJob = await workerQueue.enqueue(
"performRunExecution",
{
id: newJobExecution.id,
},
{ tx }
);
await tx.jobRunExecution.update({
where: {
id: newJobExecution.id,
},
data: {
graphileJobId: graphileJob.id,
},
});
});
async #resumeRunExecution(task: NonNullable<FoundTask>, prisma: PrismaClientOrTransaction) {
await enqueueRunExecutionV2(task.run, prisma);
}
}
@@ -305,6 +279,11 @@ async function findTask(prisma: PrismaClient, id: string) {
where: { id },
include: {
attempts: true,
run: {
include: {
queue: true,
},
},
},
});
}
+91 -41
View File
@@ -12,9 +12,8 @@ import { InvokeDispatcherService } from "./events/invokeDispatcher.server";
import { integrationAuthRepository } from "./externalApis/integrationAuthRepository.server";
import { IntegrationConnectionCreatedService } from "./externalApis/integrationConnectionCreated.server";
import { MissingConnectionCreatedService } from "./runs/missingConnectionCreated.server";
import { PerformRunExecutionService } from "./runs/performRunExecution.server";
import { RunFinishedService } from "./runs/runFinished.server";
import { StartQueuedRunsService } from "./runs/startQueuedRuns.server";
import { PerformRunExecutionV1Service } from "./runs/performRunExecutionV1.server";
import { PerformRunExecutionV2Service } from "./runs/performRunExecutionV2.server";
import { StartRunService } from "./runs/startRun.server";
import { DeliverScheduledEventService } from "./schedules/deliverScheduledEvent.server";
import { ActivateSourceService } from "./sources/activateSource.server";
@@ -30,13 +29,9 @@ const workerCatalog = {
}),
scheduleEmail: DeliverEmailSchema,
startRun: z.object({ id: z.string() }),
performRunExecution: z.object({
id: z.string(),
}),
performTaskOperation: z.object({
id: z.string(),
}),
runFinished: z.object({ id: z.string() }),
deliverHttpSourceRequest: z.object({ id: z.string() }),
refreshOAuthToken: z.object({
organizationId: z.string(),
@@ -46,7 +41,7 @@ const workerCatalog = {
id: z.string(),
orphanedEvents: z.array(z.string()).optional(),
}),
startQueuedRuns: z.object({ id: z.string() }),
deliverEvent: z.object({ id: z.string() }),
"events.invokeDispatcher": z.object({
id: z.string(),
@@ -64,10 +59,24 @@ const workerCatalog = {
}),
};
const executionWorkerCatalog = {
performRunExecution: z.object({
id: z.string(),
}),
performRunExecutionV2: z.object({
id: z.string(),
reason: z.enum(["EXECUTE_JOB", "PREPROCESS"]),
resumeTaskId: z.string().optional(),
isRetry: z.boolean(),
}),
};
let workerQueue: ZodWorker<typeof workerCatalog>;
let executionWorker: ZodWorker<typeof executionWorkerCatalog>;
declare global {
var __worker__: ZodWorker<typeof workerCatalog>;
var __executionWorker__: ZodWorker<typeof executionWorkerCatalog>;
}
// this is needed because in development we don't want to restart
@@ -76,25 +85,41 @@ declare global {
// in production we'll have a single connection to the DB.
if (env.NODE_ENV === "production") {
workerQueue = getWorkerQueue();
executionWorker = getExecutionWorkerQueue();
} else {
if (!global.__worker__) {
global.__worker__ = getWorkerQueue();
}
workerQueue = global.__worker__;
if (!global.__executionWorker__) {
global.__executionWorker__ = getExecutionWorkerQueue();
}
executionWorker = global.__executionWorker__;
}
export async function init() {
await workerQueue.initialize();
if (env.WORKER_ENABLED === "true") {
await workerQueue.initialize();
}
if (env.EXECUTION_WORKER_ENABLED === "true") {
await executionWorker.initialize();
}
}
function getWorkerQueue() {
return new ZodWorker({
name: "workerQueue",
prisma,
runnerOptions: {
connectionString: env.DATABASE_URL,
concurrency: 5,
pollInterval: 1000,
concurrency: env.WORKER_CONCURRENCY,
pollInterval: env.WORKER_POLL_INTERVAL,
noPreparedStatements: env.DATABASE_URL !== env.DIRECT_URL,
schema: env.WORKER_SCHEMA,
maxPoolSize: env.WORKER_CONCURRENCY,
},
schema: workerCatalog,
recurringTasks: {
@@ -124,6 +149,7 @@ function getWorkerQueue() {
},
tasks: {
"events.invokeDispatcher": {
priority: 0, // smaller number = higher priority
maxAttempts: 3,
handler: async (payload, job) => {
const service = new InvokeDispatcherService();
@@ -132,6 +158,7 @@ function getWorkerQueue() {
},
},
"events.deliverScheduled": {
priority: 0, // smaller number = higher priority
maxAttempts: 5,
handler: async ({ id, payload }, job) => {
const service = new DeliverScheduledEventService();
@@ -140,6 +167,7 @@ function getWorkerQueue() {
},
},
connectionCreated: {
priority: 10, // smaller number = higher priority
maxAttempts: 3,
handler: async (payload, job) => {
const service = new IntegrationConnectionCreatedService();
@@ -148,6 +176,7 @@ function getWorkerQueue() {
},
},
missingConnectionCreated: {
priority: 10, // smaller number = higher priority
maxAttempts: 3,
handler: async (payload, job) => {
const service = new MissingConnectionCreatedService();
@@ -155,24 +184,8 @@ function getWorkerQueue() {
await service.call(payload.id);
},
},
runFinished: {
maxAttempts: 3,
handler: async (payload, job) => {
const service = new RunFinishedService();
await service.call(payload.id);
},
},
startQueuedRuns: {
maxAttempts: 3,
queueName: (payload) => `queue:${payload.id}`,
handler: async (payload, job) => {
const service = new StartQueuedRunsService();
await service.call(payload.id);
},
},
activateSource: {
priority: 10, // smaller number = higher priority
maxAttempts: 3,
handler: async (payload, job) => {
const service = new ActivateSourceService();
@@ -181,7 +194,8 @@ function getWorkerQueue() {
},
},
deliverHttpSourceRequest: {
maxAttempts: 25,
priority: 1, // smaller number = higher priority
maxAttempts: 14,
handler: async (payload, job) => {
const service = new DeliverHttpSourceRequestService();
@@ -189,23 +203,16 @@ function getWorkerQueue() {
},
},
startRun: {
maxAttempts: 8,
priority: 0, // smaller number = higher priority
maxAttempts: 4,
handler: async (payload, job) => {
const service = new StartRunService();
await service.call(payload.id);
},
},
performRunExecution: {
queueName: (payload) => `runs:${payload.id}`,
maxAttempts: 1,
handler: async (payload, job) => {
const service = new PerformRunExecutionService();
await service.call(payload.id);
},
},
performTaskOperation: {
priority: 0, // smaller number = higher priority
queueName: (payload) => `tasks:${payload.id}`,
maxAttempts: 3,
handler: async (payload, job) => {
@@ -223,6 +230,8 @@ function getWorkerQueue() {
},
},
indexEndpoint: {
priority: 1, // smaller number = higher priority
maxAttempts: 7,
handler: async (payload, job) => {
const service = new IndexEndpointService();
@@ -230,6 +239,8 @@ function getWorkerQueue() {
},
},
deliverEvent: {
priority: 0, // smaller number = higher priority
maxAttempts: 5,
handler: async (payload, job) => {
const service = new DeliverEventService();
@@ -237,8 +248,9 @@ function getWorkerQueue() {
},
},
refreshOAuthToken: {
priority: 8, // smaller number = higher priority
queueName: "internal-queue",
maxAttempts: 10,
maxAttempts: 7,
handler: async (payload, job) => {
await integrationAuthRepository.refreshConnection({
connectionId: payload.connectionId,
@@ -249,4 +261,42 @@ function getWorkerQueue() {
});
}
export { workerQueue };
function getExecutionWorkerQueue() {
return new ZodWorker({
name: "executionWorker",
prisma,
runnerOptions: {
connectionString: env.DATABASE_URL,
concurrency: env.EXECUTION_WORKER_CONCURRENCY,
pollInterval: env.EXECUTION_WORKER_POLL_INTERVAL,
noPreparedStatements: env.DATABASE_URL !== env.DIRECT_URL,
schema: env.WORKER_SCHEMA,
maxPoolSize: env.EXECUTION_WORKER_CONCURRENCY,
},
schema: executionWorkerCatalog,
tasks: {
performRunExecution: {
priority: 0, // smaller number = higher priority
maxAttempts: 1,
handler: async (payload, job) => {
// This is a legacy task that we don't use anymore, but needs to be here for backwards compatibility
// TODO: remove this once all performRunExecution tasks have been processed
const service = new PerformRunExecutionV1Service();
await service.call(payload.id);
},
},
performRunExecutionV2: {
priority: 0, // smaller number = higher priority
maxAttempts: 12,
handler: async (payload, job) => {
const service = new PerformRunExecutionV2Service();
await service.call(payload.id, payload.reason, payload.isRetry, payload.resumeTaskId);
},
},
},
});
}
export { executionWorker, workerQueue };
+5 -8
View File
@@ -45,13 +45,10 @@ export function useMatchesData(
const paths = Array.isArray(id) ? id : [id];
// Get the first matching route
const route = paths.reduce(
(acc, path) => {
if (acc) return acc;
return matchingRoutes.find((route) => route.id === path);
},
undefined as RouteMatch | undefined
);
const route = paths.reduce((acc, path) => {
if (acc) return acc;
return matchingRoutes.find((route) => route.id === path);
}, undefined as RouteMatch | undefined);
return route;
}
@@ -76,7 +73,7 @@ export function hydrateDates(object: any): any {
if (
typeof object === "string" &&
object.match(/\d{4}-\d{2}-\d{2}/) &&
!isNaN(Date.parse(object))
!Number.isNaN(Date.parse(object))
) {
return new Date(object);
}
+11 -7
View File
@@ -54,14 +54,18 @@ app.all(
const port = process.env.REMIX_APP_PORT || 3000;
const server = app.listen(port, () => {
// require the built app so we're ready when the first request comes in
require(BUILD_DIR);
console.log(`✅ app ready: http://localhost:${port}`);
});
if (process.env.HTTP_SERVER_DISABLED !== "true") {
const server = app.listen(port, () => {
// require the built app so we're ready when the first request comes in
require(BUILD_DIR);
console.log(`✅ app ready: http://localhost:${port}`);
});
// Handle shutdowns gracefully
createTerminus(server, { signals: ["SIGINT", "SIGTERM"], timeout: 5000 });
// Handle shutdowns gracefully
createTerminus(server, { signals: ["SIGINT", "SIGTERM"], timeout: 5000 });
} else {
console.log(`✅ app ready (skipping http server)`);
}
function purgeRequireCache() {
// purge require cache on requests for "server side HMR" this won't let
+2 -1
View File
@@ -31,7 +31,8 @@ RUN corepack enable
ENV NODE_ENV production
RUN pnpm install --prod --no-frozen-lockfile
COPY --from=pruner --chown=node:node /triggerdotdev/packages/database/prisma/schema.prisma /triggerdotdev/packages/database/prisma/schema.prisma
RUN pnpx prisma@5.1.1 generate --schema /triggerdotdev/packages/database/prisma/schema.prisma
# RUN pnpm add @prisma/client@5.1.1 -w
RUN pnpx prisma@4.16.0 generate --schema /triggerdotdev/packages/database/prisma/schema.prisma
## Builder (builds the webapp)
FROM base AS builder
+86
View File
@@ -0,0 +1,86 @@
---
title: Introduction
---
<Snippet file="integration-getting-started.mdx" />
## Installation
<CodeGroup>
```bash npm
npm install @trigger.dev/sendgrid@latest
```
```bash pnpm
pnpm install @trigger.dev/sendgrid@latest
```
```bash yarn
yarn add @trigger.dev/sendgrid@latest
```
</CodeGroup>
## Authentication
SendGrid integration supports API Keys. To authenticate, you'll need to create an instance of the SendGrid class and provide your API key.
```ts
import { SendGrid } from "@trigger.dev/sendgrid";
const sendgrid = new SendGrid({
id: "sendgrid",
apiKey: process.env.SENDGRID_API_KEY!,
});
```
## Example
In this example we use [Zod](/documentation/guides/zod), a TypeScript-first schema declaration and validation library.
```ts
import { SendGrid } from "@trigger.dev/sendgrid";
import { Job, eventTrigger } from "@trigger.dev/sdk";
import { z } from "zod";
// Create an instance of SendGrid
const sendgrid = new SendGrid({
id: "sendgrid",
apiKey: process.env.SENDGRID_API_KEY!,
});
// Define a Trigger.dev job
client.defineJob({
id: "send-sendgrid-email",
name: "Send SendGrid Email",
version: "0.1.0",
trigger: eventTrigger({
name: "send.email",
schema: z.object({
to:z.string(),
subject: z.string(),
text: z.string(),
}),
}),
integrations: {
sendgrid,
},
run: async (payload, io, ctx) => {
await io.sendgrid.sendEmail({
to: payload.to,
from: "Trigger.dev <hello@email.trigger.dev>",
subject: payload.subject,
text: payload.text,
});
},
});
```
## Tasks
| Function Name | Description |
| ------------- | ------------- |
| `sendEmail` | Send an email |
+2 -1
View File
@@ -33,9 +33,10 @@ Navigate the menu or select Integrations from the table below.
| API | Description | Webhooks | Tasks |
| --------------------------------------- | ---------------------------------------------------------------- | -------- | ----- |
| [GitHub](/integrations/apis/github) | Subscribe to webhooks and perform actions | ✅ | ✅ |
| [Supabase](/integrations/apis/supabase) | Interact with your projects and databases | ✅ | ✅ |
| [OpenAI](/integrations/apis/openai) | Generate text and images. Including longer than 30s prompts | N/A | ✅ |
| [Plain](/integrations/apis/plain) | Perform customer support using Plain | 🕘 | ✅ |
| [Resend](/integrations/apis/resend) | Send emails using Resend | 🕘 | ✅ |
| [SendGrid](/integrations/apis/sendgrid) | Send emails using SendGrid | 🕘 | ✅ |
| [Slack](/integrations/apis/slack) | Send Slack messages | 🕘 | ✅ |
| [Supabase](/integrations/apis/supabase) | Interact with your projects and databases | ✅ | ✅ |
| [Typeform](/integrations/apis/typeform) | Interact with the Typeform API and get notified of new responses | ✅ | ✅ |
+24 -40
View File
@@ -63,10 +63,7 @@
"documentation/introduction",
{
"group": "Quick Starts",
"pages": [
"documentation/quickstarts/nextjs",
"documentation/quickstarts/supabase"
]
"pages": ["documentation/quickstarts/nextjs", "documentation/quickstarts/supabase"]
},
"documentation/guides/create-a-job",
"documentation/guides/video-walkthrough"
@@ -170,10 +167,7 @@
},
{
"group": "Overview",
"pages": [
"integrations/introduction",
"integrations/create"
]
"pages": ["integrations/introduction", "integrations/create"]
},
{
"group": "Integrations",
@@ -186,6 +180,24 @@
"integrations/apis/github-tasks"
]
},
{
"group": "OpenAI",
"pages": ["integrations/apis/openai"]
},
"integrations/apis/plain",
{
"group": "Resend",
"pages": ["integrations/apis/resend"]
},
{
"group": "SendGrid",
"pages": ["integrations/apis/sendgrid"]
},
{
"group": "Slack",
"pages": ["integrations/apis/slack"]
},
{
"group": "Supabase",
"pages": [
@@ -194,25 +206,6 @@
"integrations/apis/supabase/client"
]
},
{
"group": "OpenAI",
"pages": [
"integrations/apis/openai"
]
},
"integrations/apis/plain",
{
"group": "Resend",
"pages": [
"integrations/apis/resend"
]
},
{
"group": "Slack",
"pages": [
"integrations/apis/slack"
]
},
"integrations/apis/typeform"
]
},
@@ -265,10 +258,7 @@
"sdk/dynamictrigger/constructor",
{
"group": "Instance methods",
"pages": [
"sdk/dynamictrigger/register",
"sdk/dynamictrigger/unregister"
]
"pages": ["sdk/dynamictrigger/register", "sdk/dynamictrigger/unregister"]
}
]
},
@@ -279,10 +269,7 @@
"sdk/dynamicschedule/constructor",
{
"group": "Instance methods",
"pages": [
"sdk/dynamicschedule/register",
"sdk/dynamicschedule/unregister"
]
"pages": ["sdk/dynamicschedule/register", "sdk/dynamicschedule/unregister"]
}
]
},
@@ -303,10 +290,7 @@
},
{
"group": "Overview",
"pages": [
"examples/introduction",
"examples/examples-repository"
]
"pages": ["examples/introduction", "examples/examples-repository"]
}
],
"footerSocials": {
@@ -319,4 +303,4 @@
"apiKey": "phc_hwYmedO564b3Ik8nhA4Csrb5SueY0EwFJWCbseGwWW"
}
}
}
}
-6
View File
@@ -114,12 +114,6 @@ client.defineJob({
Imports the specified integrations into the Job. The integrations will be available on the `io` object in the `run()` function with the same name as the key. For example:
<Snippet file="how-to-pass-integrations.mdx" />
</ParamField>
<ParamField body="enabled" type="boolean">
The `enabled` property is used to enable or disable the Job. If you disable a Job, it will not run.
</ParamField>
<ParamField body="queue" type="string | QueueOptions">
The `queue` property is used to specify a custom queue. If you use an Object and specify the `maxConcurrent` option, you can control how many simulataneous runs can happen.
</ParamField>
<ParamField body="logLevel" type="log | error | warn | info | debug">
The `logLevel` property is an optional property that specifies the level of
logging for the Job. The level is inherited from the client if you omit this property.
+6 -1
View File
@@ -4,9 +4,14 @@
"private": true,
"scripts": {
"stripe": "nodemon --watch src/stripe.ts -r tsconfig-paths/register -r dotenv/config src/stripe.ts",
"slack": "nodemon --watch src/slack.ts -r tsconfig-paths/register -r dotenv/config src/slack.ts",
"openai": "nodemon --watch src/openai.ts -r tsconfig-paths/register -r dotenv/config src/openai.ts",
"sendgrid": "nodemon --watch src/sendgrid.ts -r tsconfig-paths/register -r dotenv/config src/sendgrid.ts",
"supabase": "nodemon --watch src/supabase.ts -r tsconfig-paths/register -r dotenv/config src/supabase.ts",
"supabase:types": "npx supabase gen types typescript --project-id $SUPABASE_PROJECT_ID --schema public --schema auth --schema storage > src/supabase-types.ts",
"events": "nodemon --watch src/events.ts -r tsconfig-paths/register -r dotenv/config src/events.ts",
"stressTest": "nodemon --watch src/stressTest.ts -r tsconfig-paths/register -r dotenv/config src/stressTest.ts",
"delays": "nodemon --watch src/delays.ts -r tsconfig-paths/register -r dotenv/config src/delays.ts",
"dev:trigger": "trigger-cli dev --port 8080"
},
"dependencies": {
@@ -37,4 +42,4 @@
"ts-node": "^10.9.1",
"tsconfig-paths": "^3.14.1"
}
}
}
+36
View File
@@ -0,0 +1,36 @@
import { createExpressServer } from "@trigger.dev/express";
import { TriggerClient, eventTrigger } from "@trigger.dev/sdk";
export const client = new TriggerClient({
id: "job-catalog",
apiKey: process.env["TRIGGER_API_KEY"],
apiUrl: process.env["TRIGGER_API_URL"],
verbose: false,
ioLogLocalEnabled: true,
});
client.defineJob({
id: "delays-example-1",
name: "Delays Example 1",
version: "1.0.0",
trigger: eventTrigger({
name: "delays.example",
}),
run: async (payload, io, ctx) => {
await io.wait("wait-1", 60);
},
});
client.defineJob({
id: "delays-example-2",
name: "Delays Example 2 - Long Delay",
version: "1.0.0",
trigger: eventTrigger({
name: "delays.example.long",
}),
run: async (payload, io, ctx) => {
await io.wait("wait-1", 60 * 30);
},
});
createExpressServer(client);
+32
View File
@@ -0,0 +1,32 @@
import { createExpressServer } from "@trigger.dev/express";
import { TriggerClient, eventTrigger } from "@trigger.dev/sdk";
export const client = new TriggerClient({
id: "job-catalog",
apiKey: process.env["TRIGGER_API_KEY"],
apiUrl: process.env["TRIGGER_API_URL"],
verbose: false,
ioLogLocalEnabled: true,
});
client.defineJob({
id: "event-example-1",
name: "Event Example 1",
version: "1.0.0",
trigger: eventTrigger({
name: "event.example",
}),
run: async (payload, io, ctx) => {
await io.runTask("task-example-1", { name: "Task 1" }, async () => {
return {
message: "Hello World",
};
});
await io.wait("wait-1", 1);
await io.logger.info("Hello World", { ctx });
},
});
createExpressServer(client);
+80
View File
@@ -0,0 +1,80 @@
import { createExpressServer } from "@trigger.dev/express";
import { TriggerClient, eventTrigger } from "@trigger.dev/sdk";
import { OpenAI } from "@trigger.dev/openai";
export const client = new TriggerClient({
id: "job-catalog",
apiKey: process.env["TRIGGER_API_KEY"],
apiUrl: process.env["TRIGGER_API_URL"],
verbose: false,
ioLogLocalEnabled: true,
});
const openai = new OpenAI({
id: "openai",
apiKey: process.env["OPENAI_API_KEY"]!,
});
client.defineJob({
id: "openai-tasks",
name: "OpenAI Tasks",
version: "0.0.1",
trigger: eventTrigger({
name: "openai.tasks",
}),
integrations: {
openai,
},
run: async (payload, io, ctx) => {
const models = await io.openai.listModels("list-models");
if (models.data.length > 0) {
await io.openai.retrieveModel("get-model", {
model: models.data[0].id,
});
}
await io.openai.backgroundCreateChatCompletion("background-chat-completion", {
model: "gpt-3.5-turbo",
messages: [
{
role: "user",
content: "Create a good programming joke about background jobs",
},
],
});
await io.openai.createChatCompletion("chat-completion", {
model: "gpt-3.5-turbo",
messages: [
{
role: "user",
content: "Create a good programming joke about background jobs",
},
],
});
await io.openai.backgroundCreateCompletion("background-completion", {
model: "text-davinci-003",
prompt: "Create a good programming joke about Tasks",
});
await io.openai.createCompletion("completion", {
model: "text-davinci-003",
prompt: "Create a good programming joke about Tasks",
});
await io.openai.createEdit("edit", {
model: "text-davinci-edit-001",
input: "Thsi is ridddled with erors",
instruction: "Fix the spelling errors",
});
await io.openai.createEmbedding("embedding", {
model: "text-embedding-ada-002",
input: "The food was delicious and the waiter...",
});
},
});
createExpressServer(client);
+33
View File
@@ -0,0 +1,33 @@
import { createExpressServer } from "@trigger.dev/express";
import { TriggerClient, eventTrigger } from "@trigger.dev/sdk";
import { Slack } from "@trigger.dev/slack";
export const slack = new Slack({ id: "slack" });
export const client = new TriggerClient({
id: "job-catalog",
apiKey: process.env["TRIGGER_API_KEY"],
apiUrl: process.env["TRIGGER_API_URL"],
verbose: false,
ioLogLocalEnabled: true,
});
client.defineJob({
id: "slack-example-1",
name: "Slack Example 1",
version: "1.0.0",
trigger: eventTrigger({
name: "slack.example",
}),
integrations: {
slack,
},
run: async (payload, io, ctx) => {
await io.slack.postMessage("Slack 📝", {
channel: "C04GWUTDC3W",
text: "Welcome to the team, Eric!",
});
},
});
createExpressServer(client);
+41
View File
@@ -0,0 +1,41 @@
import { createExpressServer } from "@trigger.dev/express";
import { TriggerClient, eventTrigger } from "@trigger.dev/sdk";
export const client = new TriggerClient({
id: "job-catalog",
apiKey: process.env["TRIGGER_API_KEY"],
apiUrl: process.env["TRIGGER_API_URL"],
verbose: false,
ioLogLocalEnabled: true,
});
client.defineJob({
id: "stress-test-1",
name: "Stress Test 1",
version: "1.0.0",
trigger: eventTrigger({
name: "stress.test.1",
}),
run: async (payload, io, ctx) => {
// Run 10 tasks, each with a 300KB output
for (let i = 0; i < 10; i++) {
await io.runTask(`task-${i}`, { name: `Task ${i}` }, async (task) => {
return {
output: "a".repeat(300 * 1024),
};
});
}
// Now run a single task with 5MB output
await io.runTask(`task-5mb`, { name: `Task 5MB` }, async (task) => {
return {
output: "a".repeat(5 * 1024 * 1024),
};
});
// Now do a wait for 5 seconds
await io.wait("wait", 5);
},
});
createExpressServer(client);
+3
View File
@@ -0,0 +1,3 @@
{
"compilerOptions": {}
}
+2 -1
View File
@@ -17,7 +17,8 @@
},
"devDependencies": {
"eslint": "8.44.0",
"eslint-config-next": "12.3.4"
"eslint-config-next": "12.3.4",
"@trigger.dev/cli": "workspace:*"
},
"trigger.dev": {
"endpointId": "nextjs-12"
+5 -1
View File
@@ -1,3 +1,7 @@
{
"extends": "next/core-web-vitals"
"extends": ["next/core-web-vitals"],
"plugins": ["@trigger.dev"],
"rules": {
"@trigger.dev/no-duplicated-task-keys": 2
}
}
+1
View File
@@ -19,6 +19,7 @@
"@trigger.dev/sdk": "workspace:*",
"@trigger.dev/slack": "workspace:*",
"@trigger.dev/typeform": "workspace:*",
"@trigger.dev/eslint-plugin": "workspace:*",
"@types/node": "18.15.13",
"@types/react": "18.2.17",
"@types/react-dom": "18.2.7",
+9
View File
@@ -1,5 +1,14 @@
# @trigger.dev/github
## 2.0.10
### Patch Changes
- Updated dependencies [b1b9321a]
- Updated dependencies [b1b9321a]
- @trigger.dev/sdk@2.0.10
- @trigger.dev/integration-kit@2.0.10
## 2.0.9
### Patch Changes
+3 -3
View File
@@ -1,6 +1,6 @@
{
"name": "@trigger.dev/github",
"version": "2.0.9",
"version": "2.0.10",
"description": "The official GitHub integration for Trigger.dev",
"main": "./dist/index.js",
"types": "./dist/index.d.ts",
@@ -29,8 +29,8 @@
"@octokit/request": "^6.2.5",
"@octokit/request-error": "^4.0.1",
"@octokit/webhooks": "^10.4.0",
"@trigger.dev/sdk": "workspace:^2.0.9",
"@trigger.dev/integration-kit": "workspace:^2.0.9",
"@trigger.dev/sdk": "workspace:^2.0.10",
"@trigger.dev/integration-kit": "workspace:^2.0.10",
"octokit": "^2.0.14",
"zod": "3.21.4"
},
+9
View File
@@ -1,5 +1,14 @@
# @trigger.dev/slack
## 2.0.10
### Patch Changes
- Updated dependencies [b1b9321a]
- Updated dependencies [b1b9321a]
- @trigger.dev/sdk@2.0.10
- @trigger.dev/integration-kit@2.0.10
## 2.0.9
### Patch Changes
+3 -3
View File
@@ -1,6 +1,6 @@
{
"name": "@trigger.dev/openai",
"version": "2.0.9",
"version": "2.0.10",
"description": "The official OpenAI integration for Trigger.dev",
"main": "./dist/index.js",
"types": "./dist/index.d.ts",
@@ -25,8 +25,8 @@
},
"dependencies": {
"openai": "^3.3.0",
"@trigger.dev/sdk": "workspace:^2.0.9",
"@trigger.dev/integration-kit": "workspace:^2.0.9",
"@trigger.dev/sdk": "workspace:^2.0.10",
"@trigger.dev/integration-kit": "workspace:^2.0.10",
"zod": "3.21.4"
},
"engines": {
+9
View File
@@ -1,5 +1,14 @@
# @trigger.dev/plain
## 2.0.10
### Patch Changes
- Updated dependencies [b1b9321a]
- Updated dependencies [b1b9321a]
- @trigger.dev/sdk@2.0.10
- @trigger.dev/integration-kit@2.0.10
## 2.0.9
### Patch Changes
+3 -3
View File
@@ -1,6 +1,6 @@
{
"name": "@trigger.dev/plain",
"version": "2.0.9",
"version": "2.0.10",
"description": "The official Plain.com integration for Trigger.dev",
"main": "./dist/index.js",
"types": "./dist/index.d.ts",
@@ -24,8 +24,8 @@
"build:tsup": "tsup"
},
"dependencies": {
"@trigger.dev/integration-kit": "workspace:^2.0.9",
"@trigger.dev/sdk": "workspace:^2.0.9",
"@trigger.dev/integration-kit": "workspace:^2.0.10",
"@trigger.dev/sdk": "workspace:^2.0.10",
"@team-plain/typescript-sdk": "^2.7.0"
},
"engines": {
+9
View File
@@ -1,5 +1,14 @@
# @trigger.dev/resend
## 2.0.10
### Patch Changes
- Updated dependencies [b1b9321a]
- Updated dependencies [b1b9321a]
- @trigger.dev/sdk@2.0.10
- @trigger.dev/integration-kit@2.0.10
## 2.0.9
### Patch Changes
+3 -3
View File
@@ -1,6 +1,6 @@
{
"name": "@trigger.dev/resend",
"version": "2.0.9",
"version": "2.0.10",
"description": "The official Resend.com integration for Trigger.dev",
"main": "./dist/index.js",
"types": "./dist/index.d.ts",
@@ -24,8 +24,8 @@
"build:tsup": "tsup"
},
"dependencies": {
"@trigger.dev/integration-kit": "workspace:^2.0.9",
"@trigger.dev/sdk": "workspace:^2.0.9",
"@trigger.dev/integration-kit": "workspace:^2.0.10",
"@trigger.dev/sdk": "workspace:^2.0.10",
"resend": "^0.9.1"
},
"engines": {
+9
View File
@@ -1,5 +1,14 @@
# @trigger.dev/sendgrid
## 2.0.10
### Patch Changes
- Updated dependencies [b1b9321a]
- Updated dependencies [b1b9321a]
- @trigger.dev/sdk@2.0.10
- @trigger.dev/integration-kit@2.0.10
## 2.0.9
### Patch Changes
+3 -3
View File
@@ -1,6 +1,6 @@
{
"name": "@trigger.dev/sendgrid",
"version": "2.0.9",
"version": "2.0.10",
"description": "Trigger.dev integration for @sendgrid/mail",
"main": "./dist/index.js",
"types": "./dist/index.d.ts",
@@ -26,8 +26,8 @@
},
"dependencies": {
"@sendgrid/mail": "^7.7.0",
"@trigger.dev/sdk": "workspace:^2.0.9",
"@trigger.dev/integration-kit": "workspace:^2.0.9"
"@trigger.dev/sdk": "workspace:^2.0.10",
"@trigger.dev/integration-kit": "workspace:^2.0.10"
},
"engines": {
"node": ">=16.8.0"
+8
View File
@@ -1,5 +1,13 @@
# @trigger.dev/slack
## 2.0.10
### Patch Changes
- Updated dependencies [b1b9321a]
- Updated dependencies [b1b9321a]
- @trigger.dev/sdk@2.0.10
## 2.0.9
### Patch Changes
+2 -2
View File
@@ -1,6 +1,6 @@
{
"name": "@trigger.dev/slack",
"version": "2.0.9",
"version": "2.0.10",
"description": "The official Slack integration for Trigger.dev",
"main": "./dist/index.js",
"types": "./dist/index.d.ts",
@@ -25,7 +25,7 @@
},
"dependencies": {
"@slack/web-api": "^6.8.1",
"@trigger.dev/sdk": "workspace:^2.0.9",
"@trigger.dev/sdk": "workspace:^2.0.10",
"zod": "3.21.4"
},
"engines": {
+9
View File
@@ -1,5 +1,14 @@
# @trigger.dev/stripe
## 2.0.10
### Patch Changes
- Updated dependencies [b1b9321a]
- Updated dependencies [b1b9321a]
- @trigger.dev/sdk@2.0.10
- @trigger.dev/integration-kit@2.0.10
## 2.0.9
### Patch Changes
+3 -3
View File
@@ -1,6 +1,6 @@
{
"name": "@trigger.dev/stripe",
"version": "2.0.9",
"version": "2.0.10",
"description": "Trigger.dev integration for stripe",
"main": "./dist/index.js",
"types": "./dist/index.d.ts",
@@ -26,8 +26,8 @@
"typecheck": "tsc --noEmit"
},
"dependencies": {
"@trigger.dev/integration-kit": "workspace:^2.0.9",
"@trigger.dev/sdk": "workspace:^2.0.9",
"@trigger.dev/integration-kit": "workspace:^2.0.10",
"@trigger.dev/sdk": "workspace:^2.0.10",
"stripe": "^12.14.0",
"zod": "3.21.4"
},
+9
View File
@@ -1,5 +1,14 @@
# @trigger.dev/supabase
## 2.0.10
### Patch Changes
- Updated dependencies [b1b9321a]
- Updated dependencies [b1b9321a]
- @trigger.dev/sdk@2.0.10
- @trigger.dev/integration-kit@2.0.10
## 2.0.9
### Patch Changes
+3 -3
View File
@@ -1,6 +1,6 @@
{
"name": "@trigger.dev/supabase",
"version": "2.0.9",
"version": "2.0.10",
"description": "Trigger.dev integration for @supabase/supabase-js",
"main": "./dist/index.js",
"types": "./dist/index.d.ts",
@@ -26,8 +26,8 @@
},
"dependencies": {
"@supabase/supabase-js": "^2.26.0",
"@trigger.dev/integration-kit": "workspace:^2.0.9",
"@trigger.dev/sdk": "workspace:^2.0.9",
"@trigger.dev/integration-kit": "workspace:^2.0.10",
"@trigger.dev/sdk": "workspace:^2.0.10",
"supabase-management-js": "^0.1.4",
"zod": "3.21.4"
},
+9
View File
@@ -1,5 +1,14 @@
# @trigger.dev/typeform
## 2.0.10
### Patch Changes
- Updated dependencies [b1b9321a]
- Updated dependencies [b1b9321a]
- @trigger.dev/sdk@2.0.10
- @trigger.dev/integration-kit@2.0.10
## 2.0.9
### Patch Changes
+3 -3
View File
@@ -1,6 +1,6 @@
{
"name": "@trigger.dev/typeform",
"version": "2.0.9",
"version": "2.0.10",
"description": "The official Typeform integration for Trigger.dev",
"main": "./dist/index.js",
"types": "./dist/index.d.ts",
@@ -26,8 +26,8 @@
},
"dependencies": {
"@typeform/api-client": "^1.8.0",
"@trigger.dev/sdk": "workspace:^2.0.9",
"@trigger.dev/integration-kit": "workspace:^2.0.9",
"@trigger.dev/sdk": "workspace:^2.0.10",
"@trigger.dev/integration-kit": "workspace:^2.0.10",
"zod": "3.21.4"
},
"engines": {
+1 -1
View File
@@ -66,4 +66,4 @@
"@changesets/cli": "^2.26.0",
"node-fetch": "2.6.x"
}
}
}
+8
View File
@@ -1,5 +1,13 @@
# create-trigger
## 2.0.10
### Patch Changes
- 4ca9f182: Added tests to the CLI, updated tsconfig file accordingly
- 591422b8: Added whoami command, fixed TypeScript error
- a69f756e: Updated example job for the pages router
## 2.0.9
### Patch Changes
+10
View File
@@ -0,0 +1,10 @@
/** @type {import('ts-jest').JestConfigWithTsJest} */
export default {
preset: 'ts-jest',
testEnvironment: 'node',
testMatch: [
"<rootDir>/test/**/*.ts?(x)",
"<rootDir>/test/**/?(*.)+(spec|test).ts?(x)",
"<rootDir>/src/**/?(*.)+(spec|test).ts?(x)"
],
};
+8 -2
View File
@@ -1,6 +1,6 @@
{
"name": "@trigger.dev/cli",
"version": "2.0.9",
"version": "2.0.10",
"description": "The Trigger.dev CLI",
"main": "./dist/index.js",
"types": "./dist/index.d.ts",
@@ -35,11 +35,16 @@
"trigger-cli": "./dist/index.js"
},
"devDependencies": {
"@trigger.dev/tsconfig": "workspace:*",
"@gmrchk/cli-testing-library": "^0.1.2",
"@types/gradient-string": "^1.1.2",
"@types/inquirer": "^9.0.3",
"@types/jest": "^29.5.3",
"@types/node": "16",
"@types/node-fetch": "^2.6.2",
"jest": "^29.6.2",
"rimraf": "^3.0.2",
"ts-jest": "^29.1.1",
"tsup": "^6.5.0",
"type-fest": "^3.6.0",
"typescript": "^4.9.5"
@@ -49,7 +54,8 @@
"build": "tsup",
"dev": "tsup --watch",
"clean": "rimraf dist",
"start": "node dist/index.js"
"start": "node dist/index.js",
"test": "jest"
},
"dependencies": {
"@types/degit": "^2.8.3",
+22 -6
View File
@@ -1,12 +1,13 @@
import { Command } from "commander";
import inquirer from "inquirer";
import pathModule from "node:path";
import { createIntegrationCommand } from "../commands/createIntegration.js";
import { devCommand } from "../commands/dev.js";
import { initCommand } from "../commands/init.js";
import { CLOUD_TRIGGER_URL, COMMAND_NAME } from "../consts.js";
import { telemetryClient } from "../telemetry/telemetry.js";
import { getVersion } from "../utils/getVersion.js";
import { createIntegrationCommand } from "../commands/createIntegration";
import { devCommand } from "../commands/dev";
import { whoamiCommand } from "../commands/whoami.js";
import { initCommand } from "../commands/init";
import { CLOUD_TRIGGER_URL, COMMAND_NAME } from "../consts";
import { telemetryClient } from "../telemetry/telemetry";
import { getVersion } from "../utils/getVersion";
export const program = new Command();
@@ -78,6 +79,21 @@ program
await createIntegrationCommand(path, options);
});
program
.command("whoami")
.description("display the current logged in user and project details")
.argument("[path]", "The path to the project", ".")
.option("-p, --port <port>", "The local port your server is on", "3000")
.option("-e, --env-file <name>", "The name of the env file to load", ".env.local")
.version(getVersion(), "-v, --version", "Display the version number")
.action(async (path, options) => {
try {
await whoamiCommand(path, options);
} catch (e) {
throw e;
}
});
export const promptTriggerUrl = async (): Promise<string> => {
const { instanceType } = await inquirer.prompt<{
instanceType: "cloud" | "self-hosted";
@@ -2,14 +2,14 @@ import inquirer from "inquirer";
import pathModule from "node:path";
import ora from "ora";
import { z } from "zod";
import { COMMAND_NAME } from "../consts.js";
import { getLatestPackageVersion } from "../utils/addDependencies.js";
import { createFile, pathExists, readJSONFile, writeJSONFile } from "../utils/fileSystem.js";
import { generateIntegrationFiles } from "../utils/generateIntegrationFiles.js";
import { getPackageName } from "../utils/getPackagName.js";
import { installDependencies } from "../utils/installDependencies.js";
import { logger } from "../utils/logger.js";
import { resolvePath } from "../utils/parseNameAndPath.js";
import { COMMAND_NAME } from "../consts";
import { getLatestPackageVersion } from "../utils/addDependencies";
import { createFile, pathExists, readJSONFile, writeJSONFile } from "../utils/fileSystem";
import { generateIntegrationFiles } from "../utils/generateIntegrationFiles";
import { getPackageName } from "../utils/getPackagName";
import { installDependencies } from "../utils/installDependencies";
import { logger } from "../utils/logger";
import { resolvePath } from "../utils/parseNameAndPath";
const CLIOptionsSchema = z.object({
packageName: z.string().optional(),
+9 -9
View File
@@ -8,12 +8,12 @@ import ora, { Ora } from "ora";
import pathModule from "path";
import util from "util";
import { z } from "zod";
import { CLOUD_API_URL } from "../consts.js";
import { telemetryClient } from "../telemetry/telemetry.js";
import { pathExists, readFile } from "../utils/fileSystem.js";
import { logger } from "../utils/logger.js";
import { resolvePath } from "../utils/parseNameAndPath.js";
import { TriggerApi } from "../utils/triggerApi.js";
import { CLOUD_API_URL } from "../consts";
import { telemetryClient } from "../telemetry/telemetry";
import { pathExists, readFile } from "../utils/fileSystem";
import { logger } from "../utils/logger";
import { resolvePath } from "../utils/parseNameAndPath";
import { TriggerApi } from "../utils/triggerApi";
const asyncExecFile = util.promisify(childProcess.execFile);
@@ -206,7 +206,7 @@ export async function devCommand(path: string, anyOptions: any) {
throttle(refresh, throttleTimeMs);
}
async function getEndpointIdFromPackageJson(path: string, options: DevCommandOptions) {
export async function getEndpointIdFromPackageJson(path: string, options: DevCommandOptions) {
if (options.clientId) {
return options.clientId;
}
@@ -221,7 +221,7 @@ async function getEndpointIdFromPackageJson(path: string, options: DevCommandOpt
return value as string;
}
async function readEnvFilesWithBackups(
export async function readEnvFilesWithBackups(
path: string,
envFile: string,
backups: string[]
@@ -249,7 +249,7 @@ async function readEnvFilesWithBackups(
return;
}
async function getTriggerApiDetails(path: string, envFile: string) {
export async function getTriggerApiDetails(path: string, envFile: string) {
const resolvedEnvFile = await readEnvFilesWithBackups(path, envFile, [
".env",
".env.local",
+26 -18
View File
@@ -7,17 +7,17 @@ import { pathToRegexp } from "path-to-regexp";
import { simpleGit } from "simple-git";
import { parse } from "tsconfck";
import { pathToFileURL } from "url";
import { promptApiKey, promptTriggerUrl } from "../cli/index.js";
import { CLOUD_API_URL, CLOUD_TRIGGER_URL, COMMAND_NAME } from "../consts.js";
import { TelemetryClient, telemetryClient } from "../telemetry/telemetry.js";
import { addDependencies } from "../utils/addDependencies.js";
import { detectNextJsProject } from "../utils/detectNextJsProject.js";
import { pathExists, readJSONFile } from "../utils/fileSystem.js";
import { logger } from "../utils/logger.js";
import { resolvePath } from "../utils/parseNameAndPath.js";
import { renderApiKey } from "../utils/renderApiKey.js";
import { renderTitle } from "../utils/renderTitle.js";
import { TriggerApi, WhoamiResponse } from "../utils/triggerApi.js";
import { promptApiKey, promptTriggerUrl } from "../cli/index";
import { CLOUD_API_URL, CLOUD_TRIGGER_URL, COMMAND_NAME } from "../consts";
import { TelemetryClient, telemetryClient } from "../telemetry/telemetry";
import { addDependencies } from "../utils/addDependencies";
import { detectNextJsProject } from "../utils/detectNextJsProject";
import { pathExists, readJSONFile } from "../utils/fileSystem";
import { logger } from "../utils/logger";
import { resolvePath } from "../utils/parseNameAndPath";
import { renderApiKey } from "../utils/renderApiKey";
import { renderTitle } from "../utils/renderTitle";
import { TriggerApi, WhoamiResponse } from "../utils/triggerApi";
export type InitCommandOptions = {
projectPath: string;
@@ -603,20 +603,28 @@ export const client = new TriggerClient({
import { eventTrigger } from "@trigger.dev/sdk";
import { client } from "${jobsPathPrefix}trigger";
// your first job
// Your first job
// This Job will be triggered by an event, log a joke to the console, and then wait 5 seconds before logging the punchline
client.defineJob({
// This is the unique identifier for your Job, it must be unique across all Jobs in your project
id: "example-job",
name: "Example Job",
name: "Example Job: a joke with a delay",
version: "0.0.1",
// This is triggered by an event using eventTrigger. You can also trigger Jobs with webhooks, on schedules, and more: https://trigger.dev/docs/documentation/concepts/triggers/introduction
trigger: eventTrigger({
name: "example.event",
}),
run: async (payload, io, ctx) => {
await io.logger.info("Hello world!", { payload });
return {
message: "Hello world!",
};
// This logs a message to the console
await io.logger.info("🧪 Example Job: a joke with a delay");
await io.logger.info("How do you comfort a JavaScript bug?");
// This waits for 5 seconds, the second parameter is the number of seconds to wait, you can add delays of up to a year
await io.wait("Wait 5 seconds for the punchline...", 5);
await io.logger.info("You console it! 🤦");
await io.logger.info(
"✨ Congratulations, You just ran your first successful Trigger.dev Job! ✨"
);
// To learn how to write much more complex (and probably funnier) Jobs, check out our docs: https://trigger.dev/docs/documentation/guides/create-a-job
},
});
`;
+62
View File
@@ -0,0 +1,62 @@
import { z } from "zod";
import { logger } from "../utils/logger";
import { resolvePath } from "../utils/parseNameAndPath";
import { TriggerApi } from "../utils/triggerApi";
import { DevCommandOptions, getEndpointIdFromPackageJson, getTriggerApiDetails } from "./dev";
import ora from "ora";
export const WhoAmICommandOptionsSchema = z.object({
envFile: z.string(),
});
export type WhoAmICommandOptions = z.infer<typeof WhoAmICommandOptionsSchema>;
export async function whoamiCommand(path: string, anyOptions: any) {
const loadingSpinner = ora(`Hold while we fetch your data`);
loadingSpinner.start();
const result = WhoAmICommandOptionsSchema.safeParse(anyOptions);
if (!result.success) {
logger.error(result.error.message);
return;
}
const options = result.data;
const resolvedPath = resolvePath(path);
// Read from package.json to get the endpointId
const endpointId = await getEndpointIdFromPackageJson(resolvedPath, options as DevCommandOptions);
if (!endpointId) {
logger.error(
"You must run the `init` command first to setup the project you are missing \n'trigger.dev': { 'endpointId': 'your-client-id' } from your package.json file, or pass in the --client-id option to this command"
);
loadingSpinner.stop();
return;
}
// Read from .env.local or .env to get the TRIGGER_API_KEY and TRIGGER_API_URL
const apiDetails = await getTriggerApiDetails(resolvedPath, options.envFile);
if (!apiDetails) {
return;
}
const triggerAPI = new TriggerApi(apiDetails.apiKey, apiDetails.apiUrl);
const userData = await triggerAPI.whoami(apiDetails.apiKey);
loadingSpinner.stop();
logger.info(`
environment: ${userData?.type}
Trigger Client Id: ${endpointId}
User ID: ${userData?.userId}
Project:
id: ${userData?.project.id}
slug: ${userData?.project.slug}
name: ${userData?.project.name}
Organization:
id: ${userData?.organization.id}
slug: ${userData?.organization.slug}
title: ${userData?.organization.title}
`);
process.exit(1);
}
+2 -2
View File
@@ -1,7 +1,7 @@
#!/usr/bin/env node
import { program } from "./cli/index.js";
import { logger } from "./utils/logger.js";
import { program } from "./cli/index";
import { logger } from "./utils/logger";
const main = async () => {
await program.parseAsync();
+3 -3
View File
@@ -1,8 +1,8 @@
import { PostHog } from "posthog-node";
import { InitCommandOptions } from "../commands/init.js";
import { InitCommandOptions } from "../commands/init";
import { nanoid } from "nanoid";
import { getVersion } from "../utils/getVersion.js";
import { DevCommandOptions } from "../commands/dev.js";
import { getVersion } from "../utils/getVersion";
import { DevCommandOptions } from "../commands/dev";
const postHogApiKey = "phc_hwYmedO564b3Ik8nhA4Csrb5SueY0EwFJWCbseGwWW";
+1 -1
View File
@@ -2,7 +2,7 @@ import chalk from "chalk";
import { execa } from "execa";
import ora, { type Ora } from "ora";
import pathModule from "path";
import { getUserPackageManager, type PackageManager } from "./getUserPkgManager.js";
import { getUserPackageManager, type PackageManager } from "./getUserPkgManager";
import fs from "fs/promises";
import fetch from "node-fetch";
import { z } from "zod";
@@ -1,6 +1,6 @@
import fs from "fs/promises";
import pathModule from "path";
import { readPackageJson } from "./readPackageJson.js";
import { readPackageJson } from "./readPackageJson";
/** Detects if the project is a Next.js project at path */
export async function detectNextJsProject(path: string): Promise<boolean> {
+3 -3
View File
@@ -25,11 +25,11 @@ export async function removeFile(path: string) {
}
export async function readFile(path: string) {
return await fsModule.readFile(path, "utf-8");
return await fsModule.readFile(path, "utf8");
}
export async function readJSONFile(path: string) {
const fileContents = await fsModule.readFile(path, "utf-8");
const fileContents = await fsModule.readFile(path, "utf8");
return JSON.parse(fileContents);
}
@@ -39,7 +39,7 @@ export async function writeJSONFile(path: string, json: any) {
}
export function readJSONFileSync(path: string) {
const fileContents = fsSync.readFileSync(path, "utf-8");
const fileContents = fsSync.readFileSync(path, "utf8");
return JSON.parse(fileContents);
}
@@ -0,0 +1,96 @@
import { randomUUID } from "crypto";
import { pathExists } from "./fileSystem";
import { getUserPackageManager } from "./getUserPkgManager";
import * as pathModule from "path";
jest.mock('path', () => ({
join: jest.fn().mockImplementation((...paths: string[]) => paths.join('/')),
}))
jest.mock('./fileSystem', () => ({
pathExists: jest.fn().mockResolvedValue(false),
}));
describe(getUserPackageManager.name, () => {
let path: string;
beforeEach(() => {
path = randomUUID();
});
afterEach(jest.clearAllMocks);
describe(`should use ${pathExists.name} to check for package manager artifacts`, () => {
it('should join the path with the artifact name', async () => {
await getUserPackageManager(path);
expect(pathModule.join).toBeCalledWith(path, 'yarn.lock');
expect(pathModule.join).toBeCalledWith(path, 'pnpm-lock.yaml');
expect(pathModule.join).toBeCalledWith(path, 'package-lock.json');
});
it(`should call ${pathExists.name} with the path.join result`, async () => {
const expected = randomUUID();
(pathModule.join as jest.Mock).mockReturnValueOnce(expected);
await getUserPackageManager(path);
expect(pathExists).toBeCalledWith(expected);
});
it('should return "yarn" if yarn.lock exists', async () => {
(pathExists as jest.Mock).mockImplementation((path: string) => path.endsWith('yarn.lock'));
expect(await getUserPackageManager(path)).toBe('yarn');
});
it('should return "pnpm" if pnpm-lock.yaml exists', async () => {
(pathExists as jest.Mock).mockImplementation(async (path: string) => path.endsWith('pnpm-lock.yaml'));
expect(await getUserPackageManager(path)).toBe('pnpm');
});
it('should return "npm" if package-lock.json exists', async () => {
(pathExists as jest.Mock).mockImplementation((path: string) => path.endsWith('package-lock.json'));
expect(await getUserPackageManager(path)).toBe('npm');
});
it('should return "npm" if npm-shrinkwrap.json exists', async () => {
(pathExists as jest.Mock).mockImplementation((path: string) => path.endsWith('npm-shrinkwrap.json'));
expect(await getUserPackageManager(path)).toBe('npm');
});
});
describe(`if doesn't found artifacts, should use process.env.npm_config_user_agent to detect package manager`, () => {
beforeEach(() => {
(pathExists as jest.Mock).mockResolvedValue(false);
})
it('should return "yarn" if process.env.npm_config_user_agent starts with "yarn"', async () => {
process.env.npm_config_user_agent = 'yarn';
expect(await getUserPackageManager(path)).toBe('yarn');
});
it('should return "pnpm" if process.env.npm_config_user_agent starts with "pnpm"', async () => {
process.env.npm_config_user_agent = 'pnpm';
expect(await getUserPackageManager(path)).toBe('pnpm');
});
it('if doesn\'t start with "yarn" or "pnpm", should return "npm"', async () => {
process.env.npm_config_user_agent = randomUUID();
expect(await getUserPackageManager(path)).toBe('npm');
});
it('should return "npm" if process.env.npm_config_user_agent is not set', async () => {
delete process.env.npm_config_user_agent;
expect(await getUserPackageManager(path)).toBe('npm');
});
});
})
+1 -1
View File
@@ -1,5 +1,5 @@
import pathModule from "path";
import { pathExists } from "./fileSystem.js";
import { pathExists } from "./fileSystem";
export type PackageManager = "npm" | "pnpm" | "yarn";
+2 -2
View File
@@ -1,7 +1,7 @@
import { type PackageJson } from "type-fest";
import path from "path";
import { PKG_ROOT } from "../consts.js";
import { readJSONFileSync } from "./fileSystem.js";
import { PKG_ROOT } from "../consts";
import { readJSONFileSync } from "./fileSystem";
export function getVersion() {
const packageJsonPath = path.join(PKG_ROOT, "package.json");
+2 -2
View File
@@ -4,8 +4,8 @@ import { execa } from "execa";
import inquirer from "inquirer";
import ora from "ora";
import path from "path";
import { logger } from "./logger.js";
import { pathExists, removeFile } from "./fileSystem.js";
import { logger } from "./logger";
import { pathExists, removeFile } from "./fileSystem";
const isGitInstalled = (dir: string): boolean => {
try {
@@ -1,5 +1,5 @@
import { getUserPackageManager, type PackageManager } from "./getUserPkgManager.js";
import { logger } from "./logger.js";
import { getUserPackageManager, type PackageManager } from "./getUserPkgManager";
import { logger } from "./logger";
import ora, { type Ora } from "ora";
import chalk from "chalk";
import { execa } from "execa";
+1 -1
View File
@@ -1,6 +1,6 @@
import pathModule from "path";
import { type PackageJson } from "type-fest";
import { readJSONFile } from "./fileSystem.js";
import { readJSONFile } from "./fileSystem";
export async function readPackageJson(directory: string): Promise<PackageJson | undefined> {
const packageJsonPath = pathModule.join(directory, "package.json");
+2 -2
View File
@@ -1,6 +1,6 @@
import gradient from "gradient-string";
import { TITLE_TEXT } from "../consts.js";
import { getUserPackageManager } from "./getUserPkgManager.js";
import { TITLE_TEXT } from "../consts";
import { getUserPackageManager } from "./getUserPkgManager";
// colors brought in from vscode poimandres theme
const poimandresTheme = {
+1 -1
View File
@@ -1,4 +1,4 @@
import { TEMPLATE_ORGANIZATION } from "../consts.js";
import { TEMPLATE_ORGANIZATION } from "../consts";
export function createTemplateRef(templateName: string): string {
return `github:${TEMPLATE_ORGANIZATION}/${templateName}`;
+39
View File
@@ -0,0 +1,39 @@
import { prepareEnvironment } from "@gmrchk/cli-testing-library";
import { CLITestEnvironment } from "@gmrchk/cli-testing-library/lib/types";
import { join } from "node:path";
let environment: CLITestEnvironment;
beforeAll(async () => {
// This will create a sandbox folder under `/var/folders`
environment = await prepareEnvironment();
});
afterAll(async () => {
await environment.cleanup();
});
describe('cli', () => {
// can be any path with a nextjs project
const NEXT_PROJECT_PATH = join(__dirname, '..', '..', '..', 'examples', 'nextjs-example');
it('should be able to execute cli', async () => {
const { waitForText, getStdout, wait, pressKey } = await environment.spawn('node', `${join(__dirname, '..', 'dist', 'index.js')} init -p ${NEXT_PROJECT_PATH}`)
console.log('getStdout() :>> ', getStdout());
await waitForText('Detected Next.js project');
console.log('getStdout() :>> ', getStdout());
await waitForText('Are you using the Trigger.dev cloud or self-hosted?');
console.log('getStdout() :>> ', getStdout());
await pressKey('enter');
console.log('getStdout() :>> ', getStdout());
// wait next prompt, make assertions and keep going
});
})
+11 -7
View File
@@ -1,11 +1,11 @@
{
"include": ["src", "tsup.config.ts"],
"include": ["src/globals.d.ts", "./src/**/*.ts", "tsup.config.ts", "./test/**/*.ts"],
"compilerOptions": {
/* LANGUAGE COMPILATION OPTIONS */
"target": "ES2020",
"lib": ["DOM", "DOM.Iterable", "ES2020"],
"module": "Node16",
"moduleResolution": "nodenext",
"module": "ESNext",
"moduleResolution": "node",
"resolveJsonModule": true,
"allowJs": true,
"checkJs": true,
@@ -29,8 +29,8 @@
"noFallthroughCasesInSwitch": true,
"noImplicitOverride": true,
"noImplicitReturns": true,
"noUnusedLocals": true,
"noUnusedParameters": true,
"noUnusedLocals": false,
"noUnusedParameters": false,
"useUnknownInCatchVariables": true,
"noUncheckedIndexedAccess": true, // TLDR - Checking an indexed value (array[0]) now forces type <T | undefined> as there is no confirmation that index exists
// THE BELOW ARE EXTRA STRICT OPTIONS THAT SHOULD ONLY BY CONSIDERED IN VERY SAFE PROJECTS
@@ -44,6 +44,10 @@
// "experimentalDecorators": true,
"forceConsistentCasingInFileNames": true,
"skipLibCheck": true,
"useDefineForClassFields": true
}
"useDefineForClassFields": true,
"experimentalDecorators": true,
"emitDecoratorMetadata": true,
"types": ["jest"]
},
"exclude": ["node_modules"]
}
+6
View File
@@ -1,5 +1,11 @@
# internal-platform
## 2.0.10
### Patch Changes
- b1b9321a: Deprecated queue options in the job and removed startPosition
## 2.0.9
### Patch Changes
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@trigger.dev/core",
"version": "2.0.9",
"version": "2.0.10",
"description": "Core code used across the Trigger.dev SDK and platform",
"main": "./dist/index.js",
"types": "./dist/index.d.ts",
+27
View File
@@ -166,6 +166,11 @@ function filterKeys(obj: unknown, keys: string[]): any {
for (const [key, value] of Object.entries(obj)) {
if (keys.includes(key)) {
if (value) {
filteredObj[key] = `[filtered ${prettyPrintBytes(value)}]`;
} else {
filteredObj[key] = value;
}
continue;
}
@@ -174,3 +179,25 @@ function filterKeys(obj: unknown, keys: string[]): any {
return filteredObj;
}
function prettyPrintBytes(value: unknown): string {
if (process.env.NODE_ENV === "production") {
return "skipped size";
}
const sizeInBytes = Buffer.byteLength(JSON.stringify(value), "utf8");
if (sizeInBytes < 1024) {
return `${sizeInBytes} bytes`;
}
if (sizeInBytes < 1024 * 1024) {
return `${(sizeInBytes / 1024).toFixed(2)} KB`;
}
if (sizeInBytes < 1024 * 1024 * 1024) {
return `${(sizeInBytes / (1024 * 1024)).toFixed(2)} MB`;
}
return `${(sizeInBytes / (1024 * 1024 * 1024)).toFixed(2)} GB`;
}
+1 -2
View File
@@ -160,9 +160,8 @@ export const JobMetadataSchema = z.object({
trigger: TriggerMetadataSchema,
integrations: z.record(IntegrationConfigSchema),
internal: z.boolean().default(false),
queue: z.union([QueueOptionsSchema, z.string()]).optional(),
startPosition: z.enum(["initial", "latest"]),
enabled: z.boolean(),
startPosition: z.enum(["initial", "latest"]),
preprocessRuns: z.boolean(),
});
+3 -3
View File
@@ -5,11 +5,11 @@
"main": "./src/index.ts",
"types": "./src/index.ts",
"dependencies": {
"@prisma/client": "5.1.1",
"@prisma/client": "4.16.0",
"typescript": "^4.8.4"
},
"devDependencies": {
"prisma": "^5.1.0"
"prisma": "4.16.0"
},
"scripts": {
"generate": "prisma generate",
@@ -18,4 +18,4 @@
"db:studio": "prisma studio",
"typecheck": "tsc --noEmit"
}
}
}
+23
View File
@@ -0,0 +1,23 @@
"use strict";
module.exports = {
root: true,
extends: [
"eslint:recommended",
"plugin:eslint-plugin/recommended",
"plugin:node/recommended",
],
env: {
node: true,
},
overrides: [
{
files: ["tests/**/*.js"],
env: { mocha: true },
},
],
parserOptions: {
sourceType: "module",
ecmaVersion: 2020,
},
};
+7
View File
@@ -0,0 +1,7 @@
# @trigger.dev/eslint-plugin
## 2.0.10
### Patch Changes
- 74686c00: An eslint plugin that ensures uniqueness on task keys
+52
View File
@@ -0,0 +1,52 @@
# @trigger.dev/eslint-plugin
ESLint plugin with trigger.dev best practices
## Installation
You'll first need to install [ESLint](https://eslint.org/):
```sh
npm i eslint --save-dev
```
Next, install `@trigger.dev/eslint-plugin`:
```sh
npm install @trigger.dev/eslint-plugin --save-dev
```
## Usage
Add `trigger-dev` to the plugins section of your `.eslintrc` configuration file. You can omit the `eslint-plugin-` prefix:
```json
{
"plugins": [
"trigger-dev"
]
}
```
Then configure the rules you want to use under the rules section.
```json
{
"rules": {
"trigger-dev/rule-name": 2
}
}
```
## Rules
<!-- begin auto-generated rules list -->
| Name | Description |
| :--------------------------------------------------------------- | :----------------------------------------------- |
| [no-duplicated-task-keys](docs/rules/no-duplicated-task-keys.md) | Prevent duplicated task keys on trigger.dev jobs |
<!-- end auto-generated rules list -->
@@ -0,0 +1,37 @@
# Prevent duplicated task keys on trigger.dev jobs (`trigger-dev/no-duplicated-task-keys`)
<!-- end auto-generated rule header -->
Please describe the origin of the rule here.
## Rule Details
This rule aims to...
Examples of **incorrect** code for this rule:
```js
// fill me in
```
Examples of **correct** code for this rule:
```js
// fill me in
```
### Options
If there are any options, describe them here. Otherwise, delete this section.
## When Not To Use It
Give a short description of when it would be appropriate to turn off this rule.
## Further Reading
If there are other links that describe the issue this rule addresses, please include them here in a bulleted list.
+22
View File
@@ -0,0 +1,22 @@
/**
* @fileoverview ESLint plugin with trigger.dev best practices
* @author
*/
"use strict";
//------------------------------------------------------------------------------
// Requirements
//------------------------------------------------------------------------------
const requireIndex = require("requireindex");
//------------------------------------------------------------------------------
// Plugin Definition
//------------------------------------------------------------------------------
// import all rules in lib/rules
module.exports.rules = requireIndex(__dirname + "/rules");
@@ -0,0 +1,110 @@
/**
* @fileoverview Prevent duplicated task keys on trigger.dev jobs
* @author
*/
"use strict";
//------------------------------------------------------------------------------
// Rule Definition
//------------------------------------------------------------------------------
/** @type {import('eslint').Rule.RuleModule} */
module.exports = {
meta: {
type: 'problem', // `problem`, `suggestion`, or `layout`
docs: {
description: "Prevent duplicated task keys on trigger.dev jobs",
recommended: true,
url: null, // URL to the documentation page for this rule
},
fixable: null, // Or `code` or `whitespace`
schema: [], // Add a schema if the rule has options
messages: {
duplicatedTaskKey: "Task key '{{taskKey}}' is duplicated"
}
},
create(context) {
const getArguments = (node) => node.arguments || node.argument.arguments;
const getKey = (node) => {
const args = getArguments(node);
const key = args.find((arg) => arg.type === 'Literal');
if (!key) return;
return key.value;
}
const getTaskName = (expression) => {
const callee = expression.callee || expression.argument.callee;
const property = callee.property;
// We need property to be an Identifier, otherwise it's not a task
if (property.type !== 'Identifier') return;
// for io.slack.postMessage, postMessage
return property.name;
}
const groupExpressionsByTask = (ExpressionStatements, map = new Map()) => ExpressionStatements.reduce((acc, { expression }) => {
const taskName = getTaskName(expression);
const taskKey = getKey(expression);
if (acc.has(taskName)) {
acc.get(taskName).push(taskKey);
} else {
acc.set(taskName, [taskKey]);
}
return acc;
}, map);
const groupVariableDeclarationsByTask = VariableDeclarations => VariableDeclarations.reduce((acc, { declarations }) => {
declarations.forEach((declaration) => {
if (!['AwaitExpression', 'CallExpression'].includes(declaration.init.type)) return;
const taskName = getTaskName(declaration.init);
const taskKey = getKey(declaration.init);
if (acc.has(taskName)) {
acc.get(taskName).push(taskKey);
} else {
acc.set(taskName, [taskKey]);
}
});
return acc;
}, new Map());
return {
"CallExpression[callee.property.name='defineJob'] ObjectExpression BlockStatement": (node) => {
const VariableDeclarations = node.body.filter((arg) => arg.type === 'VariableDeclaration');
const grouped = groupVariableDeclarationsByTask(VariableDeclarations);
const ExpressionStatements = node.body.filter((arg) => arg.type === 'ExpressionStatement');
// it'll be a map of taskName => [key1, key2, ...]
const groupedByTask = groupExpressionsByTask(ExpressionStatements, grouped);
groupedByTask.forEach((keys) => {
const duplicated = keys.find((key, index) => keys.indexOf(key) !== index);
if (duplicated) {
context.report({
node,
messageId: 'duplicatedTaskKey',
data: {
taskKey: duplicated
},
});
}
})
}
}
}
};
+38
View File
@@ -0,0 +1,38 @@
{
"name": "@trigger.dev/eslint-plugin",
"version": "2.0.10",
"description": "ESLint plugin with trigger.dev best practices",
"keywords": [
"eslint",
"eslintplugin",
"eslint-plugin"
],
"author": "",
"main": "./lib/index.js",
"exports": "./lib/index.js",
"scripts": {
"lint": "npm-run-all \"lint:*\"",
"lint:eslint-docs": "npm-run-all \"update:eslint-docs -- --check\"",
"lint:js": "eslint .",
"test": "mocha tests --recursive",
"update:eslint-docs": "eslint-doc-generator"
},
"dependencies": {
"requireindex": "^1.2.0"
},
"devDependencies": {
"eslint": "^8.19.0",
"eslint-doc-generator": "^1.0.0",
"eslint-plugin-eslint-plugin": "^5.0.0",
"eslint-plugin-node": "^11.1.0",
"mocha": "^10.0.0",
"npm-run-all": "^4.1.5"
},
"engines": {
"node": "^14.17.0 || ^16.0.0 || >= 18.0.0"
},
"peerDependencies": {
"eslint": ">=7"
},
"license": "ISC"
}
@@ -0,0 +1,245 @@
/**
* @fileoverview Prevent duplicated task keys on trigger.dev jobs
* @author
*/
"use strict";
//------------------------------------------------------------------------------
// Requirements
//------------------------------------------------------------------------------
const rule = require("../../../lib/rules/no-duplicated-task-keys"),
RuleTester = require("eslint").RuleTester;
//------------------------------------------------------------------------------
// Tests
//------------------------------------------------------------------------------
const ruleTester = new RuleTester({
parserOptions: {
sourceType: "module",
ecmaVersion: 2020,
}
});
ruleTester.run("no-duplicated-task-keys", rule, {
valid: [
{
code: `client.defineJob({
run: async (payload, io, ctx) => {
await io.runTask("task", { name: "My Task" }, async () => {});
}
})`
},
{
code: `client.defineJob({
run: async (payload, io, ctx) => {
await io.stripe.createCharge("charge", {})
}
})`
},
{
code: `client.defineJob({
run: async (payload, io, ctx) => {
await io.supabase.createProject("create-project", {})
}
})`
},
{
code: `client.defineJob({
run: async (payload, io, ctx) => {
await io.typeform.listForms("list-forms");
}
})`
},
],
invalid: [
{
code: `client.defineJob({
run: async (payload, io, ctx) => {
await io.runTask("duplicated-task", { name: "My Task" }, async () => {
return await longRunningCode(payload.userId);
});
await io.runTask("duplicated-task", { name: "My Task" }, async () => {
return await longRunningCode(payload.userId);
});
}
})`,
errors: [{ message: "Task key 'duplicated-task' is duplicated" }]
},
{
code: `client.defineJob({
run: async (payload, io, ctx) => {
await io.stripe.createCharge("duplicated-charge", {
amount: 100,
currency: "usd",
source: payload.source,
customer: payload.customerId,
});
await io.stripe.createCharge("duplicated-charge", {
amount: 100,
currency: "usd",
source: payload.source,
customer: payload.customerId,
});
}
})`,
errors: [{ message: "Task key 'duplicated-charge' is duplicated" }]
},
{
code: `client.defineJob({
run: async (payload, io, ctx) => {
await io.supabase.createProject("create-project", {
name: payload.name,
organization_id: payload.organization_id,
plan: payload.plan,
region: payload.region,
kps_enabled: true,
db_pass: payload.password,
})
await io.supabase.createProject("create-project", {
name: payload.name,
organization_id: payload.organization_id,
plan: payload.plan,
region: payload.region,
kps_enabled: true,
db_pass: payload.password,
})
}
});`,
errors: [{ message: "Task key 'create-project' is duplicated" }]
},
{
code: `client.defineJob({
run: async (payload, io, ctx) => {
await io.typeform.listForms("list-forms");
await io.typeform.listForms("list-forms");
}
})`,
errors: [{ message: "Task key 'list-forms' is duplicated" }]
},
{
code: `client.defineJob({
id: "github-integration-on-issue-opened",
name: "GitHub Integration - On Issue Opened",
version: "0.1.0",
integrations: { github: githubApiKey },
trigger: githubApiKey.triggers.repo({
event: events.onIssueOpened,
owner: "triggerdotdev",
repo: "empty",
}),
run: async (payload, io, ctx) => {
await io.github.addIssueAssignees("add assignee", {
owner: payload.repository.owner.login,
repo: payload.repository.name,
issueNumber: payload.issue.number,
assignees: ["matt-aitken"],
});
await io.github.addIssueAssignees("add assignee", {
owner: payload.repository.owner.login,
repo: payload.repository.name,
issueNumber: payload.issue.number,
assignees: ["matt-aitken"],
});
await io.github.addIssueLabels("add label", {
owner: payload.repository.owner.login,
repo: payload.repository.name,
issueNumber: payload.issue.number,
labels: ["bug"],
});
return { payload, ctx };
},
})`,
errors: [
{ message: "Task key 'add assignee' is duplicated" },
]
},
{
code: `client.defineJob({
id: "react-hook",
name: "React Hook test",
version: "0.0.1",
trigger: eventTrigger({
name: "react-hook",
}),
integrations: {
openai,
},
run: async (_payload, io) => {
await io.wait("Wait 2 seconds", 2);
await io.wait("Wait 1 second", 1);
await io.wait("Wait 1 second", 1);
await io.openai.backgroundCreateChatCompletion("Tell me a joke", {
model: "gpt-3.5-turbo-16k",
messages: [
{
role: "user",
content: 'Tell me a joke please',
},
],
});
const result = await io.openai.backgroundCreateChatCompletion("Tell me a joke", {
model: "gpt-3.5-turbo-16k",
messages: [
{
role: "user",
content: 'Tell me a joke please',
},
],
});
return {
summary: result?.choices[0]?.message?.content,
};
},
});`,
errors: [
{ message: "Task key 'Tell me a joke' is duplicated" },
{ message: "Task key 'Wait 1 second' is duplicated" },
]
},
{
code: `client.defineJob({
id: "github-integration-get-tag",
name: "GitHub Integration - Get Tag",
version: "0.1.0",
integrations: { github },
trigger: githubApiKey.triggers.repo({
event: events.onNewBranchOrTag,
owner: "triggerdotdev",
repo: "empty",
}),
run: async (payload, io, ctx) => {
await io.logger.info("This is a simple log info message");
if (payload.ref_type === "tag") {
const tag = io.github.getTag("Get Tag", {
owner: payload.repository.owner.login,
repo: payload.repository.name,
tagSHA: payload.ref,
});
io.github.getTag("Get Tag", {
owner: payload.repository.owner.login,
repo: payload.repository.name,
tagSHA: payload.ref,
});
await io.logger.info("Tag ", tag);
await io.logger.info("Tag ", tag);
}
return { payload, ctx };
},
});`,
errors: [
{ message: "Task key 'Get Tag' is duplicated" },
{ message: "Task key 'Tag ' is duplicated" },
]
}
]
});
+8
View File
@@ -1,5 +1,13 @@
# @trigger.dev/express
## 2.0.10
### Patch Changes
- Updated dependencies [b1b9321a]
- Updated dependencies [b1b9321a]
- @trigger.dev/sdk@2.0.10
## 2.0.9
### Patch Changes

Some files were not shown because too many files have changed in this diff Show More