github-featured/triggerdotdev--trigger.dev
2
0
Fork 0
Code Issues Pull Requests Deployments Models Agent Notes Packages Projects Releases Wiki Activity

chore(webapp,core): remove the end-of-life v3 (engine V1) execution stack (#4236)

Browse Source 
## Summary

v3 (the engine that ran the SDK v3 era, internally
`RunEngineVersion.V1`) is end-of-life. Following the removal of the v3
execution apps
([#4194](https://github.com/triggerdotdev/trigger.dev/pull/4194)) and
the legacy dev websocket
([#4198](https://github.com/triggerdotdev/trigger.dev/pull/4198)), this
removes the remaining v3 execution stack from the server.

Clients still on v3 (an old SDK or CLI that has not upgraded) keep
getting a clear "upgrade to v4" response. Triggers, batch triggers,
reschedules, and deploys that resolve to v3 are rejected with a graceful
4xx pointing at the migration guide, never a 5xx, so a stale client
cannot affect server health. Self-hosted instances still running v3
should stay on the 4.5.x release line until they migrate.

## What is removed

- The MarQS queue and its shared/dev queue consumers.
- The v3 socket.io namespaces (coordinator, provider, shared-queue) and
the v3 run lifecycle services (attempt, checkpoint, and batch-resume).
- The graphile-worker background job system; all live jobs already run
on `@trigger.dev/redis-worker`.
- The `DEPRECATE_V3_ENABLED` flag: v3 is now rejected unconditionally,
so the flag is gone.
- Unused v3 exports from `@trigger.dev/core` (the `v3/zodNamespace`
subpath and the legacy socket message catalogs) and the now-dead MarQS
environment variables.

## What stays

The v4 engine is untouched. The graceful v3 rejection boundary stays,
`determineEngineVersion` still detects a v3 project so it can reject it,
and the batch service plus batch-completion worker stay for current
clients. Live queue concurrency limits and metrics now read from the v4
run engine instead of MarQS, and a brand-new dev environment now
defaults to v4.



## Dependency cleanup

Removes webapp dependencies left unused by this change: `seedrandom` and
`semver` (only the removed v3 code used them) plus a set that was
already dead, their orphaned `@types` packages, and two dead files. Adds
a `knip:deps` script and a `knip.json` config so unused dependencies can
be found the same way going forward.
This commit is contained in:
Eric Allam
2026-07-13 11:32:06 +01:00
committed by GitHub
parent c0f7c803b1
commit 5ba8557a51
114 changed files with 715 additions and 23184 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.changeset/retire-v3-zod-namespace.md
+5
View File
@@ -0,0 +1,5 @@
---
"@trigger.dev/core": patch
---
Removed the unused `@trigger.dev/core/v3/zodNamespace` export and the legacy v3 socket message schemas. These were only used by the now-retired v3 engine and have no v4 consumers.
.claude/rules/legacy-v3-code.md
+14 -21
View File
@@ -3,31 +3,24 @@ paths:
- "apps/webapp/app/v3/**"
---
# Legacy V1 Engine Code in `app/v3/`
# v3 (engine V1) has been removed
The `v3/` directory name is misleading - most code here is actively used by the current V2 engine. Only the specific files below are legacy V1-only code.
The v3 engine (RunEngineVersion `V1`: MarQS queue + Graphile worker) is end-of-life and its execution code has been removed from the webapp. The `app/v3/` directory name is historical: everything under it now serves the current V2 engine (`@internal/run-engine` + `@trigger.dev/redis-worker`).
## V1-Only Files - Never Modify
There is no `V1` execution path anymore. If you find a `RunEngineVersion` branch, the `V1` arm should only reject or finalize gracefully (for example, mark a historical run cancelled in the DB), never run V1 work. Do not reintroduce MarQS, the graphile worker, or the v3 socket.io namespaces.
- `marqs/` directory (entire MarQS queue system: sharedQueueConsumer, devQueueConsumer, fairDequeuingStrategy, devPubSub)
- `legacyRunEngineWorker.server.ts` (V1 background job worker)
- `services/triggerTaskV1.server.ts` (deprecated V1 task triggering)
- `services/cancelTaskRunV1.server.ts` (deprecated V1 cancellation)
- `authenticatedSocketConnection.server.ts` (V1 dev WebSocket using DevQueueConsumer)
- `sharedSocketConnection.ts` (V1 shared queue socket using SharedQueueConsumer)
## The deprecation boundary (keep this)
## V1/V2 Branching Pattern
Requests from clients still on v3 (old SDK/CLI) or historical V1 runs must return a clean 4xx, never a 5xx. The boundary lives in:
Some services act as routers that branch on `RunEngineVersion`:
- `services/cancelTaskRun.server.ts` - calls V1 service or `engine.cancelRun()` for V2
- `services/batchTriggerV3.server.ts` - uses marqs for V1 path, run-engine for V2
- `engineDeprecation.server.ts` - the `V3_TRIGGER_DEPRECATION_MESSAGE` / `V3_DEV_DEPRECATION_MESSAGE` / `V3_MIGRATION_URL` upgrade messages.
- `engineVersion.server.ts` - `determineEngineVersion()` still detects a V1 project/run so callers can reject it.
- `services/triggerTask.server.ts`, `services/cancelTaskRun.server.ts`, `services/rescheduleTaskRun.server.ts` - the `V1` arm rejects or finalizes gracefully instead of executing.
- `services/initializeDeployment.server.ts` - the `DEPRECATE_V3_CLI_DEPLOYS_ENABLED`-gated v3 CLI deploy rejection.
- `handleWebsockets.server.ts` - the legacy `trigger dev` websocket closes with the upgrade message.
When editing these shared services, only modify V2 code paths.
## V2 modern stack
## V2 Modern Stack
- **Run lifecycle**: `@internal/run-engine` (internal-packages/run-engine)
- **Background jobs**: `@trigger.dev/redis-worker` (not graphile-worker/zodworker)
- **Queue operations**: RunQueue inside run-engine (not MarQS)
- **V2 engine singleton**: `runEngine.server.ts`, `runEngineHandlers.server.ts`
- **V2 workers**: `commonWorker.server.ts`, `alertsWorker.server.ts`, `batchTriggerWorker.server.ts`
- **Run lifecycle**: `@internal/run-engine` (`runEngine.server.ts`, `runEngineHandlers.server.ts`)
- **Background jobs**: `@trigger.dev/redis-worker` (`commonWorker.server.ts`, `alertsWorker.server.ts`, `batchTriggerWorker.server.ts`; `legacyRunEngineWorker.server.ts` still hosts the live batch-completion jobs)
- **Queue operations**: RunQueue inside run-engine (`runQueue.server.ts`), not MarQS
.server-changes/remove-v3-engine.md
+6
View File
@@ -0,0 +1,6 @@
---
area: webapp
type: breaking
---
Removed support for the end-of-life v3 engine. Instances or projects still on v3 must stay on the 4.5.x release line or upgrade to v4; v3 triggers, batch triggers, reschedules, and deploys now return a clear upgrade message instead of running.
AGENTS.md
+2 -3
View File
@@ -138,11 +138,10 @@ User API call -> Webapp routes -> Services -> RunEngine -> Redis Queue -> Superv
- **internal-packages/redis**: Redis client creation utilities (ioredis)
- **internal-packages/testcontainers**: Test helpers for Redis/PostgreSQL containers
- **internal-packages/schedule-engine**: Durable cron scheduling
- **internal-packages/zod-worker**: Graphile-worker wrapper (DEPRECATED - use redis-worker)
### Legacy V1 Engine Code
### v3 (engine V1) removed
The `apps/webapp/app/v3/` directory name is misleading - most code there is actively used by V2. Only specific files are V1-only legacy (MarQS queue, triggerTaskV1, cancelTaskRunV1, etc.). See `apps/webapp/CLAUDE.md` for the exact list. When you encounter V1/V2 branching in services, only modify V2 code paths. All new work uses Run Engine 2.0 (`@internal/run-engine`) and redis-worker.
v3 (engine V1: MarQS + Graphile worker) is end-of-life and its execution code has been removed. The `apps/webapp/app/v3/` directory name is historical - everything there now serves V2 (Run Engine 2.0, `@internal/run-engine` + redis-worker). There is no V1 execution path: a `RunEngineVersion` `V1` branch only rejects or finalizes gracefully so v3 clients get a clean 4xx, never a 5xx. Do not reintroduce V1. See `apps/webapp/CLAUDE.md` and `.claude/rules/legacy-v3-code.md`.
### Documentation
ai/references/repo.md
-1
View File
@@ -23,7 +23,6 @@ This is a pnpm 10.33.2 monorepo that uses turborepo @turbo.json. The following w
- <root>/internal-packages/run-engine is the `@internal/run-engine` package that is "Run Engine 2.0" and handles moving a run all the way through it's lifecycle
- <root>/internal-packages/redis is the `@internal/redis` package that exports Redis types and the `createRedisClient` function to unify how we create redis clients in the repo. It's not used everywhere yet, but it's the preferred way to create redis clients from now on.
- <root>/internal-packages/testcontainers is the `@internal/testcontainers` package that exports a few useful functions for spinning up local testcontainers when writing vitest tests. See our [tests.md](./tests.md) file for more information.
- <root>/internal-packages/zodworker is the `@internal/zodworker` package that implements a wrapper around graphile-worker that allows us to use zod to validate our background jobs. We are moving away from using graphile-worker as our background job system, replacing it with our own redis-worker package.
## References
apps/supervisor/CLAUDE.md
+1 -1
View File
@@ -7,7 +7,7 @@ Node.js app that manages task execution containers. Receives work from the platf
- `src/services/` - Core service logic
- `src/workloadManager/` - Container orchestration abstraction (Docker or Kubernetes)
- `src/workloadServer/` - HTTP server for workload communication (heartbeats, snapshots)
- `src/clients/` - Platform communication (webapp/coordinator)
- `src/clients/` - Platform communication (webapp)
- `src/env.ts` - Environment configuration
## Architecture
apps/webapp/CLAUDE.md
+2 -12
View File
@@ -91,24 +91,14 @@ Background job workers use `@trigger.dev/redis-worker`:
- `app/v3/alertsWorker.server.ts`
- `app/v3/batchTriggerWorker.server.ts`
Do NOT add new jobs using zodworker/graphile-worker (legacy).
## Real-time
- Socket.io: `app/v3/handleSocketIo.server.ts`, `app/v3/handleWebsockets.server.ts`
- Electric SQL: Powers real-time data sync for the dashboard
## Legacy V1 Code
## v3 (engine V1) removed
The `app/v3/` directory name is misleading - most code is actively used by V2. Only these specific files are V1-only legacy:
- `app/v3/marqs/` (old MarQS queue system)
- `app/v3/legacyRunEngineWorker.server.ts`
- `app/v3/services/triggerTaskV1.server.ts`
- `app/v3/services/cancelTaskRunV1.server.ts`
- `app/v3/authenticatedSocketConnection.server.ts`
- `app/v3/sharedSocketConnection.ts`
Some services (e.g., `cancelTaskRun.server.ts`, `batchTriggerV3.server.ts`) branch on `RunEngineVersion` to support both V1 and V2. When editing these, only modify V2 code paths.
v3 (engine V1: MarQS + Graphile worker) is end-of-life and its execution code is gone. The `app/v3/` directory name is historical; everything under it now serves V2. There is no V1 execution path: a `RunEngineVersion` `V1` branch (e.g. in `triggerTask.server.ts`, `cancelTaskRun.server.ts`) only rejects/finalizes gracefully so v3 clients get a clean 4xx, never a 5xx. Do not reintroduce V1. See `.claude/rules/legacy-v3-code.md` for the deprecation boundary.
## Performance: Trigger Hot Path
apps/webapp/app/components/admin/debugRun.tsx
+6 -258
View File
@@ -10,7 +10,6 @@ import { useEffect } from "react";
import { Spinner } from "../primitives/Spinner";
import * as Property from "~/components/primitives/PropertyTable";
import { ClipboardField } from "../primitives/ClipboardField";
import { MarQSShortKeyProducer } from "~/v3/marqs/marqsKeyProducer";
export function AdminDebugRun({ friendlyId }: { friendlyId: string }) {
const hasAdminAccess = useHasAdminAccess();
@@ -69,26 +68,13 @@ function DebugRunContent({ friendlyId }: { friendlyId: string }) {
function DebugRunData(props: UseDataFunctionReturn<typeof loader>) {
if (props.engine === "V1") {
return <DebugRunDataEngineV1 {...props} />;
return <DebugRunDataEngineV1 run={props.run} />;
}
return <DebugRunDataEngineV2 {...props} />;
}
function DebugRunDataEngineV1({
run,
environment,
queueConcurrencyLimit,
queueCurrentConcurrency,
envConcurrencyLimit,
envCurrentConcurrency,
queueReserveConcurrency,
envReserveConcurrency,
}: UseDataFunctionReturn<typeof loader>) {
const keys = new MarQSShortKeyProducer("marqs:");
const withPrefix = (key: string) => `marqs:${key}`;
function DebugRunDataEngineV1({ run }: { run: UseDataFunctionReturn<typeof loader>["run"] }) {
return (
<Property.Table>
<Property.Item>
@@ -98,247 +84,9 @@ function DebugRunDataEngineV1({
</Property.Value>
</Property.Item>
<Property.Item>
<Property.Label>Message key</Property.Label>
<Property.Value className="flex items-center gap-2">
<ClipboardField
value={withPrefix(keys.messageKey(run.id))}
variant="tertiary/small"
iconButton
/>
</Property.Value>
</Property.Item>
<Property.Item>
<Property.Label>GET message</Property.Label>
<Property.Value className="flex items-center gap-2">
<ClipboardField
value={`GET ${withPrefix(keys.messageKey(run.id))}`}
variant="tertiary/small"
iconButton
/>
</Property.Value>
</Property.Item>
<Property.Item>
<Property.Label>Queue key</Property.Label>
<Property.Value className="flex items-center gap-2">
<ClipboardField
value={withPrefix(
keys.queueKey(environment, run.queue, run.concurrencyKey ?? undefined)
)}
variant="tertiary/small"
iconButton
/>
</Property.Value>
</Property.Item>
<Property.Item>
<Property.Label>Get queue set</Property.Label>
<Property.Value className="flex items-center gap-2">
<ClipboardField
value={`ZRANGE ${withPrefix(
keys.queueKey(environment, run.queue, run.concurrencyKey ?? undefined)
)} 0 -1`}
variant="tertiary/small"
iconButton
/>
</Property.Value>
</Property.Item>
<Property.Item>
<Property.Label>Queue current concurrency key</Property.Label>
<Property.Value className="flex items-center gap-2">
<ClipboardField
value={withPrefix(
keys.queueCurrentConcurrencyKey(
environment,
run.queue,
run.concurrencyKey ?? undefined
)
)}
variant="tertiary/small"
iconButton
/>
</Property.Value>
</Property.Item>
<Property.Item>
<Property.Label>Get queue current concurrency</Property.Label>
<Property.Value className="flex items-center gap-2">
<ClipboardField
value={`SMEMBERS ${withPrefix(
keys.queueCurrentConcurrencyKey(
environment,
run.queue,
run.concurrencyKey ?? undefined
)
)}`}
variant="tertiary/small"
iconButton
/>
</Property.Value>
</Property.Item>
<Property.Item>
<Property.Label>Queue current concurrency</Property.Label>
<Property.Value className="flex items-center gap-2">
<span>{queueCurrentConcurrency ?? "0"}</span>
</Property.Value>
</Property.Item>
<Property.Item>
<Property.Label>Queue reserve concurrency key</Property.Label>
<Property.Value className="flex items-center gap-2">
<ClipboardField
value={withPrefix(
keys.queueReserveConcurrencyKeyFromQueue(
keys.queueKey(environment, run.queue, run.concurrencyKey ?? undefined)
)
)}
variant="tertiary/small"
iconButton
/>
</Property.Value>
</Property.Item>
<Property.Item>
<Property.Label>Get queue reserve concurrency</Property.Label>
<Property.Value className="flex items-center gap-2">
<ClipboardField
value={`SMEMBERS ${withPrefix(
keys.queueReserveConcurrencyKeyFromQueue(
keys.queueKey(environment, run.queue, run.concurrencyKey ?? undefined)
)
)}`}
variant="tertiary/small"
iconButton
/>
</Property.Value>
</Property.Item>
<Property.Item>
<Property.Label>Queue reserve concurrency</Property.Label>
<Property.Value className="flex items-center gap-2">
<span>{queueReserveConcurrency ?? "0"}</span>
</Property.Value>
</Property.Item>
<Property.Item>
<Property.Label>Queue concurrency limit key</Property.Label>
<Property.Value className="flex items-center gap-2">
<ClipboardField
value={withPrefix(keys.queueConcurrencyLimitKey(environment, run.queue))}
variant="tertiary/small"
iconButton
/>
</Property.Value>
</Property.Item>
<Property.Item>
<Property.Label>GET queue concurrency limit</Property.Label>
<Property.Value className="flex items-center gap-2">
<ClipboardField
value={`GET ${withPrefix(keys.queueConcurrencyLimitKey(environment, run.queue))}`}
variant="tertiary/small"
iconButton
/>
</Property.Value>
</Property.Item>
<Property.Item>
<Property.Label>Queue concurrency limit</Property.Label>
<Property.Value className="flex items-center gap-2">
<span>{queueConcurrencyLimit ?? "Not set"}</span>
</Property.Value>
</Property.Item>
<Property.Item>
<Property.Label>Env current concurrency key</Property.Label>
<Property.Value className="flex items-center gap-2">
<ClipboardField
value={withPrefix(keys.envCurrentConcurrencyKey(environment))}
variant="tertiary/small"
iconButton
/>
</Property.Value>
</Property.Item>
<Property.Item>
<Property.Label>Get env current concurrency</Property.Label>
<Property.Value className="flex items-center gap-2">
<ClipboardField
value={`SMEMBERS ${withPrefix(keys.envCurrentConcurrencyKey(environment))}`}
variant="tertiary/small"
iconButton
/>
</Property.Value>
</Property.Item>
<Property.Item>
<Property.Label>Env current concurrency</Property.Label>
<Property.Value className="flex items-center gap-2">
<span>{envCurrentConcurrency ?? "0"}</span>
</Property.Value>
</Property.Item>
<Property.Item>
<Property.Label>Env reserve concurrency key</Property.Label>
<Property.Value className="flex items-center gap-2">
<ClipboardField
value={withPrefix(keys.envReserveConcurrencyKey(environment.id))}
variant="tertiary/small"
iconButton
/>
</Property.Value>
</Property.Item>
<Property.Item>
<Property.Label>Get env reserve concurrency</Property.Label>
<Property.Value className="flex items-center gap-2">
<ClipboardField
value={`SMEMBERS ${withPrefix(keys.envReserveConcurrencyKey(environment.id))}`}
variant="tertiary/small"
iconButton
/>
</Property.Value>
</Property.Item>
<Property.Item>
<Property.Label>Env reserve concurrency</Property.Label>
<Property.Value className="flex items-center gap-2">
<span>{envReserveConcurrency ?? "0"}</span>
</Property.Value>
</Property.Item>
<Property.Item>
<Property.Label>Env concurrency limit key</Property.Label>
<Property.Value className="flex items-center gap-2">
<ClipboardField
value={withPrefix(keys.envConcurrencyLimitKey(environment))}
variant="tertiary/small"
iconButton
/>
</Property.Value>
</Property.Item>
<Property.Item>
<Property.Label>GET env concurrency limit</Property.Label>
<Property.Value className="flex items-center gap-2">
<ClipboardField
value={`GET ${withPrefix(keys.envConcurrencyLimitKey(environment))}`}
variant="tertiary/small"
iconButton
/>
</Property.Value>
</Property.Item>
<Property.Item>
<Property.Label>Env concurrency limit</Property.Label>
<Property.Value className="flex items-center gap-2">
<span>{envConcurrencyLimit ?? "Not set"}</span>
</Property.Value>
</Property.Item>
<Property.Item>
<Property.Label>Shared queue key</Property.Label>
<Property.Value className="flex items-center gap-2">
<ClipboardField
value={`GET ${withPrefix(keys.envSharedQueueKey(environment))}`}
variant="tertiary/small"
iconButton
/>
</Property.Value>
</Property.Item>
<Property.Item>
<Property.Label>Get shared queue set</Property.Label>
<Property.Value className="flex items-center gap-2">
<ClipboardField
value={`ZRANGEBYSCORE ${withPrefix(
keys.envSharedQueueKey(environment)
)} -inf ${Date.now()} WITHSCORES`}
variant="tertiary/small"
iconButton
/>
<Property.Label>Engine</Property.Label>
<Property.Value>
Engine V1 (v3) is retired. Queue debug data is no longer available for V1 runs.
</Property.Value>
</Property.Item>
</Property.Table>
@@ -352,7 +100,7 @@ function DebugRunDataEngineV2({
envConcurrencyLimit,
envCurrentConcurrency,
keys,
}: UseDataFunctionReturn<typeof loader>) {
}: Extract<UseDataFunctionReturn<typeof loader>, { engine: "V2" }>) {
return (
<Property.Table>
<Property.Item>
apps/webapp/app/components/primitives/SimpleSelect.tsx
-143
View File
@@ -1,143 +0,0 @@
"use client";
import * as SelectPrimitive from "@radix-ui/react-select";
import { Check, ChevronDown } from "lucide-react";
import * as React from "react";
import { cn } from "~/utils/cn";
const sizes = {
"secondary/small":
"text-xs h-6 bg-tertiary border border-tertiary group-hover:text-text-bright hover:border-border-bright pr-2 pl-1.5",
medium: "text-sm h-8 bg-tertiary border border-tertiary hover:border-border-bright px-2.5",
minimal: "text-xs h-6 bg-transparent hover:bg-tertiary pl-1.5 pr-2",
};
export type SelectProps = {
size?: keyof typeof sizes;
width?: "content" | "full";
};
const Select = SelectPrimitive.Root;
const SelectGroup = SelectPrimitive.Group;
const SelectValue = SelectPrimitive.Value;
const SelectTrigger = React.forwardRef<
React.ElementRef<typeof SelectPrimitive.Trigger>,
React.ComponentPropsWithoutRef<typeof SelectPrimitive.Trigger> & SelectProps
>(({ className, children, width = "content", size = "secondary/small", ...props }, ref) => {
const sizeClassName = sizes[size];
return (
<SelectPrimitive.Trigger
ref={ref}
className={cn(
"ring-offset-background group flex items-center justify-between gap-x-1 rounded text-text-dimmed transition placeholder:text-text-dimmed hover:text-text-bright focus-visible:focus-custom disabled:cursor-not-allowed disabled:opacity-50",
width === "full" ? "w-full" : "w-min",
sizeClassName,
className
)}
{...props}
>
{children}
<SelectPrimitive.Icon asChild>
<ChevronDown
className={cn(
"size-4 text-text-dimmed transition group-hover:text-text-bright group-focus:text-text-bright"
)}
/>
</SelectPrimitive.Icon>
</SelectPrimitive.Trigger>
);
});
SelectTrigger.displayName = SelectPrimitive.Trigger.displayName;
const SelectContent = React.forwardRef<
React.ElementRef<typeof SelectPrimitive.Content>,
React.ComponentPropsWithoutRef<typeof SelectPrimitive.Content>
>(({ className, children, position = "popper", ...props }, ref) => (
<SelectPrimitive.Portal>
<SelectPrimitive.Content
ref={ref}
className={cn(
"relative z-50 min-w-max overflow-hidden rounded-md border border-grid-bright bg-background-dimmed text-text-bright shadow-md animate-in fade-in-40",
position === "popper" && "translate-y-1",
className
)}
position={position}
{...props}
>
<SelectPrimitive.Viewport
className={cn(
"space-y-0.5 px-1 py-1",
position === "popper" &&
"h-(--radix-select-trigger-height) w-full min-w-(--radix-select-trigger-width)"
)}
>
{children}
</SelectPrimitive.Viewport>
</SelectPrimitive.Content>
</SelectPrimitive.Portal>
));
SelectContent.displayName = SelectPrimitive.Content.displayName;
const SelectLabel = React.forwardRef<
React.ElementRef<typeof SelectPrimitive.Label>,
React.ComponentPropsWithoutRef<typeof SelectPrimitive.Label>
>(({ className, ...props }, ref) => (
<SelectPrimitive.Label
ref={ref}
className={cn(
"-ml-1 -mr-1 mb-1 bg-background-deep py-1.5 pl-2 pr-2 font-sans text-xxs font-normal uppercase leading-normal tracking-wider text-text-dimmed first-of-type:mt-0",
className
)}
{...props}
/>
));
SelectLabel.displayName = SelectPrimitive.Label.displayName;
type SelectItemProps = React.ComponentPropsWithoutRef<typeof SelectPrimitive.Item> & {
contentClassName?: string;
};
const SelectItem = React.forwardRef<React.ElementRef<typeof SelectPrimitive.Item>, SelectItemProps>(
({ className, children, contentClassName, ...props }, ref) => (
<SelectPrimitive.Item
ref={ref}
className={cn(
"relative flex w-full cursor-default select-none items-center rounded-sm py-1.5 pl-2 pr-12 text-sm outline-hidden transition data-disabled:pointer-events-none data-disabled:opacity-50 hover:bg-background-hover focus:bg-background-hover/50",
className
)}
{...props}
>
<span className="absolute right-2 flex h-3.5 w-3.5 items-center justify-center">
<SelectPrimitive.ItemIndicator>
<Check className="h-4 w-4" />
</SelectPrimitive.ItemIndicator>
</span>
<SelectPrimitive.ItemText>{children}</SelectPrimitive.ItemText>
</SelectPrimitive.Item>
)
);
SelectItem.displayName = SelectPrimitive.Item.displayName;
const SelectSeparator = React.forwardRef<
React.ElementRef<typeof SelectPrimitive.Separator>,
React.ComponentPropsWithoutRef<typeof SelectPrimitive.Separator>
>(({ className, ...props }, ref) => (
<SelectPrimitive.Separator
ref={ref}
className={cn("bg-muted -mx-1 my-1 h-px", className)}
{...props}
/>
));
SelectSeparator.displayName = SelectPrimitive.Separator.displayName;
export {
Select,
SelectContent,
SelectGroup,
SelectItem,
SelectLabel,
SelectSeparator,
SelectTrigger,
SelectValue,
};
apps/webapp/app/entry.server.tsx
-9
View File
@@ -7,7 +7,6 @@ import { parseAcceptLanguage } from "intl-parse-accept-language";
import isbot from "isbot";
import { renderToPipeableStream } from "react-dom/server";
import { PassThrough } from "stream";
import * as Worker from "~/services/worker.server";
import { initMollifierDrainerWorker } from "~/v3/mollifierDrainerWorker.server";
import { initMollifierStaleSweepWorker } from "~/v3/mollifierStaleSweepWorker.server";
import { initBillingLimitWorker } from "~/v3/billingLimitWorker.server";
@@ -227,10 +226,6 @@ export const handleError = wrapHandleErrorWithSentry((error, { request }) => {
}
});
Worker.init().catch((error) => {
logError(error);
});
initMollifierDrainerWorker();
initMollifierStaleSweepWorker();
initBillingLimitWorker();
@@ -241,10 +236,6 @@ bootstrap().catch((error) => {
function logError(error: unknown, request?: Request) {
console.error(error);
if (error instanceof Error && error.message.startsWith("There are locked jobs present")) {
console.log("⚠️ graphile-worker migration issue detected!");
}
}
process.on("uncaughtException", (error, origin) => {
apps/webapp/app/env.server.ts
-111
View File
@@ -224,9 +224,6 @@ const EnvironmentSchema = z
PLAIN_CUSTOMER_CARDS_SECRET: z.string().optional(),
PLAIN_CUSTOMER_CARDS_KEY: z.string().optional(),
PLAIN_CUSTOMER_CARDS_HEADERS: z.string().optional(),
WORKER_SCHEMA: z.string().default("graphile_worker"),
WORKER_CONCURRENCY: z.coerce.number().int().default(10),
WORKER_POLL_INTERVAL: z.coerce.number().int().default(1000),
// How often each replica reloads the global flags snapshot from the DB.
// Sets kill/ramp propagation latency.
GLOBAL_FLAGS_RELOAD_INTERVAL_MS: z.coerce.number().int().min(1000).default(5000),
@@ -528,9 +525,6 @@ const EnvironmentSchema = z
API_RATE_LIMIT_JWT_WINDOW: z.string().default("1m"),
API_RATE_LIMIT_JWT_TOKENS: z.coerce.number().int().default(60),
//v3
PROVIDER_SECRET: z.string().default("provider-secret"),
COORDINATOR_SECRET: z.string().default("coordinator-secret"),
DEPOT_TOKEN: z.string().optional(),
DEPOT_ORG_ID: z.string().optional(),
DEPOT_REGION: z.string().default("us-east-1"),
@@ -618,16 +612,6 @@ const EnvironmentSchema = z
// log-only mode before enforcement.
DEPRECATE_V3_CLI_DEPLOYS_ENABLED: z.string().default("0"),
// Master switch for the v3 engine (RunEngineVersion.V1) shutdown. When
// enabled it: rejects triggers that resolve to V1 (single, batch, schedule,
// replay, triggerAndWait) with a graceful error pointing at the v4 migration
// guide; closes the legacy `trigger dev` websocket used by v3 CLIs; and turns
// the V1 run-lifecycle background jobs (heartbeat timeout, TTL expiry, retry,
// resume, scheduled fires) into no-ops so abandoned V1 runs stop generating
// database load. v4 (V2) is never affected (every gate also checks the run is
// V1). Defaults to off so self-hosted instances still on V1 keep working.
DEPRECATE_V3_ENABLED: z.string().default("0"),
// Verify the deploy image exists before promoting. Disable for out-of-band/air-gapped push. ECR only.
DEPLOY_IMAGE_VERIFICATION_ENABLED: BoolEnv.default(true),
@@ -661,11 +645,6 @@ const EnvironmentSchema = z
EVENTS_MEMORY_PRESSURE_THRESHOLD: z.coerce.number().int().default(5000),
EVENTS_LOAD_SHEDDING_THRESHOLD: z.coerce.number().int().default(100000),
EVENTS_LOAD_SHEDDING_ENABLED: z.string().default("1"),
SHARED_QUEUE_CONSUMER_POOL_SIZE: z.coerce.number().int().default(10),
SHARED_QUEUE_CONSUMER_INTERVAL_MS: z.coerce.number().int().default(100),
SHARED_QUEUE_CONSUMER_NEXT_TICK_INTERVAL_MS: z.coerce.number().int().default(100),
SHARED_QUEUE_CONSUMER_EMIT_RESUME_DEPENDENCY_TIMEOUT_MS: z.coerce.number().int().default(1000),
SHARED_QUEUE_CONSUMER_RESOLVE_PAYLOADS_BATCH_SIZE: z.coerce.number().int().default(25),
MANAGED_WORKER_SECRET: z.string().default("managed-secret"),
@@ -785,50 +764,9 @@ const EnvironmentSchema = z
LOOPS_API_KEY: z.string().optional(),
ATTIO_API_KEY: z.string().optional(),
MARQS_DISABLE_REBALANCING: BoolEnv.default(false),
MARQS_VISIBILITY_TIMEOUT_MS: z.coerce
.number()
.int()
.default(60 * 1000 * 15),
MARQS_SHARED_QUEUE_LIMIT: z.coerce.number().int().default(1000),
MARQS_MAXIMUM_QUEUE_PER_ENV_COUNT: z.coerce.number().int().default(50),
MARQS_DEV_QUEUE_LIMIT: z.coerce.number().int().default(1000),
MARQS_MAXIMUM_NACK_COUNT: z.coerce.number().int().default(64),
MARQS_CONCURRENCY_LIMIT_BIAS: z.coerce.number().default(0.75),
MARQS_AVAILABLE_CAPACITY_BIAS: z.coerce.number().default(0.3),
MARQS_QUEUE_AGE_RANDOMIZATION_BIAS: z.coerce.number().default(0.25),
MARQS_REUSE_SNAPSHOT_COUNT: z.coerce.number().int().default(0),
MARQS_MAXIMUM_ENV_COUNT: z.coerce.number().int().optional(),
MARQS_SHARED_WORKER_QUEUE_CONSUMER_INTERVAL_MS: z.coerce.number().int().default(250),
MARQS_SHARED_WORKER_QUEUE_MAX_MESSAGE_COUNT: z.coerce.number().int().default(10),
MARQS_SHARED_WORKER_QUEUE_EAGER_DEQUEUE_ENABLED: z.string().default("0"),
MARQS_WORKER_ENABLED: z.string().default("0"),
MARQS_WORKER_COUNT: z.coerce.number().int().default(2),
MARQS_WORKER_CONCURRENCY_LIMIT: z.coerce.number().int().default(50),
MARQS_WORKER_CONCURRENCY_TASKS_PER_WORKER: z.coerce.number().int().default(5),
MARQS_WORKER_POLL_INTERVAL_MS: z.coerce.number().int().default(100),
MARQS_WORKER_IMMEDIATE_POLL_INTERVAL_MS: z.coerce.number().int().default(100),
MARQS_WORKER_SHUTDOWN_TIMEOUT_MS: z.coerce.number().int().default(60_000),
MARQS_SHARED_WORKER_QUEUE_COOLOFF_COUNT_THRESHOLD: z.coerce.number().int().default(10),
MARQS_SHARED_WORKER_QUEUE_COOLOFF_PERIOD_MS: z.coerce.number().int().default(5_000),
PROD_TASK_HEARTBEAT_INTERVAL_MS: z.coerce.number().int().optional(),
VERBOSE_GRAPHILE_LOGGING: z.string().default("false"),
V2_MARQS_ENABLED: z.string().default("0"),
V2_MARQS_CONSUMER_POOL_ENABLED: z.string().default("0"),
V2_MARQS_CONSUMER_POOL_SIZE: z.coerce.number().int().default(10),
V2_MARQS_CONSUMER_POLL_INTERVAL_MS: z.coerce.number().int().default(1000),
V2_MARQS_QUEUE_SELECTION_COUNT: z.coerce.number().int().default(36),
V2_MARQS_VISIBILITY_TIMEOUT_MS: z.coerce
.number()
.int()
.default(60 * 1000 * 15),
V2_MARQS_DEFAULT_ENV_CONCURRENCY: z.coerce.number().int().default(100),
V2_MARQS_VERBOSE: z.string().default("0"),
V3_MARQS_CONCURRENCY_MONITOR_ENABLED: z.string().default("0"),
V2_MARQS_CONCURRENCY_MONITOR_ENABLED: z.string().default("0"),
/* Usage settings */
USAGE_EVENT_URL: z.string().optional(),
PROD_USAGE_HEARTBEAT_INTERVAL_MS: z.coerce.number().int().optional(),
@@ -1168,55 +1106,6 @@ const EnvironmentSchema = z
/** The CLI should connect to this for dev runs */
DEV_ENGINE_URL: z.string().default(process.env.APP_ORIGIN ?? "http://localhost:3030"),
LEGACY_RUN_ENGINE_WORKER_ENABLED: z.string().default(process.env.WORKER_ENABLED ?? "true"),
LEGACY_RUN_ENGINE_WORKER_CONCURRENCY_WORKERS: z.coerce.number().int().default(2),
LEGACY_RUN_ENGINE_WORKER_CONCURRENCY_TASKS_PER_WORKER: z.coerce.number().int().default(1),
LEGACY_RUN_ENGINE_WORKER_POLL_INTERVAL: z.coerce.number().int().default(1000),
LEGACY_RUN_ENGINE_WORKER_IMMEDIATE_POLL_INTERVAL: z.coerce.number().int().default(50),
LEGACY_RUN_ENGINE_WORKER_CONCURRENCY_LIMIT: z.coerce.number().int().default(50),
LEGACY_RUN_ENGINE_WORKER_SHUTDOWN_TIMEOUT_MS: z.coerce.number().int().default(60_000),
LEGACY_RUN_ENGINE_WORKER_LOG_LEVEL: z
.enum(["log", "error", "warn", "info", "debug"])
.default("info"),
LEGACY_RUN_ENGINE_WORKER_REDIS_HOST: z
.string()
.optional()
.transform((v) => v ?? process.env.REDIS_HOST),
LEGACY_RUN_ENGINE_WORKER_REDIS_READER_HOST: z
.string()
.optional()
.transform((v) => v ?? process.env.REDIS_READER_HOST),
LEGACY_RUN_ENGINE_WORKER_REDIS_READER_PORT: z.coerce
.number()
.optional()
.transform(
(v) =>
v ?? (process.env.REDIS_READER_PORT ? parseInt(process.env.REDIS_READER_PORT) : undefined)
),
LEGACY_RUN_ENGINE_WORKER_REDIS_PORT: z.coerce
.number()
.optional()
.transform(
(v) => v ?? (process.env.REDIS_PORT ? parseInt(process.env.REDIS_PORT) : undefined)
),
LEGACY_RUN_ENGINE_WORKER_REDIS_USERNAME: z
.string()
.optional()
.transform((v) => v ?? process.env.REDIS_USERNAME),
LEGACY_RUN_ENGINE_WORKER_REDIS_PASSWORD: z
.string()
.optional()
.transform((v) => v ?? process.env.REDIS_PASSWORD),
LEGACY_RUN_ENGINE_WORKER_REDIS_TLS_DISABLED: z
.string()
.default(process.env.REDIS_TLS_DISABLED ?? "false"),
LEGACY_RUN_ENGINE_WORKER_REDIS_CLUSTER_MODE_ENABLED: z.string().default("0"),
LEGACY_RUN_ENGINE_WAITING_FOR_DEPLOY_BATCH_SIZE: z.coerce.number().int().default(100),
LEGACY_RUN_ENGINE_WAITING_FOR_DEPLOY_BATCH_STAGGER_MS: z.coerce.number().int().default(1_000),
LEGACY_RUN_ENGINE_WAITING_FOR_DEPLOY_DISABLED: z.string().default("0"),
COMMON_WORKER_ENABLED: z.string().default(process.env.WORKER_ENABLED ?? "true"),
COMMON_WORKER_CONCURRENCY_WORKERS: z.coerce.number().int().default(2),
COMMON_WORKER_CONCURRENCY_TASKS_PER_WORKER: z.coerce.number().int().default(10),
apps/webapp/app/models/admin.server.ts
-1
View File
@@ -130,7 +130,6 @@ export async function adminGetOrganizations(userId: string, { page, search }: Se
id: true,
slug: true,
title: true,
v2Enabled: true,
isActivated: true,
deletedAt: true,
members: {
apps/webapp/app/models/project.server.ts
+4
View File
@@ -101,6 +101,10 @@ export async function createProject(
},
externalRef: `proj_${externalRefGenerator()}`,
version: version === "v3" ? "V3" : "V2",
// New projects run on the v2 engine. The Prisma column still defaults to V1
// for historical rows; the V1->V2 upgrade guards on worker-register / deploy
// stay in place to migrate existing legacy projects.
engine: "V2",
onboardingData,
},
include: {
apps/webapp/app/presenters/v3/EnvironmentQueuePresenter.server.ts
+4 -11
View File
@@ -1,5 +1,4 @@
import { type AuthenticatedEnvironment } from "~/services/apiAuth.server";
import { marqs } from "~/v3/marqs/index.server";
import { engine } from "~/v3/runEngine.server";
import { getQueueSizeLimit } from "~/v3/utils/queueLimits.server";
import { BasePresenter } from "./basePresenter.server";
@@ -15,16 +14,10 @@ export type Environment = {
export class EnvironmentQueuePresenter extends BasePresenter {
async call(environment: AuthenticatedEnvironment): Promise<Environment> {
const [engineV1Executing, engineV2Executing, engineV1Queued, engineV2Queued] =
await Promise.all([
marqs.currentConcurrencyOfEnvironment(environment),
engine.concurrencyOfEnvQueue(environment),
marqs.lengthOfEnvQueue(environment),
engine.lengthOfEnvQueue(environment),
]);
const running = (engineV1Executing ?? 0) + (engineV2Executing ?? 0);
const queued = (engineV1Queued ?? 0) + (engineV2Queued ?? 0);
const [running, queued] = await Promise.all([
engine.concurrencyOfEnvQueue(environment),
engine.lengthOfEnvQueue(environment),
]);
const organization = await this._replica.organization.findFirst({
where: {
apps/webapp/app/routes/_app.orgs.$organizationSlug_.projects.new/route.tsx
-4
View File
@@ -119,8 +119,6 @@ export async function loader({ params, request }: LoaderFunctionArgs) {
id: true,
title: true,
isActivated: true,
v2Enabled: true,
hasRequestedV3: true,
_count: {
select: {
projects: {
@@ -152,8 +150,6 @@ export async function loader({ params, request }: LoaderFunctionArgs) {
slug: organizationSlug,
projectsCount: organization._count.projects,
isActivated: organization.isActivated,
v2Enabled: organization.v2Enabled,
hasRequestedV3: organization.hasRequestedV3,
},
defaultVersion: url.searchParams.get("version") ?? "v2",
message: message ? decodeURIComponent(message) : undefined,
apps/webapp/app/routes/admin.concurrency.tsx
-39
View File
@@ -1,39 +0,0 @@
import { InformationCircleIcon } from "@heroicons/react/20/solid";
import { typedjson, useTypedLoaderData } from "remix-typedjson";
import { Header1 } from "~/components/primitives/Headers";
import { InfoPanel } from "~/components/primitives/InfoPanel";
import { Paragraph } from "~/components/primitives/Paragraph";
import { dashboardLoader } from "~/services/routeBuilders/dashboardBuilder";
import { concurrencyTracker } from "~/v3/services/taskRunConcurrencyTracker.server";
export const loader = dashboardLoader({ authorization: { requireSuper: true } }, async () => {
const deployedConcurrency = await concurrencyTracker.globalConcurrentRunCount(true);
const devConcurrency = await concurrencyTracker.globalConcurrentRunCount(false);
return typedjson({ deployedConcurrency, devConcurrency });
});
export default function AdminDashboardRoute() {
const { deployedConcurrency, devConcurrency } = useTypedLoaderData<typeof loader>();
return (
<main
aria-labelledby="primary-heading"
className="flex h-full w-fit min-w-0 flex-1 flex-col gap-4 overflow-y-auto px-4 pb-4 lg:order-last"
>
<div className="flex items-center divide-x divide-grid-bright rounded border border-grid-bright">
<div className="w-1/2 p-3">
<Paragraph spacing>Dev</Paragraph>
<Header1>{devConcurrency}</Header1>
</div>
<div className="w-1/2 p-3">
<Paragraph spacing>Deployed</Paragraph>
<Header1>{deployedConcurrency}</Header1>
</div>
</div>
<InfoPanel icon={InformationCircleIcon}>
This refers to the number of 'Dequeued' runs, which are either currently executing or about
to begin execution.
</InfoPanel>
</main>
);
}
apps/webapp/app/routes/admin.tsx
-4
View File
@@ -26,10 +26,6 @@ export default function Page() {
label: "Organizations",
to: `/admin/orgs${searchSuffix}`,
},
{
label: "Concurrency",
to: "/admin/concurrency",
},
{
label: "LLM Models",
to: "/admin/llm-models",
apps/webapp/app/routes/api.v1.runs.$runParam.attempts.ts
-47
View File
@@ -1,47 +0,0 @@
import type { ActionFunctionArgs } from "@remix-run/server-runtime";
import { json } from "@remix-run/server-runtime";
import { z } from "zod";
import { authenticateApiRequest } from "~/services/apiAuth.server";
import { logger } from "~/services/logger.server";
import { ServiceValidationError } from "~/v3/services/baseService.server";
import { CreateTaskRunAttemptService } from "~/v3/services/createTaskRunAttempt.server";
const ParamsSchema = z.object({
/* This is the run friendly ID */
runParam: z.string(),
});
export async function action({ request, params }: ActionFunctionArgs) {
// Authenticate the request
const authenticationResult = await authenticateApiRequest(request);
if (!authenticationResult) {
return json({ error: "Invalid or Missing API Key" }, { status: 401 });
}
const parsed = ParamsSchema.safeParse(params);
if (!parsed.success) {
return json({ error: "Invalid or missing run ID" }, { status: 400 });
}
const { runParam } = parsed.data;
const service = new CreateTaskRunAttemptService();
try {
const { execution } = await service.call({
runId: runParam,
authenticatedEnv: authenticationResult.environment,
});
return json(execution, { status: 200 });
} catch (error) {
if (error instanceof ServiceValidationError) {
return json({ error: error.message }, { status: error.status ?? 422 });
}
logger.error("Failed to create run attempt", { error });
return json({ error: "Something went wrong, please try again." }, { status: 500 });
}
}
apps/webapp/app/routes/projects.v3.$projectRef.metrics/registerProjectMetrics.server.ts
+11 -11
View File
@@ -3,7 +3,7 @@ import type { Registry } from "prom-client";
import { Gauge } from "prom-client";
import { prisma } from "~/db.server";
import type { AuthenticatedEnvironment } from "~/services/apiAuth.server";
import { marqs } from "~/v3/marqs/index.server";
import { engine } from "~/v3/runEngine.server";
export async function registerProjectMetrics(
registry: Registry,
@@ -44,7 +44,7 @@ async function registerEnvironmentMetrics(
help: `The number of tasks currently being executed in the dev environment queue`,
registers: [registry],
async collect() {
const length = await marqs?.currentConcurrencyOfEnvironment(env);
const length = await engine.runQueue.currentConcurrencyOfEnvironment(env);
if (length) {
this.set(length);
@@ -57,7 +57,7 @@ async function registerEnvironmentMetrics(
help: `The concurrency limit for the dev environment queue`,
registers: [registry],
async collect() {
const length = await marqs?.getEnvConcurrencyLimit(env);
const length = await engine.runQueue.getEnvConcurrencyLimit(env);
if (length) {
this.set(length);
@@ -70,8 +70,8 @@ async function registerEnvironmentMetrics(
help: `The capacity of the dev environment queue`,
registers: [registry],
async collect() {
const concurrencyLimit = await marqs?.getEnvConcurrencyLimit(env);
const currentConcurrency = await marqs?.currentConcurrencyOfEnvironment(env);
const concurrencyLimit = await engine.runQueue.getEnvConcurrencyLimit(env);
const currentConcurrency = await engine.runQueue.currentConcurrencyOfEnvironment(env);
if (typeof concurrencyLimit === "number" && typeof currentConcurrency === "number") {
this.set(concurrencyLimit - currentConcurrency);
@@ -94,7 +94,7 @@ function registerTaskQueueMetrics(
help: `The number of tasks in the ${queue.name} queue`,
registers: [registry],
async collect() {
const length = await marqs?.lengthOfQueue(env, queue.name);
const length = await engine.runQueue.lengthOfQueue(env, queue.name);
if (length) {
this.set(length);
@@ -107,7 +107,7 @@ function registerTaskQueueMetrics(
help: `The number of tasks currently being executed in the ${queue.name} queue`,
registers: [registry],
async collect() {
const length = await marqs?.currentConcurrencyOfQueue(env, queue.name);
const length = await engine.runQueue.currentConcurrencyOfQueue(env, queue.name);
if (length) {
this.set(length);
@@ -120,7 +120,7 @@ function registerTaskQueueMetrics(
help: `The concurrency limit for the ${queue.name} queue`,
registers: [registry],
async collect() {
const length = await marqs?.getQueueConcurrencyLimit(env, queue.name);
const length = await engine.runQueue.getQueueConcurrencyLimit(env, queue.name);
if (length) {
this.set(length);
@@ -133,8 +133,8 @@ function registerTaskQueueMetrics(
help: `The capacity of the ${queue.name} queue`,
registers: [registry],
async collect() {
const concurrencyLimit = await marqs?.getQueueConcurrencyLimit(env, queue.name);
const currentConcurrency = await marqs?.currentConcurrencyOfQueue(env, queue.name);
const concurrencyLimit = await engine.runQueue.getQueueConcurrencyLimit(env, queue.name);
const currentConcurrency = await engine.runQueue.currentConcurrencyOfQueue(env, queue.name);
if (typeof concurrencyLimit === "number" && typeof currentConcurrency === "number") {
this.set(concurrencyLimit - currentConcurrency);
@@ -147,7 +147,7 @@ function registerTaskQueueMetrics(
help: `The age of the oldest message in the ${queue.name} queue`,
registers: [registry],
async collect() {
const oldestMessage = await marqs?.oldestMessageInQueue(env, queue.name);
const oldestMessage = await engine.runQueue.oldestMessageInQueue(env, queue.name);
if (oldestMessage) {
this.set(oldestMessage);
apps/webapp/app/routes/resources.batches.$batchId.check-completion.ts
+4 -30
View File
@@ -1,7 +1,6 @@
import { parseWithZod } from "@conform-to/zod";
import type { ActionFunction } from "@remix-run/node";
import { json } from "@remix-run/node";
import { assertExhaustive } from "@trigger.dev/core/utils";
import { z } from "zod";
import { prisma } from "~/db.server";
import { redirectWithErrorMessage, redirectWithSuccessMessage } from "~/models/message.server";
@@ -10,7 +9,7 @@ import { requireUserId } from "~/services/session.server";
import { sanitizeRedirectPath } from "~/utils";
import { runStore } from "~/v3/runStore.server";
import { findBatchRunIdForUser } from "~/v3/services/batchRunAccess.server";
import { ResumeBatchRunService } from "~/v3/services/resumeBatchRun.server";
import { tryCompleteBatchV3 } from "~/v3/services/batchTriggerV3.server";
export const checkCompletionSchema = z.object({
redirectUrl: z.string(),
@@ -44,35 +43,10 @@ export const action: ActionFunction = async ({ request, params }) => {
}
try {
const resumeBatchRunService = new ResumeBatchRunService();
// Resume by the resolved internal id: the service looks up strictly by
// `{ id }`, so passing a friendlyId param would resolve to nothing.
const resumeResult = await resumeBatchRunService.call(ownedBatchRunId);
// v3 (engine V1) is retired; finalize the batch through the v2 completion path (no-op if not ready).
await tryCompleteBatchV3(ownedBatchRunId, prisma, true);
let message: string | undefined;
switch (resumeResult) {
case "ERROR": {
throw "Unknown error during batch completion check";
}
case "ALREADY_COMPLETED": {
message = "Batch already completed.";
break;
}
case "COMPLETED": {
message = "Batch completed and parent tasks resumed.";
break;
}
case "PENDING": {
message = "Child runs still in progress. Please try again later.";
break;
}
default: {
assertExhaustive(resumeResult);
}
}
return redirectWithSuccessMessage(safeRedirectUrl, request, message);
return redirectWithSuccessMessage(safeRedirectUrl, request, "Batch completion checked.");
} catch (error) {
if (error instanceof Error) {
logger.error("Failed to check batch completion", {
apps/webapp/app/routes/resources.taskruns.$runParam.debug.ts
+4 -27
View File
@@ -3,7 +3,6 @@ import { typedjson } from "remix-typedjson";
import { z } from "zod";
import { prisma } from "~/db.server";
import { requireUserId } from "~/services/session.server";
import { marqs } from "~/v3/marqs/index.server";
import { engine } from "~/v3/runEngine.server";
import { runStore } from "~/v3/runStore.server";
import { controlPlaneResolver } from "~/v3/runOpsMigration/controlPlaneResolver.server";
@@ -58,34 +57,12 @@ export async function loader({ request, params }: LoaderFunctionArgs) {
}
if (run.engine === "V1") {
const queueConcurrencyLimit = await marqs.getQueueConcurrencyLimit(environment, run.queue);
const envConcurrencyLimit = await marqs.getEnvConcurrencyLimit(environment);
const queueCurrentConcurrency = await marqs.currentConcurrencyOfQueue(
environment,
run.queue,
run.concurrencyKey ?? undefined
);
const envCurrentConcurrency = await marqs.currentConcurrencyOfEnvironment(environment);
const queueReserveConcurrency = await marqs.reserveConcurrencyOfQueue(
environment,
run.queue,
run.concurrencyKey ?? undefined
);
const envReserveConcurrency = await marqs.reserveConcurrencyOfEnvironment(environment);
// v3 (engine V1) is retired: there are no marqs queues left to introspect for a
// historical V1 run, so return a minimal payload instead of querying marqs.
return typedjson({
engine: "V1",
engine: "V1" as const,
run,
environment,
queueConcurrencyLimit,
envConcurrencyLimit,
queueCurrentConcurrency,
envCurrentConcurrency,
queueReserveConcurrency,
envReserveConcurrency,
keys: [],
});
} else {
const queueConcurrencyLimit = await engine.runQueue.getQueueConcurrencyLimit(
@@ -141,7 +118,7 @@ export async function loader({ request, params }: LoaderFunctionArgs) {
];
return typedjson({
engine: "V2",
engine: "V2" as const,
run,
environment,
queueConcurrencyLimit,
apps/webapp/app/services/db/graphileMigrationHelper.server.ts
-96
View File
@@ -1,96 +0,0 @@
import { runMigrations } from "graphile-worker";
import type { PrismaClient } from "~/db.server";
import { prisma } from "~/db.server";
import { env } from "~/env.server";
import { logger } from "~/services/logger.server";
import { PgNotifyService } from "./pgNotify.server";
import { z } from "zod";
export class GraphileMigrationHelperService {
#prismaClient: PrismaClient;
constructor(prismaClient: PrismaClient = prisma) {
this.#prismaClient = prismaClient;
}
public async call() {
this.#logDebug("GraphileMigrationHelperService.call");
await this.#detectAndPrepareForMigrations();
await runMigrations({
connectionString: env.DATABASE_URL,
schema: env.WORKER_SCHEMA,
});
}
#logDebug(message: string, args?: any) {
logger.debug(`[migrationHelper] ${message}`, args);
}
async #getLatestMigration() {
const migrationQueryResult = await this.#prismaClient.$queryRawUnsafe(`
SELECT id FROM ${env.WORKER_SCHEMA}.migrations
ORDER BY id DESC LIMIT 1
`);
const MigrationQueryResultSchema = z.array(z.object({ id: z.number() }));
const migrationResults = MigrationQueryResultSchema.parse(migrationQueryResult);
if (!migrationResults.length) {
// no migrations applied yet
return -1;
}
return migrationResults[0].id;
}
async #graphileSchemaExists() {
const schemaCount = await this.#prismaClient.$executeRaw`
SELECT schema_name FROM information_schema.schemata
WHERE schema_name = ${env.WORKER_SCHEMA}
`;
return schemaCount === 1;
}
/** Helper for graphile-worker v0.14.0 migration. No-op if already migrated. */
async #detectAndPrepareForMigrations() {
if (!(await this.#graphileSchemaExists())) {
// no schema yet, likely first start
return;
}
const latestMigration = await this.#getLatestMigration();
if (latestMigration < 0) {
// no migrations found
return;
}
// the first v0.14.0 migration has ID 11
if (latestMigration > 10) {
// already migrated
return;
}
// add 15s to graceful shutdown timeout, just to be safe
const migrationDelayInMs = env.GRACEFUL_SHUTDOWN_TIMEOUT + 15000;
this.#logDebug("Delaying worker startup due to pending migration", {
latestMigration,
migrationDelayInMs,
});
console.log(`⚠️ detected pending graphile migration`);
console.log(`⚠️ notifying running workers`);
const pgNotify = new PgNotifyService();
await pgNotify.call("trigger:graphile:migrate", { latestMigration });
console.log(`⚠️ delaying worker startup by ${migrationDelayInMs}ms`);
await new Promise((resolve) => setTimeout(resolve, migrationDelayInMs));
}
}
apps/webapp/app/services/db/pgNotify.server.ts
-28
View File
@@ -1,28 +0,0 @@
import type { z } from "zod";
import type { PrismaClient } from "~/db.server";
import { prisma } from "~/db.server";
import { logger } from "~/services/logger.server";
import type { NotificationCatalog, NotificationChannel } from "./types";
export class PgNotifyService {
#prismaClient: PrismaClient;
constructor(prismaClient: PrismaClient = prisma) {
this.#prismaClient = prismaClient;
}
public async call<TChannel extends NotificationChannel>(
channelName: TChannel,
payload: z.infer<NotificationCatalog[TChannel]>
) {
this.#logDebug("Sending notification", { channelName, notifyPayload: payload });
await this.#prismaClient.$executeRaw`
SELECT pg_notify(${channelName}, ${JSON.stringify(payload)})
`;
}
#logDebug(message: string, args?: any) {
logger.debug(`[pgNotify] ${message}`, args);
}
}
apps/webapp/app/services/db/types.ts
-11
View File
@@ -1,11 +0,0 @@
import { z } from "zod";
export const notificationCatalog = {
"trigger:graphile:migrate": z.object({
latestMigration: z.number(),
}),
};
export type NotificationCatalog = typeof notificationCatalog;
export type NotificationChannel = keyof NotificationCatalog;
apps/webapp/app/services/deleteProject.server.ts
-6
View File
@@ -1,6 +1,5 @@
import type { PrismaClient } from "@trigger.dev/database";
import { prisma } from "~/db.server";
import { marqs } from "~/v3/marqs/index.server";
import { engine } from "~/v3/runEngine.server";
import { controlPlaneResolver } from "~/v3/runOpsMigration/controlPlaneResolver.server";
@@ -36,11 +35,6 @@ export class DeleteProjectService {
return;
}
// Remove queues from MARQS
for (const environment of project.environments) {
await marqs?.removeEnvironmentQueuesFromMasterQueue(project.organization.id, environment.id);
}
// Delete all queues from the RunEngine 2 prod master queues
for (const environment of project.environments) {
await engine.removeEnvironmentQueuesFromMasterQueue({
apps/webapp/app/services/ulid.server.ts
-7
View File
@@ -1,7 +0,0 @@
import { monotonicFactory } from "ulid";
const factory = monotonicFactory();
export function ulid(): ReturnType<typeof factory> {
return factory().toLowerCase();
}
apps/webapp/app/services/worker.server.ts
-366
View File
@@ -1,366 +0,0 @@
/**
* ⚠️ LEGACY — Graphile-worker / ZodWorker setup. Do not touch.
*
* This file wires the original background-job system the webapp was
* built on (`@internal/zod-worker` → graphile-worker → Postgres). It is
* now in deprecation mode: every task in `workerCatalog` below is
* annotated with `@deprecated, moved to <new home>` and the live jobs
* for new features all run on `@trigger.dev/redis-worker` instead.
*
* Where to put new things:
* - Background jobs / queues → use redis-worker, alongside
* `~/v3/commonWorker.server.ts`, `~/v3/alertsWorker.server.ts`, or
* `~/v3/batchTriggerWorker.server.ts`.
* - Run lifecycle → `@internal/run-engine` via `~/v3/runEngine.server`.
* - Custom polling loops with their own Redis connection → keep them
* in their own lifecycle module (e.g. `~/v3/mollifierDrainerWorker.server.ts`)
* and wire the bootstrap from `entry.server.tsx`. Don't reach into
* `init()` below.
*
* Edit only when removing legacy paths.
*/
import { ZodWorker } from "@internal/zod-worker";
import { DeliverEmailSchema } from "emails";
import { z } from "zod";
import { $replica, prisma } from "~/db.server";
import { env } from "~/env.server";
import {
BatchProcessingOptions as RunEngineBatchProcessingOptions,
RunEngineBatchTriggerService,
} from "~/runEngine/services/batchTrigger.server";
import { MarqsConcurrencyMonitor } from "~/v3/marqs/concurrencyMonitor.server";
import { scheduleEngine } from "~/v3/scheduleEngine.server";
import { DeliverAlertService } from "~/v3/services/alerts/deliverAlert.server";
import { PerformDeploymentAlertsService } from "~/v3/services/alerts/performDeploymentAlerts.server";
import { PerformTaskRunAlertsService } from "~/v3/services/alerts/performTaskRunAlerts.server";
import { BatchProcessingOptions, BatchTriggerV3Service } from "~/v3/services/batchTriggerV3.server";
import { PerformBulkActionService } from "~/v3/services/bulk/performBulkAction.server";
import {
CancelDevSessionRunsService,
CancelDevSessionRunsServiceOptions,
} from "~/v3/services/cancelDevSessionRuns.server";
import { CancelTaskAttemptDependenciesService } from "~/v3/services/cancelTaskAttemptDependencies.server";
import { EnqueueDelayedRunService } from "~/v3/services/enqueueDelayedRun.server";
import { ExecuteTasksWaitingForDeployService } from "~/v3/services/executeTasksWaitingForDeploy";
import { ExpireEnqueuedRunService } from "~/v3/services/expireEnqueuedRun.server";
import { ResumeBatchRunService } from "~/v3/services/resumeBatchRun.server";
import { ResumeTaskDependencyService } from "~/v3/services/resumeTaskDependency.server";
import { RetryAttemptService } from "~/v3/services/retryAttempt.server";
import { TimeoutDeploymentService } from "~/v3/services/timeoutDeployment.server";
import { GraphileMigrationHelperService } from "./db/graphileMigrationHelper.server";
import { sendEmail } from "./email.server";
import { logger } from "./logger.server";
const workerCatalog = {
// @deprecated, moved to commonWorker.server.ts
scheduleEmail: DeliverEmailSchema,
// @deprecated, but still used when resuming batch runs in a transaction
"v3.resumeBatchRun": z.object({
batchRunId: z.string(),
}),
// @deprecated, moved to commonWorker.server.ts
"v3.resumeTaskDependency": z.object({
dependencyId: z.string(),
sourceTaskAttemptId: z.string(),
}),
// @deprecated, moved to commonWorker.server.ts
"v3.timeoutDeployment": z.object({
deploymentId: z.string(),
fromStatus: z.string(),
errorMessage: z.string(),
}),
// @deprecated, moved to commonWorker.server.ts
"v3.executeTasksWaitingForDeploy": z.object({
backgroundWorkerId: z.string(),
}),
// @deprecated, moved to ScheduleEngine
"v3.triggerScheduledTask": z.object({
instanceId: z.string(),
}),
// @deprecated, moved to commonWorker.server.ts
"v3.performTaskRunAlerts": z.object({
runId: z.string(),
}),
// @deprecated, moved to commonWorker.server.ts
"v3.deliverAlert": z.object({
alertId: z.string(),
}),
// @deprecated, moved to commonWorker.server.ts
"v3.performDeploymentAlerts": z.object({
deploymentId: z.string(),
}),
"v3.performBulkAction": z.object({
bulkActionGroupId: z.string(),
}),
"v3.performBulkActionItem": z.object({
bulkActionItemId: z.string(),
}),
// @deprecated, moved to legacyRunEngineWorker.server.ts
"v3.requeueTaskRun": z.object({
runId: z.string(),
}),
// @deprecated, moved to commonWorker.server.ts
"v3.retryAttempt": z.object({
runId: z.string(),
}),
// @deprecated, moved to commonWorker.server.ts
"v3.enqueueDelayedRun": z.object({
runId: z.string(),
}),
// @deprecated, moved to commonWorker.server.ts
"v3.expireRun": z.object({
runId: z.string(),
}),
// @deprecated, moved to commonWorker.server.ts
"v3.cancelTaskAttemptDependencies": z.object({
attemptId: z.string(),
}),
// @deprecated, moved to commonWorker.server.ts
"v3.cancelDevSessionRuns": CancelDevSessionRunsServiceOptions,
// @deprecated, moved to commonWorker.server.ts
"v3.processBatchTaskRun": BatchProcessingOptions,
// @deprecated, moved to commonWorker.server.ts
"runengine.processBatchTaskRun": RunEngineBatchProcessingOptions,
};
let workerQueue: ZodWorker<typeof workerCatalog>;
declare global {
var __worker__: ZodWorker<typeof workerCatalog>;
}
// this is needed because in development we don't want to restart
// the server with every change, but we want to make sure we don't
// create a new connection to the DB with every change either.
// in production we'll have a single connection to the DB.
if (env.NODE_ENV === "production") {
workerQueue = getWorkerQueue();
} else {
if (!global.__worker__) {
global.__worker__ = getWorkerQueue();
}
workerQueue = global.__worker__;
}
export async function init() {
const migrationHelper = new GraphileMigrationHelperService();
await migrationHelper.call();
if (env.WORKER_ENABLED === "true") {
await workerQueue.initialize();
}
}
function getWorkerQueue() {
return new ZodWorker({
name: "workerQueue",
prisma,
replica: $replica,
runnerOptions: {
connectionString: env.DATABASE_URL,
concurrency: env.WORKER_CONCURRENCY,
pollInterval: env.WORKER_POLL_INTERVAL,
noPreparedStatements: env.DATABASE_URL !== env.DIRECT_URL,
schema: env.WORKER_SCHEMA,
maxPoolSize: env.WORKER_CONCURRENCY + 1,
},
logger: logger,
shutdownTimeoutInMs: env.GRACEFUL_SHUTDOWN_TIMEOUT,
schema: workerCatalog,
recurringTasks: {
"marqs.v3.queueConcurrencyMonitor": {
// run every 5 minutes
match: "*/5 * * * *",
handler: async (payload, job, helpers) => {
await MarqsConcurrencyMonitor.initiateV3Monitoring(helpers.abortSignal);
},
},
},
tasks: {
// @deprecated, moved to commonWorker.server.ts
scheduleEmail: {
priority: 0,
maxAttempts: 3,
handler: async (payload, job) => {
await sendEmail(payload);
},
},
// @deprecated, moved to commonWorker.server.ts but still used when resuming batch runs in a transaction
"v3.resumeBatchRun": {
priority: 0,
maxAttempts: 5,
handler: async (payload, job) => {
const service = new ResumeBatchRunService();
await service.call(payload.batchRunId);
},
},
// @deprecated, moved to commonWorker.server.ts
"v3.resumeTaskDependency": {
priority: 0,
maxAttempts: 5,
handler: async (payload, job) => {
const service = new ResumeTaskDependencyService();
return await service.call(payload.dependencyId, payload.sourceTaskAttemptId);
},
},
// @deprecated, moved to commonWorker.server.ts
"v3.timeoutDeployment": {
priority: 0,
maxAttempts: 5,
handler: async (payload, job) => {
const service = new TimeoutDeploymentService();
return await service.call(payload.deploymentId, payload.fromStatus, payload.errorMessage);
},
},
// @deprecated, moved to commonWorker.server.ts
"v3.executeTasksWaitingForDeploy": {
priority: 0,
maxAttempts: 5,
handler: async (payload, job) => {
const service = new ExecuteTasksWaitingForDeployService();
return await service.call(payload.backgroundWorkerId);
},
},
// @deprecated, moved to ScheduleEngine
"v3.triggerScheduledTask": {
priority: 0,
maxAttempts: 3, // total delay of 30 seconds
handler: async (payload, job) => {
await scheduleEngine.triggerScheduledTask({
instanceId: payload.instanceId,
finalAttempt: job.attempts === job.max_attempts,
});
},
},
// @deprecated, moved to alertsWorker.server.ts
"v3.performTaskRunAlerts": {
priority: 0,
maxAttempts: 3,
handler: async (payload, job) => {
const service = new PerformTaskRunAlertsService();
return await service.call(payload.runId);
},
},
// @deprecated, moved to alertsWorker.server.ts
"v3.deliverAlert": {
priority: 0,
maxAttempts: 8,
handler: async (payload, job) => {
const service = new DeliverAlertService();
return await service.call(payload.alertId);
},
},
// @deprecated, moved to alertsWorker.server.ts
"v3.performDeploymentAlerts": {
priority: 0,
maxAttempts: 3,
handler: async (payload, job) => {
const service = new PerformDeploymentAlertsService();
return await service.call(payload.deploymentId);
},
},
// @deprecated, new bulk actions use the new bulk actions worker
"v3.performBulkAction": {
priority: 0,
maxAttempts: 3,
handler: async (payload, job) => {
const service = new PerformBulkActionService();
return await service.call(payload.bulkActionGroupId);
},
},
// @deprecated, new bulk actions use the new bulk actions worker
"v3.performBulkActionItem": {
priority: 0,
maxAttempts: 3,
handler: async (payload, job) => {
const service = new PerformBulkActionService();
await service.performBulkActionItem(payload.bulkActionItemId);
},
},
"v3.requeueTaskRun": {
priority: 0,
maxAttempts: 3,
handler: async (payload, job) => {}, // This is now handled by redisWorker
},
// @deprecated, moved to commonWorker.server.ts
"v3.retryAttempt": {
priority: 0,
maxAttempts: 3,
handler: async (payload, job) => {
const service = new RetryAttemptService();
return await service.call(payload.runId);
},
},
// @deprecated, moved to commonWorker.server.ts
"v3.enqueueDelayedRun": {
priority: 0,
maxAttempts: 8,
handler: async (payload, job) => {
const service = new EnqueueDelayedRunService();
return await service.call(payload.runId);
},
},
// @deprecated, moved to commonWorker.server.ts
"v3.expireRun": {
priority: 0,
maxAttempts: 8,
handler: async (payload, job) => {
const service = new ExpireEnqueuedRunService();
return await service.call(payload.runId);
},
},
// @deprecated, moved to commonWorker.server.ts
"v3.cancelTaskAttemptDependencies": {
priority: 0,
maxAttempts: 8,
handler: async (payload, job) => {
const service = new CancelTaskAttemptDependenciesService();
return await service.call(payload.attemptId);
},
},
// @deprecated, moved to commonWorker.server.ts
"v3.cancelDevSessionRuns": {
priority: 0,
maxAttempts: 5,
handler: async (payload, job) => {
const service = new CancelDevSessionRunsService();
return await service.call(payload);
},
},
// @deprecated, moved to commonWorker.server.ts
"v3.processBatchTaskRun": {
priority: 0,
maxAttempts: 5,
handler: async (payload, job) => {
const service = new BatchTriggerV3Service(payload.strategy);
await service.processBatchTaskRun(payload);
},
},
// @deprecated, moved to commonWorker.server.ts
"runengine.processBatchTaskRun": {
priority: 0,
maxAttempts: 5,
handler: async (payload, job) => {
const service = new RunEngineBatchTriggerService(payload.strategy);
await service.processBatchTaskRun(payload);
},
},
},
});
}
export { workerQueue };
apps/webapp/app/v3/commonWorker.server.ts
-118
View File
@@ -17,14 +17,6 @@ import { DeliverAlertService } from "./services/alerts/deliverAlert.server";
import { PerformDeploymentAlertsService } from "./services/alerts/performDeploymentAlerts.server";
import { PerformTaskRunAlertsService } from "./services/alerts/performTaskRunAlerts.server";
import { BatchTriggerV3Service } from "./services/batchTriggerV3.server";
import { CancelDevSessionRunsService } from "./services/cancelDevSessionRuns.server";
import { CancelTaskAttemptDependenciesService } from "./services/cancelTaskAttemptDependencies.server";
import { EnqueueDelayedRunService } from "./services/enqueueDelayedRun.server";
import { ExecuteTasksWaitingForDeployService } from "./services/executeTasksWaitingForDeploy";
import { ExpireEnqueuedRunService } from "./services/expireEnqueuedRun.server";
import { ResumeBatchRunService } from "./services/resumeBatchRun.server";
import { ResumeTaskDependencyService } from "./services/resumeTaskDependency.server";
import { RetryAttemptService } from "./services/retryAttempt.server";
import { TimeoutDeploymentService } from "./services/timeoutDeployment.server";
import { BulkActionService } from "./services/bulk/BulkActionV2.server";
@@ -66,25 +58,6 @@ function initializeWorker() {
maxAttempts: 3,
},
},
"v3.resumeBatchRun": {
schema: z.object({
batchRunId: z.string(),
}),
visibilityTimeoutMs: 60_000,
retry: {
maxAttempts: 5,
},
},
"v3.resumeTaskDependency": {
schema: z.object({
dependencyId: z.string(),
sourceTaskAttemptId: z.string(),
}),
visibilityTimeoutMs: 60_000,
retry: {
maxAttempts: 5,
},
},
"v3.timeoutDeployment": {
schema: z.object({
deploymentId: z.string(),
@@ -96,45 +69,6 @@ function initializeWorker() {
maxAttempts: 5,
},
},
"v3.executeTasksWaitingForDeploy": {
schema: z.object({
backgroundWorkerId: z.string(),
}),
visibilityTimeoutMs: 60_000,
retry: {
maxAttempts: 5,
},
},
"v3.retryAttempt": {
schema: z.object({
runId: z.string(),
}),
visibilityTimeoutMs: 60_000,
retry: {
maxAttempts: 3,
},
},
"v3.cancelTaskAttemptDependencies": {
schema: z.object({
attemptId: z.string(),
}),
visibilityTimeoutMs: 60_000,
retry: {
maxAttempts: 8,
},
},
"v3.cancelDevSessionRuns": {
schema: z.object({
runIds: z.array(z.string()),
cancelledAt: z.coerce.date(),
reason: z.string(),
cancelledSessionId: z.string().optional(),
}),
visibilityTimeoutMs: 60_000,
retry: {
maxAttempts: 5,
},
},
// @deprecated, moved to batchTriggerWorker.server.ts
"v3.processBatchTaskRun": {
schema: z.object({
@@ -192,24 +126,6 @@ function initializeWorker() {
maxAttempts: 3,
},
},
"v3.expireRun": {
schema: z.object({
runId: z.string(),
}),
visibilityTimeoutMs: 60_000,
retry: {
maxAttempts: 6,
},
},
"v3.enqueueDelayedRun": {
schema: z.object({
runId: z.string(),
}),
visibilityTimeoutMs: 60_000,
retry: {
maxAttempts: 6,
},
},
processBulkAction: {
schema: z.object({
bulkActionId: z.string(),
@@ -239,34 +155,10 @@ function initializeWorker() {
"attio.syncUser": async ({ payload }) => {
await runAttioUserSync(payload);
},
"v3.resumeBatchRun": async ({ payload }) => {
const service = new ResumeBatchRunService();
await service.call(payload.batchRunId);
},
"v3.resumeTaskDependency": async ({ payload }) => {
const service = new ResumeTaskDependencyService();
await service.call(payload.dependencyId, payload.sourceTaskAttemptId);
},
"v3.timeoutDeployment": async ({ payload }) => {
const service = new TimeoutDeploymentService();
await service.call(payload.deploymentId, payload.fromStatus, payload.errorMessage);
},
"v3.executeTasksWaitingForDeploy": async ({ payload }) => {
const service = new ExecuteTasksWaitingForDeployService();
await service.call(payload.backgroundWorkerId);
},
"v3.retryAttempt": async ({ payload }) => {
const service = new RetryAttemptService();
await service.call(payload.runId);
},
"v3.cancelTaskAttemptDependencies": async ({ payload }) => {
const service = new CancelTaskAttemptDependenciesService();
await service.call(payload.attemptId);
},
"v3.cancelDevSessionRuns": async ({ payload }) => {
const service = new CancelDevSessionRunsService();
await service.call(payload);
},
// @deprecated, moved to batchTriggerWorker.server.ts
"v3.processBatchTaskRun": async ({ payload }) => {
const service = new BatchTriggerV3Service(payload.strategy);
@@ -294,16 +186,6 @@ function initializeWorker() {
const service = new PerformTaskRunAlertsService();
await service.call(payload.runId);
},
"v3.expireRun": async ({ payload }) => {
const service = new ExpireEnqueuedRunService();
await service.call(payload.runId);
},
"v3.enqueueDelayedRun": async ({ payload }) => {
const service = new EnqueueDelayedRunService();
await service.call(payload.runId);
},
processBulkAction: async ({ payload }) => {
const service = new BulkActionService();
await service.process(payload.bulkActionId);
apps/webapp/app/v3/engineDeprecation.server.ts
+2 -27
View File
@@ -1,22 +1,5 @@
import { env } from "~/env.server";
/**
* Graceful sunset of the v3 engine (RunEngineVersion.V1).
*
* v3 maps to engine V1 (MarQS + Graphile); v4 is engine V2 (run-engine). A
* single master flag (DEPRECATE_V3_ENABLED, default off) gates every shutdown
* behaviour so the cloud can flip the switch while self-hosted instances still
* on V1 keep working until they migrate. This mirrors
* DEPRECATE_V3_CLI_DEPLOYS_ENABLED, which already gates deploys.
*
* The flag controls three surfaces:
* 1. Triggers that resolve to V1 are rejected with a graceful error.
* 2. The legacy `trigger dev` websocket (v3 CLIs only) is closed.
* 3. V1 run-lifecycle background jobs become no-ops to shed database load.
*
* Every call site also checks the run/project is actually V1, so v4 (V2) is
* never affected.
*/
// User-facing deprecation messages returned when a retired v3 (engine V1) SDK/CLI
// still triggers, reschedules, or opens the legacy dev websocket.
export const V3_MIGRATION_URL = "https://trigger.dev/docs/migrating-from-v3";
@@ -24,11 +7,3 @@ export const V3_TRIGGER_DEPRECATION_MESSAGE = `Trigger.dev v3 is no longer suppo
// Sent as a websocket close reason, which is capped at 123 bytes, so keep it short.
export const V3_DEV_DEPRECATION_MESSAGE = `Trigger.dev v3 is no longer supported. Upgrade to v4: ${V3_MIGRATION_URL}`;
/**
* Whether the v3 (engine V1) shutdown is being enforced. Guard every V1-only
* code path with `isV3Disabled() && <run/project is V1>` so v4 is untouched.
*/
export function isV3Disabled(): boolean {
return env.DEPRECATE_V3_ENABLED === "1";
}
apps/webapp/app/v3/engineVersion.server.ts
+3 -2
View File
@@ -63,10 +63,11 @@ export async function determineEngineVersion({
return worker.engine;
}
// Dev: use the latest BackgroundWorker
// Dev: use the latest BackgroundWorker. Default to V2 when there is no current
// worker: v3 (engine V1) is retired, so a fresh/idle dev env must resolve to V2.
if (environment.type === "DEVELOPMENT") {
const backgroundWorker = await findCurrentWorkerFromEnvironment(environment);
return backgroundWorker?.engine ?? "V1";
return backgroundWorker?.engine ?? "V2";
}
// Deployed: use the latest deployed BackgroundWorker
apps/webapp/app/v3/failedTaskRun.server.ts
-304
View File
@@ -1,304 +0,0 @@
import type {
TaskRunExecution,
TaskRunExecutionRetry,
TaskRunFailedExecutionResult,
V3TaskRunExecution,
} from "@trigger.dev/core/v3";
import { calculateNextRetryDelay, RetryOptions } from "@trigger.dev/core/v3";
import type { Prisma, TaskRun } from "@trigger.dev/database";
import * as semver from "semver";
import { logger } from "~/services/logger.server";
import { sharedQueueTasks } from "./marqs/sharedQueueConsumer.server";
import { BaseService } from "./services/baseService.server";
import { CompleteAttemptService } from "./services/completeAttempt.server";
import { CreateTaskRunAttemptService } from "./services/createTaskRunAttempt.server";
import { isFailableRunStatus, isFinalAttemptStatus } from "./taskStatus";
const FailedTaskRunRetryGetPayload = {
select: {
id: true,
attempts: {
orderBy: {
createdAt: "desc",
},
take: 1,
},
lockedById: true, // task
lockedToVersionId: true, // worker
},
} as const;
type TaskRunWithAttempts = Prisma.TaskRunGetPayload<typeof FailedTaskRunRetryGetPayload>;
export class FailedTaskRunService extends BaseService {
public async call(anyRunId: string, completion: TaskRunFailedExecutionResult) {
logger.debug("[FailedTaskRunService] Handling failed task run", { anyRunId, completion });
const isFriendlyId = anyRunId.startsWith("run_");
const taskRun = await this.runStore.findRun(
{
friendlyId: isFriendlyId ? anyRunId : undefined,
id: !isFriendlyId ? anyRunId : undefined,
},
this._prisma
);
if (!taskRun) {
logger.error("[FailedTaskRunService] Task run not found", {
anyRunId,
completion,
});
return;
}
if (!isFailableRunStatus(taskRun.status)) {
logger.error("[FailedTaskRunService] Task run is not in a failable state", {
taskRun,
completion,
});
return;
}
const retryHelper = new FailedTaskRunRetryHelper(this._prisma);
const retryResult = await retryHelper.call({
runId: taskRun.id,
completion,
});
logger.debug("[FailedTaskRunService] Completion result", {
runId: taskRun.id,
result: retryResult,
});
}
}
interface TaskRunWithWorker extends TaskRun {
lockedBy: { retryConfig: Prisma.JsonValue } | null;
lockedToVersion: { sdkVersion: string } | null;
}
export class FailedTaskRunRetryHelper extends BaseService {
async call({
runId,
completion,
isCrash,
}: {
runId: string;
completion: TaskRunFailedExecutionResult;
isCrash?: boolean;
}) {
const taskRun = await this.runStore.findRun(
{
id: runId,
},
FailedTaskRunRetryGetPayload,
this._prisma
);
if (!taskRun) {
logger.error("[FailedTaskRunRetryHelper] Task run not found", {
runId,
completion,
});
return "NO_TASK_RUN";
}
const retriableExecution = await this.#getRetriableAttemptExecution(taskRun, completion);
if (!retriableExecution) {
return "NO_EXECUTION";
}
logger.debug("[FailedTaskRunRetryHelper] Completing attempt", { taskRun, completion });
const completeAttempt = new CompleteAttemptService({
prisma: this._prisma,
isSystemFailure: !isCrash,
isCrash,
});
const completeResult = await completeAttempt.call({
completion,
execution: retriableExecution,
});
return completeResult;
}
async #getRetriableAttemptExecution(
run: TaskRunWithAttempts,
completion: TaskRunFailedExecutionResult
): Promise<V3TaskRunExecution | undefined> {
let attempt = run.attempts[0];
// We need to create an attempt if:
// - None exists yet
// - The last attempt has a final status, e.g. we failed between attempts
if (!attempt || isFinalAttemptStatus(attempt.status)) {
logger.debug("[FailedTaskRunRetryHelper] No attempts found", {
run,
completion,
});
const createAttempt = new CreateTaskRunAttemptService(this._prisma);
try {
const { execution } = await createAttempt.call({
runId: run.id,
// This ensures we correctly respect `maxAttempts = 1` when failing before the first attempt was created
startAtZero: true,
});
return execution;
} catch (error) {
logger.error("[FailedTaskRunRetryHelper] Failed to create attempt", {
run,
completion,
error,
});
return;
}
}
// We already have an attempt with non-final status, let's use it
try {
const executionPayload = await sharedQueueTasks.getExecutionPayloadFromAttempt({
id: attempt.id,
skipStatusChecks: true,
});
return executionPayload?.execution;
} catch (error) {
logger.error("[FailedTaskRunRetryHelper] Failed to get execution payload", {
run,
completion,
error,
});
return;
}
}
static getExecutionRetry({
run,
execution,
}: {
run: TaskRunWithWorker;
execution: TaskRunExecution;
}): TaskRunExecutionRetry | undefined {
try {
const retryConfig = FailedTaskRunRetryHelper.getRetryConfig({ run, execution });
if (!retryConfig) {
return;
}
const delay = calculateNextRetryDelay(retryConfig, execution.attempt.number);
if (!delay) {
logger.debug("[FailedTaskRunRetryHelper] No more retries", {
run,
execution,
});
return;
}
return {
timestamp: Date.now() + delay,
delay,
};
} catch (error) {
logger.error("[FailedTaskRunRetryHelper] Failed to get execution retry", {
run,
execution,
error,
});
return;
}
}
static getRetryConfig({
run,
execution,
}: {
run: TaskRunWithWorker;
execution: TaskRunExecution;
}): RetryOptions | undefined {
try {
const retryConfig = run.lockedBy?.retryConfig;
if (!retryConfig) {
if (!run.lockedToVersion) {
logger.error("[FailedTaskRunRetryHelper] Run not locked to version", {
run,
execution,
});
return;
}
const sdkVersion = run.lockedToVersion.sdkVersion ?? "0.0.0";
const isValid = semver.valid(sdkVersion);
if (!isValid) {
logger.error("[FailedTaskRunRetryHelper] Invalid SDK version", {
run,
execution,
});
return;
}
// With older SDK versions, tasks only have a retry config stored in the DB if it's explicitly defined on the task itself
// It won't get populated with retry.default in trigger.config.ts
if (semver.lt(sdkVersion, FailedTaskRunRetryHelper.DEFAULT_RETRY_CONFIG_SINCE_VERSION)) {
logger.warn(
"[FailedTaskRunRetryHelper] SDK version not recent enough to determine retry config",
{
run,
execution,
}
);
return;
}
}
const parsedRetryConfig = RetryOptions.nullable().safeParse(retryConfig);
if (!parsedRetryConfig.success) {
logger.error("[FailedTaskRunRetryHelper] Invalid retry config", {
run,
execution,
});
return;
}
if (!parsedRetryConfig.data) {
logger.debug("[FailedTaskRunRetryHelper] No retry config", {
run,
execution,
});
return;
}
return parsedRetryConfig.data;
} catch (error) {
logger.error("[FailedTaskRunRetryHelper] Failed to get execution retry", {
run,
execution,
error,
});
return;
}
}
static DEFAULT_RETRY_CONFIG_SINCE_VERSION = "3.1.0";
}
apps/webapp/app/v3/handleSocketIo.server.ts
-372
View File
@@ -1,43 +1,21 @@
import type { EventBusEventArgs } from "@internal/run-engine";
import { createAdapter } from "@socket.io/redis-adapter";
import {
ClientToSharedQueueMessages,
CoordinatorSocketData,
CoordinatorToPlatformMessages,
PlatformToCoordinatorMessages,
PlatformToProviderMessages,
ProviderToPlatformMessages,
SharedQueueToClientMessages,
} from "@trigger.dev/core/v3";
import { RunId } from "@trigger.dev/core/v3/isomorphic";
import type {
WorkerClientToServerEvents,
WorkerServerToClientEvents,
} from "@trigger.dev/core/v3/workers";
import { ZodNamespace } from "@trigger.dev/core/v3/zodNamespace";
import { defaultReconnectOnError } from "@internal/redis";
import { Redis } from "ioredis";
import type { Namespace, Socket } from "socket.io";
import { Server } from "socket.io";
import { env } from "~/env.server";
import { findEnvironmentById } from "~/models/runtimeEnvironment.server";
import { authenticateApiRequestWithFailure } from "~/services/apiAuth.server";
import { logger } from "~/services/logger.server";
import { singleton } from "~/utils/singleton";
import { recordRunDebugLog } from "./eventRepository/index.server";
import { sharedQueueTasks } from "./marqs/sharedQueueConsumer.server";
import { engine } from "./runEngine.server";
import { CompleteAttemptService } from "./services/completeAttempt.server";
import { CrashTaskRunService } from "./services/crashTaskRun.server";
import { CreateCheckpointService } from "./services/createCheckpoint.server";
import { CreateDeploymentBackgroundWorkerServiceV3 } from "./services/createDeploymentBackgroundWorkerV3.server";
import { CreateTaskRunAttemptService } from "./services/createTaskRunAttempt.server";
import { DeploymentIndexFailed } from "./services/deploymentIndexFailed.server";
import { ResumeAttemptService } from "./services/resumeAttempt.server";
import { UpdateFatalRunErrorService } from "./services/updateFatalRunError.server";
import { WorkerGroupTokenService } from "./services/worker/workerGroupTokenService.server";
import { SharedSocketConnection } from "./sharedSocketConnection";
import { isV3Disabled } from "./engineDeprecation.server";
export const socketIo = singleton("socketIo", initalizeIoServer);
@@ -48,9 +26,6 @@ function initalizeIoServer() {
logger.log(`[socket.io][${socket.id}] connection at url: ${socket.request.url}`);
});
const coordinatorNamespace = createCoordinatorNamespace(io);
const providerNamespace = createProviderNamespace(io);
const sharedQueueConsumerNamespace = createSharedQueueConsumerNamespace(io);
const workerNamespace = createWorkerNamespace({
io,
namespace: "/worker",
@@ -80,9 +55,6 @@ function initalizeIoServer() {
return {
io,
coordinatorNamespace,
providerNamespace,
sharedQueueConsumerNamespace,
workerNamespace,
devWorkerNamespace,
};
@@ -114,350 +86,6 @@ function initializeSocketIOServerInstance() {
return new Server();
}
function createCoordinatorNamespace(io: Server) {
const coordinator = new ZodNamespace({
// @ts-ignore - for some reason the built ZodNamespace Server type is not compatible with the Server type here, but only when doing typechecking
io,
name: "coordinator",
authToken: env.COORDINATOR_SECRET,
clientMessages: CoordinatorToPlatformMessages,
serverMessages: PlatformToCoordinatorMessages,
socketData: CoordinatorSocketData,
handlers: {
READY_FOR_EXECUTION: async (message) => {
const payload = await sharedQueueTasks.getLatestExecutionPayloadFromRun(
message.runId,
true,
!!message.totalCompletions
);
if (!payload) {
logger.error("Failed to retrieve execution payload", message);
return { success: false };
} else {
return { success: true, payload };
}
},
READY_FOR_LAZY_ATTEMPT: async (message) => {
try {
const payload = await sharedQueueTasks.getLazyAttemptPayload(
message.envId,
message.runId
);
if (!payload) {
logger.error(
"READY_FOR_LAZY_ATTEMPT: Failed to retrieve lazy attempt payload",
message
);
return { success: false, reason: "READY_FOR_LAZY_ATTEMPT: Failed to retrieve payload" };
}
return { success: true, lazyPayload: payload };
} catch (error) {
logger.error("READY_FOR_LAZY_ATTEMPT: Error while creating lazy attempt", {
runId: message.runId,
envId: message.envId,
totalCompletions: message.totalCompletions,
error,
});
return { success: false };
}
},
READY_FOR_RESUME: async (message) => {
const resumeAttempt = new ResumeAttemptService();
await resumeAttempt.call(message);
},
TASK_RUN_COMPLETED: async (message) => {
const completeAttempt = new CompleteAttemptService({
supportsRetryCheckpoints: message.version === "v1",
});
await completeAttempt.call({
completion: message.completion,
execution: message.execution,
checkpoint: message.checkpoint,
});
},
TASK_RUN_COMPLETED_WITH_ACK: async (message) => {
try {
const completeAttempt = new CompleteAttemptService({
supportsRetryCheckpoints: message.version === "v1",
});
await completeAttempt.call({
completion: message.completion,
execution: message.execution,
checkpoint: message.checkpoint,
});
return {
success: true,
};
} catch (error) {
const friendlyError =
error instanceof Error
? {
name: error.name,
message: error.message,
stack: error.stack,
}
: {
name: "UnknownError",
message: String(error),
};
logger.error("Error while completing attempt with ack", {
error: friendlyError,
message,
});
return {
success: false,
error: friendlyError,
};
}
},
TASK_RUN_FAILED_TO_RUN: async (message) => {
await sharedQueueTasks.taskRunFailed(message.completion);
},
TASK_HEARTBEAT: async (message) => {
await sharedQueueTasks.taskHeartbeat(message.attemptFriendlyId);
},
TASK_RUN_HEARTBEAT: async (message) => {
await sharedQueueTasks.taskRunHeartbeat(message.runId);
},
CHECKPOINT_CREATED: async (message) => {
try {
const createCheckpoint = new CreateCheckpointService();
const result = await createCheckpoint.call(message);
return { keepRunAlive: result?.keepRunAlive ?? false };
} catch (error) {
logger.error("Error while creating checkpoint", {
rawMessage: message,
error: error instanceof Error ? error.message : error,
});
return { keepRunAlive: false };
}
},
CREATE_WORKER: async (message) => {
try {
const environment = await findEnvironmentById(message.envId);
if (!environment) {
logger.error("Environment not found", { id: message.envId });
return { success: false };
}
const service = new CreateDeploymentBackgroundWorkerServiceV3();
const worker = await service.call(message.projectRef, environment, message.deploymentId, {
localOnly: false,
metadata: message.metadata,
supportsLazyAttempts: message.version !== "v1" && message.supportsLazyAttempts,
});
return { success: !!worker };
} catch (error) {
logger.error("Error while creating worker", {
error,
envId: message.envId,
projectRef: message.projectRef,
deploymentId: message.deploymentId,
version: message.version,
});
return { success: false };
}
},
CREATE_TASK_RUN_ATTEMPT: async (message) => {
try {
const environment = await findEnvironmentById(message.envId);
if (!environment) {
logger.error("CREATE_TASK_RUN_ATTEMPT: Environment not found", message);
return { success: false, reason: "Environment not found" };
}
const service = new CreateTaskRunAttemptService();
const { attempt } = await service.call({
runId: message.runId,
authenticatedEnv: environment,
setToExecuting: false,
});
const payload = await sharedQueueTasks.getExecutionPayloadFromAttempt({
id: attempt.id,
setToExecuting: true,
skipStatusChecks: true,
});
if (!payload) {
logger.error(
"CREATE_TASK_RUN_ATTEMPT: Failed to retrieve payload after attempt creation",
message
);
return {
success: false,
reason: "CREATE_TASK_RUN_ATTEMPT: Failed to retrieve payload",
};
}
return { success: true, executionPayload: payload };
} catch (error) {
logger.error("CREATE_TASK_RUN_ATTEMPT: Error while creating attempt", {
...message,
error,
});
return { success: false };
}
},
INDEXING_FAILED: async (message) => {
try {
const service = new DeploymentIndexFailed();
await service.call(message.deploymentId, message.error);
} catch (error) {
logger.error("Error while processing index failure", {
deploymentId: message.deploymentId,
error,
});
}
},
RUN_CRASHED: async (message) => {
try {
const service = new CrashTaskRunService();
await service.call(message.runId, {
reason: `${message.error.name}: ${message.error.message}`,
logs: message.error.stack,
});
} catch (error) {
logger.error("Error while processing run failure", {
runId: message.runId,
error,
});
}
},
},
onConnection: async (socket, handler, sender, logger) => {
if (socket.data.supportsDynamicConfig) {
socket.emit("DYNAMIC_CONFIG", {
version: "v1",
checkpointThresholdInMs: env.CHECKPOINT_THRESHOLD_IN_MS,
});
}
},
postAuth: async (socket, next, logger) => {
function setSocketDataFromHeader(
dataKey: keyof typeof socket.data,
headerName: string,
required: boolean = true
) {
const value = socket.handshake.headers[headerName];
if (value) {
socket.data[dataKey] = Array.isArray(value) ? value[0] : value;
return;
}
if (required) {
logger.error("missing required header", { headerName });
throw new Error("missing header");
}
}
try {
setSocketDataFromHeader("supportsDynamicConfig", "x-supports-dynamic-config", false);
} catch (error) {
logger.error("setSocketDataFromHeader error", { error });
socket.disconnect(true);
return;
}
logger.debug("success", socket.data);
next();
},
});
return coordinator.namespace;
}
function createProviderNamespace(io: Server) {
const provider = new ZodNamespace({
// @ts-ignore - for some reason the built ZodNamespace Server type is not compatible with the Server type here, but only when doing typechecking
io,
name: "provider",
authToken: env.PROVIDER_SECRET,
clientMessages: ProviderToPlatformMessages,
serverMessages: PlatformToProviderMessages,
handlers: {
WORKER_CRASHED: async (message) => {
try {
if (message.overrideCompletion) {
const updateErrorService = new UpdateFatalRunErrorService();
await updateErrorService.call(message.runId, { ...message });
} else {
const crashRunService = new CrashTaskRunService();
await crashRunService.call(message.runId, { ...message });
}
} catch (error) {
logger.error("Error while handling crashed worker", { error });
}
},
INDEXING_FAILED: async (message) => {
try {
const service = new DeploymentIndexFailed();
await service.call(message.deploymentId, message.error, message.overrideCompletion);
} catch (e) {
logger.error("Error while indexing", { error: e });
}
},
},
});
return provider.namespace;
}
function createSharedQueueConsumerNamespace(io: Server) {
const sharedQueue = new ZodNamespace({
// @ts-ignore - for some reason the built ZodNamespace Server type is not compatible with the Server type here, but only when doing typechecking
io,
name: "shared-queue",
authToken: env.PROVIDER_SECRET,
clientMessages: ClientToSharedQueueMessages,
serverMessages: SharedQueueToClientMessages,
onConnection: async (socket, handler, sender, logger) => {
// v3 (engine V1) shutdown: don't start the MarQS shared-queue consumer, so no
// deployed V1 runs are dequeued. This namespace is V1-only; v4 dequeues through
// the run-engine worker path. This is the code-level equivalent of taking the
// v3 coordinator offline.
if (isV3Disabled()) {
logger.warn("Refusing /shared-queue connection: v3 engine is shut down");
socket.disconnect(true);
return;
}
const sharedSocketConnection = new SharedSocketConnection({
// @ts-ignore - for some reason the built ZodNamespace Server type is not compatible with the Server type here, but only when doing typechecking
namespace: sharedQueue.namespace,
// @ts-ignore - for some reason the built ZodNamespace Server type is not compatible with the Server type here, but only when doing typechecking
socket,
logger,
poolSize: env.SHARED_QUEUE_CONSUMER_POOL_SIZE,
});
sharedSocketConnection.onClose.attach((closeEvent) => {
logger.info("Socket closed", { closeEvent });
});
await sharedSocketConnection.initialize();
},
});
return sharedQueue.namespace;
}
function headersFromHandshake(handshake: Socket["handshake"]) {
const headers = new Headers();
apps/webapp/app/v3/legacyRunEngineWorker.server.ts
-118
View File
@@ -1,118 +0,0 @@
import { Worker as RedisWorker } from "@trigger.dev/redis-worker";
import { Logger } from "@trigger.dev/core/logger";
import { z } from "zod";
import { env } from "~/env.server";
import { logger } from "~/services/logger.server";
import { singleton } from "~/utils/singleton";
import { TaskRunHeartbeatFailedService } from "./taskRunHeartbeatFailed.server";
import { completeBatchTaskRunItemV3, tryCompleteBatchV3 } from "./services/batchTriggerV3.server";
import { prisma } from "~/db.server";
import { marqs } from "./marqs/index.server";
function initializeWorker() {
const redisOptions = {
keyPrefix: "legacy-run-engine:worker:",
host: env.LEGACY_RUN_ENGINE_WORKER_REDIS_HOST,
port: env.LEGACY_RUN_ENGINE_WORKER_REDIS_PORT,
username: env.LEGACY_RUN_ENGINE_WORKER_REDIS_USERNAME,
password: env.LEGACY_RUN_ENGINE_WORKER_REDIS_PASSWORD,
enableAutoPipelining: true,
...(env.LEGACY_RUN_ENGINE_WORKER_REDIS_TLS_DISABLED === "true" ? {} : { tls: {} }),
};
logger.debug(
`👨‍🏭 Initializing legacy run engine worker at host ${env.LEGACY_RUN_ENGINE_WORKER_REDIS_HOST}`
);
const worker = new RedisWorker({
name: "legacy-run-engine-worker",
redisOptions,
catalog: {
runHeartbeat: {
schema: z.object({
runId: z.string(),
}),
visibilityTimeoutMs: 60_000,
retry: {
maxAttempts: 3,
},
},
completeBatchTaskRunItem: {
schema: z.object({
itemId: z.string(),
batchTaskRunId: z.string(),
scheduleResumeOnComplete: z.boolean(),
taskRunAttemptId: z.string().optional(),
attempt: z.number().optional(),
}),
visibilityTimeoutMs: 60_000,
retry: {
maxAttempts: 10,
},
},
tryCompleteBatchV3: {
schema: z.object({
batchId: z.string(),
scheduleResumeOnComplete: z.boolean(),
}),
visibilityTimeoutMs: 30_000,
retry: {
maxAttempts: 5,
},
},
scheduleRequeueMessage: {
schema: z.object({
messageId: z.string(),
}),
visibilityTimeoutMs: 60_000,
retry: {
maxAttempts: 5,
},
},
},
concurrency: {
workers: env.LEGACY_RUN_ENGINE_WORKER_CONCURRENCY_WORKERS,
tasksPerWorker: env.LEGACY_RUN_ENGINE_WORKER_CONCURRENCY_TASKS_PER_WORKER,
limit: env.LEGACY_RUN_ENGINE_WORKER_CONCURRENCY_LIMIT,
},
pollIntervalMs: env.LEGACY_RUN_ENGINE_WORKER_POLL_INTERVAL,
immediatePollIntervalMs: env.LEGACY_RUN_ENGINE_WORKER_IMMEDIATE_POLL_INTERVAL,
shutdownTimeoutMs: env.LEGACY_RUN_ENGINE_WORKER_SHUTDOWN_TIMEOUT_MS,
logger: new Logger("LegacyRunEngineWorker", env.LEGACY_RUN_ENGINE_WORKER_LOG_LEVEL),
jobs: {
runHeartbeat: async ({ payload }) => {
const service = new TaskRunHeartbeatFailedService();
await service.call(payload.runId);
},
completeBatchTaskRunItem: async ({ payload, attempt }) => {
await completeBatchTaskRunItemV3(
payload.itemId,
payload.batchTaskRunId,
prisma,
payload.scheduleResumeOnComplete,
payload.taskRunAttemptId,
attempt
);
},
tryCompleteBatchV3: async ({ payload }) => {
await tryCompleteBatchV3(payload.batchId, prisma, payload.scheduleResumeOnComplete);
},
scheduleRequeueMessage: async ({ payload }) => {
await marqs.requeueMessageById(payload.messageId);
},
},
});
if (env.LEGACY_RUN_ENGINE_WORKER_ENABLED === "true") {
logger.debug(
`👨‍🏭 Starting legacy run engine worker at host ${env.LEGACY_RUN_ENGINE_WORKER_REDIS_HOST}, pollInterval = ${env.LEGACY_RUN_ENGINE_WORKER_POLL_INTERVAL}, immediatePollInterval = ${env.LEGACY_RUN_ENGINE_WORKER_IMMEDIATE_POLL_INTERVAL}, workers = ${env.LEGACY_RUN_ENGINE_WORKER_CONCURRENCY_WORKERS}, tasksPerWorker = ${env.LEGACY_RUN_ENGINE_WORKER_CONCURRENCY_TASKS_PER_WORKER}, concurrencyLimit = ${env.LEGACY_RUN_ENGINE_WORKER_CONCURRENCY_LIMIT}`
);
worker.start();
}
return worker;
}
export const legacyRunEngineWorker = singleton("legacyRunEngineWorker", initializeWorker);
apps/webapp/app/v3/marqs/asyncWorker.server.ts
-37
View File
@@ -1,37 +0,0 @@
export class AsyncWorker {
private running = false;
private timeout?: NodeJS.Timeout;
constructor(
private readonly fn: () => Promise<void>,
private readonly interval: number
) {}
start() {
if (this.running) {
return;
}
this.running = true;
this.#run();
}
stop() {
this.running = false;
}
async #run() {
if (!this.running) {
return;
}
try {
await this.fn();
} catch (e) {
console.error(e);
}
this.timeout = setTimeout(this.#run.bind(this), this.interval);
}
}
apps/webapp/app/v3/marqs/concurrencyMonitor.server.ts
-204
View File
@@ -1,204 +0,0 @@
import type { Logger } from "@trigger.dev/core/logger";
import type { Redis } from "ioredis";
import { prisma } from "~/db.server";
import { logger } from "~/services/logger.server";
import type { MarQS } from "./index.server";
import { marqs as marqsv3 } from "./index.server";
import { env } from "~/env.server";
export type MarqsConcurrencyMonitorOptions = {
dryRun?: boolean;
abortSignal?: AbortSignal;
};
export interface MarqsConcurrencyResolveCompletedRunsCallback {
(candidateRunIds: string[]): Promise<Array<{ id: string }>>;
}
export class MarqsConcurrencyMonitor {
private _logger: Logger;
constructor(
private marqs: MarQS,
private callback: MarqsConcurrencyResolveCompletedRunsCallback,
private options: MarqsConcurrencyMonitorOptions = {}
) {
this._logger = logger.child({
component: "marqs",
operation: "concurrencyMonitor",
dryRun: this.dryRun,
marqs: marqs.name,
});
}
get dryRun() {
return typeof this.options.dryRun === "boolean" ? this.options.dryRun : false;
}
get keys() {
return this.marqs.keys;
}
get signal() {
return this.options.abortSignal;
}
public async call() {
this._logger.debug("[MarqsConcurrencyMonitor] Initiating monitoring");
const stats = {
streamCallbacks: 0,
processedKeys: 0,
};
const { stream, redis } = this.marqs.queueConcurrencyScanStream(
10,
() => {
this._logger.debug("[MarqsConcurrencyMonitor] stream closed", {
stats,
});
},
(error) => {
this._logger.debug("[MarqsConcurrencyMonitor] stream error", {
stats,
error: {
name: error.name,
message: error.message,
stack: error.stack,
},
});
}
);
stream.on("data", async (keys) => {
stream.pause();
if (this.signal?.aborted) {
stream.destroy();
return;
}
stats.streamCallbacks++;
const uniqueKeys = Array.from(new Set<string>(keys));
if (uniqueKeys.length === 0) {
stream.resume();
return;
}
this._logger.debug("[MarqsConcurrencyMonitor] correcting queues concurrency", {
keys: uniqueKeys,
});
stats.processedKeys += uniqueKeys.length;
await Promise.allSettled(uniqueKeys.map((key) => this.#processKey(key, redis))).finally(
() => {
stream.resume();
}
);
});
}
async #processKey(key: string, redis: Redis) {
key = this.keys.stripKeyPrefix(key);
const envKey = this.keys.envCurrentConcurrencyKeyFromQueue(key);
let runIds: string[] = [];
try {
// Next, we need to get all the items from the key, and any parent keys (org, env, queue) using sunion.
runIds = await redis.sunion(envKey, key);
} catch (e) {
this._logger.error("[MarqsConcurrencyMonitor] error during sunion", {
key,
envKey,
runIds,
error: e,
});
}
if (runIds.length === 0) {
return;
}
const perfNow = performance.now();
const completeRuns = await this.callback(runIds);
const durationMs = performance.now() - perfNow;
const completedRunIds = completeRuns.map((run) => run.id);
if (completedRunIds.length === 0) {
this._logger.debug("[MarqsConcurrencyMonitor] no completed runs found", {
key,
envKey,
runIds,
durationMs,
});
return;
}
this._logger.debug("[MarqsConcurrencyMonitor] removing completed runs from queue", {
key,
envKey,
completedRunIds,
durationMs,
});
if (this.dryRun) {
return;
}
const pipeline = redis.pipeline();
pipeline.srem(key, ...completedRunIds);
pipeline.srem(envKey, ...completedRunIds);
try {
await pipeline.exec();
} catch (e) {
this._logger.error("[MarqsConcurrencyMonitor] error removing completed runs from queue", {
key,
envKey,
completedRunIds,
error: e,
});
}
}
static async initiateV3Monitoring(abortSignal?: AbortSignal) {
if (!marqsv3) {
return;
}
const instance = new MarqsConcurrencyMonitor(
marqsv3,
(runIds) =>
prisma.taskRun.findMany({
select: { id: true },
where: {
id: {
in: runIds,
},
status: {
in: [
"CANCELED",
"COMPLETED_SUCCESSFULLY",
"COMPLETED_WITH_ERRORS",
"CRASHED",
"SYSTEM_FAILURE",
"INTERRUPTED",
],
},
},
}),
{ dryRun: env.V3_MARQS_CONCURRENCY_MONITOR_ENABLED === "0", abortSignal }
);
await instance.call();
}
}
apps/webapp/app/v3/marqs/constants.server.ts
-4
View File
@@ -1,4 +0,0 @@
export const MARQS_RESUME_PRIORITY_TIMESTAMP_OFFSET = 31_556_952 * 1000; // 1 year
export const MARQS_RETRY_PRIORITY_TIMESTAMP_OFFSET = 15_778_476 * 1000; // 6 months
export const MARQS_DELAYED_REQUEUE_THRESHOLD_IN_MS = 500;
export const MARQS_SCHEDULED_REQUEUE_AVAILABLE_AT_THRESHOLD_IN_MS = 500;
apps/webapp/app/v3/marqs/devPubSub.server.ts
-45
View File
@@ -1,45 +0,0 @@
import { z } from "zod";
import { singleton } from "~/utils/singleton";
import type { ZodSubscriber } from "../utils/zodPubSub.server";
import { ZodPubSub } from "../utils/zodPubSub.server";
import { env } from "~/env.server";
import { Gauge } from "prom-client";
import { metricsRegister } from "~/metrics.server";
const messageCatalog = {
CANCEL_ATTEMPT: z.object({
version: z.literal("v1").default("v1"),
backgroundWorkerId: z.string(),
attemptId: z.string(),
taskRunId: z.string(),
}),
};
export type DevSubscriber = ZodSubscriber<typeof messageCatalog>;
export const devPubSub = singleton("devPubSub", initializeDevPubSub);
function initializeDevPubSub() {
const pubSub = new ZodPubSub({
redis: {
port: env.PUBSUB_REDIS_PORT,
host: env.PUBSUB_REDIS_HOST,
username: env.PUBSUB_REDIS_USERNAME,
password: env.PUBSUB_REDIS_PASSWORD,
tlsDisabled: env.PUBSUB_REDIS_TLS_DISABLED === "true",
clusterMode: env.PUBSUB_REDIS_CLUSTER_MODE_ENABLED === "1",
},
schema: messageCatalog,
});
new Gauge({
name: "dev_pub_sub_subscribers",
help: "Number of dev pub sub subscribers",
collect() {
this.set(pubSub.subscriberCount);
},
registers: [metricsRegister],
});
return pubSub;
}
apps/webapp/app/v3/marqs/devQueueConsumer.server.ts
-623
View File
@@ -1,623 +0,0 @@
import type { Context, Span } from "@opentelemetry/api";
import { ROOT_CONTEXT, SpanKind, context, trace } from "@opentelemetry/api";
import type {
V3TaskRunExecution,
TaskRunExecutionLazyAttemptPayload,
TaskRunExecutionResult,
TaskRunFailedExecutionResult,
serverWebsocketMessages,
} from "@trigger.dev/core/v3";
import { getMaxDuration } from "@trigger.dev/core/v3/isomorphic";
import type { ZodMessageSender } from "@trigger.dev/core/v3/zodMessageHandler";
import type { BackgroundWorker, BackgroundWorkerTask } from "@trigger.dev/database";
import { z } from "zod";
import { prisma } from "~/db.server";
import { createNewSession, disconnectSession } from "~/models/runtimeEnvironment.server";
import { findQueueInEnvironment, sanitizeQueueName } from "~/models/taskQueue.server";
import type { RedisClient } from "~/redis.server";
import { createRedisClient } from "~/redis.server";
import type { AuthenticatedEnvironment } from "~/services/apiAuth.server";
import { logger } from "~/services/logger.server";
import { marqs } from "~/v3/marqs/index.server";
import { resolveVariablesForEnvironment } from "../environmentVariables/environmentVariablesRepository.server";
import { FailedTaskRunService } from "../failedTaskRun.server";
import { CancelDevSessionRunsService } from "../services/cancelDevSessionRuns.server";
import { CompleteAttemptService } from "../services/completeAttempt.server";
import { attributesFromAuthenticatedEnv, tracer } from "../tracer.server";
import type { DevSubscriber } from "./devPubSub.server";
import { devPubSub } from "./devPubSub.server";
const MessageBody = z.discriminatedUnion("type", [
z.object({
type: z.literal("EXECUTE"),
taskIdentifier: z.string(),
}),
]);
type BackgroundWorkerWithTasks = BackgroundWorker & { tasks: BackgroundWorkerTask[] };
export type DevQueueConsumerOptions = {
maximumItemsPerTrace?: number;
traceTimeoutSeconds?: number;
ipAddress?: string;
};
export class DevQueueConsumer {
private _backgroundWorkers: Map<string, BackgroundWorkerWithTasks> = new Map();
private _backgroundWorkerSubscriber: Map<string, DevSubscriber> = new Map();
private _deprecatedWorkers: Map<string, BackgroundWorkerWithTasks> = new Map();
private _enabled = false;
private _maximumItemsPerTrace: number;
private _traceTimeoutSeconds: number;
private _perTraceCountdown: number | undefined;
private _lastNewTrace: Date | undefined;
private _currentSpanContext: Context | undefined;
private _taskFailures: number = 0;
private _taskSuccesses: number = 0;
private _currentSpan: Span | undefined;
private _endSpanInNextIteration = false;
private _inProgressRuns: Map<string, string> = new Map(); // Keys are task run friendly IDs, values are TaskRun internal ids/queue message ids
private _connectionLostAt?: Date;
private _redisClient: RedisClient;
constructor(
public id: string,
public env: AuthenticatedEnvironment,
private _sender: ZodMessageSender<typeof serverWebsocketMessages>,
private _options: DevQueueConsumerOptions = {}
) {
this._traceTimeoutSeconds = _options.traceTimeoutSeconds ?? 60;
this._maximumItemsPerTrace = _options.maximumItemsPerTrace ?? 1_000;
this._redisClient = createRedisClient("tr:devQueueConsumer", {
keyPrefix: "tr:devQueueConsumer:",
...devPubSub.redisOptions,
});
}
// This method is called when a background worker is deprecated and will no longer be used unless a run is locked to it
public async deprecateBackgroundWorker(id: string) {
const backgroundWorker = this._backgroundWorkers.get(id);
if (!backgroundWorker) {
return;
}
logger.debug("[DevQueueConsumer] Deprecating background worker", {
backgroundWorker: backgroundWorker.id,
env: this.env.id,
});
this._deprecatedWorkers.set(id, backgroundWorker);
this._backgroundWorkers.delete(id);
}
public async registerBackgroundWorker(id: string, inProgressRuns: string[] = []) {
const backgroundWorker = await prisma.backgroundWorker.findFirst({
where: { friendlyId: id, runtimeEnvironmentId: this.env.id },
include: {
tasks: true,
},
});
if (!backgroundWorker) {
return;
}
if (this._backgroundWorkers.has(backgroundWorker.id)) {
return;
}
this._backgroundWorkers.set(backgroundWorker.id, backgroundWorker);
logger.debug("[DevQueueConsumer] Registered background worker", {
backgroundWorker: backgroundWorker.id,
inProgressRuns,
env: this.env.id,
});
const subscriber = await devPubSub.subscribe(`backgroundWorker:${backgroundWorker.id}:*`);
subscriber.on("CANCEL_ATTEMPT", async (message) => {
await this._sender.send("BACKGROUND_WORKER_MESSAGE", {
backgroundWorkerId: backgroundWorker.friendlyId,
data: {
type: "CANCEL_ATTEMPT",
taskAttemptId: message.attemptId,
taskRunId: message.taskRunId,
},
});
});
this._backgroundWorkerSubscriber.set(backgroundWorker.id, subscriber);
for (const runId of inProgressRuns) {
this._inProgressRuns.set(runId, runId);
}
// Start reading from the queue if we haven't already
await this.#enable();
}
public async taskAttemptCompleted(
workerId: string,
completion: TaskRunExecutionResult,
execution: V3TaskRunExecution
) {
if (completion.ok) {
this._taskSuccesses++;
} else {
this._taskFailures++;
}
logger.debug("[DevQueueConsumer] taskAttemptCompleted()", {
taskRunCompletion: completion,
execution,
env: this.env.id,
});
const service = new CompleteAttemptService();
const result = await service.call({ completion, execution, env: this.env });
if (result === "COMPLETED") {
this._inProgressRuns.delete(execution.run.id);
}
}
public async taskRunFailed(workerId: string, completion: TaskRunFailedExecutionResult) {
this._taskFailures++;
logger.debug("[DevQueueConsumer] taskRunFailed()", { completion, env: this.env.id });
this._inProgressRuns.delete(completion.id);
const service = new FailedTaskRunService();
await service.call(completion.id, completion);
}
/**
* @deprecated Use `taskRunHeartbeat` instead
*/
public async taskHeartbeat(workerId: string, id: string) {
logger.debug("[DevQueueConsumer] taskHeartbeat()", { id });
const taskRunAttempt = await prisma.taskRunAttempt.findFirst({
where: { friendlyId: id },
});
if (!taskRunAttempt) {
return;
}
await marqs?.heartbeatMessage(taskRunAttempt.taskRunId);
}
public async taskRunHeartbeat(workerId: string, id: string) {
logger.debug("[DevQueueConsumer] taskRunHeartbeat()", { id });
await marqs?.heartbeatMessage(id);
}
public async stop(reason: string = "CLI disconnected") {
if (!this._enabled) {
return;
}
logger.debug("[DevQueueConsumer] Stopping dev queue consumer", { env: this.env });
this._enabled = false;
// Create the session
const session = await disconnectSession(this.env.id);
const runIds = Array.from(this._inProgressRuns.values());
this._inProgressRuns.clear();
if (runIds.length > 0) {
await CancelDevSessionRunsService.enqueue(
{
runIds,
cancelledAt: new Date(),
reason,
cancelledSessionId: session?.id,
},
new Date(Date.now() + 1000 * 10) // 10 seconds from now
);
}
// We need to unsubscribe from the background worker channels
for (const [id, subscriber] of this._backgroundWorkerSubscriber) {
logger.debug("Unsubscribing from background worker channel", { id });
await subscriber.stopListening();
this._backgroundWorkerSubscriber.delete(id);
logger.debug("Unsubscribed from background worker channel", { id });
}
// We need to end the current span
if (this._currentSpan) {
this._currentSpan.end();
}
}
async #enable() {
if (this._enabled) {
return;
}
await this._redisClient.set(`connection:${this.env.id}`, this.id, "EX", 60 * 60 * 24); // 24 hours
this._enabled = true;
// Create the session
await createNewSession(this.env, this._options.ipAddress ?? "unknown");
this._perTraceCountdown = this._options.maximumItemsPerTrace;
this._lastNewTrace = new Date();
this._taskFailures = 0;
this._taskSuccesses = 0;
this.#doWork().finally(() => {});
}
async #doWork() {
if (!this._enabled) {
return;
}
const canSendMessage = await this._sender.validateCanSendMessage();
if (!canSendMessage) {
this._connectionLostAt ??= new Date();
if (Date.now() - this._connectionLostAt.getTime() > 60 * 1000) {
logger.debug("Connection lost for more than 60 seconds, stopping the consumer", {
env: this.env,
});
await this.stop("Connection lost for more than 60 seconds");
return;
}
setTimeout(() => this.#doWork(), 1000);
return;
}
this._connectionLostAt = undefined;
const currentConnection = await this._redisClient.get(`connection:${this.env.id}`);
if (currentConnection && currentConnection !== this.id) {
logger.debug("Another connection is active, stopping the consumer", {
currentConnection,
env: this.env,
});
await this.stop("Another connection is active");
return;
}
// Check if the trace has expired
if (
this._perTraceCountdown === 0 ||
Date.now() - this._lastNewTrace!.getTime() > this._traceTimeoutSeconds * 1000 ||
this._currentSpanContext === undefined ||
this._endSpanInNextIteration
) {
if (this._currentSpan) {
this._currentSpan.setAttribute("tasks.period.failures", this._taskFailures);
this._currentSpan.setAttribute("tasks.period.successes", this._taskSuccesses);
logger.debug("Ending DevQueueConsumer.doWork() trace", {
isRecording: this._currentSpan.isRecording(),
});
this._currentSpan.end();
}
// Create a new trace
this._currentSpan = tracer.startSpan(
"DevQueueConsumer.doWork()",
{
kind: SpanKind.CONSUMER,
attributes: {
...attributesFromAuthenticatedEnv(this.env),
},
},
ROOT_CONTEXT
);
// Get the span trace context
this._currentSpanContext = trace.setSpan(ROOT_CONTEXT, this._currentSpan);
this._perTraceCountdown = this._options.maximumItemsPerTrace;
this._lastNewTrace = new Date();
this._taskFailures = 0;
this._taskSuccesses = 0;
this._endSpanInNextIteration = false;
}
return context.with(this._currentSpanContext ?? ROOT_CONTEXT, async () => {
await this.#doWorkInternal();
this._perTraceCountdown = this._perTraceCountdown! - 1;
});
}
async #doWorkInternal() {
// Attempt to dequeue a message from the environment's queue
// If no message is available, reschedule the worker to run again in 1 second
// If a message is available, find the BackgroundWorkerTask that matches the message's taskIdentifier
// If no matching task is found, nack the message and reschedule the worker to run again in 1 second
// If the matching task is found, create the task attempt and lock the task run, then send the task run to the client
// Store the message as a processing message
// If the websocket connection disconnects before the task run is completed, nack the message
// When the task run completes, ack the message
// Using a heartbeat mechanism, if the client keeps responding with a heartbeat, we'll keep the message processing and increase the visibility timeout.
const message = await marqs?.dequeueMessageInEnv(this.env);
if (!message) {
setTimeout(() => this.#doWork(), 1000);
return;
}
const dequeuedStart = Date.now();
const messageBody = MessageBody.safeParse(message.data);
if (!messageBody.success) {
logger.error("Failed to parse message", {
queueMessage: message.data,
error: messageBody.error,
env: this.env,
});
await marqs?.acknowledgeMessage(
message.messageId,
"Failed to parse message.data with MessageBody schema in DevQueueConsumer"
);
setTimeout(() => this.#doWork(), 100);
return;
}
const existingTaskRun = await prisma.taskRun.findFirst({
where: {
id: message.messageId,
},
});
if (!existingTaskRun) {
logger.debug("Failed to find existing task run, acking", {
messageId: message.messageId,
});
await marqs?.acknowledgeMessage(
message.messageId,
"Failed to find task run in DevQueueConsumer"
);
setTimeout(() => this.#doWork(), 100);
return;
}
const backgroundWorker = existingTaskRun.lockedToVersionId
? (this._deprecatedWorkers.get(existingTaskRun.lockedToVersionId) ??
this._backgroundWorkers.get(existingTaskRun.lockedToVersionId))
: this.#getLatestBackgroundWorker();
if (!backgroundWorker) {
logger.debug("Failed to find background worker, acking", {
messageId: message.messageId,
lockedToVersionId: existingTaskRun.lockedToVersionId,
deprecatedWorkers: Array.from(this._deprecatedWorkers.keys()),
backgroundWorkers: Array.from(this._backgroundWorkers.keys()),
latestWorker: this.#getLatestBackgroundWorker(),
});
await marqs?.acknowledgeMessage(
message.messageId,
"Failed to find background worker in DevQueueConsumer"
);
setTimeout(() => this.#doWork(), 100);
return;
}
const backgroundTask = backgroundWorker.tasks.find(
(task) => task.slug === existingTaskRun.taskIdentifier
);
if (!backgroundTask) {
logger.warn("No matching background task found for task run", {
taskRun: existingTaskRun.id,
taskIdentifier: existingTaskRun.taskIdentifier,
backgroundWorker: backgroundWorker.id,
taskSlugs: backgroundWorker.tasks.map((task) => task.slug),
});
await marqs?.acknowledgeMessage(
message.messageId,
"No matching background task found in DevQueueConsumer"
);
setTimeout(() => this.#doWork(), 100);
return;
}
const lockedAt = new Date();
const startedAt = existingTaskRun.startedAt ?? new Date();
const lockedTaskRun = await prisma.taskRun.update({
where: {
id: message.messageId,
},
data: {
lockedAt,
lockedById: backgroundTask.id,
status: "EXECUTING",
lockedToVersionId: backgroundWorker.id,
taskVersion: backgroundWorker.version,
sdkVersion: backgroundWorker.sdkVersion,
cliVersion: backgroundWorker.cliVersion,
startedAt,
maxDurationInSeconds: getMaxDuration(
existingTaskRun.maxDurationInSeconds,
backgroundTask.maxDurationInSeconds
),
},
});
if (!lockedTaskRun) {
logger.warn("Failed to lock task run", {
taskRun: existingTaskRun.id,
taskIdentifier: existingTaskRun.taskIdentifier,
backgroundWorker: backgroundWorker.id,
messageId: message.messageId,
});
await marqs?.acknowledgeMessage(
message.messageId,
"Failed to lock task run in DevQueueConsumer"
);
setTimeout(() => this.#doWork(), 100);
return;
}
const queue = await findQueueInEnvironment(
lockedTaskRun.queue,
this.env.id,
backgroundTask.id,
backgroundTask
);
if (!queue) {
logger.debug("[DevQueueConsumer] Failed to find queue", {
queueName: lockedTaskRun.queue,
sanitizedName: sanitizeQueueName(lockedTaskRun.queue),
taskRun: lockedTaskRun.id,
messageId: message.messageId,
});
await marqs?.nackMessage(message.messageId);
setTimeout(() => this.#doWork(), 1000);
return;
}
if (!this._enabled) {
logger.debug("Dev queue consumer is disabled", { env: this.env, queueMessage: message });
await marqs?.nackMessage(message.messageId);
return;
}
const variables = await resolveVariablesForEnvironment(this.env);
if (backgroundWorker.supportsLazyAttempts) {
const payload: TaskRunExecutionLazyAttemptPayload = {
traceContext: lockedTaskRun.traceContext as Record<string, unknown>,
environment: variables.reduce((acc: Record<string, string>, curr) => {
acc[curr.key] = curr.value;
return acc;
}, {}),
runId: lockedTaskRun.friendlyId,
messageId: lockedTaskRun.id,
isTest: lockedTaskRun.isTest,
isReplay: !!lockedTaskRun.replayedFromTaskRunFriendlyId,
metrics: [
{
name: "start",
event: "dequeue",
timestamp: dequeuedStart,
duration: Date.now() - dequeuedStart,
},
],
};
try {
await this._sender.send("BACKGROUND_WORKER_MESSAGE", {
backgroundWorkerId: backgroundWorker.friendlyId,
data: {
type: "EXECUTE_RUN_LAZY_ATTEMPT",
payload,
},
});
logger.debug("Executing the run", {
messageId: message.messageId,
});
this._inProgressRuns.set(lockedTaskRun.friendlyId, message.messageId);
} catch (e) {
if (e instanceof Error) {
this._currentSpan?.recordException(e);
} else {
this._currentSpan?.recordException(new Error(String(e)));
}
this._endSpanInNextIteration = true;
// We now need to unlock the task run and delete the task run attempt
await prisma.$transaction([
prisma.taskRun.update({
where: {
id: lockedTaskRun.id,
},
data: {
lockedAt: null,
lockedById: null,
status: "PENDING",
startedAt: existingTaskRun.startedAt,
},
}),
]);
this._inProgressRuns.delete(lockedTaskRun.friendlyId);
// Finally we need to nack the message so it can be retried
await marqs?.nackMessage(message.messageId);
} finally {
setTimeout(() => this.#doWork(), 100);
}
} else {
logger.debug("We no longer support non-lazy attempts, aborting this run", {
messageId: message.messageId,
backgroundWorker,
});
await marqs?.acknowledgeMessage(
message.messageId,
"Non-lazy attempts are no longer supported in DevQueueConsumer"
);
setTimeout(() => this.#doWork(), 100);
}
}
// Get the latest background worker based on the version.
// Versions are in the format of 20240101.1 and 20240101.2, or even 20240101.10, 20240101.11, etc.
#getLatestBackgroundWorker() {
const workers = Array.from(this._backgroundWorkers.values());
if (workers.length === 0) {
return;
}
return workers.reduce((acc, curr) => {
const accParts = acc.version.split(".").map(Number);
const currParts = curr.version.split(".").map(Number);
// Compare the major part
if (accParts[0] < currParts[0]) {
return curr;
} else if (accParts[0] > currParts[0]) {
return acc;
}
// Compare the minor part (assuming all versions have two parts)
if (accParts[1] < currParts[1]) {
return curr;
} else {
return acc;
}
});
}
}
apps/webapp/app/v3/marqs/fairDequeuingStrategy.server.ts
-598
View File
@@ -1,598 +0,0 @@
import type { Cache as UnkeyCache } from "@unkey/cache";
import { createCache, DefaultStatefulContext, Namespace } from "@unkey/cache";
import { createLRUMemoryStore } from "@internal/cache";
import { randomUUID } from "crypto";
import type { Redis } from "ioredis";
import type { EnvQueues, MarQSFairDequeueStrategy, MarQSKeyProducer } from "./types";
import seedrandom from "seedrandom";
import type { Tracer } from "@opentelemetry/api";
import { startSpan } from "../tracing.server";
export type FairDequeuingStrategyBiases = {
/**
* How much to bias towards environments with higher concurrency limits
* 0 = no bias, 1 = full bias based on limit differences
*/
concurrencyLimitBias: number;
/**
* How much to bias towards environments with more available capacity
* 0 = no bias, 1 = full bias based on available capacity
*/
availableCapacityBias: number;
/**
* Controls randomization of queue ordering within environments
* 0 = strict age-based ordering (oldest first)
* 1 = completely random ordering
* Values between 0-1 blend between age-based and random ordering
*/
queueAgeRandomization: number;
};
export type FairDequeuingStrategyOptions = {
redis: Redis;
keys: MarQSKeyProducer;
defaultEnvConcurrency: number;
parentQueueLimit: number;
tracer: Tracer;
seed?: string;
/**
* Configure biasing for environment shuffling
* If not provided, no biasing will be applied (completely random shuffling)
*/
biases?: FairDequeuingStrategyBiases;
reuseSnapshotCount?: number;
maximumEnvCount?: number;
/**
* Maximum number of queues to process per environment
* If not provided, all queues in an environment will be processed
*/
maximumQueuePerEnvCount?: number;
};
type FairQueueConcurrency = {
current: number;
limit: number;
reserve: number;
};
type FairQueue = { id: string; age: number; org: string; env: string };
type FairQueueSnapshot = {
id: string;
envs: Record<string, { concurrency: FairQueueConcurrency }>;
queues: Array<FairQueue>;
};
type WeightedEnv = {
envId: string;
weight: number;
};
type WeightedQueue = {
queue: FairQueue;
weight: number;
};
const emptyFairQueueSnapshot: FairQueueSnapshot = {
id: "empty",
envs: {},
queues: [],
};
const defaultBiases: FairDequeuingStrategyBiases = {
concurrencyLimitBias: 0,
availableCapacityBias: 0,
queueAgeRandomization: 0, // Default to completely age-based ordering
};
export class FairDequeuingStrategy implements MarQSFairDequeueStrategy {
private _cache: UnkeyCache<{
concurrencyLimit: number;
}>;
private _rng: seedrandom.PRNG;
private _reusedSnapshotForConsumer: Map<
string,
{ snapshot: FairQueueSnapshot; reuseCount: number }
> = new Map();
constructor(private options: FairDequeuingStrategyOptions) {
const ctx = new DefaultStatefulContext();
const memory = createLRUMemoryStore(500);
this._cache = createCache({
concurrencyLimit: new Namespace<number>(ctx, {
stores: [memory],
fresh: 60_000, // The time in milliseconds that a value is considered fresh. Cache hits within this time will return the cached value.
stale: 180_000, // The time in milliseconds that a value is considered stale. Cache hits within this time will return the cached value and trigger a background refresh.
}),
});
this._rng = seedrandom(options.seed);
}
async distributeFairQueuesFromParentQueue(
parentQueue: string,
consumerId: string
): Promise<Array<EnvQueues>> {
return await startSpan(
this.options.tracer,
"distributeFairQueuesFromParentQueue",
async (span) => {
span.setAttribute("consumer_id", consumerId);
span.setAttribute("parent_queue", parentQueue);
const snapshot = await this.#createQueueSnapshot(parentQueue, consumerId);
span.setAttributes({
snapshot_env_count: Object.keys(snapshot.envs).length,
snapshot_queue_count: snapshot.queues.length,
});
const queues = snapshot.queues;
if (queues.length === 0) {
return [];
}
const envQueues = this.#shuffleQueuesByEnv(snapshot);
span.setAttribute(
"shuffled_queue_count",
envQueues.reduce((sum, env) => sum + env.queues.length, 0)
);
if (envQueues[0]?.queues[0]) {
span.setAttribute("winning_env", envQueues[0].envId);
span.setAttribute(
"winning_org",
this.options.keys.orgIdFromQueue(envQueues[0].queues[0])
);
}
return envQueues;
}
);
}
#shuffleQueuesByEnv(snapshot: FairQueueSnapshot): Array<EnvQueues> {
const envs = Object.keys(snapshot.envs);
const biases = this.options.biases ?? defaultBiases;
if (biases.concurrencyLimitBias === 0 && biases.availableCapacityBias === 0) {
const shuffledEnvs = this.#shuffle(envs);
return this.#orderQueuesByEnvs(shuffledEnvs, snapshot);
}
// Find the maximum concurrency limit for normalization
const maxLimit = Math.max(...envs.map((envId) => snapshot.envs[envId].concurrency.limit));
// Calculate weights for each environment
const weightedEnvs: WeightedEnv[] = envs.map((envId) => {
const env = snapshot.envs[envId];
// Start with base weight of 1
let weight = 1;
// Add normalized concurrency limit bias if configured
if (biases.concurrencyLimitBias > 0) {
const normalizedLimit = env.concurrency.limit / maxLimit;
// Square or cube the bias to make it more pronounced at higher values
weight *= 1 + Math.pow(normalizedLimit * biases.concurrencyLimitBias, 2);
}
// Add available capacity bias if configured
if (biases.availableCapacityBias > 0) {
const usedCapacityPercentage = env.concurrency.current / env.concurrency.limit;
const availableCapacityBonus = 1 - usedCapacityPercentage;
// Square or cube the bias to make it more pronounced at higher values
weight *= 1 + Math.pow(availableCapacityBonus * biases.availableCapacityBias, 2);
}
return { envId, weight };
});
const shuffledEnvs = this.#weightedShuffle(weightedEnvs);
return this.#orderQueuesByEnvs(shuffledEnvs, snapshot);
}
#weightedShuffle(weightedItems: WeightedEnv[]): string[] {
const totalWeight = weightedItems.reduce((sum, item) => sum + item.weight, 0);
const result: string[] = [];
const items = [...weightedItems];
while (items.length > 0) {
let random = this._rng() * totalWeight;
let index = 0;
// Find item based on weighted random selection
while (random > 0 && index < items.length) {
random -= items[index].weight;
index++;
}
index = Math.max(0, index - 1);
// Add selected item to result and remove from items
result.push(items[index].envId);
items.splice(index, 1);
}
return result;
}
#orderQueuesByEnvs(envs: string[], snapshot: FairQueueSnapshot): Array<EnvQueues> {
const queuesByEnv = snapshot.queues.reduce(
(acc, queue) => {
if (!acc[queue.env]) {
acc[queue.env] = [];
}
acc[queue.env].push(queue);
return acc;
},
{} as Record<string, Array<FairQueue>>
);
return envs.reduce((acc, envId) => {
if (queuesByEnv[envId]) {
// Get ordered queues for this env
const orderedQueues = this.#weightedRandomQueueOrder(queuesByEnv[envId]);
// Apply queue limit if maximumQueuePerEnvCount is set
const limitedQueues = this.options.maximumQueuePerEnvCount
? orderedQueues.slice(0, this.options.maximumQueuePerEnvCount)
: orderedQueues;
// Only add the env if it has queues
if (limitedQueues.length > 0) {
acc.push({
envId,
queues: limitedQueues.map((queue) => queue.id),
});
}
}
return acc;
}, [] as Array<EnvQueues>);
}
#weightedRandomQueueOrder(queues: FairQueue[]): FairQueue[] {
if (queues.length <= 1) return queues;
const biases = this.options.biases ?? defaultBiases;
// When queueAgeRandomization is 0, use strict age-based ordering
if (biases.queueAgeRandomization === 0) {
return [...queues].sort((a, b) => b.age - a.age);
}
// Find the maximum age for normalization
const maxAge = Math.max(...queues.map((q) => q.age));
// Calculate weights for each queue
const weightedQueues: WeightedQueue[] = queues.map((queue) => {
// Normalize age to be between 0 and 1
const normalizedAge = queue.age / maxAge;
// Calculate weight: combine base weight with configurable age influence
const baseWeight = 1;
const weight = baseWeight + normalizedAge * biases.queueAgeRandomization;
return { queue, weight };
});
// Perform weighted random selection for ordering
const result: FairQueue[] = [];
let remainingQueues = [...weightedQueues];
let totalWeight = remainingQueues.reduce((sum, wq) => sum + wq.weight, 0);
while (remainingQueues.length > 0) {
let random = this._rng() * totalWeight;
let index = 0;
// Find queue based on weighted random selection
while (random > 0 && index < remainingQueues.length) {
random -= remainingQueues[index].weight;
index++;
}
index = Math.max(0, index - 1);
// Add selected queue to result and remove from remaining
result.push(remainingQueues[index].queue);
totalWeight -= remainingQueues[index].weight;
remainingQueues.splice(index, 1);
}
return result;
}
#shuffle<T>(array: Array<T>): Array<T> {
let currentIndex = array.length;
let temporaryValue;
let randomIndex;
const newArray = [...array];
while (currentIndex !== 0) {
randomIndex = Math.floor(this._rng() * currentIndex);
currentIndex -= 1;
temporaryValue = newArray[currentIndex];
newArray[currentIndex] = newArray[randomIndex];
newArray[randomIndex] = temporaryValue;
}
return newArray;
}
async #createQueueSnapshot(parentQueue: string, consumerId: string): Promise<FairQueueSnapshot> {
return await startSpan(this.options.tracer, "createQueueSnapshot", async (span) => {
span.setAttribute("consumer_id", consumerId);
span.setAttribute("parent_queue", parentQueue);
if (
typeof this.options.reuseSnapshotCount === "number" &&
this.options.reuseSnapshotCount > 0
) {
const key = `${parentQueue}:${consumerId}`;
const reusedSnapshot = this._reusedSnapshotForConsumer.get(key);
if (reusedSnapshot) {
if (reusedSnapshot.reuseCount < this.options.reuseSnapshotCount) {
span.setAttribute("reused_snapshot", true);
this._reusedSnapshotForConsumer.set(key, {
snapshot: reusedSnapshot.snapshot,
reuseCount: reusedSnapshot.reuseCount + 1,
});
return reusedSnapshot.snapshot;
} else {
this._reusedSnapshotForConsumer.delete(key);
}
}
}
span.setAttribute("reused_snapshot", false);
const now = Date.now();
let queues = await this.#allChildQueuesByScore(parentQueue, consumerId, now);
span.setAttribute("parent_queue_count", queues.length);
if (queues.length === 0) {
return emptyFairQueueSnapshot;
}
// Apply env selection if maximumEnvCount is specified
let selectedEnvIds: Set<string>;
if (this.options.maximumEnvCount && this.options.maximumEnvCount > 0) {
selectedEnvIds = this.#selectTopEnvs(queues, this.options.maximumEnvCount);
// Filter queues to only include selected envs
queues = queues.filter((queue) => selectedEnvIds.has(queue.env));
span.setAttribute("selected_env_count", selectedEnvIds.size);
}
span.setAttribute("selected_queue_count", queues.length);
const envIds = new Set<string>();
for (const queue of queues) {
envIds.add(queue.env);
}
const envs = await Promise.all(
Array.from(envIds).map(async (envId) => {
return { id: envId, concurrency: await this.#getEnvConcurrency(envId) };
})
);
const envsAtFullConcurrency = envs.filter(
(env) => env.concurrency.current >= env.concurrency.limit + env.concurrency.reserve
);
const envIdsAtFullConcurrency = new Set(envsAtFullConcurrency.map((env) => env.id));
const envsSnapshot = envs.reduce(
(acc, env) => {
if (!envIdsAtFullConcurrency.has(env.id)) {
acc[env.id] = env;
}
return acc;
},
{} as Record<string, { concurrency: FairQueueConcurrency }>
);
span.setAttributes({
env_count: envs.length,
envs_at_full_concurrency_count: envsAtFullConcurrency.length,
});
const queuesSnapshot = queues.filter((queue) => !envIdsAtFullConcurrency.has(queue.env));
const snapshot = {
id: randomUUID(),
envs: envsSnapshot,
queues: queuesSnapshot,
};
if (
typeof this.options.reuseSnapshotCount === "number" &&
this.options.reuseSnapshotCount > 0
) {
this._reusedSnapshotForConsumer.set(`${parentQueue}:${consumerId}`, {
snapshot,
reuseCount: 0,
});
}
return snapshot;
});
}
#selectTopEnvs(queues: FairQueue[], maximumEnvCount: number): Set<string> {
// Group queues by env
const queuesByEnv = queues.reduce(
(acc, queue) => {
if (!acc[queue.env]) {
acc[queue.env] = [];
}
acc[queue.env].push(queue);
return acc;
},
{} as Record<string, FairQueue[]>
);
// Calculate average age for each env
const envAverageAges = Object.entries(queuesByEnv).map(([envId, envQueues]) => {
const averageAge = envQueues.reduce((sum, q) => sum + q.age, 0) / envQueues.length;
return { envId, averageAge };
});
// Perform weighted shuffle based on average ages
const maxAge = Math.max(...envAverageAges.map((e) => e.averageAge));
const weightedEnvs = envAverageAges.map((env) => ({
envId: env.envId,
weight: env.averageAge / maxAge, // Normalize weights
}));
// Select top N envs using weighted shuffle
const selectedEnvs = new Set<string>();
let remainingEnvs = [...weightedEnvs];
let totalWeight = remainingEnvs.reduce((sum, env) => sum + env.weight, 0);
while (selectedEnvs.size < maximumEnvCount && remainingEnvs.length > 0) {
let random = this._rng() * totalWeight;
let index = 0;
while (random > 0 && index < remainingEnvs.length) {
random -= remainingEnvs[index].weight;
index++;
}
index = Math.max(0, index - 1);
selectedEnvs.add(remainingEnvs[index].envId);
totalWeight -= remainingEnvs[index].weight;
remainingEnvs.splice(index, 1);
}
return selectedEnvs;
}
async #getEnvConcurrency(envId: string): Promise<FairQueueConcurrency> {
return await startSpan(this.options.tracer, "getEnvConcurrency", async (span) => {
span.setAttribute("env_id", envId);
const [currentValue, limitValue, reserveValue] = await Promise.all([
this.#getEnvCurrentConcurrency(envId),
this.#getEnvConcurrencyLimit(envId),
this.#getEnvReserveConcurrency(envId),
]);
span.setAttribute("current_value", currentValue);
span.setAttribute("limit_value", limitValue);
span.setAttribute("reserve_value", reserveValue);
return { current: currentValue, limit: limitValue, reserve: reserveValue };
});
}
async #allChildQueuesByScore(
parentQueue: string,
consumerId: string,
now: number
): Promise<Array<FairQueue>> {
return await startSpan(this.options.tracer, "allChildQueuesByScore", async (span) => {
span.setAttribute("consumer_id", consumerId);
span.setAttribute("parent_queue", parentQueue);
const valuesWithScores = await this.options.redis.zrangebyscore(
parentQueue,
"-inf",
now,
"WITHSCORES",
"LIMIT",
0,
this.options.parentQueueLimit
);
const result: Array<FairQueue> = [];
for (let i = 0; i < valuesWithScores.length; i += 2) {
result.push({
id: valuesWithScores[i],
age: now - Number(valuesWithScores[i + 1]),
env: this.options.keys.envIdFromQueue(valuesWithScores[i]),
org: this.options.keys.orgIdFromQueue(valuesWithScores[i]),
});
}
span.setAttribute("queue_count", result.length);
if (result.length === this.options.parentQueueLimit) {
span.setAttribute("parent_queue_limit_reached", true);
}
return result;
});
}
async #getEnvConcurrencyLimit(envId: string) {
return await startSpan(this.options.tracer, "getEnvConcurrencyLimit", async (span) => {
span.setAttribute("env_id", envId);
const key = this.options.keys.envConcurrencyLimitKey(envId);
const result = await this._cache.concurrencyLimit.swr(key, async () => {
const value = await this.options.redis.get(key);
if (!value) {
return this.options.defaultEnvConcurrency;
}
return Number(value);
});
return result.val ?? this.options.defaultEnvConcurrency;
});
}
async #getEnvCurrentConcurrency(envId: string) {
return await startSpan(this.options.tracer, "getEnvCurrentConcurrency", async (span) => {
span.setAttribute("env_id", envId);
const key = this.options.keys.envCurrentConcurrencyKey(envId);
const result = await this.options.redis.scard(key);
span.setAttribute("current_value", result);
return result;
});
}
async #getEnvReserveConcurrency(envId: string) {
return await startSpan(this.options.tracer, "getEnvReserveConcurrency", async (span) => {
span.setAttribute("env_id", envId);
const key = this.options.keys.envReserveConcurrencyKey(envId);
const result = await this.options.redis.scard(key);
span.setAttribute("current_value", result);
return result;
});
}
}
export class NoopFairDequeuingStrategy implements MarQSFairDequeueStrategy {
async distributeFairQueuesFromParentQueue(
parentQueue: string,
consumerId: string
): Promise<Array<EnvQueues>> {
return [];
}
}
apps/webapp/app/v3/marqs/index.server.ts
-2691
View File
File diff suppressed because it is too large Load Diff
apps/webapp/app/v3/marqs/marqsKeyProducer.ts
-287
View File
@@ -1,287 +0,0 @@
import type { MarQSKeyProducer, MarQSKeyProducerEnv, QueueDescriptor } from "./types";
const constants = {
SHARED_QUEUE: "sharedQueue",
SHARED_WORKER_QUEUE: "sharedWorkerQueue",
CURRENT_CONCURRENCY_PART: "currentConcurrency",
CONCURRENCY_LIMIT_PART: "concurrency",
DISABLED_CONCURRENCY_LIMIT_PART: "disabledConcurrency",
ENV_PART: "env",
ORG_PART: "org",
QUEUE_PART: "queue",
CONCURRENCY_KEY_PART: "ck",
MESSAGE_PART: "message",
RESERVE_CONCURRENCY_PART: "reserveConcurrency",
} as const;
const ORG_REGEX = /org:([^:]+):/;
const ENV_REGEX = /env:([^:]+):/;
const QUEUE_REGEX = /queue:([^:]+)(?::|$)/;
const CONCURRENCY_KEY_REGEX = /ck:([^:]+)(?::|$)/;
export class MarQSShortKeyProducer implements MarQSKeyProducer {
constructor(private _prefix: string) {}
sharedQueueScanPattern() {
return `${this._prefix}*${constants.SHARED_QUEUE}`;
}
queueCurrentConcurrencyScanPattern() {
return `${this._prefix}${constants.ORG_PART}:*:${constants.ENV_PART}:*:queue:*:${constants.CURRENT_CONCURRENCY_PART}`;
}
stripKeyPrefix(key: string): string {
if (key.startsWith(this._prefix)) {
return key.slice(this._prefix.length);
}
return key;
}
queueConcurrencyLimitKey(env: MarQSKeyProducerEnv, queue: string) {
return [this.queueKey(env, queue), constants.CONCURRENCY_LIMIT_PART].join(":");
}
envConcurrencyLimitKey(envId: string): string;
envConcurrencyLimitKey(env: MarQSKeyProducerEnv): string;
envConcurrencyLimitKey(envOrId: MarQSKeyProducerEnv | string): string {
return [
this.envKeySection(typeof envOrId === "string" ? envOrId : envOrId.id),
constants.CONCURRENCY_LIMIT_PART,
].join(":");
}
queueKey(orgId: string, envId: string, queue: string, concurrencyKey?: string): string;
queueKey(env: MarQSKeyProducerEnv, queue: string, concurrencyKey?: string): string;
queueKey(
envOrOrgId: MarQSKeyProducerEnv | string,
queueOrEnvId: string,
queueOrConcurrencyKey: string,
concurrencyKeyOrPriority?: string | number
): string {
if (typeof envOrOrgId === "string") {
return [
this.orgKeySection(envOrOrgId),
this.envKeySection(queueOrEnvId),
this.queueSection(queueOrConcurrencyKey),
]
.concat(
typeof concurrencyKeyOrPriority === "string"
? this.concurrencyKeySection(concurrencyKeyOrPriority)
: []
)
.join(":");
} else {
return [
this.orgKeySection(envOrOrgId.organizationId),
this.envKeySection(envOrOrgId.id),
this.queueSection(queueOrEnvId),
]
.concat(queueOrConcurrencyKey ? this.concurrencyKeySection(queueOrConcurrencyKey) : [])
.join(":");
}
}
queueKeyFromQueue(queue: string): string {
const descriptor = this.queueDescriptorFromQueue(queue);
return this.queueKey(
descriptor.organization,
descriptor.environment,
descriptor.name,
descriptor.concurrencyKey
);
}
envSharedQueueKey(env: MarQSKeyProducerEnv) {
if (env.type === "DEVELOPMENT") {
return [
this.orgKeySection(env.organizationId),
this.envKeySection(env.id),
constants.SHARED_QUEUE,
].join(":");
}
return this.sharedQueueKey();
}
sharedQueueKey(): string {
return constants.SHARED_QUEUE;
}
sharedWorkerQueueKey(): string {
return constants.SHARED_WORKER_QUEUE;
}
queueConcurrencyLimitKeyFromQueue(queue: string) {
const descriptor = this.queueDescriptorFromQueue(queue);
return this.queueConcurrencyLimitKeyFromDescriptor(descriptor);
}
queueCurrentConcurrencyKeyFromQueue(queue: string) {
const descriptor = this.queueDescriptorFromQueue(queue);
return this.currentConcurrencyKeyFromDescriptor(descriptor);
}
queueReserveConcurrencyKeyFromQueue(queue: string) {
const descriptor = this.queueDescriptorFromQueue(queue);
return this.queueReserveConcurrencyKeyFromDescriptor(descriptor);
}
queueCurrentConcurrencyKey(
env: MarQSKeyProducerEnv,
queue: string,
concurrencyKey?: string
): string {
return [this.queueKey(env, queue, concurrencyKey), constants.CURRENT_CONCURRENCY_PART].join(
":"
);
}
envConcurrencyLimitKeyFromQueue(queue: string) {
const descriptor = this.queueDescriptorFromQueue(queue);
return `${constants.ENV_PART}:${descriptor.environment}:${constants.CONCURRENCY_LIMIT_PART}`;
}
envCurrentConcurrencyKeyFromQueue(queue: string) {
const descriptor = this.queueDescriptorFromQueue(queue);
return `${constants.ENV_PART}:${descriptor.environment}:${constants.CURRENT_CONCURRENCY_PART}`;
}
envReserveConcurrencyKeyFromQueue(queue: string) {
const descriptor = this.queueDescriptorFromQueue(queue);
return this.envReserveConcurrencyKey(descriptor.environment);
}
envReserveConcurrencyKey(envId: string): string {
return `${constants.ENV_PART}:${this.shortId(envId)}:${constants.RESERVE_CONCURRENCY_PART}`;
}
envCurrentConcurrencyKey(envId: string): string;
envCurrentConcurrencyKey(env: MarQSKeyProducerEnv): string;
envCurrentConcurrencyKey(envOrId: MarQSKeyProducerEnv | string): string {
return [
this.envKeySection(typeof envOrId === "string" ? envOrId : envOrId.id),
constants.CURRENT_CONCURRENCY_PART,
].join(":");
}
envQueueKeyFromQueue(queue: string) {
const descriptor = this.queueDescriptorFromQueue(queue);
return `${constants.ENV_PART}:${descriptor.environment}:${constants.QUEUE_PART}`;
}
envQueueKey(env: MarQSKeyProducerEnv): string {
return [constants.ENV_PART, this.shortId(env.id), constants.QUEUE_PART].join(":");
}
messageKey(messageId: string) {
return `${constants.MESSAGE_PART}:${messageId}`;
}
nackCounterKey(messageId: string): string {
return `${constants.MESSAGE_PART}:${messageId}:nacks`;
}
orgIdFromQueue(queue: string) {
const descriptor = this.queueDescriptorFromQueue(queue);
return descriptor.organization;
}
envIdFromQueue(queue: string) {
const descriptor = this.queueDescriptorFromQueue(queue);
return descriptor.environment;
}
queueDescriptorFromQueue(queue: string): QueueDescriptor {
const match = queue.match(QUEUE_REGEX);
if (!match) {
throw new Error(`Invalid queue: ${queue}, no queue name found`);
}
const [, queueName] = match;
const envMatch = queue.match(ENV_REGEX);
if (!envMatch) {
throw new Error(`Invalid queue: ${queue}, no environment found`);
}
const [, envId] = envMatch;
const orgMatch = queue.match(ORG_REGEX);
if (!orgMatch) {
throw new Error(`Invalid queue: ${queue}, no organization found`);
}
const [, orgId] = orgMatch;
const concurrencyKeyMatch = queue.match(CONCURRENCY_KEY_REGEX);
const concurrencyKey = concurrencyKeyMatch ? concurrencyKeyMatch[1] : undefined;
return {
name: queueName,
environment: envId,
organization: orgId,
concurrencyKey,
};
}
private shortId(id: string) {
// Return the last 12 characters of the id
return id.slice(-12);
}
private envKeySection(envId: string) {
return `${constants.ENV_PART}:${this.shortId(envId)}`;
}
private orgKeySection(orgId: string) {
return `${constants.ORG_PART}:${this.shortId(orgId)}`;
}
private queueSection(queue: string) {
return `${constants.QUEUE_PART}:${queue}`;
}
private concurrencyKeySection(concurrencyKey: string) {
return `${constants.CONCURRENCY_KEY_PART}:${concurrencyKey}`;
}
private currentConcurrencyKeyFromDescriptor(descriptor: QueueDescriptor) {
return [
this.queueKey(
descriptor.organization,
descriptor.environment,
descriptor.name,
descriptor.concurrencyKey
),
constants.CURRENT_CONCURRENCY_PART,
].join(":");
}
private queueReserveConcurrencyKeyFromDescriptor(descriptor: QueueDescriptor) {
return [
this.queueKey(descriptor.organization, descriptor.environment, descriptor.name),
constants.RESERVE_CONCURRENCY_PART,
].join(":");
}
private queueConcurrencyLimitKeyFromDescriptor(descriptor: QueueDescriptor) {
return [
this.queueKey(descriptor.organization, descriptor.environment, descriptor.name),
constants.CONCURRENCY_LIMIT_PART,
].join(":");
}
}
apps/webapp/app/v3/marqs/sharedQueueConsumer.server.ts
-2215
View File
File diff suppressed because it is too large Load Diff
apps/webapp/app/v3/marqs/types.ts
-111
View File
@@ -1,111 +0,0 @@
import type { RuntimeEnvironmentType } from "@trigger.dev/database";
import { z } from "zod";
export type QueueRange = { offset: number; count: number };
export type QueueDescriptor = {
organization: string;
environment: string;
name: string;
concurrencyKey?: string;
};
export type MarQSKeyProducerEnv = {
id: string;
organizationId: string;
type: RuntimeEnvironmentType;
};
export interface MarQSKeyProducer {
queueConcurrencyLimitKey(env: MarQSKeyProducerEnv, queue: string): string;
envConcurrencyLimitKey(envId: string): string;
envConcurrencyLimitKey(env: MarQSKeyProducerEnv): string;
envCurrentConcurrencyKey(envId: string): string;
envCurrentConcurrencyKey(env: MarQSKeyProducerEnv): string;
envReserveConcurrencyKey(envId: string): string;
queueKey(orgId: string, envId: string, queue: string, concurrencyKey?: string): string;
queueKey(env: MarQSKeyProducerEnv, queue: string, concurrencyKey?: string): string;
queueKeyFromQueue(queue: string): string;
envQueueKey(env: MarQSKeyProducerEnv): string;
envSharedQueueKey(env: MarQSKeyProducerEnv): string;
sharedQueueKey(): string;
sharedQueueScanPattern(): string;
sharedWorkerQueueKey(): string;
queueCurrentConcurrencyScanPattern(): string;
queueConcurrencyLimitKeyFromQueue(queue: string): string;
queueCurrentConcurrencyKeyFromQueue(queue: string): string;
queueCurrentConcurrencyKey(
env: MarQSKeyProducerEnv,
queue: string,
concurrencyKey?: string
): string;
envConcurrencyLimitKeyFromQueue(queue: string): string;
envCurrentConcurrencyKeyFromQueue(queue: string): string;
envReserveConcurrencyKeyFromQueue(queue: string): string;
envQueueKeyFromQueue(queue: string): string;
messageKey(messageId: string): string;
nackCounterKey(messageId: string): string;
stripKeyPrefix(key: string): string;
orgIdFromQueue(queue: string): string;
envIdFromQueue(queue: string): string;
queueReserveConcurrencyKeyFromQueue(queue: string): string;
queueDescriptorFromQueue(queue: string): QueueDescriptor;
}
export type EnvQueues = {
envId: string;
queues: string[];
};
const MarQSPriorityLevel = z.enum(["resume", "retry"]);
export type MarQSPriorityLevel = z.infer<typeof MarQSPriorityLevel>;
export interface MarQSFairDequeueStrategy {
distributeFairQueuesFromParentQueue(
parentQueue: string,
consumerId: string
): Promise<Array<EnvQueues>>;
}
export const MessagePayload = z.object({
version: z.literal("1"),
data: z.record(z.unknown()),
queue: z.string(),
messageId: z.string(),
timestamp: z.number(),
parentQueue: z.string(),
concurrencyKey: z.string().optional(),
priority: MarQSPriorityLevel.optional(),
availableAt: z.number().optional(),
enqueueMethod: z.enum(["enqueue", "requeue", "replace"]).default("enqueue"),
});
export type MessagePayload = z.infer<typeof MessagePayload>;
export interface MessageQueueSubscriber {
messageEnqueued(message: MessagePayload): Promise<void>;
messageDequeued(message: MessagePayload): Promise<void>;
messageAcked(message: MessagePayload): Promise<void>;
messageNacked(message: MessagePayload): Promise<void>;
messageReplaced(message: MessagePayload): Promise<void>;
messageRequeued(message: MessagePayload): Promise<void>;
}
export interface VisibilityTimeoutStrategy {
startHeartbeat(messageId: string, timeoutInMs: number): Promise<void>;
heartbeat(messageId: string, timeoutInMs: number): Promise<void>;
cancelHeartbeat(messageId: string): Promise<void>;
}
export type EnqueueMessageReserveConcurrencyOptions = {
messageId: string;
recursiveQueue: boolean;
};
apps/webapp/app/v3/marqs/v3VisibilityTimeout.server.ts
-39
View File
@@ -1,39 +0,0 @@
import { legacyRunEngineWorker } from "../legacyRunEngineWorker.server";
import { TaskRunHeartbeatFailedService } from "../taskRunHeartbeatFailed.server";
import type { VisibilityTimeoutStrategy } from "./types";
export class V3GraphileVisibilityTimeout implements VisibilityTimeoutStrategy {
async startHeartbeat(messageId: string, timeoutInMs: number): Promise<void> {
await TaskRunHeartbeatFailedService.enqueue(messageId, new Date(Date.now() + timeoutInMs));
}
async heartbeat(messageId: string, timeoutInMs: number): Promise<void> {
await TaskRunHeartbeatFailedService.enqueue(messageId, new Date(Date.now() + timeoutInMs));
}
async cancelHeartbeat(messageId: string): Promise<void> {
await TaskRunHeartbeatFailedService.dequeue(messageId);
}
}
export class V3LegacyRunEngineWorkerVisibilityTimeout implements VisibilityTimeoutStrategy {
async startHeartbeat(messageId: string, timeoutInMs: number): Promise<void> {
await legacyRunEngineWorker.enqueue({
id: `heartbeat:${messageId}`,
job: "runHeartbeat",
payload: { runId: messageId },
availableAt: new Date(Date.now() + timeoutInMs),
});
}
async heartbeat(messageId: string, timeoutInMs: number): Promise<void> {
await legacyRunEngineWorker.reschedule(
`heartbeat:${messageId}`,
new Date(Date.now() + timeoutInMs)
);
}
async cancelHeartbeat(messageId: string): Promise<void> {
await legacyRunEngineWorker.ack(`heartbeat:${messageId}`);
}
}
apps/webapp/app/v3/mollifierDrainerWorker.server.ts
+3 -6
View File
@@ -26,12 +26,9 @@ declare global {
* factory) guarantees a signal landing during boot can never find
* the polling loop running without a graceful-stop path.
*
* The drainer is intentionally NOT wired through `~/services/worker.server`
* — that file is the legacy ZodWorker / graphile-worker setup. The
* mollifier drainer is a custom polling loop over `MollifierBuffer`, not
* a graphile-worker job, so it gets its own lifecycle file alongside the
* redis-worker workers (`commonWorker`, `alertsWorker`,
* `batchTriggerWorker`).
* The mollifier drainer is a custom polling loop over `MollifierBuffer`, not a
* redis-worker job, so it gets its own lifecycle file alongside the redis-worker
* workers (`commonWorker`, `alertsWorker`, `batchTriggerWorker`).
*
* Gating order:
* - `TRIGGER_MOLLIFIER_DRAINER_ENABLED !== "1"` → early return. Unset defaults
apps/webapp/app/v3/queueSizeLimits.server.ts
+2 -7
View File
@@ -1,6 +1,6 @@
import type { AuthenticatedEnvironment } from "~/services/apiAuth.server";
import { env } from "~/env.server";
import type { MarQS } from "./marqs/index.server";
import { engine } from "./runEngine.server";
export type QueueSizeGuardResult = {
isWithinLimits: boolean;
@@ -10,7 +10,6 @@ export type QueueSizeGuardResult = {
export async function guardQueueSizeLimitsForEnv(
environment: AuthenticatedEnvironment,
marqs?: MarQS,
itemsToAdd: number = 1
): Promise<QueueSizeGuardResult> {
const maximumSize = getMaximumSizeForEnvironment(environment);
@@ -19,11 +18,7 @@ export async function guardQueueSizeLimitsForEnv(
return { isWithinLimits: true };
}
if (!marqs) {
return { isWithinLimits: true, maximumSize };
}
const queueSize = await marqs.lengthOfEnvQueue(environment);
const queueSize = await engine.lengthOfEnvQueue(environment);
const projectedSize = queueSize + itemsToAdd;
return {
apps/webapp/app/v3/runQueue.server.ts
+6 -18
View File
@@ -1,10 +1,7 @@
import { type AuthenticatedEnvironment } from "~/services/apiAuth.server";
import { marqs } from "./marqs/index.server";
import { engine } from "./runEngine.server";
//This allows us to update MARQS and the RunQueue
/** Updates MARQS and the RunQueue limits */
/** Updates the RunQueue env concurrency limits */
export async function updateEnvConcurrencyLimits(
environment: AuthenticatedEnvironment,
maximumConcurrencyLimit?: number
@@ -14,31 +11,22 @@ export async function updateEnvConcurrencyLimits(
updatedEnvironment.maximumConcurrencyLimit = maximumConcurrencyLimit;
}
await Promise.allSettled([
marqs?.updateEnvConcurrencyLimits(updatedEnvironment),
engine.runQueue.updateEnvConcurrencyLimits(updatedEnvironment),
]);
await engine.runQueue.updateEnvConcurrencyLimits(updatedEnvironment);
}
/** Updates MARQS and the RunQueue limits for a queue */
/** Updates the RunQueue limits for a queue */
export async function updateQueueConcurrencyLimits(
environment: AuthenticatedEnvironment,
queueName: string,
concurrency: number
) {
await Promise.allSettled([
marqs?.updateQueueConcurrencyLimits(environment, queueName, concurrency),
engine.runQueue.updateQueueConcurrencyLimits(environment, queueName, concurrency),
]);
await engine.runQueue.updateQueueConcurrencyLimits(environment, queueName, concurrency);
}
/** Removes MARQS and the RunQueue limits for a queue */
/** Removes the RunQueue limits for a queue */
export async function removeQueueConcurrencyLimits(
environment: AuthenticatedEnvironment,
queueName: string
) {
await Promise.allSettled([
marqs?.removeQueueConcurrencyLimits(environment, queueName),
engine.runQueue.removeQueueConcurrencyLimits(environment, queueName),
]);
await engine.runQueue.removeQueueConcurrencyLimits(environment, queueName);
}
apps/webapp/app/v3/scheduleEngine.server.ts
+2 -24
View File
@@ -8,9 +8,7 @@ import { logger } from "~/services/logger.server";
import { singleton } from "~/utils/singleton";
import { OutOfEntitlementError, TriggerTaskService } from "./services/triggerTask.server";
import { meter, tracer } from "./tracer.server";
import { workerQueue } from "~/services/worker.server";
import { ServiceValidationError } from "./services/common.server";
import { isV3Disabled } from "./engineDeprecation.server";
export const scheduleEngine = singleton("ScheduleEngine", createScheduleEngine);
@@ -85,11 +83,8 @@ function createScheduleEngine() {
exactScheduleTime,
}) => {
try {
// v3 (engine V1) shutdown: skip firing schedules for V1 projects so the
// cron doesn't keep doing trigger work just to be rejected. Return success
// so the schedule engine treats it as handled and doesn't retry. v4 is
// unaffected.
if (isV3Disabled() && environment.project.engine === "V1") {
// v3 (engine V1) is retired: skip firing V1 schedules instead of triggering into a guaranteed rejection every tick.
if (environment.project.engine === "V1") {
logger.debug("[ScheduleEngine] Skipping scheduled fire for shut-down v3 project", {
taskIdentifier,
scheduleId,
@@ -151,24 +146,7 @@ function createScheduleEngine() {
}
},
isDevEnvironmentConnectedHandler: isDevEnvironmentConnectedHandler,
onRegisterScheduleInstance: removeDeprecatedWorkerQueueItem,
});
return engine;
}
async function removeDeprecatedWorkerQueueItem(instanceId: string) {
// We need to dequeue the instance from the existing workerQueue
try {
await workerQueue.dequeue(`scheduled-task-instance:${instanceId}`);
logger.debug("Removed deprecated worker queue item", {
instanceId,
});
} catch (error) {
logger.error("Error dequeuing scheduled task instance from deprecated queue", {
instanceId,
error: error instanceof Error ? error.message : String(error),
});
}
}
apps/webapp/app/v3/services/batchTriggerV3.server.ts
+5 -94
View File
@@ -5,12 +5,7 @@ import type {
} from "@trigger.dev/core/v3";
import { packetRequiresOffloading, parsePacket } from "@trigger.dev/core/v3";
import type { BatchTaskRun, TaskRunAttempt } from "@trigger.dev/database";
import {
isPrismaRaceConditionError,
isPrismaRetriableError,
isUniqueConstraintError,
Prisma,
} from "@trigger.dev/database";
import { isUniqueConstraintError, Prisma } from "@trigger.dev/database";
import type { RunStore } from "@internal/run-store";
import { z } from "zod";
import type { PrismaClientOrTransaction } from "~/db.server";
@@ -29,14 +24,11 @@ import { resolveInheritedMintKind } from "~/v3/runOpsMigration/resolveInheritedM
import { mintFriendlyIdForKind } from "~/v3/runOpsMigration/mintAnchoredRunFriendlyId.server";
import { mintBatchFriendlyId } from "~/v3/runOpsMigration/mintBatchFriendlyId.server";
import { batchTriggerWorker } from "../batchTriggerWorker.server";
import { legacyRunEngineWorker } from "../legacyRunEngineWorker.server";
import { marqs } from "../marqs/index.server";
import { guardQueueSizeLimitsForEnv } from "../queueSizeLimits.server";
import { downloadPacketFromObjectStore, uploadPacketToObjectStore } from "../objectStore.server";
import { isFinalAttemptStatus, isFinalRunStatus } from "../taskStatus";
import { startActiveSpan } from "../tracer.server";
import { BaseService, ServiceValidationError } from "./baseService.server";
import { ResumeBatchRunService } from "./resumeBatchRun.server";
import { OutOfEntitlementError, TriggerTaskService } from "./triggerTask.server";
const PROCESSING_BATCH_SIZE = 50;
@@ -88,9 +80,8 @@ type RunItemData = {
* we increment the BatchTaskRun's completed count. Once the completed count is equal to the expected count, and the
* batch is sealed, we can consider the batch completed.
*
* So now when the v3 batch is considered completed, we will enqueue the ResumeBatchRunService to resume the dependent
* task attempt if there is one. This is in contrast to v2 batches where every time a task was completed, we would schedule
* the ResumeBatchRunService to check if the batch was completed and set it to completed if it was.
* When the v3 batch is considered completed it is marked COMPLETED. (Dependent-attempt
* batches from batchTriggerAndWait only existed on the retired V1 engine.)
*
* We've also introduced a new column "resumedAt" that will be set when the batch is resumed. Previously in v2 batches, the status == "COMPLETED" was overloaded
* to mean that the batch was completed and resumed. Now we have a separate column to track when the batch was resumed (and to make sure it's only resumed once).
@@ -260,7 +251,7 @@ export class BatchTriggerV3Service extends BaseService {
};
}
const queueSizeGuard = await guardQueueSizeLimitsForEnv(environment, marqs, newRunCount);
const queueSizeGuard = await guardQueueSizeLimitsForEnv(environment, newRunCount);
logger.debug("Queue size guard result", {
newRunCount,
@@ -1050,85 +1041,5 @@ export async function tryCompleteBatchV3(
logger.debug("tryCompleteBatchV3: Batch completed", { batchId, completedCount });
if (scheduleResumeOnComplete && batch.dependentTaskAttemptId) {
await ResumeBatchRunService.enqueue(batchId, true, tx);
}
}
export async function completeBatchTaskRunItemV3(
itemId: string,
batchTaskRunId: string,
tx: PrismaClientOrTransaction,
scheduleResumeOnComplete = false,
taskRunAttemptId?: string,
retryAttempt?: number,
// Threaded in so a run-ops id (NEW-resident) batch's item lands on the owning store; route by
// batchTaskRunId (items co-reside with their batch). Defaults to the singleton.
runStore: RunStore = defaultRunStore
) {
const isRetry = retryAttempt !== undefined;
logger.debug("completeBatchTaskRunItemV3", {
itemId,
batchTaskRunId,
scheduleResumeOnComplete,
taskRunAttemptId,
retryAttempt,
isRetry,
});
try {
// Update item to COMPLETED (no transaction needed, no contention). Routed by
// batchTaskRunId so the item write lands on the batch's owning DB.
const updated = await runStore.updateManyBatchTaskRunItems({
where: { id: itemId, batchTaskRunId, status: "PENDING" },
data: { status: "COMPLETED", taskRunAttemptId },
});
if (updated.count === 0) {
logger.debug("completeBatchTaskRunItemV3: Item already completed", {
itemId,
batchTaskRunId,
});
return;
}
// Schedule debounced completion check
// enqueue with same ID overwrites, resetting the 200ms timer (debounce behavior)
await legacyRunEngineWorker.enqueue({
id: `tryCompleteBatchV3:${batchTaskRunId}`,
job: "tryCompleteBatchV3",
payload: { batchId: batchTaskRunId, scheduleResumeOnComplete },
availableAt: new Date(Date.now() + 200),
});
} catch (error) {
if (isPrismaRetriableError(error) || isPrismaRaceConditionError(error)) {
logger.error("completeBatchTaskRunItemV3 failed, scheduling retry", {
itemId,
batchTaskRunId,
error,
retryAttempt,
isRetry,
});
if (isRetry) {
throw error;
} else {
await legacyRunEngineWorker.enqueue({
id: `completeBatchTaskRunItem:${itemId}`,
job: "completeBatchTaskRunItem",
payload: { itemId, batchTaskRunId, scheduleResumeOnComplete, taskRunAttemptId },
availableAt: new Date(Date.now() + 2_000),
});
}
} else {
logger.error("completeBatchTaskRunItemV3 failed with non-retriable error", {
itemId,
batchTaskRunId,
error,
retryAttempt,
isRetry,
});
}
}
// Dependent-attempt batches (batchTriggerAndWait) only exist on the retired V1 engine, so there is no parent to resume here.
}
apps/webapp/app/v3/services/bulk/createBulkAction.server.ts
-60
View File
@@ -1,60 +0,0 @@
import { type BulkActionType } from "@trigger.dev/database";
import { bulkActionVerb } from "~/components/runs/v3/BulkAction";
import { BULK_ACTION_RUN_LIMIT } from "~/consts";
import { logger } from "~/services/logger.server";
import { generateFriendlyId } from "../../friendlyIdentifiers";
import { BaseService } from "../baseService.server";
import { PerformBulkActionService } from "./performBulkAction.server";
type BulkAction = {
projectId: string;
action: BulkActionType;
runIds: string[];
};
export class CreateBulkActionService extends BaseService {
public async call({ projectId, action, runIds }: BulkAction) {
const group = await this._prisma.bulkActionGroup.create({
data: {
friendlyId: generateFriendlyId("bulk"),
projectId,
type: action,
},
});
//limit to the first X runs
const passedTooManyRuns = runIds.length > BULK_ACTION_RUN_LIMIT;
runIds = runIds.slice(0, BULK_ACTION_RUN_LIMIT);
const _items = await this._prisma.bulkActionItem.createMany({
data: runIds.map((runId) => ({
friendlyId: generateFriendlyId("bulkitem"),
type: action,
groupId: group.id,
sourceRunId: runId,
})),
});
logger.debug("Created bulk action group", {
groupId: group.id,
action,
runIds,
});
await PerformBulkActionService.enqueue(group.id, this._prisma);
let message = bulkActionVerb(action);
if (passedTooManyRuns) {
message += ` the first ${BULK_ACTION_RUN_LIMIT} runs`;
} else {
message += ` ${runIds.length} runs`;
}
return {
id: group.id,
friendlyId: group.friendlyId,
runCount: runIds.length,
message,
};
}
}
apps/webapp/app/v3/services/bulk/performBulkAction.server.ts
-127
View File
@@ -1,127 +0,0 @@
import assertNever from "assert-never";
import type { PrismaClientOrTransaction } from "~/db.server";
import { workerQueue } from "~/services/worker.server";
import { BaseService } from "../baseService.server";
import { CancelTaskRunService } from "../cancelTaskRun.server";
import { ReplayTaskRunService } from "../replayTaskRun.server";
export class PerformBulkActionService extends BaseService {
public async performBulkActionItem(bulkActionItemId: string) {
const item = await this._prisma.bulkActionItem.findFirst({
where: { id: bulkActionItemId },
include: {
sourceRun: true,
destinationRun: true,
},
});
if (!item) {
return;
}
if (item.status !== "PENDING") {
return;
}
switch (item.type) {
case "REPLAY": {
const service = new ReplayTaskRunService(this._prisma);
const result = await service.call(item.sourceRun, { triggerSource: "dashboard" });
await this._prisma.bulkActionItem.update({
where: { id: item.id },
data: {
destinationRunId: result?.id,
status: result ? "COMPLETED" : "FAILED",
error: result ? undefined : "Failed to replay task run",
},
});
break;
}
case "CANCEL": {
const service = new CancelTaskRunService(this._prisma);
const result = await service.call(item.sourceRun);
await this._prisma.bulkActionItem.update({
where: { id: item.id },
data: {
destinationRunId: item.sourceRun.id,
status: result ? "COMPLETED" : "FAILED",
error: result ? undefined : "Task wasn't cancelable",
},
});
break;
}
default: {
assertNever(item.type);
}
}
const groupItems = await this._prisma.bulkActionItem.findMany({
where: { groupId: item.groupId },
select: {
status: true,
},
});
const isGroupCompleted = groupItems.every((item) => item.status !== "PENDING");
if (isGroupCompleted) {
await this._prisma.bulkActionItem.update({
where: { id: item.id },
data: {
status: "COMPLETED",
},
});
}
}
public async enqueueBulkActionItem(bulkActionItemId: string, groupId: string) {
await workerQueue.enqueue(
"v3.performBulkActionItem",
{
bulkActionItemId,
},
{
jobKey: `performBulkActionItem:${bulkActionItemId}`,
}
);
}
public async call(bulkActionGroupId: string) {
const actionGroup = await this._prisma.bulkActionGroup.findFirst({
where: { id: bulkActionGroupId },
select: { id: true },
});
if (!actionGroup) {
return;
}
const items = await this._prisma.bulkActionItem.findMany({
where: { groupId: bulkActionGroupId },
select: { id: true },
});
for (const item of items) {
await this.enqueueBulkActionItem(item.id, bulkActionGroupId);
}
}
static async enqueue(bulkActionGroupId: string, tx: PrismaClientOrTransaction, runAt?: Date) {
return await workerQueue.enqueue(
"v3.performBulkAction",
{
bulkActionGroupId,
},
{
tx,
runAt,
jobKey: `performBulkAction:${bulkActionGroupId}`,
}
);
}
}
apps/webapp/app/v3/services/cancelAttempt.server.ts
-100
View File
@@ -1,100 +0,0 @@
import { $transaction, type PrismaClientOrTransaction, prisma } from "~/db.server";
import { type AuthenticatedEnvironment } from "~/services/apiAuth.server";
import { logger } from "~/services/logger.server";
import { isCancellableRunStatus } from "../taskStatus";
import { BaseService } from "./baseService.server";
import { FinalizeTaskRunService } from "./finalizeTaskRun.server";
export class CancelAttemptService extends BaseService {
public async call(
attemptId: string,
taskRunId: string,
cancelledAt: Date,
reason: string,
env?: AuthenticatedEnvironment
) {
let environment: AuthenticatedEnvironment | undefined = env;
if (!environment) {
environment = await getAuthenticatedEnvironmentFromAttempt(attemptId);
if (!environment) {
return;
}
}
return await this.traceWithEnv("call()", environment, async (span) => {
span.setAttribute("taskRunId", taskRunId);
span.setAttribute("attemptId", attemptId);
const taskRunAttempt = await this._prisma.taskRunAttempt.findFirst({
where: {
friendlyId: attemptId,
},
include: {
taskRun: true,
},
});
if (!taskRunAttempt) {
return;
}
if (taskRunAttempt.status === "CANCELED") {
logger.warn("Task run attempt is already cancelled", {
attemptId,
});
return;
}
await $transaction(this._prisma, "cancel attempt", async (tx) => {
await tx.taskRunAttempt.update({
where: {
friendlyId: attemptId,
},
data: {
status: "CANCELED",
completedAt: cancelledAt,
},
});
const isCancellable = isCancellableRunStatus(taskRunAttempt.taskRun.status);
const finalizeService = new FinalizeTaskRunService(tx);
await finalizeService.call({
id: taskRunId,
status: isCancellable ? "INTERRUPTED" : undefined,
completedAt: isCancellable ? cancelledAt : undefined,
attemptStatus: isCancellable ? "CANCELED" : undefined,
error: isCancellable ? { type: "STRING_ERROR", raw: reason } : undefined,
});
});
});
}
}
async function getAuthenticatedEnvironmentFromAttempt(
friendlyId: string,
prismaClient?: PrismaClientOrTransaction
) {
const taskRunAttempt = await (prismaClient ?? prisma).taskRunAttempt.findFirst({
where: {
friendlyId,
},
include: {
runtimeEnvironment: {
include: {
organization: true,
project: true,
},
},
},
});
if (!taskRunAttempt) {
return;
}
return taskRunAttempt?.runtimeEnvironment;
}
apps/webapp/app/v3/services/cancelDevSessionRuns.server.ts
-158
View File
@@ -1,158 +0,0 @@
import { type RunStore } from "@internal/run-store";
import { z } from "zod";
import { type PrismaClientOrTransaction } from "~/db.server";
import { findLatestSession } from "~/models/runtimeEnvironment.server";
import { logger } from "~/services/logger.server";
import { commonWorker } from "../commonWorker.server";
import { type ReadThroughDeps, readThroughRun } from "../runOpsMigration/readThrough.server";
import { BaseService } from "./baseService.server";
import { type CancelableTaskRun, CancelTaskRunService } from "./cancelTaskRun.server";
export const CancelDevSessionRunsServiceOptions = z.object({
runIds: z.array(z.string()),
cancelledAt: z.coerce.date(),
reason: z.string(),
cancelledSessionId: z.string().optional(),
});
export type CancelDevSessionRunsServiceOptions = z.infer<typeof CancelDevSessionRunsServiceOptions>;
export class CancelDevSessionRunsService extends BaseService {
// Injectable read-through deps for the run-ops TaskRun read. Undefined in production:
// readThroughRun then uses its ~/db.server singleton handles and the boot split flag,
// so single-DB is unchanged. Tests inject the hetero new/legacy handles + splitEnabled.
readonly #readThroughDeps?: ReadThroughDeps;
constructor(
opts: {
prisma?: PrismaClientOrTransaction;
replica?: PrismaClientOrTransaction;
runStore?: RunStore;
readThroughDeps?: ReadThroughDeps;
} = {}
) {
super(opts.prisma, opts.replica, opts.runStore);
this.#readThroughDeps = opts.readThroughDeps;
}
public async call(options: CancelDevSessionRunsServiceOptions) {
const cancelledSession = options.cancelledSessionId
? await this._prisma.runtimeEnvironmentSession.findFirst({
where: { id: options.cancelledSessionId },
})
: undefined;
if (cancelledSession) {
const latestSession = await findLatestSession(cancelledSession.environmentId, this._replica);
if (
latestSession &&
latestSession.id !== cancelledSession.id &&
!latestSession.disconnectedAt
) {
logger.debug("Not cancelling runs because there is a newer session", {
cancelledSessionId: cancelledSession.id,
latestSessionId: latestSession.id,
});
return;
}
}
logger.debug(
"Cancelling in progress runs for dev session because there isn't a newer connected session",
{
options,
cancelledSession,
}
);
const cancelTaskRunService = new CancelTaskRunService();
// readThroughRun resolves residency from the run id alone; an env scope is only
// available when a cancelled session was resolved.
const environmentId = cancelledSession?.environmentId ?? "";
for (const runId of options.runIds) {
await this.#cancelInProgressRun(
runId,
cancelTaskRunService,
options.cancelledAt,
options.reason,
environmentId
);
}
}
async #cancelInProgressRun(
runId: string,
service: CancelTaskRunService,
cancelledAt: Date,
reason: string,
environmentId: string
) {
logger.debug("Cancelling in progress run", { runId });
// Read-through: new store first, legacy read replica for an old
// in-retention run; single plain read in single-DB passthrough.
const where = runId.startsWith("run_") ? { friendlyId: runId } : { id: runId };
const result = await readThroughRun<CancelableTaskRun>({
runId,
environmentId,
readNew: (client) =>
client.taskRun.findFirst({
where,
select: {
id: true,
engine: true,
status: true,
friendlyId: true,
taskEventStore: true,
createdAt: true,
completedAt: true,
},
}),
readLegacy: (replica) =>
replica.taskRun.findFirst({
where,
select: {
id: true,
engine: true,
status: true,
friendlyId: true,
taskEventStore: true,
createdAt: true,
completedAt: true,
},
}),
deps: this.#readThroughDeps,
});
if (result.source === "not-found" || result.source === "past-retention") {
return;
}
const taskRun = result.value;
try {
await service.call(taskRun, { reason, cancelAttempts: true, cancelledAt });
} catch (e) {
logger.error("Failed to cancel in progress run", {
runId,
error: e,
});
}
}
static async enqueue(options: CancelDevSessionRunsServiceOptions, runAt?: Date) {
return await commonWorker.enqueue({
id: options.cancelledSessionId
? `cancelDevSessionRuns:${options.cancelledSessionId}`
: undefined,
job: "v3.cancelDevSessionRuns",
payload: options,
availableAt: runAt,
});
}
}
apps/webapp/app/v3/services/cancelTaskAttemptDependencies.server.ts
-108
View File
@@ -1,108 +0,0 @@
import { logger } from "~/services/logger.server";
import { commonWorker } from "../commonWorker.server";
import { BaseService } from "./baseService.server";
import { CancelTaskRunService } from "./cancelTaskRun.server";
export class CancelTaskAttemptDependenciesService extends BaseService {
public async call(attemptId: string) {
const taskAttempt = await this._prisma.taskRunAttempt.findFirst({
where: { id: attemptId },
include: {
dependencies: {
select: {
taskRunId: true,
},
},
batchDependencies: {
include: {
runDependencies: {
select: {
taskRunId: true,
},
},
},
},
},
});
if (!taskAttempt) {
return;
}
if (taskAttempt.status !== "CANCELED") {
logger.debug("Task attempt is not cancelled, continuing anyway", {
attemptId,
status: taskAttempt.status,
});
}
const cancelRunService = new CancelTaskRunService();
logger.debug("Cancelling task attempt dependencies", {
taskAttempt,
dependencies: taskAttempt.dependencies,
batchDependencies: taskAttempt.batchDependencies,
});
// Hydrate the dependent runs from both relation paths in a single batched read,
// deduping the ids that feed the query while preserving the original iteration order.
const taskRunIds = new Set<string>();
for (const dependency of taskAttempt.dependencies) {
taskRunIds.add(dependency.taskRunId);
}
for (const batchDependency of taskAttempt.batchDependencies) {
for (const runDependency of batchDependency.runDependencies) {
taskRunIds.add(runDependency.taskRunId);
}
}
const runs =
taskRunIds.size > 0
? await this.runStore.findRuns(
{
where: { id: { in: [...taskRunIds] } },
select: {
id: true,
engine: true,
status: true,
friendlyId: true,
taskEventStore: true,
createdAt: true,
completedAt: true,
},
},
this._prisma
)
: [];
const runMap = new Map(runs.map((run) => [run.id, run]));
// TaskAttempt will either have dependencies or batchDependencies
for (const dependency of taskAttempt.dependencies) {
const run = runMap.get(dependency.taskRunId);
if (run) {
await cancelRunService.call(run);
}
}
for (const batchDependency of taskAttempt.batchDependencies) {
for (const runDependency of batchDependency.runDependencies) {
const run = runMap.get(runDependency.taskRunId);
if (run) {
await cancelRunService.call(run);
}
}
}
}
static async enqueue(attemptId: string, runAt?: Date) {
return await commonWorker.enqueue({
id: `cancelTaskAttemptDependencies:${attemptId}`,
job: "v3.cancelTaskAttemptDependencies",
payload: {
attemptId,
},
availableAt: runAt,
});
}
}
apps/webapp/app/v3/services/cancelTaskRun.server.ts
+22 -10
View File
@@ -1,7 +1,7 @@
import { RunEngineVersion, type TaskRun } from "@trigger.dev/database";
import { engine } from "../runEngine.server";
import { isCancellableRunStatus } from "../taskStatus";
import { BaseService } from "./baseService.server";
import { CancelTaskRunServiceV1 } from "./cancelTaskRunV1.server";
export type CancelTaskRunServiceOptions = {
reason?: string;
@@ -38,17 +38,29 @@ export class CancelTaskRunService extends BaseService {
taskRun: CancelableTaskRun,
options?: CancelTaskRunServiceOptions
): Promise<CancelTaskRunServiceResult | undefined> {
const service = new CancelTaskRunServiceV1(this._prisma);
const result = await service.call(taskRun, options);
if (!result) {
return;
// v3 (engine V1) execution is retired: there are no V1 workers or coordinator
// left to signal. A historical V1 run can still be cancelled by finalizing its
// DB row directly. Never throw here: the cancel route returns 500 on any throw.
if (!isCancellableRunStatus(taskRun.status)) {
if (options?.bulkActionId) {
await this._prisma.taskRun.update({
where: { id: taskRun.id },
data: { bulkActionGroupIds: { push: options.bulkActionId } },
});
}
return { id: taskRun.id, alreadyFinished: true };
}
return {
id: result.id,
alreadyFinished: false,
};
await this._prisma.taskRun.update({
where: { id: taskRun.id },
data: {
status: "CANCELED",
completedAt: options?.cancelledAt ?? new Date(),
bulkActionGroupIds: options?.bulkActionId ? { push: options.bulkActionId } : undefined,
},
});
return { id: taskRun.id, alreadyFinished: false };
}
private async callV2(
apps/webapp/app/v3/services/cancelTaskRunV1.server.ts
-210
View File
@@ -1,210 +0,0 @@
import { type Prisma } from "@trigger.dev/database";
import assertNever from "assert-never";
import { logger } from "~/services/logger.server";
import { socketIo } from "../handleSocketIo.server";
import { devPubSub } from "../marqs/devPubSub.server";
import { CANCELLABLE_ATTEMPT_STATUSES, isCancellableRunStatus } from "../taskStatus";
import { BaseService } from "./baseService.server";
import { CancelAttemptService } from "./cancelAttempt.server";
import { CancelTaskAttemptDependenciesService } from "./cancelTaskAttemptDependencies.server";
import type { CancelableTaskRun } from "./cancelTaskRun.server";
import { FinalizeTaskRunService } from "./finalizeTaskRun.server";
import { tryCatch } from "@trigger.dev/core/utils";
import { getEventRepositoryForStore } from "../eventRepository/index.server";
type ExtendedTaskRun = Prisma.TaskRunGetPayload<{
include: {
runtimeEnvironment: true;
lockedToVersion: true;
};
}>;
type ExtendedTaskRunAttempt = Prisma.TaskRunAttemptGetPayload<{
include: {
backgroundWorker: true;
};
}>;
export type CancelTaskRunServiceOptions = {
reason?: string;
cancelAttempts?: boolean;
cancelledAt?: Date;
bulkActionId?: string;
};
export class CancelTaskRunServiceV1 extends BaseService {
public async call(taskRun: CancelableTaskRun, options?: CancelTaskRunServiceOptions) {
const opts = {
reason: "Task run was cancelled by user",
cancelAttempts: true,
cancelledAt: new Date(),
...options,
};
// Make sure the task run is in a cancellable state
if (!isCancellableRunStatus(taskRun.status)) {
logger.info("Task run is not in a cancellable state", {
runId: taskRun.id,
status: taskRun.status,
});
//add the bulk action id to the run
if (opts.bulkActionId) {
await this._prisma.taskRun.update({
where: { id: taskRun.id },
data: {
bulkActionGroupIds: {
push: opts.bulkActionId,
},
},
});
}
return;
}
const finalizeService = new FinalizeTaskRunService();
const cancelledTaskRun = await finalizeService.call({
id: taskRun.id,
status: "CANCELED",
completedAt: opts.cancelledAt,
bulkActionId: opts.bulkActionId,
include: {
attempts: {
where: {
status: {
in: CANCELLABLE_ATTEMPT_STATUSES,
},
},
include: {
backgroundWorker: true,
dependencies: {
include: {
taskRun: true,
},
},
batchTaskRunItems: {
include: {
taskRun: true,
},
},
},
},
runtimeEnvironment: true,
lockedToVersion: true,
project: true,
},
attemptStatus: "CANCELED",
error: {
type: "STRING_ERROR",
raw: opts.reason,
},
});
const eventRepository = await getEventRepositoryForStore(
cancelledTaskRun.taskEventStore,
cancelledTaskRun.runtimeEnvironment.organizationId
);
const [cancelRunEventError] = await tryCatch(
eventRepository.cancelRunEvent({
reason: opts.reason,
run: cancelledTaskRun,
cancelledAt: opts.cancelledAt,
})
);
if (cancelRunEventError) {
logger.error("[CancelTaskRunServiceV1] Failed to cancel run event", {
error: cancelRunEventError,
runId: cancelledTaskRun.id,
});
}
// Cancel any in progress attempts
if (opts.cancelAttempts) {
await this.#cancelPotentiallyRunningAttempts(cancelledTaskRun, cancelledTaskRun.attempts);
await this.#cancelRemainingRunWorkers(cancelledTaskRun);
}
return {
id: cancelledTaskRun.id,
};
}
async #cancelPotentiallyRunningAttempts(
run: ExtendedTaskRun,
attempts: ExtendedTaskRunAttempt[]
) {
for (const attempt of attempts) {
await CancelTaskAttemptDependenciesService.enqueue(attempt.id);
if (run.runtimeEnvironment.type === "DEVELOPMENT") {
// Signal the task run attempt to stop
await devPubSub.publish(
`backgroundWorker:${attempt.backgroundWorkerId}:${attempt.id}`,
"CANCEL_ATTEMPT",
{
attemptId: attempt.friendlyId,
backgroundWorkerId: attempt.backgroundWorker.friendlyId,
taskRunId: run.friendlyId,
}
);
} else {
switch (attempt.status) {
case "EXECUTING": {
// We need to send a cancel message to the coordinator
socketIo.coordinatorNamespace.emit("REQUEST_ATTEMPT_CANCELLATION", {
version: "v1",
attemptId: attempt.id,
attemptFriendlyId: attempt.friendlyId,
});
break;
}
case "PENDING":
case "PAUSED": {
logger.debug("Cancelling pending or paused attempt", {
attempt,
});
const service = new CancelAttemptService();
await service.call(
attempt.friendlyId,
run.id,
new Date(),
"Task run was cancelled by user"
);
break;
}
case "CANCELED":
case "COMPLETED":
case "FAILED": {
// Do nothing
break;
}
default: {
assertNever(attempt.status);
}
}
}
}
}
async #cancelRemainingRunWorkers(run: ExtendedTaskRun) {
if (run.runtimeEnvironment.type === "DEVELOPMENT") {
// Nothing to do
return;
}
// Broadcast cancel message to all coordinators
socketIo.coordinatorNamespace.emit("REQUEST_RUN_CANCELLATION", {
version: "v1",
runId: run.id,
// Give the attempts some time to exit gracefully. If the runs supports lazy attempts, it also supports exit delays.
delayInMs: run.lockedToVersion?.supportsLazyAttempts ? 5_000 : undefined,
});
}
}
apps/webapp/app/v3/services/changeCurrentDeployment.server.ts
-13
View File
@@ -10,7 +10,6 @@ import {
import { taskMetadataCacheInstance } from "~/services/taskMetadataCacheInstance.server";
import { BaseService, ServiceValidationError } from "./baseService.server";
import { syncDeclarativeSchedules } from "./createBackgroundWorker.server";
import { ExecuteTasksWaitingForDeployService } from "./executeTasksWaitingForDeploy";
import { compareDeploymentVersions } from "../utils/deploymentVersions";
export type ChangeCurrentDeploymentDirection = "promote" | "rollback";
@@ -174,18 +173,6 @@ export class ChangeCurrentDeploymentService extends BaseService {
error: scheduleSyncError,
});
}
// Only V1 engine workers need the WAITING_FOR_DEPLOY drain — V2 runs sit
// in PENDING_VERSION and are handled out of band, so enqueuing here for V2
// just produces empty scans of the TaskRun status index.
const worker = await this._prisma.backgroundWorker.findFirst({
where: { id: deployment.workerId },
select: { engine: true },
});
if (worker?.engine === "V1") {
await ExecuteTasksWaitingForDeployService.enqueue(deployment.workerId);
}
}
async #syncSchedulesForDeployment(deployment: WorkerDeployment) {
apps/webapp/app/v3/services/completeAttempt.server.ts
-728
View File
@@ -1,728 +0,0 @@
import { tryCatch } from "@trigger.dev/core/utils";
import type {
MachinePresetName,
TaskRunExecution,
TaskRunExecutionResult,
TaskRunExecutionRetry,
TaskRunFailedExecutionResult,
TaskRunSuccessfulExecutionResult,
V3TaskRunExecution,
} from "@trigger.dev/core/v3";
import {
TaskRunContext,
TaskRunErrorCodes,
flattenAttributes,
isOOMRunError,
sanitizeError,
shouldRetryError,
taskRunErrorEnhancer,
} from "@trigger.dev/core/v3";
import type { TaskRun } from "@trigger.dev/database";
import { MAX_TASK_RUN_ATTEMPTS } from "~/consts";
import type { PrismaClientOrTransaction } from "~/db.server";
import { env } from "~/env.server";
import type { AuthenticatedEnvironment } from "~/services/apiAuth.server";
import { logger } from "~/services/logger.server";
import { marqs } from "~/v3/marqs/index.server";
import { FailedTaskRunRetryHelper } from "../failedTaskRun.server";
import { socketIo } from "../handleSocketIo.server";
import { createExceptionPropertiesFromError } from "../eventRepository/common.server";
import type { FAILED_RUN_STATUSES } from "../taskStatus";
import { isFinalAttemptStatus, isFinalRunStatus } from "../taskStatus";
import { BaseService } from "./baseService.server";
import { CancelAttemptService } from "./cancelAttempt.server";
import { CreateCheckpointService } from "./createCheckpoint.server";
import { FinalizeTaskRunService } from "./finalizeTaskRun.server";
import { RetryAttemptService } from "./retryAttempt.server";
import { getEventRepositoryForStore } from "../eventRepository/index.server";
type FoundAttempt = Awaited<ReturnType<typeof findAttempt>>;
type CheckpointData = {
docker: boolean;
location: string;
};
type CompleteAttemptServiceOptions = {
prisma?: PrismaClientOrTransaction;
supportsRetryCheckpoints?: boolean;
isSystemFailure?: boolean;
isCrash?: boolean;
};
export class CompleteAttemptService extends BaseService {
constructor(private opts: CompleteAttemptServiceOptions = {}) {
super(opts.prisma);
}
public async call({
completion,
execution,
env,
checkpoint,
}: {
completion: TaskRunExecutionResult;
execution: V3TaskRunExecution;
env?: AuthenticatedEnvironment;
checkpoint?: CheckpointData;
}): Promise<"COMPLETED" | "RETRIED"> {
const taskRunAttempt = await findAttempt(this._prisma, execution.attempt.id);
if (!taskRunAttempt) {
logger.error("[CompleteAttemptService] Task run attempt not found", {
id: execution.attempt.id,
});
const run = await this.runStore.findRun(
{
friendlyId: execution.run.id,
},
{
select: {
id: true,
},
},
this._prisma
);
if (!run) {
logger.error("[CompleteAttemptService] Task run not found", {
friendlyId: execution.run.id,
});
return "COMPLETED";
}
const finalizeService = new FinalizeTaskRunService();
await finalizeService.call({
id: run.id,
status: "SYSTEM_FAILURE",
completedAt: new Date(),
attemptStatus: "FAILED",
error: {
type: "INTERNAL_ERROR",
code: TaskRunErrorCodes.TASK_EXECUTION_FAILED,
message: "Tried to complete attempt but it doesn't exist",
},
metadata: completion.metadata,
env,
});
// No attempt, so there's no message to ACK
return "COMPLETED";
}
if (
isFinalAttemptStatus(taskRunAttempt.status) ||
isFinalRunStatus(taskRunAttempt.taskRun.status)
) {
// We don't want to retry a task run that has already been marked as failed, cancelled, or completed
logger.debug("[CompleteAttemptService] Attempt or run is already in a final state", {
taskRunAttempt,
completion,
});
return "COMPLETED";
}
if (completion.ok) {
return await this.#completeAttemptSuccessfully(completion, taskRunAttempt, env);
} else {
return await this.#completeAttemptFailed({
completion,
execution,
taskRunAttempt,
env,
checkpoint,
});
}
}
async #completeAttemptSuccessfully(
completion: TaskRunSuccessfulExecutionResult,
taskRunAttempt: NonNullable<FoundAttempt>,
env?: AuthenticatedEnvironment
): Promise<"COMPLETED"> {
await this._prisma.taskRunAttempt.update({
where: { id: taskRunAttempt.id },
data: {
status: "COMPLETED",
completedAt: new Date(),
output: completion.output,
outputType: completion.outputType,
usageDurationMs: completion.usage?.durationMs,
taskRun: {
update: {
output: completion.output,
outputType: completion.outputType,
},
},
},
});
const finalizeService = new FinalizeTaskRunService();
await finalizeService.call({
id: taskRunAttempt.taskRunId,
status: "COMPLETED_SUCCESSFULLY",
completedAt: new Date(),
metadata: completion.metadata,
env,
});
const eventRepository = await getEventRepositoryForStore(
taskRunAttempt.taskRun.taskEventStore,
taskRunAttempt.taskRun.organizationId ?? ""
);
const [completeSuccessfulRunEventError] = await tryCatch(
eventRepository.completeSuccessfulRunEvent({
run: taskRunAttempt.taskRun,
endTime: new Date(),
})
);
if (completeSuccessfulRunEventError) {
logger.error("[CompleteAttemptService] Failed to complete successful run event", {
error: completeSuccessfulRunEventError,
runId: taskRunAttempt.taskRunId,
});
}
return "COMPLETED";
}
async #completeAttemptFailed({
completion,
execution,
taskRunAttempt,
env,
checkpoint,
}: {
completion: TaskRunFailedExecutionResult;
execution: V3TaskRunExecution;
taskRunAttempt: NonNullable<FoundAttempt>;
env?: AuthenticatedEnvironment;
checkpoint?: CheckpointData;
}): Promise<"COMPLETED" | "RETRIED"> {
if (
completion.error.type === "INTERNAL_ERROR" &&
completion.error.code === "TASK_RUN_CANCELLED"
) {
// We need to cancel the task run instead of fail it
const cancelService = new CancelAttemptService();
// TODO: handle usages
await cancelService.call(
taskRunAttempt.friendlyId,
taskRunAttempt.taskRunId,
new Date(),
"Canceled by user",
env
);
return "COMPLETED";
}
const failedAt = new Date();
const sanitizedError = sanitizeError(completion.error);
await this._prisma.taskRunAttempt.update({
where: { id: taskRunAttempt.id },
data: {
status: "FAILED",
completedAt: failedAt,
error: sanitizedError,
usageDurationMs: completion.usage?.durationMs,
},
});
const environment = env ?? (await this.#getEnvironment(execution.environment.id));
// This means that tasks won't know they are being retried
let executionRetryInferred = false;
let executionRetry = completion.retry;
const shouldInfer = this.opts.isCrash || this.opts.isSystemFailure;
if (!executionRetry && shouldInfer) {
executionRetryInferred = true;
executionRetry = FailedTaskRunRetryHelper.getExecutionRetry({
run: {
...taskRunAttempt.taskRun,
lockedBy: taskRunAttempt.backgroundWorkerTask,
lockedToVersion: taskRunAttempt.backgroundWorker,
},
execution,
});
}
let retriableError = shouldRetryError(taskRunErrorEnhancer(completion.error));
let isOOMRetry = false;
let isOOMAttempt = isOOMRunError(completion.error);
let isOnMaxOOMMachine = false;
let oomMachine: MachinePresetName | undefined;
//OOM errors should retry (if an OOM machine is specified, and we're not already on it)
if (isOOMAttempt) {
const retryConfig = FailedTaskRunRetryHelper.getRetryConfig({
run: {
...taskRunAttempt.taskRun,
lockedBy: taskRunAttempt.backgroundWorkerTask,
lockedToVersion: taskRunAttempt.backgroundWorker,
},
execution,
});
oomMachine = retryConfig?.outOfMemory?.machine;
isOnMaxOOMMachine = oomMachine === taskRunAttempt.taskRun.machinePreset;
if (oomMachine && !isOnMaxOOMMachine) {
//we will retry
isOOMRetry = true;
retriableError = true;
executionRetry = FailedTaskRunRetryHelper.getExecutionRetry({
run: {
...taskRunAttempt.taskRun,
lockedBy: taskRunAttempt.backgroundWorkerTask,
lockedToVersion: taskRunAttempt.backgroundWorker,
},
execution,
});
//update the machine on the run
await this._prisma.taskRun.update({
where: {
id: taskRunAttempt.taskRunId,
},
data: {
machinePreset: oomMachine,
},
});
}
}
if (
retriableError &&
executionRetry !== undefined &&
taskRunAttempt.number < MAX_TASK_RUN_ATTEMPTS
) {
return await this.#retryAttempt({
execution,
executionRetry,
executionRetryInferred,
taskRunAttempt,
environment,
checkpoint,
forceRequeue: isOOMRetry,
oomMachine,
});
}
// The attempt has failed and we won't retry
if (isOOMAttempt && isOnMaxOOMMachine && environment.type !== "DEVELOPMENT") {
// The attempt failed due to an OOM error but we're already on the machine we should retry on
exitRun(taskRunAttempt.taskRunId);
}
const eventRepository = await getEventRepositoryForStore(
taskRunAttempt.taskRun.taskEventStore,
taskRunAttempt.taskRun.organizationId ?? ""
);
const [completeFailedRunEventError] = await tryCatch(
eventRepository.completeFailedRunEvent({
run: taskRunAttempt.taskRun,
endTime: failedAt,
exception: createExceptionPropertiesFromError(sanitizedError),
})
);
if (completeFailedRunEventError) {
logger.error("[CompleteAttemptService] Failed to complete failed run event", {
error: completeFailedRunEventError,
runId: taskRunAttempt.taskRunId,
});
}
await this._prisma.taskRun.update({
where: {
id: taskRunAttempt.taskRunId,
},
data: {
error: sanitizedError,
},
});
let status: FAILED_RUN_STATUSES;
// Set the correct task run status
if (this.opts.isSystemFailure) {
status = "SYSTEM_FAILURE";
} else if (this.opts.isCrash) {
status = "CRASHED";
} else if (
sanitizedError.type === "INTERNAL_ERROR" &&
sanitizedError.code === "MAX_DURATION_EXCEEDED"
) {
status = "TIMED_OUT";
} else if (sanitizedError.type === "INTERNAL_ERROR") {
status = "CRASHED";
} else {
status = "COMPLETED_WITH_ERRORS";
}
const finalizeService = new FinalizeTaskRunService();
await finalizeService.call({
id: taskRunAttempt.taskRunId,
status,
completedAt: failedAt,
metadata: completion.metadata,
env,
});
if (status !== "CRASHED" && status !== "SYSTEM_FAILURE") {
return "COMPLETED";
}
// Handle in-progress events
switch (status) {
case "CRASHED": {
const [createAttemptFailedEventError] = await tryCatch(
eventRepository.createAttemptFailedRunEvent({
run: taskRunAttempt.taskRun,
endTime: failedAt,
attemptNumber: taskRunAttempt.number,
exception: createExceptionPropertiesFromError(sanitizedError),
})
);
if (createAttemptFailedEventError) {
logger.error("[CompleteAttemptService] Failed to create attempt failed run event", {
error: createAttemptFailedEventError,
runId: taskRunAttempt.taskRunId,
});
}
break;
}
case "SYSTEM_FAILURE": {
const [createAttemptFailedEventError] = await tryCatch(
eventRepository.createAttemptFailedRunEvent({
run: taskRunAttempt.taskRun,
endTime: failedAt,
attemptNumber: taskRunAttempt.number,
exception: createExceptionPropertiesFromError(sanitizedError),
})
);
if (createAttemptFailedEventError) {
logger.error("[CompleteAttemptService] Failed to create attempt failed run event", {
error: createAttemptFailedEventError,
runId: taskRunAttempt.taskRunId,
});
}
}
}
return "COMPLETED";
}
async #enqueueReattempt({
run,
executionRetry,
executionRetryInferred,
checkpointEventId,
supportsLazyAttempts,
forceRequeue = false,
}: {
run: TaskRun;
executionRetry: TaskRunExecutionRetry;
executionRetryInferred: boolean;
checkpointEventId?: string;
supportsLazyAttempts: boolean;
forceRequeue?: boolean;
}) {
const retryViaQueue = () => {
logger.debug("[CompleteAttemptService] Enqueuing retry attempt", { runId: run.id });
return marqs.requeueMessage(
run.id,
{
type: "EXECUTE",
taskIdentifier: run.taskIdentifier,
checkpointEventId: this.opts.supportsRetryCheckpoints ? checkpointEventId : undefined,
retryCheckpointsDisabled: !this.opts.supportsRetryCheckpoints,
},
executionRetry.timestamp,
"retry"
);
};
const retryDirectly = () => {
logger.debug("[CompleteAttemptService] Retrying attempt directly", { runId: run.id });
return RetryAttemptService.enqueue(run.id, new Date(executionRetry.timestamp));
};
// There's a checkpoint, so we need to go through the queue
if (checkpointEventId) {
if (!this.opts.supportsRetryCheckpoints) {
logger.error(
"[CompleteAttemptService] Worker does not support retry checkpoints, but a checkpoint was created",
{
runId: run.id,
checkpointEventId,
}
);
}
logger.debug("[CompleteAttemptService] Enqueuing retry attempt with checkpoint", {
runId: run.id,
});
await retryViaQueue();
return;
}
// Workers without lazy attempt support always need to go through the queue, which is where the attempt is created
if (!supportsLazyAttempts) {
logger.debug("[CompleteAttemptService] Worker does not support lazy attempts", {
runId: run.id,
});
await retryViaQueue();
return;
}
if (forceRequeue) {
logger.debug("[CompleteAttemptService] Forcing retry via queue", { runId: run.id });
// The run won't know it should shut down as we make the decision to force requeue here
// This also ensures that this change is backwards compatible with older workers
exitRun(run.id);
await retryViaQueue();
return;
}
// Workers that never checkpoint between attempts will exit after completing their current attempt if the retry delay exceeds the threshold
if (
!this.opts.supportsRetryCheckpoints &&
executionRetry.delay >= env.CHECKPOINT_THRESHOLD_IN_MS
) {
logger.debug(
"[CompleteAttemptService] Worker does not support retry checkpoints and the delay exceeds the threshold",
{ runId: run.id }
);
await retryViaQueue();
return;
}
if (executionRetryInferred) {
logger.debug("[CompleteAttemptService] Execution retry inferred, forcing retry via queue", {
runId: run.id,
});
await retryViaQueue();
return;
}
// The worker is still running and waiting for a retry message
await retryDirectly();
}
async #retryAttempt({
execution,
executionRetry,
executionRetryInferred,
taskRunAttempt,
environment,
checkpoint,
forceRequeue = false,
oomMachine,
}: {
execution: V3TaskRunExecution;
executionRetry: TaskRunExecutionRetry;
executionRetryInferred: boolean;
taskRunAttempt: NonNullable<FoundAttempt>;
environment: AuthenticatedEnvironment;
checkpoint?: CheckpointData;
forceRequeue?: boolean;
/** Setting this will also alter the retry span message */
oomMachine?: MachinePresetName;
}) {
const retryAt = new Date(executionRetry.timestamp);
const eventRepository = await getEventRepositoryForStore(
taskRunAttempt.taskRun.taskEventStore,
taskRunAttempt.taskRun.organizationId ?? ""
);
// Retry the task run
await eventRepository.recordEvent(
`Retry #${execution.attempt.number} delay${oomMachine ? " after OOM" : ""}`,
{
taskSlug: taskRunAttempt.taskRun.taskIdentifier,
environment,
attributes: {
metadata: this.#generateMetadataAttributesForNextAttempt(execution),
properties: {
retryAt: retryAt.toISOString(),
previousMachine: oomMachine
? (taskRunAttempt.taskRun.machinePreset ?? undefined)
: undefined,
nextMachine: oomMachine,
},
runId: taskRunAttempt.taskRun.friendlyId,
style: {
icon: "schedule-attempt",
},
},
context: taskRunAttempt.taskRun.traceContext as Record<string, string | undefined>,
spanIdSeed: `retry-${taskRunAttempt.number + 1}`,
endTime: retryAt,
}
);
logger.debug("[CompleteAttemptService] Retrying", {
taskRun: taskRunAttempt.taskRun.friendlyId,
retry: executionRetry,
});
await this._prisma.taskRun.update({
where: {
id: taskRunAttempt.taskRunId,
},
data: {
status: "RETRYING_AFTER_FAILURE",
},
});
if (environment.type === "DEVELOPMENT") {
await marqs.requeueMessage(taskRunAttempt.taskRunId, {}, executionRetry.timestamp, "retry");
return "RETRIED";
}
if (checkpoint) {
// This is only here for backwards compat - we don't checkpoint between attempts anymore
return await this.#retryAttemptWithCheckpoint({
execution,
taskRunAttempt,
executionRetry,
executionRetryInferred,
checkpoint,
});
}
await this.#enqueueReattempt({
run: taskRunAttempt.taskRun,
executionRetry,
supportsLazyAttempts: taskRunAttempt.backgroundWorker.supportsLazyAttempts,
executionRetryInferred,
forceRequeue,
});
return "RETRIED";
}
async #retryAttemptWithCheckpoint({
execution,
taskRunAttempt,
executionRetry,
executionRetryInferred,
checkpoint,
}: {
execution: V3TaskRunExecution;
taskRunAttempt: NonNullable<FoundAttempt>;
executionRetry: TaskRunExecutionRetry;
executionRetryInferred: boolean;
checkpoint: CheckpointData;
}) {
const createCheckpoint = new CreateCheckpointService(this._prisma);
const checkpointCreateResult = await createCheckpoint.call({
attemptFriendlyId: execution.attempt.id,
docker: checkpoint.docker,
location: checkpoint.location,
reason: {
type: "RETRYING_AFTER_FAILURE",
attemptNumber: execution.attempt.number,
},
});
if (!checkpointCreateResult.success) {
logger.error("[CompleteAttemptService] Failed to create reattempt checkpoint", {
checkpoint,
runId: execution.run.id,
attemptId: execution.attempt.id,
});
const finalizeService = new FinalizeTaskRunService();
await finalizeService.call({
id: taskRunAttempt.taskRunId,
status: "SYSTEM_FAILURE",
completedAt: new Date(),
error: {
type: "STRING_ERROR",
raw: "Failed to create reattempt checkpoint",
},
});
return "COMPLETED" as const;
}
await this.#enqueueReattempt({
run: taskRunAttempt.taskRun,
executionRetry,
checkpointEventId: checkpointCreateResult.event.id,
supportsLazyAttempts: taskRunAttempt.backgroundWorker.supportsLazyAttempts,
executionRetryInferred,
});
return "RETRIED" as const;
}
#generateMetadataAttributesForNextAttempt(execution: TaskRunExecution) {
const context = TaskRunContext.parse(execution);
// @ts-ignore
context.attempt = {
number: context.attempt.number + 1,
};
return flattenAttributes(context, "ctx");
}
async #getEnvironment(id: string) {
return await this._prisma.runtimeEnvironment.findFirstOrThrow({
where: {
id,
},
include: {
project: true,
organization: true,
},
});
}
}
async function findAttempt(prismaClient: PrismaClientOrTransaction, friendlyId: string) {
return prismaClient.taskRunAttempt.findFirst({
where: { friendlyId },
include: {
taskRun: true,
backgroundWorkerTask: true,
backgroundWorker: {
select: {
id: true,
supportsLazyAttempts: true,
sdkVersion: true,
},
},
},
});
}
function exitRun(runId: string) {
socketIo.coordinatorNamespace.emit("REQUEST_RUN_CANCELLATION", {
version: "v1",
runId,
});
}
apps/webapp/app/v3/services/crashTaskRun.server.ts
-200
View File
@@ -1,200 +0,0 @@
import { tryCatch } from "@trigger.dev/core/utils";
import type { TaskRunInternalError } from "@trigger.dev/core/v3";
import { sanitizeError, TaskRunErrorCodes } from "@trigger.dev/core/v3";
import type { TaskRun, TaskRunAttempt } from "@trigger.dev/database";
import type { AuthenticatedEnvironment } from "~/services/apiAuth.server";
import { logger } from "~/services/logger.server";
import { FailedTaskRunRetryHelper } from "../failedTaskRun.server";
import { CRASHABLE_ATTEMPT_STATUSES, isCrashableRunStatus } from "../taskStatus";
import { BaseService } from "./baseService.server";
import { FinalizeTaskRunService } from "./finalizeTaskRun.server";
import { getEventRepositoryForStore } from "../eventRepository/index.server";
export type CrashTaskRunServiceOptions = {
reason?: string;
exitCode?: number;
logs?: string;
crashAttempts?: boolean;
crashedAt?: Date;
overrideCompletion?: boolean;
errorCode?: TaskRunInternalError["code"];
};
export class CrashTaskRunService extends BaseService {
public async call(runId: string, options?: CrashTaskRunServiceOptions) {
const opts = {
reason: "Worker crashed",
crashAttempts: true,
crashedAt: new Date(),
...options,
};
logger.debug("CrashTaskRunService.call", { runId, opts });
if (options?.overrideCompletion) {
logger.error("CrashTaskRunService.call: overrideCompletion is deprecated", { runId });
return;
}
const taskRun = await this.runStore.findRun({ id: runId }, this._prisma);
if (!taskRun) {
logger.error("[CrashTaskRunService] Task run not found", { runId });
return;
}
// Make sure the task run is in a crashable state
if (!opts.overrideCompletion && !isCrashableRunStatus(taskRun.status)) {
logger.error("[CrashTaskRunService] Task run is not in a crashable state", {
runId,
status: taskRun.status,
});
return;
}
logger.debug("[CrashTaskRunService] Completing attempt", { runId, options });
const retryHelper = new FailedTaskRunRetryHelper(this._prisma);
const retryResult = await retryHelper.call({
runId,
completion: {
ok: false,
id: runId,
error: {
type: "INTERNAL_ERROR",
code: opts.errorCode ?? TaskRunErrorCodes.TASK_RUN_CRASHED,
message: opts.reason,
stackTrace: opts.logs,
},
},
isCrash: true,
});
logger.debug("[CrashTaskRunService] Completion result", { runId, retryResult });
if (retryResult === "RETRIED") {
logger.debug("[CrashTaskRunService] Retried task run", { runId });
return;
}
if (!opts.overrideCompletion) {
return;
}
logger.debug("[CrashTaskRunService] Overriding completion", { runId, options });
const finalizeService = new FinalizeTaskRunService();
const crashedTaskRun = await finalizeService.call({
id: taskRun.id,
status: "CRASHED",
completedAt: new Date(),
include: {
attempts: {
where: {
status: {
in: CRASHABLE_ATTEMPT_STATUSES,
},
},
include: {
backgroundWorker: true,
runtimeEnvironment: true,
},
},
dependency: true,
runtimeEnvironment: {
include: {
organization: true,
project: true,
},
},
},
attemptStatus: "FAILED",
error: {
type: "INTERNAL_ERROR",
code: opts.errorCode ?? TaskRunErrorCodes.TASK_RUN_CRASHED,
message: opts.reason,
stackTrace: opts.logs,
},
});
const eventRepository = await getEventRepositoryForStore(
crashedTaskRun.taskEventStore,
crashedTaskRun.runtimeEnvironment.organizationId
);
const [createAttemptFailedEventError] = await tryCatch(
eventRepository.completeFailedRunEvent({
run: crashedTaskRun,
endTime: opts.crashedAt,
exception: {
type: opts.errorCode ?? TaskRunErrorCodes.TASK_RUN_CRASHED,
message: opts.reason,
stacktrace: opts.logs,
},
})
);
if (createAttemptFailedEventError) {
logger.error("[CrashTaskRunService] Failed to complete failed run event", {
error: createAttemptFailedEventError,
runId: crashedTaskRun.id,
});
}
if (!opts.crashAttempts) {
return;
}
// Cancel any in progress attempts
for (const attempt of crashedTaskRun.attempts) {
await this.#failAttempt(
attempt,
crashedTaskRun,
new Date(),
crashedTaskRun.runtimeEnvironment,
{
reason: opts.reason,
logs: opts.logs,
code: opts.errorCode,
}
);
}
}
async #failAttempt(
attempt: TaskRunAttempt,
run: TaskRun,
failedAt: Date,
environment: AuthenticatedEnvironment,
error: {
reason: string;
logs?: string;
code?: TaskRunInternalError["code"];
}
) {
return await this.traceWithEnv(
"[CrashTaskRunService] failAttempt()",
environment,
async (span) => {
span.setAttribute("taskRunId", run.id);
span.setAttribute("attemptId", attempt.id);
await this._prisma.taskRunAttempt.update({
where: {
id: attempt.id,
},
data: {
status: "FAILED",
completedAt: failedAt,
error: sanitizeError({
type: "INTERNAL_ERROR",
code: error.code ?? TaskRunErrorCodes.TASK_RUN_CRASHED,
message: error.reason,
stackTrace: error.logs,
}),
},
});
}
);
}
}
apps/webapp/app/v3/services/createCheckpoint.server.ts
-444
View File
@@ -1,444 +0,0 @@
import type { CoordinatorToPlatformMessages, ManualCheckpointMetadata } from "@trigger.dev/core/v3";
import type { InferSocketMessageSchema } from "@trigger.dev/core/v3/zodSocket";
import type { Checkpoint, CheckpointRestoreEvent } from "@trigger.dev/database";
import { logger } from "~/services/logger.server";
import { marqs } from "~/v3/marqs/index.server";
import { isFreezableAttemptStatus, isFreezableRunStatus } from "../taskStatus";
import { BaseService } from "./baseService.server";
import { CreateCheckpointRestoreEventService } from "./createCheckpointRestoreEvent.server";
import { ResumeBatchRunService } from "./resumeBatchRun.server";
import { ResumeDependentParentsService } from "./resumeDependentParents.server";
import { CheckpointId } from "@trigger.dev/core/v3/isomorphic";
export class CreateCheckpointService extends BaseService {
public async call(
params: Omit<
InferSocketMessageSchema<typeof CoordinatorToPlatformMessages, "CHECKPOINT_CREATED">,
"version"
>
): Promise<
| {
success: true;
checkpoint: Checkpoint;
event: CheckpointRestoreEvent;
keepRunAlive: boolean;
}
| {
success: false;
keepRunAlive?: boolean;
}
> {
logger.debug(`Creating checkpoint`, params);
const attempt = await this._prisma.taskRunAttempt.findFirst({
where: {
friendlyId: params.attemptFriendlyId,
},
include: {
taskRun: true,
backgroundWorker: {
select: {
id: true,
deployment: {
select: {
imageReference: true,
},
},
},
},
},
});
if (!attempt) {
logger.error("Attempt not found", params);
return {
success: false,
};
}
if (
!isFreezableAttemptStatus(attempt.status) ||
!isFreezableRunStatus(attempt.taskRun.status)
) {
logger.error("Unfreezable state", {
attempt: {
id: attempt.id,
status: attempt.status,
},
run: {
id: attempt.taskRunId,
status: attempt.taskRun.status,
},
params,
});
return {
success: false,
keepRunAlive: true,
};
}
const imageRef = attempt.backgroundWorker.deployment?.imageReference;
if (!imageRef) {
logger.error("Missing deployment or image ref", {
attemptId: attempt.id,
workerId: attempt.backgroundWorker.id,
params,
});
return {
success: false,
};
}
const { reason } = params;
// Check if we should accept this checkpoint
switch (reason.type) {
case "MANUAL": {
// Always accept manual checkpoints
break;
}
case "WAIT_FOR_DURATION": {
// Always accept duration checkpoints
break;
}
case "WAIT_FOR_TASK": {
const childRun = await this._prisma.taskRun.findFirst({
where: {
friendlyId: reason.friendlyId,
},
select: {
dependency: {
select: {
resumedAt: true,
},
},
},
});
if (!childRun) {
logger.error("CreateCheckpointService: Pre-check - WAIT_FOR_TASK child run not found", {
friendlyId: reason.friendlyId,
params,
});
return {
success: false,
keepRunAlive: false,
};
}
if (childRun.dependency?.resumedAt) {
logger.info("CreateCheckpointService: Child run already resumed", {
childRun,
params,
});
return {
success: false,
keepRunAlive: true,
};
}
break;
}
case "WAIT_FOR_BATCH": {
// Routed by friendlyId so a run-ops id (NEW-resident) batch is found on the owning DB;
// env-scoped to the dependent attempt's run (a batch shares its dependent's env). Read the
// primary: a batch that just resumed the parent may lag the replica, and a stale resumedAt
// (null) would checkpoint (suspend) an already-resumed run -> it stalls until a sweep.
const batchRun = await this.runStore.findBatchTaskRunByFriendlyId(
reason.batchFriendlyId,
attempt.taskRun.runtimeEnvironmentId,
undefined,
this._prisma
);
if (!batchRun) {
logger.error("CreateCheckpointService: Pre-check - Batch not found", {
batchFriendlyId: reason.batchFriendlyId,
params,
});
return {
success: false,
keepRunAlive: false,
};
}
if (batchRun.resumedAt) {
logger.info("CreateCheckpointService: Batch already resumed", {
batchRun,
params,
});
return {
success: false,
keepRunAlive: true,
};
}
break;
}
default: {
break;
}
}
//sleep to test slow checkpoints
// Sleep a random value between 4 and 30 seconds
// await new Promise((resolve) => {
// const waitSeconds = Math.floor(Math.random() * 26) + 4;
// logger.log(`Sleep for ${waitSeconds} seconds`);
// setTimeout(resolve, waitSeconds * 1000);
// });
let metadata: string;
if (params.reason.type === "MANUAL") {
metadata = JSON.stringify({
...params.reason,
attemptId: attempt.id,
previousAttemptStatus: attempt.status,
previousRunStatus: attempt.taskRun.status,
} satisfies ManualCheckpointMetadata);
} else {
metadata = JSON.stringify(params.reason);
}
const checkpoint = await this._prisma.checkpoint.create({
data: {
...CheckpointId.generate(),
runtimeEnvironmentId: attempt.taskRun.runtimeEnvironmentId,
projectId: attempt.taskRun.projectId,
attemptId: attempt.id,
attemptNumber: attempt.number,
runId: attempt.taskRunId,
location: params.location,
type: params.docker ? "DOCKER" : "KUBERNETES",
reason: params.reason.type,
metadata,
imageRef,
},
});
const eventService = new CreateCheckpointRestoreEventService(this._prisma);
await this._prisma.taskRunAttempt.update({
where: {
id: attempt.id,
},
data: {
status: params.reason.type === "RETRYING_AFTER_FAILURE" ? undefined : "PAUSED",
taskRun: {
update: {
status: "WAITING_TO_RESUME",
},
},
},
});
let checkpointEvent: CheckpointRestoreEvent | undefined;
switch (reason.type) {
case "MANUAL":
case "WAIT_FOR_DURATION": {
let restoreAtUnixTimeMs: number;
if (reason.type === "MANUAL") {
// Restore immediately if not specified, useful for live migration
restoreAtUnixTimeMs = reason.restoreAtUnixTimeMs ?? Date.now();
} else {
restoreAtUnixTimeMs = reason.now + reason.ms;
}
checkpointEvent = await eventService.checkpoint({
checkpointId: checkpoint.id,
});
if (checkpointEvent) {
await marqs.requeueMessage(
attempt.taskRunId,
{
type: "RESUME_AFTER_DURATION",
resumableAttemptId: attempt.id,
checkpointEventId: checkpointEvent.id,
},
restoreAtUnixTimeMs,
"resume"
);
return {
success: true,
checkpoint,
event: checkpointEvent,
keepRunAlive: false,
};
}
break;
}
case "WAIT_FOR_TASK": {
checkpointEvent = await eventService.checkpoint({
checkpointId: checkpoint.id,
dependencyFriendlyRunId: reason.friendlyId,
});
if (checkpointEvent) {
//heartbeats will start again when the run resumes
logger.log("CreateCheckpointService: Canceling heartbeat", {
attemptId: attempt.id,
taskRunId: attempt.taskRunId,
type: "WAIT_FOR_TASK",
reason,
params,
});
await marqs?.cancelHeartbeat(attempt.taskRunId);
const childRun = await this._prisma.taskRun.findFirst({
where: {
friendlyId: reason.friendlyId,
},
});
if (!childRun) {
logger.error("CreateCheckpointService: WAIT_FOR_TASK child run not found", {
friendlyId: reason.friendlyId,
params,
});
return {
success: true,
checkpoint,
event: checkpointEvent,
keepRunAlive: false,
};
}
const resumeService = new ResumeDependentParentsService(this._prisma);
const result = await resumeService.call({ id: childRun.id });
if (result.success) {
logger.log("CreateCheckpointService: Resumed dependent parents", {
result,
childRun,
attempt,
checkpointEvent,
params,
});
} else {
logger.error("CreateCheckpointService: Failed to resume dependent parents", {
result,
childRun,
attempt,
checkpointEvent,
params,
});
}
return {
success: true,
checkpoint,
event: checkpointEvent,
keepRunAlive: false,
};
}
break;
}
case "WAIT_FOR_BATCH": {
checkpointEvent = await eventService.checkpoint({
checkpointId: checkpoint.id,
batchDependencyFriendlyId: reason.batchFriendlyId,
});
if (checkpointEvent) {
//heartbeats will start again when the run resumes
logger.log("CreateCheckpointService: Canceling heartbeat", {
attemptId: attempt.id,
taskRunId: attempt.taskRunId,
type: "WAIT_FOR_BATCH",
params,
});
await marqs?.cancelHeartbeat(attempt.taskRunId);
// Routed by friendlyId; read the primary (this._prisma) so a just-resumed batch that still
// lags the replica doesn't leave a stale resumedAt and suspend an already-resumed run.
const batchRun = await this.runStore.findBatchTaskRunByFriendlyId(
reason.batchFriendlyId,
attempt.taskRun.runtimeEnvironmentId,
undefined,
this._prisma
);
if (!batchRun) {
logger.error("CreateCheckpointService: Batch not found", {
friendlyId: reason.batchFriendlyId,
params,
});
return {
success: true,
checkpoint,
event: checkpointEvent,
keepRunAlive: false,
};
}
//if there's a message in the queue, we make sure the checkpoint event is on it
await marqs.replaceMessage(attempt.taskRun.id, {
checkpointEventId: checkpointEvent.id,
});
await ResumeBatchRunService.enqueue(batchRun.id, batchRun.batchVersion === "v3");
return {
success: true,
checkpoint,
event: checkpointEvent,
keepRunAlive: false,
};
}
break;
}
case "RETRYING_AFTER_FAILURE": {
checkpointEvent = await eventService.checkpoint({
checkpointId: checkpoint.id,
});
// ACK is already handled by attempt completion
break;
}
default: {
break;
}
}
if (!checkpointEvent) {
logger.error("No checkpoint event", {
attemptId: attempt.id,
checkpointId: checkpoint.id,
params,
});
await marqs?.acknowledgeMessage(
attempt.taskRunId,
"No checkpoint event in CreateCheckpointService"
);
return {
success: false,
};
}
return {
success: true,
checkpoint,
event: checkpointEvent,
keepRunAlive: false,
};
}
}
apps/webapp/app/v3/services/createCheckpointRestoreEvent.server.ts
-196
View File
@@ -1,196 +0,0 @@
import { ManualCheckpointMetadata } from "@trigger.dev/core/v3";
import type {
Checkpoint,
CheckpointRestoreEvent,
CheckpointRestoreEventType,
} from "@trigger.dev/database";
import { isTaskRunAttemptStatus, isTaskRunStatus } from "~/database-types";
import { logger } from "~/services/logger.server";
import { safeJsonParse } from "~/utils/json";
import { BaseService } from "./baseService.server";
interface CheckpointRestoreEventCallParams {
checkpointId: string;
type: CheckpointRestoreEventType;
dependencyFriendlyRunId?: string;
batchDependencyFriendlyId?: string;
}
type CheckpointRestoreEventParams = Omit<CheckpointRestoreEventCallParams, "type">;
export class CreateCheckpointRestoreEventService extends BaseService {
async checkpoint(params: CheckpointRestoreEventParams) {
return this.#call({ ...params, type: "CHECKPOINT" });
}
async restore(params: CheckpointRestoreEventParams) {
return this.#call({ ...params, type: "RESTORE" });
}
async #call(
params: CheckpointRestoreEventCallParams
): Promise<CheckpointRestoreEvent | undefined> {
if (params.dependencyFriendlyRunId && params.batchDependencyFriendlyId) {
logger.error("Only one dependency can be set", { params });
return;
}
const checkpoint = await this._prisma.checkpoint.findFirst({
where: {
id: params.checkpointId,
},
});
if (!checkpoint) {
logger.error("Checkpoint not found", { id: params.checkpointId });
return;
}
if (params.type === "RESTORE" && checkpoint.reason === "MANUAL") {
const manualRestoreSuccess = await this.#handleManualCheckpointRestore(checkpoint);
if (!manualRestoreSuccess) {
return;
}
}
logger.debug(`Creating checkpoint/restore event`, { params });
let taskRunDependencyId: string | undefined;
if (params.dependencyFriendlyRunId) {
const run = await this.runStore.findRun(
{
friendlyId: params.dependencyFriendlyRunId,
},
{
select: {
id: true,
dependency: {
select: {
id: true,
},
},
},
},
this._prisma
);
taskRunDependencyId = run?.dependency?.id;
if (!taskRunDependencyId) {
logger.error("Dependency or run not found", { runId: params.dependencyFriendlyRunId });
return;
}
}
const checkpointEvent = await this._prisma.checkpointRestoreEvent.create({
data: {
checkpointId: checkpoint.id,
runtimeEnvironmentId: checkpoint.runtimeEnvironmentId,
projectId: checkpoint.projectId,
attemptId: checkpoint.attemptId,
runId: checkpoint.runId,
type: params.type,
reason: checkpoint.reason,
metadata: checkpoint.metadata,
...(taskRunDependencyId
? {
taskRunDependency: {
connect: {
id: taskRunDependencyId,
},
},
}
: undefined),
...(params.batchDependencyFriendlyId
? {
batchTaskRunDependency: {
connect: {
friendlyId: params.batchDependencyFriendlyId,
},
},
}
: undefined),
},
});
return checkpointEvent;
}
async #handleManualCheckpointRestore(checkpoint: Checkpoint): Promise<boolean> {
const json = checkpoint.metadata ? safeJsonParse(checkpoint.metadata) : undefined;
// We need to restore the previous run and attempt status as saved in the metadata
const metadata = ManualCheckpointMetadata.safeParse(json);
if (!metadata.success) {
logger.error("Invalid metadata", { metadata });
return false;
}
const { attemptId, previousAttemptStatus, previousRunStatus } = metadata.data;
if (!isTaskRunAttemptStatus(previousAttemptStatus)) {
logger.error("Invalid previous attempt status", { previousAttemptStatus });
return false;
}
if (!isTaskRunStatus(previousRunStatus)) {
logger.error("Invalid previous run status", { previousRunStatus });
return false;
}
try {
const updatedAttempt = await this._prisma.taskRunAttempt.update({
where: {
id: attemptId,
},
data: {
status: previousAttemptStatus,
taskRun: {
update: {
data: {
status: previousRunStatus,
},
},
},
},
select: {
id: true,
status: true,
taskRun: {
select: {
id: true,
status: true,
},
},
},
});
logger.debug("Set post resume statuses after manual checkpoint", {
run: {
id: updatedAttempt.taskRun.id,
status: updatedAttempt.taskRun.status,
},
attempt: {
id: updatedAttempt.id,
status: updatedAttempt.status,
},
});
return true;
} catch (error) {
logger.error("Failed to set post resume statuses", {
error:
error instanceof Error
? {
name: error.name,
message: error.message,
stack: error.stack,
}
: error,
});
return false;
}
}
}
apps/webapp/app/v3/services/createDeploymentBackgroundWorkerV3.server.ts
-219
View File
@@ -1,219 +0,0 @@
import type { CreateBackgroundWorkerRequestBody } from "@trigger.dev/core/v3";
import { tryCatch } from "@trigger.dev/core/v3";
import type { BackgroundWorker, PrismaClientOrTransaction } from "@trigger.dev/database";
import type { AuthenticatedEnvironment } from "~/services/apiAuth.server";
import { logger } from "~/services/logger.server";
import { syncTaskIdentifiers } from "~/services/taskIdentifierRegistry.server";
import { type TaskMetadataCache } from "~/services/taskMetadataCache.server";
import { taskMetadataCacheInstance } from "~/services/taskMetadataCacheInstance.server";
import { socketIo } from "../handleSocketIo.server";
import { updateEnvConcurrencyLimits } from "../runQueue.server";
import { PerformDeploymentAlertsService } from "./alerts/performDeploymentAlerts.server";
import { BaseService } from "./baseService.server";
import {
createWorkerResources,
stripBackgroundWorkerMetadataForStorage,
syncDeclarativeSchedules,
} from "./createBackgroundWorker.server";
import { ExecuteTasksWaitingForDeployService } from "./executeTasksWaitingForDeploy";
import { projectPubSub } from "./projectPubSub.server";
import { TimeoutDeploymentService } from "./timeoutDeployment.server";
import { CURRENT_DEPLOYMENT_LABEL, BackgroundWorkerId } from "@trigger.dev/core/v3/isomorphic";
/**
* This service was only used before the new build system was introduced in v3.
* It's now replaced by the CreateDeploymentBackgroundWorkerServiceV4.
*
* @deprecated
*/
export class CreateDeploymentBackgroundWorkerServiceV3 extends BaseService {
private readonly _taskMetaCache: TaskMetadataCache;
constructor(
prisma?: PrismaClientOrTransaction,
replica?: PrismaClientOrTransaction,
taskMetaCache: TaskMetadataCache = taskMetadataCacheInstance
) {
super(prisma, replica);
this._taskMetaCache = taskMetaCache;
}
public async call(
projectRef: string,
environment: AuthenticatedEnvironment,
deploymentId: string,
body: CreateBackgroundWorkerRequestBody
): Promise<BackgroundWorker | undefined> {
return this.traceWithEnv("call", environment, async (span) => {
span.setAttribute("projectRef", projectRef);
const deployment = await this._prisma.workerDeployment.findFirst({
where: {
friendlyId: deploymentId,
},
});
if (!deployment) {
return;
}
if (deployment.status !== "DEPLOYING") {
return;
}
const backgroundWorker = await this._prisma.backgroundWorker.create({
data: {
...BackgroundWorkerId.generate(),
version: deployment.version,
runtimeEnvironmentId: environment.id,
projectId: environment.projectId,
metadata: stripBackgroundWorkerMetadataForStorage(body.metadata),
contentHash: body.metadata.contentHash,
cliVersion: body.metadata.cliPackageVersion,
sdkVersion: body.metadata.packageVersion,
supportsLazyAttempts: body.supportsLazyAttempts,
engine: body.engine,
},
});
//upgrade the project to engine "V2" if it's not already
if (environment.project.engine === "V1" && body.engine === "V2") {
await this._prisma.project.update({
where: {
id: environment.project.id,
},
data: {
engine: "V2",
},
});
}
let workerTaskEntries: Awaited<ReturnType<typeof createWorkerResources>> = [];
try {
workerTaskEntries = await createWorkerResources(
body.metadata,
backgroundWorker,
environment,
this._prisma
);
await syncDeclarativeSchedules(
body.metadata.tasks,
backgroundWorker,
environment,
this._prisma
);
} catch (error) {
const name = error instanceof Error ? error.name : "UnknownError";
const message = error instanceof Error ? error.message : JSON.stringify(error);
await this._prisma.workerDeployment.update({
where: {
id: deployment.id,
},
data: {
status: "FAILED",
failedAt: new Date(),
errorData: {
name,
message,
},
},
});
throw error;
}
// Link the deployment with the background worker
await this._prisma.workerDeployment.update({
where: {
id: deployment.id,
},
data: {
status: "DEPLOYED",
workerId: backgroundWorker.id,
deployedAt: new Date(),
type: backgroundWorker.engine === "V2" ? "MANAGED" : "V1",
},
});
//set this deployment as the current deployment for this environment
await this._prisma.workerDeploymentPromotion.upsert({
where: {
environmentId_label: {
environmentId: environment.id,
label: CURRENT_DEPLOYMENT_LABEL,
},
},
create: {
deploymentId: deployment.id,
environmentId: environment.id,
label: CURRENT_DEPLOYMENT_LABEL,
},
update: {
deploymentId: deployment.id,
},
});
const [syncIdError] = await tryCatch(
syncTaskIdentifiers(
environment.id,
environment.projectId,
backgroundWorker.id,
body.metadata.tasks.map((t) => ({ id: t.id, triggerSource: t.triggerSource }))
)
);
if (syncIdError) {
logger.error("Error syncing task identifiers", { error: syncIdError });
}
// V3 promotes the deployment immediately above, so this worker is now
// current for the env — write both keyspaces atomically. Cache calls
// log+swallow internally. Empty `workerTaskEntries` is intentional: the
// populate methods clear stale hashes for zero-task deploys.
await this._taskMetaCache.populateByCurrentWorker(
environment.id,
backgroundWorker.id,
workerTaskEntries
);
try {
//send a notification that a new worker has been created
await projectPubSub.publish(
`project:${environment.projectId}:env:${environment.id}`,
"WORKER_CREATED",
{
environmentId: environment.id,
environmentType: environment.type,
createdAt: backgroundWorker.createdAt,
taskCount: body.metadata.tasks.length,
type: "deployed",
}
);
await updateEnvConcurrencyLimits(environment);
} catch (err) {
logger.error("Failed to publish WORKER_CREATED event", { err });
}
if (deployment.imageReference) {
socketIo.providerNamespace.emit("PRE_PULL_DEPLOYMENT", {
version: "v1",
imageRef: deployment.imageReference,
shortCode: deployment.shortCode,
// identifiers
deploymentId: deployment.id,
envId: environment.id,
envType: environment.type,
orgId: environment.organizationId,
projectId: deployment.projectId,
});
}
await ExecuteTasksWaitingForDeployService.enqueue(backgroundWorker.id);
await PerformDeploymentAlertsService.enqueue(deployment.id);
await TimeoutDeploymentService.dequeue(deployment.id, this._prisma);
return backgroundWorker;
});
}
}
apps/webapp/app/v3/services/createTaskRunAttempt.server.ts
-278
View File
@@ -1,278 +0,0 @@
import type { V3TaskRunExecution } from "@trigger.dev/core/v3";
import { parsePacket } from "@trigger.dev/core/v3";
import type { TaskRun, TaskRunAttempt } from "@trigger.dev/database";
import { MAX_TASK_RUN_ATTEMPTS } from "~/consts";
import type { PrismaClientOrTransaction } from "~/db.server";
import { $transaction, prisma } from "~/db.server";
import { findQueueInEnvironment } from "~/models/taskQueue.server";
import type { AuthenticatedEnvironment } from "~/services/apiAuth.server";
import { logger } from "~/services/logger.server";
import { reportInvocationUsage } from "~/services/platform.v3.server";
import { generateFriendlyId } from "../friendlyIdentifiers";
import { machinePresetFromConfig, machinePresetFromRun } from "../machinePresets.server";
import { FINAL_RUN_STATUSES } from "../taskStatus";
import { BaseService, ServiceValidationError } from "./baseService.server";
import { CrashTaskRunService } from "./crashTaskRun.server";
import { ExpireEnqueuedRunService } from "./expireEnqueuedRun.server";
import { runStore } from "../runStore.server";
import { controlPlaneResolver } from "~/v3/runOpsMigration/controlPlaneResolver.server";
export class CreateTaskRunAttemptService extends BaseService {
public async call({
runId,
authenticatedEnv,
setToExecuting = true,
startAtZero = false,
}: {
runId: string;
authenticatedEnv?: AuthenticatedEnvironment;
setToExecuting?: boolean;
startAtZero?: boolean;
}): Promise<{
execution: V3TaskRunExecution;
run: TaskRun;
attempt: TaskRunAttempt;
}> {
const environment =
authenticatedEnv ?? (await getAuthenticatedEnvironmentFromRun(runId, this._prisma));
if (!environment) {
throw new ServiceValidationError("Environment not found", 404);
}
const isFriendlyId = runId.startsWith("run_");
return await this.traceWithEnv("call()", environment, async (span) => {
if (isFriendlyId) {
span.setAttribute("taskRunFriendlyId", runId);
} else {
span.setAttribute("taskRunId", runId);
}
const taskRun = await this.runStore.findRun(
{
id: !isFriendlyId ? runId : undefined,
friendlyId: isFriendlyId ? runId : undefined,
runtimeEnvironmentId: environment.id,
},
{
include: {
attempts: {
take: 1,
orderBy: {
number: "desc",
},
},
batchItems: {
include: {
batchTaskRun: {
select: {
friendlyId: true,
},
},
},
},
},
},
this._prisma
);
logger.debug("Creating a task run attempt", { taskRun });
if (!taskRun) {
throw new ServiceValidationError("Task run not found", 404);
}
span.setAttribute("taskRunId", taskRun.id);
span.setAttribute("taskRunFriendlyId", taskRun.friendlyId);
span.setAttribute("taskRunStatus", taskRun.status);
if (taskRun.status === "CANCELED") {
throw new ServiceValidationError("Task run is cancelled", 400);
}
// If the run is finalized, it's pointless to create another attempt
if (FINAL_RUN_STATUSES.includes(taskRun.status)) {
throw new ServiceValidationError("Task run is already finished", 400);
}
const lockedWorker = await controlPlaneResolver.resolveRunLockedWorker({
lockedById: taskRun.lockedById,
});
const lockedBy = lockedWorker?.lockedBy;
if (!lockedBy) {
throw new ServiceValidationError("Task run is not locked", 400);
}
const queue = await findQueueInEnvironment(taskRun.queue, environment.id, lockedBy.id);
if (!queue) {
throw new ServiceValidationError("Queue not found", 404);
}
const nextAttemptNumber = taskRun.attempts[0]
? taskRun.attempts[0].number + 1
: startAtZero
? 0
: 1;
if (nextAttemptNumber > MAX_TASK_RUN_ATTEMPTS) {
const service = new CrashTaskRunService(this._prisma);
await service.call(taskRun.id, {
reason: lockedBy.worker.supportsLazyAttempts
? "Max attempts reached."
: "Max attempts reached. Please upgrade your CLI and SDK.",
});
throw new ServiceValidationError("Max attempts reached", 400);
}
const taskRunAttempt = await $transaction(this._prisma, "create attempt", async (tx) => {
const taskRunAttempt = await tx.taskRunAttempt.create({
data: {
number: nextAttemptNumber,
friendlyId: generateFriendlyId("attempt"),
taskRunId: taskRun.id,
startedAt: new Date(),
backgroundWorkerId: lockedBy.worker.id,
backgroundWorkerTaskId: lockedBy.id,
status: setToExecuting ? "EXECUTING" : "PENDING",
queueId: queue.id,
runtimeEnvironmentId: environment.id,
},
});
await tx.taskRun.update({
where: {
id: taskRun.id,
},
data: {
status: setToExecuting ? "EXECUTING" : undefined,
executedAt: taskRun.executedAt ?? new Date(),
attemptNumber: nextAttemptNumber,
},
});
if (taskRun.ttl) {
await ExpireEnqueuedRunService.ack(taskRun.id, tx);
}
return taskRunAttempt;
});
if (!taskRunAttempt) {
logger.error("Failed to create task run attempt", { runId: taskRun.id, nextAttemptNumber });
throw new ServiceValidationError("Failed to create task run attempt", 500);
}
if (taskRunAttempt.number === 1 && taskRun.baseCostInCents > 0) {
await reportInvocationUsage(environment.organizationId, taskRun.baseCostInCents, {
runId: taskRun.id,
});
}
const machinePreset =
machinePresetFromRun(taskRun) ?? machinePresetFromConfig(lockedBy.machineConfig ?? {});
const metadata = await parsePacket({
data: taskRun.metadata ?? undefined,
dataType: taskRun.metadataType,
});
const execution: V3TaskRunExecution = {
task: {
id: lockedBy.slug,
filePath: lockedBy.filePath,
exportName: lockedBy.exportName ?? "@deprecated",
},
attempt: {
id: taskRunAttempt.friendlyId,
number: taskRunAttempt.number,
startedAt: taskRunAttempt.startedAt ?? taskRunAttempt.createdAt,
backgroundWorkerId: lockedBy.worker.id,
backgroundWorkerTaskId: lockedBy.id,
status: "EXECUTING" as const,
},
run: {
id: taskRun.friendlyId,
payload: taskRun.payload,
payloadType: taskRun.payloadType,
context: taskRun.context,
createdAt: taskRun.createdAt,
tags: taskRun.runTags ?? [],
isTest: taskRun.isTest,
isReplay: !!taskRun.replayedFromTaskRunFriendlyId,
idempotencyKey: taskRun.idempotencyKey ?? undefined,
startedAt: taskRun.startedAt ?? taskRun.createdAt,
durationMs: taskRun.usageDurationMs,
costInCents: taskRun.costInCents,
baseCostInCents: taskRun.baseCostInCents,
maxAttempts: taskRun.maxAttempts ?? undefined,
version: lockedBy.worker.version,
metadata,
maxDuration: taskRun.maxDurationInSeconds ?? undefined,
},
queue: {
id: queue.friendlyId,
name: queue.name,
},
environment: {
id: environment.id,
slug: environment.slug,
type: environment.type,
},
organization: {
id: environment.organization.id,
slug: environment.organization.slug,
name: environment.organization.title,
},
project: {
id: environment.project.id,
ref: environment.project.externalRef,
slug: environment.project.slug,
name: environment.project.name,
},
batch:
taskRun.batchItems[0] && taskRun.batchItems[0].batchTaskRun
? { id: taskRun.batchItems[0].batchTaskRun.friendlyId }
: undefined,
machine: machinePreset,
};
return {
execution,
run: taskRun,
attempt: taskRunAttempt,
};
});
}
}
async function getAuthenticatedEnvironmentFromRun(
friendlyId: string,
prismaClient?: PrismaClientOrTransaction
) {
const isFriendlyId = friendlyId.startsWith("run_");
const taskRun = await runStore.findRun(
{
id: !isFriendlyId ? friendlyId : undefined,
friendlyId: isFriendlyId ? friendlyId : undefined,
},
{
select: {
runtimeEnvironmentId: true,
},
},
prismaClient ?? prisma
);
if (!taskRun) {
return;
}
return (
(await controlPlaneResolver.resolveAuthenticatedEnv(taskRun.runtimeEnvironmentId)) ?? undefined
);
}
apps/webapp/app/v3/services/deploymentIndexFailed.server.ts
-107
View File
@@ -1,107 +0,0 @@
import { PerformDeploymentAlertsService } from "./alerts/performDeploymentAlerts.server";
import { BaseService } from "./baseService.server";
import { logger } from "~/services/logger.server";
import { type WorkerDeploymentStatus } from "@trigger.dev/database";
import { DeploymentService } from "./deployment.server";
import { recordDeploymentOutcome } from "./recordDeploymentOutcome.server";
const FINAL_DEPLOYMENT_STATUSES: WorkerDeploymentStatus[] = [
"CANCELED",
"DEPLOYED",
"FAILED",
"TIMED_OUT",
];
export class DeploymentIndexFailed extends BaseService {
public async call(
maybeFriendlyId: string,
error: {
name: string;
message: string;
stack?: string;
stderr?: string;
},
overrideCompletion = false
) {
const isFriendlyId = maybeFriendlyId.startsWith("deployment_");
const deployment = await this._prisma.workerDeployment.findFirst({
where: isFriendlyId
? {
friendlyId: maybeFriendlyId,
}
: {
id: maybeFriendlyId,
},
include: {
environment: {
include: {
project: true,
},
},
},
});
if (!deployment) {
logger.error("Worker deployment not found", { maybeFriendlyId });
return;
}
if (FINAL_DEPLOYMENT_STATUSES.includes(deployment.status)) {
if (overrideCompletion) {
logger.error("No support for overriding final deployment statuses just yet", {
id: deployment.id,
status: deployment.status,
previousError: deployment.errorData,
incomingError: error,
});
}
logger.error("Worker deployment already in final state", {
id: deployment.id,
status: deployment.status,
});
return;
}
const failedDeployment = await this._prisma.workerDeployment.update({
where: {
id: deployment.id,
},
data: {
status: "FAILED",
failedAt: new Date(),
errorData: error,
},
});
recordDeploymentOutcome({
status: "FAILED",
deploymentFriendlyId: deployment.friendlyId,
organizationId: deployment.environment.project.organizationId,
projectId: deployment.environment.projectId,
environmentId: deployment.environmentId,
environmentType: deployment.environment.type,
reason: error.message,
});
const deploymentService = new DeploymentService();
await deploymentService
.appendToEventLog(deployment.environment.project, failedDeployment, [
{
type: "finalized",
data: {
result: "failed",
message: error.message,
},
},
])
.orTee((error) => {
logger.error("Failed to append failed deployment event to event log", { error });
});
await PerformDeploymentAlertsService.enqueue(failedDeployment.id);
return failedDeployment;
}
}
apps/webapp/app/v3/services/enqueueDelayedRun.server.ts
-121
View File
@@ -1,121 +0,0 @@
import { parseNaturalLanguageDuration } from "@trigger.dev/core/v3/isomorphic";
import { logger } from "~/services/logger.server";
import { workerQueue } from "~/services/worker.server";
import { commonWorker } from "../commonWorker.server";
import { BaseService } from "./baseService.server";
import { enqueueRun } from "./enqueueRun.server";
import { ExpireEnqueuedRunService } from "./expireEnqueuedRun.server";
import { controlPlaneResolver } from "~/v3/runOpsMigration/controlPlaneResolver.server";
import { isV3Disabled } from "../engineDeprecation.server";
export class EnqueueDelayedRunService extends BaseService {
public static async enqueue(runId: string, runAt?: Date) {
await commonWorker.enqueue({
job: "v3.enqueueDelayedRun",
payload: { runId },
availableAt: runAt,
id: `v3.enqueueDelayed:${runId}`,
});
}
public static async reschedule(runId: string, runAt?: Date) {
// We have to do this for now because it's possible that the workerQueue
// was used when the run was first delayed, and EnqueueDelayedRunService.reschedule
// is called from RescheduleTaskRunService, which allows the runAt to be changed
// so if we don't dequeue the old job, we might end up with multiple jobs
await workerQueue.dequeue(`v3.enqueueDelayedRun.${runId}`);
await commonWorker.enqueue({
job: "v3.enqueueDelayedRun",
payload: { runId },
availableAt: runAt,
id: `v3.enqueueDelayed:${runId}`,
});
}
public async call(runId: string) {
const run = await this.runStore.findRun(
{
id: runId,
},
{
include: {
dependency: {
include: {
dependentBatchRun: {
include: {
dependentTaskAttempt: {
include: {
taskRun: true,
},
},
},
},
dependentAttempt: {
include: {
taskRun: true,
},
},
},
},
},
},
this._prisma
);
if (!run) {
logger.debug("Could not find delayed run to enqueue", {
runId,
});
return;
}
// v3 (engine V1) shutdown: don't enqueue delayed V1 runs into MarQS. v4 is unaffected.
if (isV3Disabled() && run.engine === "V1") {
logger.debug("[EnqueueDelayedRunService] Skipping enqueue for shut-down v3 run", { runId });
return;
}
const env = await controlPlaneResolver.resolveAuthenticatedEnv(run.runtimeEnvironmentId);
if (!env) {
logger.debug("EnqueueDelayedRunService: environment not found", { runId });
return;
}
if (run.status !== "DELAYED") {
logger.debug("Delayed run cannot be enqueued because it's not in DELAYED status", {
run,
});
return;
}
await this._prisma.taskRun.update({
where: {
id: run.id,
},
data: {
status: "PENDING",
queuedAt: new Date(),
},
});
if (run.ttl) {
const expireAt = parseNaturalLanguageDuration(run.ttl);
if (expireAt) {
await ExpireEnqueuedRunService.enqueue(run.id, expireAt);
}
}
await enqueueRun({
env,
run: run,
dependentRun:
run.dependency?.dependentAttempt?.taskRun ??
run.dependency?.dependentBatchRun?.dependentTaskAttempt?.taskRun,
});
}
}
apps/webapp/app/v3/services/enqueueRun.server.ts
-67
View File
@@ -1,67 +0,0 @@
import type { TaskRunError } from "@trigger.dev/core/v3/schemas";
import { TaskRunErrorCodes } from "@trigger.dev/core/v3/schemas";
import type { TaskRun } from "@trigger.dev/database";
import type { AuthenticatedEnvironment } from "~/services/apiAuth.server";
import { marqs } from "../marqs/index.server";
export type EnqueueRunOptions = {
env: AuthenticatedEnvironment;
run: TaskRun;
dependentRun?: { queue: string; id: string };
};
export type EnqueueRunResult =
| {
ok: true;
}
| {
ok: false;
error: TaskRunError;
};
export async function enqueueRun({
env,
run,
dependentRun,
}: EnqueueRunOptions): Promise<EnqueueRunResult> {
// If this is a triggerAndWait or batchTriggerAndWait,
// we need to add the parent run to the reserve concurrency set
// to free up concurrency for the children to run
// In the case of a recursive queue, reserving concurrency can fail, which means there is a deadlock and we need to fail the run
// TODO: reserveConcurrency can fail because of a deadlock, we need to handle that case
const wasEnqueued = await marqs.enqueueMessage(
env,
run.queue,
run.id,
{
type: "EXECUTE",
taskIdentifier: run.taskIdentifier,
projectId: env.projectId,
environmentId: env.id,
environmentType: env.type,
},
run.concurrencyKey ?? undefined,
run.queueTimestamp ?? undefined,
dependentRun
? { messageId: dependentRun.id, recursiveQueue: dependentRun.queue === run.queue }
: undefined
);
if (!wasEnqueued) {
const error = {
type: "INTERNAL_ERROR",
code: TaskRunErrorCodes.RECURSIVE_WAIT_DEADLOCK,
message: `This run will never execute because it was triggered recursively and the task has no remaining concurrency available`,
} satisfies TaskRunError;
return {
ok: false,
error,
};
}
return {
ok: true,
};
}
apps/webapp/app/v3/services/executeTasksWaitingForDeploy.ts
-142
View File
@@ -1,142 +0,0 @@
import { ownerEngine } from "@trigger.dev/core/v3/isomorphic";
import { env } from "~/env.server";
import { logger } from "~/services/logger.server";
import { marqs } from "~/v3/marqs/index.server";
import { commonWorker } from "../commonWorker.server";
import { BaseService } from "./baseService.server";
export class ExecuteTasksWaitingForDeployService extends BaseService {
public async call(backgroundWorkerId: string) {
// Kill-switch for the legacy V1 WAITING_FOR_DEPLOY drain. Set to "1" to
// neuter any jobs already enqueued (V2 has its own PENDING_VERSION path).
if (env.LEGACY_RUN_ENGINE_WAITING_FOR_DEPLOY_DISABLED === "1") {
return;
}
const backgroundWorker = await this._prisma.backgroundWorker.findFirst({
where: {
id: backgroundWorkerId,
},
include: {
runtimeEnvironment: {
include: {
project: true,
organization: true,
},
},
tasks: {
select: {
slug: true,
},
},
},
});
if (!backgroundWorker) {
logger.error("Background worker not found", { id: backgroundWorkerId });
return;
}
const maxCount = env.LEGACY_RUN_ENGINE_WAITING_FOR_DEPLOY_BATCH_SIZE;
const runsWaitingForDeploy = await this.runStore.findRuns(
{
where: {
runtimeEnvironmentId: backgroundWorker.runtimeEnvironmentId,
projectId: backgroundWorker.projectId,
status: "WAITING_FOR_DEPLOY",
taskIdentifier: {
in: backgroundWorker.tasks.map((task) => task.slug),
},
},
orderBy: {
createdAt: "asc",
},
select: {
id: true,
status: true,
taskIdentifier: true,
concurrencyKey: true,
queue: true,
updatedAt: true,
createdAt: true,
},
take: maxCount + 1,
},
this._replica
);
if (!runsWaitingForDeploy.length) {
return;
}
// Defense-in-depth: the open-predicate findRuns fan-out can select runs from
// either DB, but the status flip below is a single control-plane updateMany. A
// run-ops id (NEW-resident) run can only reach WAITING_FOR_DEPLOY via a misconfiguration
// (it is a V1/cuid-only status — V2 uses PENDING_VERSION). Surface it loudly rather
// than silently strand the run, and only mutate the LEGACY-resident runs the
// control-plane client can actually reach.
const newResidentRuns = runsWaitingForDeploy.filter((run) => ownerEngine(run.id) === "NEW");
if (newResidentRuns.length) {
logger.error(
"WAITING_FOR_DEPLOY selected NEW-resident runs; skipping their control-plane status flip",
{ runIds: newResidentRuns.map((run) => run.id) }
);
}
const legacyRuns = runsWaitingForDeploy.filter((run) => !newResidentRuns.includes(run));
const pendingRuns = await this._prisma.taskRun.updateMany({
where: {
id: {
in: legacyRuns.map((run) => run.id),
},
},
data: {
status: "PENDING",
},
});
if (pendingRuns.count) {
logger.debug("Task runs waiting for deploy are now ready for execution", {
tasks: legacyRuns.map((run) => run.id),
total: pendingRuns.count,
});
}
// Only enqueue the runs whose status was actually flipped (the legacy set) — never
// marqs-enqueue a NEW-resident run we couldn't transition out of WAITING_FOR_DEPLOY.
for (const run of legacyRuns) {
await marqs?.enqueueMessage(
backgroundWorker.runtimeEnvironment,
run.queue,
run.id,
{
type: "EXECUTE",
taskIdentifier: run.taskIdentifier,
projectId: backgroundWorker.runtimeEnvironment.projectId,
environmentId: backgroundWorker.runtimeEnvironment.id,
environmentType: backgroundWorker.runtimeEnvironment.type,
},
run.concurrencyKey ?? undefined
);
}
if (runsWaitingForDeploy.length > maxCount) {
await ExecuteTasksWaitingForDeployService.enqueue(
backgroundWorkerId,
new Date(Date.now() + env.LEGACY_RUN_ENGINE_WAITING_FOR_DEPLOY_BATCH_STAGGER_MS)
);
}
}
static async enqueue(backgroundWorkerId: string, runAt?: Date) {
return await commonWorker.enqueue({
id: `v3.executeTasksWaitingForDeploy:${backgroundWorkerId}`,
job: "v3.executeTasksWaitingForDeploy",
payload: {
backgroundWorkerId,
},
availableAt: runAt,
});
}
}
apps/webapp/app/v3/services/expireEnqueuedRun.server.ts
-132
View File
@@ -1,132 +0,0 @@
import type { PrismaClientOrTransaction } from "~/db.server";
import { logger } from "~/services/logger.server";
import { commonWorker } from "../commonWorker.server";
import { BaseService } from "./baseService.server";
import { FinalizeTaskRunService } from "./finalizeTaskRun.server";
import { tryCatch } from "@trigger.dev/core/utils";
import { getEventRepositoryForStore } from "../eventRepository/index.server";
import { controlPlaneResolver } from "~/v3/runOpsMigration/controlPlaneResolver.server";
import { isV3Disabled } from "../engineDeprecation.server";
export class ExpireEnqueuedRunService extends BaseService {
public static async ack(runId: string, tx?: PrismaClientOrTransaction) {
// We don't "dequeue" from the workerQueue here because it would be redundant and if this service
// is called for a run that has already started, nothing happens
await commonWorker.ack(`v3.expireRun:${runId}`);
}
public static async enqueue(runId: string, runAt?: Date) {
return await commonWorker.enqueue({
job: "v3.expireRun",
payload: { runId },
availableAt: runAt,
id: `v3.expireRun:${runId}`,
});
}
public async call(runId: string) {
const run = await this.runStore.findRun(
{
id: runId,
},
{
select: {
id: true,
status: true,
engine: true,
lockedAt: true,
ttl: true,
taskEventStore: true,
runtimeEnvironmentId: true,
friendlyId: true,
traceId: true,
spanId: true,
parentSpanId: true,
createdAt: true,
completedAt: true,
taskIdentifier: true,
projectId: true,
organizationId: true,
isTest: true,
},
},
this._prisma
);
if (!run) {
logger.debug("Could not find enqueued run to expire", {
runId,
});
return;
}
// v3 (engine V1) shutdown: skip expiring abandoned V1 runs. v4 is unaffected.
if (isV3Disabled() && run.engine === "V1") {
logger.debug("[ExpireEnqueuedRunService] Skipping expiry for shut-down v3 run", { runId });
return;
}
const env = await controlPlaneResolver.resolveEnv(run.runtimeEnvironmentId);
if (!env) {
logger.debug("ExpireEnqueuedRunService: environment not found", { runId });
return;
}
if (run.status !== "PENDING") {
logger.debug("Run cannot be expired because it's not in PENDING status", {
run,
});
return;
}
if (run.lockedAt) {
logger.debug("Run cannot be expired because it's locked", {
run,
});
return;
}
logger.debug("Expiring enqueued run", {
run,
});
const finalizeService = new FinalizeTaskRunService();
await finalizeService.call({
id: run.id,
status: "EXPIRED",
expiredAt: new Date(),
completedAt: new Date(),
attemptStatus: "FAILED",
error: {
type: "STRING_ERROR",
raw: `Run expired because the TTL (${run.ttl}) was reached`,
},
});
const eventRepository = await getEventRepositoryForStore(
run.taskEventStore,
env.organizationId
);
if (run.ttl) {
const [completeExpiredRunEventError] = await tryCatch(
eventRepository.completeExpiredRunEvent({
run,
endTime: new Date(),
ttl: run.ttl,
})
);
if (completeExpiredRunEventError) {
logger.error("[ExpireEnqueuedRunService] Failed to complete expired run event", {
error: completeExpiredRunEventError,
runId: run.id,
});
}
}
}
}
apps/webapp/app/v3/services/finalizeDeployment.server.ts
-15
View File
@@ -1,7 +1,6 @@
import type { FinalizeDeploymentRequestBody } from "@trigger.dev/core/v3/schemas";
import type { AuthenticatedEnvironment } from "~/services/apiAuth.server";
import { logger } from "~/services/logger.server";
import { socketIo } from "../handleSocketIo.server";
import { updateEnvConcurrencyLimits } from "../runQueue.server";
import { PerformDeploymentAlertsService } from "./alerts/performDeploymentAlerts.server";
import { BaseService, ServiceValidationError } from "./baseService.server";
@@ -129,20 +128,6 @@ export class FinalizeDeploymentService extends BaseService {
logger.error("Failed to publish WORKER_CREATED event", { err });
}
if (finalizedDeployment.imageReference) {
socketIo.providerNamespace.emit("PRE_PULL_DEPLOYMENT", {
version: "v1",
imageRef: finalizedDeployment.imageReference,
shortCode: finalizedDeployment.shortCode,
// identifiers
deploymentId: finalizedDeployment.id,
envId: authenticatedEnv.id,
envType: authenticatedEnv.type,
orgId: authenticatedEnv.organizationId,
projectId: finalizedDeployment.projectId,
});
}
if (deployment.worker.engine === "V2") {
const [schedulePendingVersionsError] = await tryCatch(
engine.scheduleEnqueueRunsForBackgroundWorker(deployment.worker.id)
apps/webapp/app/v3/services/finalizeTaskRun.server.ts
-361
View File
@@ -1,361 +0,0 @@
import { type FlushedRunMetadata, type TaskRunError, sanitizeError } from "@trigger.dev/core/v3";
import { type Prisma, type TaskRun } from "@trigger.dev/database";
import { findQueueInEnvironment } from "~/models/taskQueue.server";
import type { AuthenticatedEnvironment } from "~/services/apiAuth.server";
import { logger } from "~/services/logger.server";
import { updateMetadataService } from "~/services/metadata/updateMetadataInstance.server";
import { marqs } from "~/v3/marqs/index.server";
import { generateFriendlyId } from "../friendlyIdentifiers";
import { socketIo } from "../handleSocketIo.server";
import {
type FINAL_ATTEMPT_STATUSES,
isFailedRunStatus,
isFatalRunStatus,
type FINAL_RUN_STATUSES,
} from "../taskStatus";
import { PerformTaskRunAlertsService } from "./alerts/performTaskRunAlerts.server";
import { BaseService } from "./baseService.server";
import { completeBatchTaskRunItemV3 } from "./batchTriggerV3.server";
import { ExpireEnqueuedRunService } from "./expireEnqueuedRun.server";
import { ResumeBatchRunService } from "./resumeBatchRun.server";
import { ResumeDependentParentsService } from "./resumeDependentParents.server";
import { controlPlaneResolver } from "~/v3/runOpsMigration/controlPlaneResolver.server";
type BaseInput = {
id: string;
status?: FINAL_RUN_STATUSES;
expiredAt?: Date;
completedAt?: Date;
attemptStatus?: FINAL_ATTEMPT_STATUSES;
error?: TaskRunError;
metadata?: FlushedRunMetadata;
env?: AuthenticatedEnvironment;
bulkActionId?: string;
};
type InputWithInclude<T extends Prisma.TaskRunInclude> = BaseInput & {
include: T;
};
type InputWithoutInclude = BaseInput & {
include?: undefined;
};
type Output<T extends Prisma.TaskRunInclude | undefined> = T extends Prisma.TaskRunInclude
? Prisma.TaskRunGetPayload<{ include: T }>
: TaskRun;
export class FinalizeTaskRunService extends BaseService {
public async call<T extends Prisma.TaskRunInclude | undefined>({
id,
status,
expiredAt,
completedAt,
bulkActionId,
include,
attemptStatus,
error,
metadata,
env,
}: T extends Prisma.TaskRunInclude ? InputWithInclude<T> : InputWithoutInclude): Promise<
Output<T>
> {
logger.debug("Finalizing run marqs ack", {
id,
status,
expiredAt,
completedAt,
});
await marqs?.acknowledgeMessage(id, "FinalTaskRunService call");
logger.debug("Finalizing run updating run status", {
id,
status,
expiredAt,
completedAt,
});
if (metadata) {
try {
await updateMetadataService.call(id, metadata, env);
} catch (e) {
logger.error("[FinalizeTaskRunService] Failed to update metadata", {
taskRun: id,
error:
e instanceof Error
? {
name: e.name,
message: e.message,
stack: e.stack,
}
: e,
});
}
}
// Error is written in the same update as the status: a separate later write races realtime,
// which shuts the stream down on the final status before the error lands, losing it.
const taskRunError = error ? sanitizeError(error) : undefined;
const run = await this._prisma.taskRun.update({
where: { id },
data: {
status,
expiredAt,
completedAt,
error: taskRunError,
bulkActionGroupIds: bulkActionId
? {
push: bulkActionId,
}
: undefined,
},
...(include ? { include } : {}),
});
if (run.ttl) {
await ExpireEnqueuedRunService.ack(run.id);
}
if (attemptStatus || error) {
await this.finalizeAttempt({ attemptStatus, error, run });
}
try {
await this.#finalizeBatch(run);
} catch (finalizeBatchError) {
logger.error("FinalizeTaskRunService: Failed to finalize batch", {
runId: run.id,
error: finalizeBatchError,
});
}
const resumeService = new ResumeDependentParentsService(this._prisma);
const result = await resumeService.call({ id: run.id });
if (result.success) {
logger.log("FinalizeTaskRunService: Resumed dependent parents", { result, run: run.id });
} else {
logger.error("FinalizeTaskRunService: Failed to resume dependent parents", {
result,
run: run.id,
});
}
if (isFailedRunStatus(run.status)) {
await PerformTaskRunAlertsService.enqueue(run.id);
}
if (isFatalRunStatus(run.status)) {
logger.warn("FinalizeTaskRunService: Fatal status", { runId: run.id, status: run.status });
const extendedRun = await this.runStore.findRun(
{ id: run.id },
{
select: {
id: true,
runtimeEnvironmentId: true,
lockedToVersionId: true,
},
},
this._prisma
);
const extendedEnv = extendedRun
? await controlPlaneResolver.resolveEnv(extendedRun.runtimeEnvironmentId)
: null;
const extendedLockedWorker = extendedRun
? await controlPlaneResolver.resolveRunLockedWorker({
lockedToVersionId: extendedRun.lockedToVersionId,
})
: null;
if (extendedRun && extendedEnv && extendedEnv.type !== "DEVELOPMENT") {
logger.warn("FinalizeTaskRunService: Fatal status, requesting worker exit", {
runId: run.id,
status: run.status,
});
// Signal to exit any leftover containers
socketIo.coordinatorNamespace.emit("REQUEST_RUN_CANCELLATION", {
version: "v1",
runId: run.id,
// Give the run a few seconds to exit to complete any flushing etc
delayInMs: extendedLockedWorker?.lockedToVersion?.supportsLazyAttempts
? 5_000
: undefined,
});
}
}
return run as Output<T>;
}
async #finalizeBatch(run: TaskRun) {
if (!run.batchId) {
return;
}
logger.debug("FinalizeTaskRunService: Finalizing batch", { runId: run.id });
const environment = await this._prisma.runtimeEnvironment.findFirst({
where: {
id: run.runtimeEnvironmentId,
},
});
if (!environment) {
return;
}
const batchItems = await this._prisma.batchTaskRunItem.findMany({
where: {
taskRunId: run.id,
},
include: {
batchTaskRun: {
select: {
id: true,
dependentTaskAttemptId: true,
batchVersion: true,
},
},
},
});
if (batchItems.length === 0) {
return;
}
if (batchItems.length > 10) {
logger.error("FinalizeTaskRunService: More than 10 batch items", {
runId: run.id,
batchItems: batchItems.length,
});
return;
}
for (const item of batchItems) {
// Don't do anything if this is a batchTriggerAndWait in a deployed task
// As that is being handled in resumeDependentParents and resumeTaskRunDependencies
if (environment.type !== "DEVELOPMENT" && item.batchTaskRun.dependentTaskAttemptId) {
continue;
}
if (item.batchTaskRun.batchVersion === "v3") {
await completeBatchTaskRunItemV3(item.id, item.batchTaskRunId, this._prisma);
} else {
// THIS IS DEPRECATED and only happens with batchVersion != v3
await this._prisma.batchTaskRunItem.update({
where: {
id: item.id,
},
data: {
status: "COMPLETED",
},
});
// This won't resume because this batch does not have a dependent task attempt ID
// or is in development, but this service will mark the batch as completed
await ResumeBatchRunService.enqueue(item.batchTaskRunId, false);
}
}
}
async finalizeAttempt({
attemptStatus,
error,
run,
}: {
attemptStatus?: FINAL_ATTEMPT_STATUSES;
error?: TaskRunError;
run: TaskRun;
}) {
if (!attemptStatus && !error) {
logger.error("FinalizeTaskRunService: No attemptStatus or error provided", { runId: run.id });
return;
}
const latestAttempt = await this._prisma.taskRunAttempt.findFirst({
where: { taskRunId: run.id },
orderBy: { id: "desc" },
take: 1,
});
if (latestAttempt) {
logger.debug("Finalizing run attempt", {
id: latestAttempt.id,
status: attemptStatus,
error,
});
await this._prisma.taskRunAttempt.update({
where: { id: latestAttempt.id },
data: { status: attemptStatus, error: error ? sanitizeError(error) : undefined },
});
return;
}
// There's no attempt, so create one
logger.debug("Finalizing run no attempt found", {
runId: run.id,
attemptStatus,
error,
});
if (!run.lockedById) {
// This happens when a run is expired or was cancelled before an attempt, it's not a problem
logger.info(
"FinalizeTaskRunService: No lockedById, so can't get the BackgroundWorkerTask. Not creating an attempt.",
{ runId: run.id, status: run.status }
);
return;
}
const workerTask = await this._prisma.backgroundWorkerTask.findFirst({
select: {
id: true,
workerId: true,
runtimeEnvironmentId: true,
queueConfig: true,
},
where: {
id: run.lockedById,
},
});
if (!workerTask) {
logger.error("FinalizeTaskRunService: No worker task found", { runId: run.id });
return;
}
const queue = await findQueueInEnvironment(
run.queue,
workerTask.runtimeEnvironmentId,
workerTask.id,
workerTask
);
if (!queue) {
logger.error("FinalizeTaskRunService: No queue found", { runId: run.id });
return;
}
await this._prisma.taskRunAttempt.create({
data: {
number: 1,
friendlyId: generateFriendlyId("attempt"),
taskRunId: run.id,
backgroundWorkerId: workerTask?.workerId,
backgroundWorkerTaskId: workerTask?.id,
queueId: queue.id,
runtimeEnvironmentId: workerTask.runtimeEnvironmentId,
status: attemptStatus,
error: error ? sanitizeError(error) : undefined,
},
});
}
}
apps/webapp/app/v3/services/rescheduleTaskRun.server.ts
+9 -8
View File
@@ -1,12 +1,18 @@
import type { RescheduleRunRequestBody } from "@trigger.dev/core/v3";
import type { TaskRun } from "@trigger.dev/database";
import { parseDelay } from "~/utils/delays";
import { V3_TRIGGER_DEPRECATION_MESSAGE } from "../engineDeprecation.server";
import { BaseService, ServiceValidationError } from "./baseService.server";
import { EnqueueDelayedRunService } from "./enqueueDelayedRun.server";
import { engine } from "../runEngine.server";
export class RescheduleTaskRunService extends BaseService {
public async call(taskRun: TaskRun, body: RescheduleRunRequestBody) {
// v3 (engine V1) is retired: reject rescheduling a legacy V1 delayed run
// gracefully instead of enqueuing into the removed V1 worker.
if (taskRun.engine === "V1") {
throw new ServiceValidationError(V3_TRIGGER_DEPRECATION_MESSAGE);
}
if (taskRun.status !== "DELAYED") {
throw new ServiceValidationError("Cannot reschedule a run that is not delayed");
}
@@ -17,7 +23,7 @@ export class RescheduleTaskRunService extends BaseService {
throw new ServiceValidationError(`Invalid delay: ${body.delay}`);
}
const updatedRun = await this.runStore.rescheduleRun(
await this.runStore.rescheduleRun(
taskRun.id,
{
delayUntil: delay,
@@ -26,11 +32,6 @@ export class RescheduleTaskRunService extends BaseService {
this._prisma
);
if (updatedRun.engine === "V1") {
await EnqueueDelayedRunService.reschedule(taskRun.id, delay);
return updatedRun;
} else {
return engine.rescheduleDelayedRun({ runId: taskRun.id, delayUntil: delay });
}
return engine.rescheduleDelayedRun({ runId: taskRun.id, delayUntil: delay });
}
}
apps/webapp/app/v3/services/restoreCheckpoint.server.ts
-137
View File
@@ -1,137 +0,0 @@
import { type Checkpoint } from "@trigger.dev/database";
import { logger } from "~/services/logger.server";
import { socketIo } from "../handleSocketIo.server";
import { machinePresetFromConfig, machinePresetFromRun } from "../machinePresets.server";
import { BaseService } from "./baseService.server";
import { CreateCheckpointRestoreEventService } from "./createCheckpointRestoreEvent.server";
import { isRestorableAttemptStatus, isRestorableRunStatus } from "../taskStatus";
export class RestoreCheckpointService extends BaseService {
public async call(params: {
eventId: string;
isRetry?: boolean;
}): Promise<Checkpoint | undefined> {
logger.debug(`Restoring checkpoint`, params);
const checkpointEvent = await this._prisma.checkpointRestoreEvent.findFirst({
where: {
id: params.eventId,
type: "CHECKPOINT",
},
include: {
checkpoint: {
include: {
run: {
select: {
status: true,
machinePreset: true,
},
},
attempt: {
select: {
status: true,
backgroundWorkerTask: {
select: {
machineConfig: true,
},
},
},
},
runtimeEnvironment: true,
},
},
},
});
if (!checkpointEvent) {
logger.error("Checkpoint event not found", { eventId: params.eventId });
return;
}
const checkpoint = checkpointEvent.checkpoint;
if (!isRestorableRunStatus(checkpoint.run.status)) {
logger.error("Run is unrestorable", {
eventId: params.eventId,
runId: checkpoint.runId,
runStatus: checkpoint.run.status,
attemptId: checkpoint.attemptId,
});
return;
}
if (!isRestorableAttemptStatus(checkpoint.attempt.status) && !params.isRetry) {
logger.error("Attempt is unrestorable", {
eventId: params.eventId,
runId: checkpoint.runId,
attemptId: checkpoint.attemptId,
attemptStatus: checkpoint.attempt.status,
});
return;
}
const machine =
machinePresetFromRun(checkpoint.run) ??
machinePresetFromConfig(checkpoint.attempt.backgroundWorkerTask.machineConfig ?? {});
const restoreEvent = await this._prisma.checkpointRestoreEvent.findFirst({
where: {
checkpointId: checkpoint.id,
type: "RESTORE",
},
});
if (restoreEvent) {
logger.warn("Restore event already exists", {
runId: checkpoint.runId,
attemptId: checkpoint.attemptId,
checkpointId: checkpoint.id,
restoreEventId: restoreEvent.id,
});
return;
}
const eventService = new CreateCheckpointRestoreEventService(this._prisma);
await eventService.restore({ checkpointId: checkpoint.id });
socketIo.providerNamespace.emit("RESTORE", {
version: "v1",
type: checkpoint.type,
location: checkpoint.location,
reason: checkpoint.reason ?? undefined,
imageRef: checkpoint.imageRef,
machine,
attemptNumber: checkpoint.attemptNumber ?? undefined,
// identifiers
checkpointId: checkpoint.id,
envId: checkpoint.runtimeEnvironment.id,
envType: checkpoint.runtimeEnvironment.type,
orgId: checkpoint.runtimeEnvironment.organizationId,
projectId: checkpoint.runtimeEnvironment.projectId,
runId: checkpoint.runId,
});
return checkpoint;
}
async getLastCheckpointEventIfUnrestored(runId: string) {
const event = await this._prisma.checkpointRestoreEvent.findFirst({
where: {
runId,
},
take: 1,
orderBy: {
createdAt: "desc",
},
});
if (!event) {
return;
}
if (event.type === "CHECKPOINT") {
return event;
}
}
}
apps/webapp/app/v3/services/resumeAttempt.server.ts
-290
View File
@@ -1,290 +0,0 @@
import type {
CoordinatorToPlatformMessages,
TaskRunExecution,
TaskRunExecutionResult,
} from "@trigger.dev/core/v3";
import type { InferSocketMessageSchema } from "@trigger.dev/core/v3/zodSocket";
import type { Prisma, TaskRunAttempt } from "@trigger.dev/database";
import { logger } from "~/services/logger.server";
import { marqs } from "~/v3/marqs/index.server";
import { socketIo } from "../handleSocketIo.server";
import { sharedQueueTasks } from "../marqs/sharedQueueConsumer.server";
import { FINAL_ATTEMPT_STATUSES, isFinalRunStatus } from "../taskStatus";
import { BaseService } from "./baseService.server";
export class ResumeAttemptService extends BaseService {
private _logger = logger;
public async call(
params: InferSocketMessageSchema<typeof CoordinatorToPlatformMessages, "READY_FOR_RESUME">
): Promise<void> {
this._logger.debug(`ResumeAttemptService.call()`, params);
const latestAttemptSelect = {
orderBy: {
number: "desc",
},
take: 1,
select: {
id: true,
number: true,
status: true,
},
} satisfies Prisma.TaskRunInclude["attempts"];
const attempt = await this._prisma.taskRunAttempt.findFirst({
where: {
friendlyId: params.attemptFriendlyId,
},
include: {
taskRun: true,
dependencies: {
select: {
taskRun: {
select: {
attempts: latestAttemptSelect,
},
},
},
orderBy: {
createdAt: "desc",
},
take: 1,
},
batchDependencies: {
select: {
items: {
select: {
taskRun: {
select: {
attempts: latestAttemptSelect,
},
},
},
},
},
orderBy: {
createdAt: "desc",
},
take: 1,
},
},
});
if (!attempt) {
this._logger.error("Could not find attempt", params);
return;
}
this._logger = logger.child({
attemptId: attempt.id,
attemptFriendlyId: attempt.friendlyId,
taskRun: attempt.taskRun,
});
if (isFinalRunStatus(attempt.taskRun.status)) {
this._logger.error("Run is not resumable");
return;
}
let completedAttemptIds: string[] = [];
switch (params.type) {
case "WAIT_FOR_DURATION": {
this._logger.debug("Sending duration wait resume message");
await this.#setPostResumeStatuses(attempt);
socketIo.coordinatorNamespace.emit("RESUME_AFTER_DURATION", {
version: "v1",
attemptId: attempt.id,
attemptFriendlyId: attempt.friendlyId,
});
break;
}
case "WAIT_FOR_TASK": {
if (attempt.dependencies.length) {
// We only care about the latest dependency
const dependentAttempt = attempt.dependencies[0].taskRun.attempts[0];
if (!dependentAttempt) {
this._logger.error("No dependent attempt");
return;
}
completedAttemptIds = [dependentAttempt.id];
} else {
this._logger.error("No task dependency");
return;
}
await this.#handleDependencyResume(attempt, completedAttemptIds);
break;
}
case "WAIT_FOR_BATCH": {
if (attempt.batchDependencies) {
// We only care about the latest batch dependency
const dependentBatchItems = attempt.batchDependencies[0].items;
if (!dependentBatchItems) {
this._logger.error("No dependent batch items");
return;
}
//find the best attempt for each batch item
//it should be the most recent one in a final state
const finalAttempts = dependentBatchItems
.map((item) => {
return item.taskRun.attempts
.filter((a) => FINAL_ATTEMPT_STATUSES.includes(a.status))
.sort((a, b) => b.number - a.number)
.at(0);
})
.filter(Boolean);
completedAttemptIds = finalAttempts.map((a) => a.id);
if (completedAttemptIds.length !== dependentBatchItems.length) {
this._logger.error("[ResumeAttemptService] not all batch items have attempts", {
runId: attempt.taskRunId,
completedAttemptIds,
finalAttempts,
dependentBatchItems,
});
return;
}
} else {
this._logger.error("No batch dependency");
return;
}
await this.#handleDependencyResume(attempt, completedAttemptIds);
break;
}
default: {
break;
}
}
}
async #handleDependencyResume(attempt: TaskRunAttempt, completedAttemptIds: string[]) {
if (completedAttemptIds.length === 0) {
this._logger.error("No completed attempt IDs");
return;
}
const completions: TaskRunExecutionResult[] = [];
const executions: TaskRunExecution[] = [];
for (const completedAttemptId of completedAttemptIds) {
const completedAttempt = await this._prisma.taskRunAttempt.findFirst({
where: {
id: completedAttemptId,
taskRun: {
lockedAt: {
not: null,
},
lockedById: {
not: null,
},
},
},
});
if (!completedAttempt) {
this._logger.error("Completed attempt not found", { completedAttemptId });
await marqs?.acknowledgeMessage(
attempt.taskRunId,
"Cannot find completed attempt in ResumeAttemptService"
);
return;
}
const logger = this._logger.child({
completedAttemptId: completedAttempt.id,
completedAttemptFriendlyId: completedAttempt.friendlyId,
completedRunId: completedAttempt.taskRunId,
});
const resumePayload = await sharedQueueTasks.getResumePayload(completedAttempt.id);
if (!resumePayload) {
logger.error("Failed to get resume payload");
await marqs?.acknowledgeMessage(
attempt.taskRunId,
"Failed to get resume payload in ResumeAttemptService"
);
return;
}
completions.push(resumePayload.completion);
executions.push(resumePayload.execution);
}
await this.#setPostResumeStatuses(attempt);
socketIo.coordinatorNamespace.emit("RESUME_AFTER_DEPENDENCY", {
version: "v1",
runId: attempt.taskRunId,
attemptId: attempt.id,
attemptFriendlyId: attempt.friendlyId,
completions,
executions,
});
}
async #setPostResumeStatuses(attempt: TaskRunAttempt) {
try {
const updatedAttempt = await this._prisma.taskRunAttempt.update({
where: {
id: attempt.id,
},
data: {
status: "EXECUTING",
taskRun: {
update: {
data: {
status: attempt.number > 1 ? "RETRYING_AFTER_FAILURE" : "EXECUTING",
},
},
},
},
select: {
id: true,
status: true,
taskRun: {
select: {
id: true,
status: true,
},
},
},
});
this._logger.debug("Set post resume statuses", {
run: {
id: updatedAttempt.taskRun.id,
status: updatedAttempt.taskRun.status,
},
attempt: {
id: updatedAttempt.id,
status: updatedAttempt.status,
},
});
} catch (error) {
this._logger.error("Failed to set post resume statuses", {
error:
error instanceof Error
? {
name: error.name,
message: error.message,
stack: error.stack,
}
: error,
});
}
}
}
apps/webapp/app/v3/services/resumeBatchRun.server.ts
-399
View File
@@ -1,399 +0,0 @@
import type { PrismaClientOrTransaction } from "~/db.server";
import { commonWorker } from "../commonWorker.server";
import { marqs } from "~/v3/marqs/index.server";
import { BaseService } from "./baseService.server";
import { logger } from "~/services/logger.server";
import type { BatchTaskRun, Prisma } from "@trigger.dev/database";
import { findEnvironmentById } from "~/models/runtimeEnvironment.server";
import type { AuthenticatedEnvironment } from "~/services/apiAuth.server";
import { workerQueue } from "~/services/worker.server";
import { isV3Disabled } from "../engineDeprecation.server";
const finishedBatchRunStatuses = ["COMPLETED", "FAILED", "CANCELED"];
const BATCH_RUN_INCLUDE = {
items: {
select: {
status: true,
taskRunAttemptId: true,
},
},
} satisfies Prisma.BatchTaskRunInclude;
type RetrieveBatchRunResult = Prisma.BatchTaskRunGetPayload<{
include: typeof BATCH_RUN_INCLUDE;
}>;
export class ResumeBatchRunService extends BaseService {
public async call(batchRunId: string) {
const batchRun = await this.runStore.findBatchTaskRunById(batchRunId, {
include: BATCH_RUN_INCLUDE,
});
if (!batchRun) {
logger.error(
"ResumeBatchRunService: Batch run doesn't exist or doesn't have a dependent attempt",
{
batchRunId,
}
);
return "ERROR";
}
// BatchTaskRun -> RuntimeEnvironment FK is dropped; resolve the env from the scalar id.
const environment = await findEnvironmentById(batchRun.runtimeEnvironmentId);
if (!environment) {
logger.error("ResumeBatchRunService: Environment not found", {
batchRunId,
runtimeEnvironmentId: batchRun.runtimeEnvironmentId,
});
return "ERROR";
}
// v3 (engine V1) shutdown: don't resume batches for abandoned V1 projects. v4 is unaffected.
// The BatchTaskRun -> RuntimeEnvironment relation is dropped, so read the engine from the
// resolved environment's project rather than the unloaded batchRun.runtimeEnvironment relation.
if (isV3Disabled() && environment.project.engine === "V1") {
logger.debug("[ResumeBatchRunService] Skipping resume for shut-down v3 batch", {
batchRunId,
});
return "ERROR";
}
if (batchRun.batchVersion === "v3") {
return await this.#handleV3BatchRun(batchRun, environment);
} else {
return await this.#handleLegacyBatchRun(batchRun, environment);
}
}
async #handleV3BatchRun(batchRun: RetrieveBatchRunResult, environment: AuthenticatedEnvironment) {
// V3 batch runs should already be complete by the time this is called
if (batchRun.status !== "COMPLETED") {
logger.debug("ResumeBatchRunService: Batch run is already completed", {
batchRunId: batchRun.id,
batchRun: {
id: batchRun.id,
status: batchRun.status,
},
});
return "ERROR";
}
// Even though we are in v3, we still need to check if the batch run has a dependent attempt
if (!batchRun.dependentTaskAttemptId) {
logger.debug("ResumeBatchRunService: Batch run doesn't have a dependent attempt", {
batchRunId: batchRun.id,
});
return "ERROR";
}
return await this.#handleDependentTaskAttempt(
batchRun,
batchRun.dependentTaskAttemptId,
environment
);
}
async #handleLegacyBatchRun(
batchRun: RetrieveBatchRunResult,
environment: AuthenticatedEnvironment
) {
if (batchRun.status === "COMPLETED") {
logger.debug("ResumeBatchRunService: Batch run is already completed", {
batchRunId: batchRun.id,
batchRun: {
id: batchRun.id,
status: batchRun.status,
},
});
return "ERROR";
}
if (batchRun.batchVersion === "v2") {
if (batchRun.items.length < batchRun.runCount) {
logger.debug("ResumeBatchRunService: All items aren't yet completed [v2]", {
batchRunId: batchRun.id,
batchRun: {
id: batchRun.id,
status: batchRun.status,
itemsLength: batchRun.items.length,
runCount: batchRun.runCount,
},
});
return "PENDING";
}
}
if (batchRun.items.some((item) => !finishedBatchRunStatuses.includes(item.status))) {
logger.debug("ResumeBatchRunService: All items aren't yet completed [v1]", {
batchRunId: batchRun.id,
batchRun: {
id: batchRun.id,
status: batchRun.status,
},
});
return "PENDING";
}
// If we are in development, or there is no dependent attempt, we can just mark the batch as completed and return
if (environment.type === "DEVELOPMENT" || !batchRun.dependentTaskAttemptId) {
// We need to update the batchRun status so we don't resume it again
await this.runStore.updateBatchTaskRun({
where: {
id: batchRun.id,
},
data: {
status: "COMPLETED",
},
select: { id: true },
});
return "COMPLETED";
}
return await this.#handleDependentTaskAttempt(
batchRun,
batchRun.dependentTaskAttemptId,
environment
);
}
async #handleDependentTaskAttempt(
batchRun: RetrieveBatchRunResult,
dependentTaskAttemptId: string,
environment: AuthenticatedEnvironment
) {
const dependentTaskAttempt = await this._prisma.taskRunAttempt.findFirst({
where: {
id: dependentTaskAttemptId,
},
select: {
status: true,
id: true,
taskRun: {
select: {
id: true,
queue: true,
taskIdentifier: true,
concurrencyKey: true,
createdAt: true,
queueTimestamp: true,
},
},
},
});
if (!dependentTaskAttempt) {
logger.error("ResumeBatchRunService: Dependent attempt not found", {
batchRunId: batchRun.id,
dependentTaskAttemptId: batchRun.dependentTaskAttemptId,
});
return "ERROR";
}
// This batch has a dependent attempt and just finalized, we should resume that attempt
const dependentRun = dependentTaskAttempt.taskRun;
if (dependentTaskAttempt.status === "PAUSED" && batchRun.checkpointEventId) {
logger.debug("ResumeBatchRunService: Attempt is paused and has a checkpoint event", {
batchRunId: batchRun.id,
dependentTaskAttempt: dependentTaskAttempt,
checkpointEventId: batchRun.checkpointEventId,
});
// We need to update the batchRun status so we don't resume it again
const wasUpdated = await this.#setBatchToResumedOnce(batchRun);
if (wasUpdated) {
logger.debug("ResumeBatchRunService: Resuming dependent run with checkpoint", {
batchRunId: batchRun.id,
dependentTaskAttemptId: dependentTaskAttempt.id,
});
await marqs.enqueueMessage(
environment,
dependentRun.queue,
dependentRun.id,
{
type: "RESUME",
completedAttemptIds: [],
resumableAttemptId: dependentTaskAttempt.id,
checkpointEventId: batchRun.checkpointEventId,
taskIdentifier: dependentTaskAttempt.taskRun.taskIdentifier,
projectId: environment.projectId,
environmentId: environment.id,
environmentType: environment.type,
},
dependentRun.concurrencyKey ?? undefined,
dependentRun.queueTimestamp ?? dependentRun.createdAt,
undefined,
"resume"
);
return "COMPLETED";
} else {
logger.debug("ResumeBatchRunService: with checkpoint was already completed", {
batchRunId: batchRun.id,
dependentTaskAttempt: dependentTaskAttempt,
checkpointEventId: batchRun.checkpointEventId,
hasCheckpointEvent: !!batchRun.checkpointEventId,
});
return "ALREADY_COMPLETED";
}
} else {
logger.debug("ResumeBatchRunService: attempt is not paused or there's no checkpoint event", {
batchRunId: batchRun.id,
dependentTaskAttempt: dependentTaskAttempt,
checkpointEventId: batchRun.checkpointEventId,
hasCheckpointEvent: !!batchRun.checkpointEventId,
});
if (dependentTaskAttempt.status === "PAUSED" && !batchRun.checkpointEventId) {
// In case of race conditions the status can be PAUSED without a checkpoint event
// When the checkpoint is created, it will continue the run
logger.error("ResumeBatchRunService: attempt is paused but there's no checkpoint event", {
batchRunId: batchRun.id,
dependentTaskAttempt: dependentTaskAttempt,
checkpointEventId: batchRun.checkpointEventId,
hasCheckpointEvent: !!batchRun.checkpointEventId,
});
return "ERROR";
}
// We need to update the batchRun status so we don't resume it again
const wasUpdated = await this.#setBatchToResumedOnce(batchRun);
if (wasUpdated) {
logger.debug("ResumeBatchRunService: Resuming dependent run without checkpoint", {
batchRunId: batchRun.id,
dependentTaskAttempt: dependentTaskAttempt,
checkpointEventId: batchRun.checkpointEventId,
hasCheckpointEvent: !!batchRun.checkpointEventId,
});
await marqs.requeueMessage(
dependentRun.id,
{
type: "RESUME",
completedAttemptIds: batchRun.items
.map((item) => item.taskRunAttemptId)
.filter(Boolean),
resumableAttemptId: dependentTaskAttempt.id,
checkpointEventId: batchRun.checkpointEventId ?? undefined,
taskIdentifier: dependentTaskAttempt.taskRun.taskIdentifier,
projectId: environment.projectId,
environmentId: environment.id,
environmentType: environment.type,
},
(
dependentTaskAttempt.taskRun.queueTimestamp ?? dependentTaskAttempt.taskRun.createdAt
).getTime(),
"resume"
);
return "COMPLETED";
} else {
logger.debug("ResumeBatchRunService: without checkpoint was already completed", {
batchRunId: batchRun.id,
dependentTaskAttempt: dependentTaskAttempt,
checkpointEventId: batchRun.checkpointEventId,
hasCheckpointEvent: !!batchRun.checkpointEventId,
});
return "ALREADY_COMPLETED";
}
}
}
async #setBatchToResumedOnce(batchRun: BatchTaskRun) {
// v3 batches don't use the status for deciding whether a batch has been resumed
if (batchRun.batchVersion === "v3") {
const result = await this.runStore.updateManyBatchTaskRun({
where: {
id: batchRun.id,
resumedAt: null,
},
data: {
resumedAt: new Date(),
},
});
if (result.count > 0) {
return true;
} else {
return false;
}
}
const result = await this.runStore.updateManyBatchTaskRun({
where: {
id: batchRun.id,
status: {
not: "COMPLETED", // Ensure the status is not already "COMPLETED"
},
},
data: {
status: "COMPLETED",
},
});
if (result.count > 0) {
return true;
} else {
return false;
}
}
static async enqueue(
batchRunId: string,
skipJobKey: boolean,
tx?: PrismaClientOrTransaction,
runAt?: Date
) {
if (tx) {
logger.debug("ResumeBatchRunService: Enqueuing resume batch run using workerQueue", {
batchRunId,
skipJobKey,
runAt,
});
return await workerQueue.enqueue(
"v3.resumeBatchRun",
{
batchRunId,
},
{
jobKey: skipJobKey ? undefined : `resumeBatchRun-${batchRunId}`,
runAt,
tx,
}
);
} else {
logger.debug("ResumeBatchRunService: Enqueuing resume batch run using commonWorker", {
batchRunId,
skipJobKey,
runAt,
});
return await commonWorker.enqueue({
id: skipJobKey ? undefined : `resumeBatchRun-${batchRunId}`,
job: "v3.resumeBatchRun",
payload: {
batchRunId,
},
availableAt: runAt,
});
}
}
}
apps/webapp/app/v3/services/resumeDependentParents.server.ts
-296
View File
@@ -1,296 +0,0 @@
import type { Prisma } from "@trigger.dev/database";
import { logger } from "~/services/logger.server";
import { isFinalAttemptStatus, isFinalRunStatus } from "../taskStatus";
import { BaseService } from "./baseService.server";
import { ResumeBatchRunService } from "./resumeBatchRun.server";
import { ResumeTaskDependencyService } from "./resumeTaskDependency.server";
import { $transaction } from "~/db.server";
import { completeBatchTaskRunItemV3 } from "./batchTriggerV3.server";
type Output =
| {
success: true;
action:
| "resume-scheduled"
| "batch-resume-scheduled"
| "no-dependencies"
| "not-finished"
| "dev";
}
| {
success: false;
error: string;
};
const taskRunDependencySelect = {
select: {
id: true,
taskRunId: true,
taskRun: {
select: {
id: true,
status: true,
friendlyId: true,
runtimeEnvironment: {
select: {
type: true,
},
},
},
},
dependentAttempt: {
select: {
id: true,
},
},
dependentBatchRun: {
select: {
id: true,
batchVersion: true,
},
},
},
} as const;
type Dependency = Prisma.TaskRunDependencyGetPayload<typeof taskRunDependencySelect>;
/** This will resume a dependent (parent) run if there is one and it makes sense. */
export class ResumeDependentParentsService extends BaseService {
public async call({ id }: { id: string }): Promise<Output> {
try {
const dependency = await this._prisma.taskRunDependency.findFirst({
...taskRunDependencySelect,
where: {
taskRunId: id,
},
});
logger.log("ResumeDependentParentsService: tried to find dependency", {
runId: id,
dependency: dependency,
});
if (!dependency) {
logger.log("ResumeDependentParentsService: dependency not found", {
runId: id,
});
//no dependency, that's fine most runs won't have one.
return {
success: true,
action: "no-dependencies",
};
}
if (dependency.taskRun.runtimeEnvironment.type === "DEVELOPMENT") {
return {
success: true,
action: "dev",
};
}
if (!isFinalRunStatus(dependency.taskRun.status)) {
logger.debug(
"ResumeDependentParentsService: run not finished yet, can't resume parent yet",
{
runId: id,
dependency,
}
);
// the child run isn't finished yet, so we can't resume the parent yet.
return {
success: true,
action: "not-finished",
};
}
if (dependency.dependentAttempt) {
return this.#singleRunDependency(dependency);
} else if (dependency.dependentBatchRun) {
return this.#batchRunDependency(dependency);
} else {
logger.error("ResumeDependentParentsService: dependency has no dependencies", {
runId: id,
dependency,
});
return {
success: false,
error: `Dependency has no dependencies (single or batch)`,
};
}
} catch (error) {
return {
success: false,
error: error instanceof Error ? error.message : JSON.stringify(error),
};
}
}
async #singleRunDependency(dependency: Dependency): Promise<Output> {
logger.debug(
`ResumeDependentParentsService.singleRunDependency(): Resuming dependent parent for run`,
{
dependency,
}
);
const lastAttempt = await this._prisma.taskRunAttempt.findFirst({
select: {
id: true,
status: true,
},
where: {
taskRunId: dependency.taskRunId,
},
orderBy: {
id: "desc",
},
});
if (!lastAttempt) {
logger.error(
"ResumeDependentParentsService.singleRunDependency(): dependency child attempt not found",
{
dependency,
}
);
return {
success: false,
error: `Dependency child attempt not found for run ${dependency.taskRunId}`,
};
}
if (!isFinalAttemptStatus(lastAttempt.status)) {
//We still want to continue if this happens because the run is final but log it
logger.error(
"ResumeDependentParentsService.singleRunDependency(): dependency child attempt not final, but the run is.",
{
dependency,
lastAttempt,
}
);
return {
success: false,
error: `Dependency child attempt not final, but the run is`,
};
}
//resume the dependent task
await ResumeTaskDependencyService.enqueue(dependency.id, lastAttempt.id);
return {
success: true,
action: "resume-scheduled",
};
}
async #batchRunDependency(dependency: Dependency): Promise<Output> {
logger.debug(
`ResumeDependentParentsService.batchRunDependency(): Resuming dependent batch for run`,
{
dependency,
}
);
if (!dependency.dependentBatchRun) {
logger.error(
"ResumeDependentParentsService.batchRunDependency(): dependency has no dependent batch",
{
dependency,
}
);
return {
success: false,
error: `Dependency has no dependent batch`,
};
}
const lastAttempt = await this._prisma.taskRunAttempt.findFirst({
select: {
id: true,
status: true,
},
where: {
taskRunId: dependency.taskRunId,
},
orderBy: {
id: "desc",
},
});
if (!lastAttempt) {
logger.error(
"ResumeDependentParentsService.singleRunDependency(): dependency child attempt not found",
{
dependency,
}
);
return {
success: false,
error: `Dependency child attempt not found for run ${dependency.taskRunId}`,
};
}
logger.log(
"ResumeDependentParentsService.batchRunDependency(): Setting the batchTaskRunItem to COMPLETED",
{
dependency,
lastAttempt,
}
);
if (dependency.dependentBatchRun!.batchVersion === "v3") {
const batchTaskRunItem = await this._prisma.batchTaskRunItem.findFirst({
where: {
batchTaskRunId: dependency.dependentBatchRun!.id,
taskRunId: dependency.taskRunId,
},
});
if (batchTaskRunItem) {
await completeBatchTaskRunItemV3(
batchTaskRunItem.id,
batchTaskRunItem.batchTaskRunId,
this._prisma,
true,
lastAttempt.id
);
} else {
logger.debug(
"ResumeDependentParentsService.batchRunDependency() v3: batchTaskRunItem not found",
{
dependency,
lastAttempt,
}
);
}
} else {
await $transaction(this._prisma, async (tx) => {
await tx.batchTaskRunItem.update({
where: {
batchTaskRunId_taskRunId: {
batchTaskRunId: dependency.dependentBatchRun!.id,
taskRunId: dependency.taskRunId,
},
},
data: {
status: "COMPLETED",
taskRunAttemptId: lastAttempt.id,
},
});
await ResumeBatchRunService.enqueue(dependency.dependentBatchRun!.id, false, tx);
});
}
return {
success: true,
action: "batch-resume-scheduled",
};
}
}
apps/webapp/app/v3/services/resumeTaskDependency.server.ts
-175
View File
@@ -1,175 +0,0 @@
import type { TaskRunDependency } from "@trigger.dev/database";
import { logger } from "~/services/logger.server";
import { marqs } from "~/v3/marqs/index.server";
import { commonWorker } from "../commonWorker.server";
import { BaseService } from "./baseService.server";
import { isV3Disabled } from "../engineDeprecation.server";
export class ResumeTaskDependencyService extends BaseService {
public async call(dependencyId: string, sourceTaskAttemptId: string) {
const dependency = await this._prisma.taskRunDependency.findFirst({
where: { id: dependencyId },
include: {
taskRun: {
include: {
runtimeEnvironment: {
include: {
project: true,
organization: true,
},
},
},
},
dependentAttempt: {
include: {
taskRun: true,
},
},
},
});
// Dependencies with a dependentBatchRun are handled already by the ResumeBatchRunService
if (!dependency || !dependency.dependentAttempt) {
return;
}
// v3 (engine V1) shutdown: don't resume dependencies for abandoned V1 runs. v4 is unaffected.
if (isV3Disabled() && dependency.taskRun.engine === "V1") {
logger.debug("[ResumeTaskDependencyService] Skipping resume for shut-down v3 run", {
dependencyId,
});
return;
}
if (dependency.taskRun.runtimeEnvironment.type === "DEVELOPMENT") {
return;
}
const dependentRun = dependency.dependentAttempt.taskRun;
if (dependency.dependentAttempt.status === "PAUSED" && dependency.checkpointEventId) {
logger.debug(
"Task dependency resume: Attempt is paused and there's a checkpoint. Enqueuing resume with checkpoint.",
{
attemptId: dependency.id,
dependentAttempt: dependency.dependentAttempt,
checkpointEventId: dependency.checkpointEventId,
hasCheckpointEvent: !!dependency.checkpointEventId,
runId: dependentRun.id,
}
);
const wasUpdated = await this.#setDependencyToResumedOnce(dependency);
if (!wasUpdated) {
logger.debug("Task dependency resume: Attempt with checkpoint was already resumed", {
attemptId: dependency.id,
dependentAttempt: dependency.dependentAttempt,
checkpointEventId: dependency.checkpointEventId,
hasCheckpointEvent: !!dependency.checkpointEventId,
runId: dependentRun.id,
});
return;
}
// TODO: use the new priority queue thingie
await marqs?.enqueueMessage(
dependency.taskRun.runtimeEnvironment,
dependentRun.queue,
dependentRun.id,
{
type: "RESUME",
completedAttemptIds: [sourceTaskAttemptId],
resumableAttemptId: dependency.dependentAttempt.id,
checkpointEventId: dependency.checkpointEventId,
taskIdentifier: dependency.taskRun.taskIdentifier,
projectId: dependency.taskRun.runtimeEnvironment.projectId,
environmentId: dependency.taskRun.runtimeEnvironment.id,
environmentType: dependency.taskRun.runtimeEnvironment.type,
},
dependentRun.concurrencyKey ?? undefined,
dependentRun.queueTimestamp ?? dependentRun.createdAt,
undefined,
"resume"
);
} else {
logger.debug("Task dependency resume: Attempt is not paused or there's no checkpoint event", {
attemptId: dependency.id,
dependentAttempt: dependency.dependentAttempt,
checkpointEventId: dependency.checkpointEventId,
hasCheckpointEvent: !!dependency.checkpointEventId,
runId: dependentRun.id,
});
if (dependency.dependentAttempt.status === "PAUSED" && !dependency.checkpointEventId) {
// In case of race conditions the status can be PAUSED without a checkpoint event
// When the checkpoint is created, it will continue the run
logger.error("Task dependency resume: Attempt is paused but there's no checkpoint event", {
attemptId: dependency.id,
dependentAttemptId: dependency.dependentAttempt.id,
});
return;
}
const wasUpdated = await this.#setDependencyToResumedOnce(dependency);
if (!wasUpdated) {
logger.debug("Task dependency resume: Attempt without checkpoint was already resumed", {
attemptId: dependency.id,
dependentAttempt: dependency.dependentAttempt,
checkpointEventId: dependency.checkpointEventId,
hasCheckpointEvent: !!dependency.checkpointEventId,
runId: dependentRun.id,
});
return;
}
await marqs.requeueMessage(
dependentRun.id,
{
type: "RESUME",
completedAttemptIds: [sourceTaskAttemptId],
resumableAttemptId: dependency.dependentAttempt.id,
checkpointEventId: dependency.checkpointEventId ?? undefined,
taskIdentifier: dependency.taskRun.taskIdentifier,
projectId: dependency.taskRun.runtimeEnvironment.projectId,
environmentId: dependency.taskRun.runtimeEnvironment.id,
environmentType: dependency.taskRun.runtimeEnvironment.type,
},
(dependentRun.queueTimestamp ?? dependentRun.createdAt).getTime(),
"resume"
);
}
}
async #setDependencyToResumedOnce(dependency: TaskRunDependency) {
const result = await this._prisma.taskRunDependency.updateMany({
where: {
id: dependency.id,
resumedAt: null,
},
data: {
resumedAt: new Date(),
},
});
// Check if any records were updated
if (result.count > 0) {
// The status was changed, so we return true
return true;
} else {
return false;
}
}
static async enqueue(dependencyId: string, sourceTaskAttemptId: string, runAt?: Date) {
return await commonWorker.enqueue({
job: "v3.resumeTaskDependency",
payload: {
dependencyId,
sourceTaskAttemptId,
},
availableAt: runAt,
});
}
}
apps/webapp/app/v3/services/retryAttempt.server.ts
-38
View File
@@ -1,38 +0,0 @@
import { logger } from "~/services/logger.server";
import { commonWorker } from "../commonWorker.server";
import { socketIo } from "../handleSocketIo.server";
import { BaseService } from "./baseService.server";
import { isV3Disabled } from "../engineDeprecation.server";
export class RetryAttemptService extends BaseService {
public async call(runId: string) {
const taskRun = await this.runStore.findRun({ id: runId }, this._prisma);
if (!taskRun) {
logger.error("Task run not found", { runId });
return;
}
// v3 (engine V1) shutdown: don't retry abandoned V1 runs. v4 is unaffected.
if (isV3Disabled() && taskRun.engine === "V1") {
logger.debug("[RetryAttemptService] Skipping retry for shut-down v3 run", { runId });
return;
}
socketIo.coordinatorNamespace.emit("READY_FOR_RETRY", {
version: "v1",
runId,
});
}
static async enqueue(runId: string, runAt?: Date) {
return await commonWorker.enqueue({
id: `retryAttempt:${runId}`,
job: "v3.retryAttempt",
payload: {
runId,
},
availableAt: runAt,
});
}
}
apps/webapp/app/v3/services/taskRunConcurrencyTracker.server.ts
-338
View File
@@ -1,338 +0,0 @@
import { env } from "~/env.server";
import Redis, { type RedisOptions } from "ioredis";
import { singleton } from "~/utils/singleton";
import { type MessagePayload, type MessageQueueSubscriber } from "../marqs/types";
import { z } from "zod";
import { logger } from "~/services/logger.server";
type Options = {
redis: RedisOptions;
};
const ConcurrentMessageData = z.object({
taskIdentifier: z.string(),
projectId: z.string(),
environmentId: z.string(),
environmentType: z.string(),
});
class TaskRunConcurrencyTracker implements MessageQueueSubscriber {
private redis: Redis;
constructor(config: Options) {
this.redis = new Redis(config.redis);
}
async messageEnqueued(message: MessagePayload): Promise<void> {}
async messageDequeued(message: MessagePayload): Promise<void> {
logger.debug("TaskRunConcurrencyTracker.messageDequeued()", {
data: message.data,
messageId: message.messageId,
});
const data = this.getMessageData(message);
if (!data) {
logger.info(
`TaskRunConcurrencyTracker.messageDequeued(): could not parse message data`,
message
);
return;
}
await this.executionStarted({
projectId: data.projectId,
taskId: data.taskIdentifier,
runId: message.messageId,
environmentId: data.environmentId,
deployed: data.environmentType !== "DEVELOPMENT",
});
}
async messageAcked(message: MessagePayload): Promise<void> {
logger.debug("TaskRunConcurrencyTracker.messageAcked()", {
data: message.data,
messageId: message.messageId,
});
const data = this.getMessageData(message);
if (!data) {
logger.info(
`TaskRunConcurrencyTracker.messageAcked(): could not parse message data`,
message
);
return;
}
await this.executionFinished({
projectId: data.projectId,
taskId: data.taskIdentifier,
runId: message.messageId,
environmentId: data.environmentId,
deployed: data.environmentType !== "DEVELOPMENT",
});
}
async messageNacked(message: MessagePayload): Promise<void> {
logger.debug("TaskRunConcurrencyTracker.messageNacked()", {
data: message.data,
messageId: message.messageId,
});
const data = this.getMessageData(message);
if (!data) {
logger.info(
`TaskRunConcurrencyTracker.messageNacked(): could not parse message data`,
message
);
return;
}
await this.executionFinished({
projectId: data.projectId,
taskId: data.taskIdentifier,
runId: message.messageId,
environmentId: data.environmentId,
deployed: data.environmentType !== "DEVELOPMENT",
});
}
async messageReplaced(message: MessagePayload): Promise<void> {
logger.debug("TaskRunConcurrencyTracker.messageReplaced()", {
data: message.data,
messageId: message.messageId,
});
const data = this.getMessageData(message);
if (!data) {
logger.info(
`TaskRunConcurrencyTracker.messageReplaced(): could not parse message data`,
message
);
return;
}
await this.executionFinished({
projectId: data.projectId,
taskId: data.taskIdentifier,
runId: message.messageId,
environmentId: data.environmentId,
deployed: data.environmentType !== "DEVELOPMENT",
});
}
async messageRequeued(message: MessagePayload): Promise<void> {
logger.debug("TaskRunConcurrencyTracker.messageRequeued()", {
data: message.data,
messageId: message.messageId,
});
const data = this.getMessageData(message);
if (!data) {
logger.info(
`TaskRunConcurrencyTracker.messageReplaced(): could not parse message data`,
message
);
return;
}
await this.executionFinished({
projectId: data.projectId,
taskId: data.taskIdentifier,
runId: message.messageId,
environmentId: data.environmentId,
deployed: data.environmentType !== "DEVELOPMENT",
});
}
private getMessageData(message: MessagePayload) {
const result = ConcurrentMessageData.safeParse(message.data);
if (result.success) {
return result.data;
}
return;
}
private async executionStarted({
projectId,
taskId,
runId,
environmentId,
deployed,
}: {
projectId: string;
taskId: string;
runId: string;
environmentId: string;
deployed: boolean;
}): Promise<void> {
try {
const pipeline = this.redis.pipeline();
pipeline.sadd(this.getTaskKey(projectId, taskId), runId);
pipeline.sadd(this.getTaskEnvironmentKey(projectId, taskId, environmentId), runId);
pipeline.sadd(this.getEnvironmentKey(projectId, environmentId), runId);
pipeline.sadd(this.getGlobalKey(deployed), runId);
await pipeline.exec();
} catch (error) {
logger.error("TaskRunConcurrencyTracker.executionStarted() error", { error });
}
}
private async executionFinished({
projectId,
taskId,
runId,
environmentId,
deployed,
}: {
projectId: string;
taskId: string;
runId: string;
environmentId: string;
deployed: boolean;
}): Promise<void> {
try {
const pipeline = this.redis.pipeline();
pipeline.srem(this.getTaskKey(projectId, taskId), runId);
pipeline.srem(this.getTaskEnvironmentKey(projectId, taskId, environmentId), runId);
pipeline.srem(this.getEnvironmentKey(projectId, environmentId), runId);
pipeline.srem(this.getGlobalKey(deployed), runId);
await pipeline.exec();
} catch (error) {
logger.error("TaskRunConcurrencyTracker.executionFinished() error", { error });
}
}
async taskConcurrentRunCount(projectId: string, taskId: string): Promise<number> {
return await this.redis.scard(this.getTaskKey(projectId, taskId));
}
async globalConcurrentRunCount(deployed: boolean): Promise<number> {
return await this.redis.scard(this.getGlobalKey(deployed));
}
async currentlyExecutingRuns(projectId: string, taskId: string): Promise<string[]> {
return await this.redis.smembers(this.getTaskKey(projectId, taskId));
}
private async getTaskCounts(projectId: string, taskIds: string[]): Promise<number[]> {
try {
const pipeline = this.redis.pipeline();
taskIds.forEach((taskId) => {
pipeline.scard(this.getTaskKey(projectId, taskId));
});
const results = await pipeline.exec();
if (!results) {
return [];
}
return results.map(([err, count]) => {
if (err) {
console.error("Error in getTaskCounts:", err);
return 0;
}
return count as number;
});
} catch (error) {
logger.error("TaskRunConcurrencyTracker.getTaskCounts() error", { error });
return [];
}
}
async projectTotalConcurrentRunCount(projectId: string, taskIds: string[]): Promise<number> {
const counts = await this.getTaskCounts(projectId, taskIds);
return counts.reduce((total, count) => total + count, 0);
}
async taskConcurrentRunCounts(
projectId: string,
taskIds: string[]
): Promise<Record<string, number>> {
const counts = await this.getTaskCounts(projectId, taskIds);
return taskIds.reduce(
(acc, taskId, index) => {
acc[taskId] = counts[index] ?? 0;
return acc;
},
{} as Record<string, number>
);
}
async environmentConcurrentRunCounts(
projectId: string,
environmentIds: string[]
): Promise<Record<string, number>> {
try {
const pipeline = this.redis.pipeline();
environmentIds.forEach((environmentId) => {
pipeline.scard(this.getEnvironmentKey(projectId, environmentId));
});
const results = await pipeline.exec();
if (!results) {
return Object.fromEntries(environmentIds.map((id) => [id, 0]));
}
return results.reduce(
(acc, [err, count], index) => {
if (err) {
console.error("Error in environmentConcurrentRunCounts:", err);
return acc;
}
acc[environmentIds[index]] = count as number;
return acc;
},
{} as Record<string, number>
);
} catch (error) {
logger.error("TaskRunConcurrencyTracker.environmentConcurrentRunCounts() error", { error });
return Object.fromEntries(environmentIds.map((id) => [id, 0]));
}
}
private getTaskKey(projectId: string, taskId: string): string {
return `project:${projectId}:task:${taskId}`;
}
private getTaskEnvironmentKey(projectId: string, taskId: string, environmentId: string): string {
return `project:${projectId}:task:${taskId}:env:${environmentId}`;
}
private getGlobalKey(deployed: boolean): string {
return `global:${deployed ? "deployed" : "dev"}`;
}
private getEnvironmentKey(projectId: string, environmentId: string): string {
return `project:${projectId}:env:${environmentId}`;
}
}
export const concurrencyTracker = singleton("concurrency-tracker", getTracker);
function getTracker() {
if (!env.REDIS_HOST || !env.REDIS_PORT) {
throw new Error(
"Could not initialize TaskRunConcurrencyTracker because process.env.REDIS_HOST and process.env.REDIS_PORT are required to be set. "
);
}
logger.debug("Initializing TaskRunConcurrencyTracker", {
redisHost: env.REDIS_HOST,
redisPort: env.REDIS_PORT,
});
return new TaskRunConcurrencyTracker({
redis: {
keyPrefix: "concurrencytracker:",
port: env.REDIS_PORT,
host: env.REDIS_HOST,
username: env.REDIS_USERNAME,
password: env.REDIS_PASSWORD,
enableAutoPipelining: true,
...(env.REDIS_TLS_DISABLED === "true" ? {} : { tls: {} }),
},
});
}
apps/webapp/app/v3/services/timeoutDeployment.server.ts
-3
View File
@@ -3,7 +3,6 @@ import { BaseService } from "./baseService.server";
import { commonWorker } from "../commonWorker.server";
import { PerformDeploymentAlertsService } from "./alerts/performDeploymentAlerts.server";
import { type PrismaClientOrTransaction } from "~/db.server";
import { workerQueue } from "~/services/worker.server";
import { DeploymentService } from "./deployment.server";
import { recordDeploymentOutcome } from "./recordDeploymentOutcome.server";
@@ -96,8 +95,6 @@ export class TimeoutDeploymentService extends BaseService {
}
static async dequeue(deploymentId: string, tx?: PrismaClientOrTransaction) {
// For backwards compatibility during transition, we need to dequeue/ack from both workers
await workerQueue.dequeue(`timeoutDeployment:${deploymentId}`, { tx });
await commonWorker.ack(`timeoutDeployment:${deploymentId}`);
}
}
apps/webapp/app/v3/services/triggerTask.server.ts
+5 -21
View File
@@ -10,9 +10,8 @@ import { DefaultTriggerTaskValidator } from "~/runEngine/validators/triggerTaskV
import type { AuthenticatedEnvironment } from "~/services/apiAuth.server";
import { determineEngineVersion } from "../engineVersion.server";
import { tracer } from "../tracer.server";
import { isV3Disabled, V3_TRIGGER_DEPRECATION_MESSAGE } from "../engineDeprecation.server";
import { V3_TRIGGER_DEPRECATION_MESSAGE } from "../engineDeprecation.server";
import { ServiceValidationError, WithRunEngine } from "./baseService.server";
import { TriggerTaskServiceV1 } from "./triggerTaskV1.server";
export type TriggerTaskServiceOptions = {
idempotencyKey?: string;
@@ -74,15 +73,10 @@ export class TriggerTaskService extends WithRunEngine {
switch (v) {
case "V1": {
// v3 (engine V1) is being sunset. When the shutdown is on, reject the
// trigger with a graceful, actionable error instead of creating a V1
// run. Covers single, batch, schedule, replay, and triggerAndWait,
// which all route through here.
if (isV3Disabled()) {
throw new ServiceValidationError(V3_TRIGGER_DEPRECATION_MESSAGE);
}
return await this.callV1(taskId, environment, body, options);
// v3 (engine V1) is retired. Reject the trigger with a graceful,
// actionable error instead of executing. Covers single, batch,
// schedule, replay, and triggerAndWait, which all route through here.
throw new ServiceValidationError(V3_TRIGGER_DEPRECATION_MESSAGE);
}
case "V2": {
return await this.callV2(taskId, environment, body, options);
@@ -91,16 +85,6 @@ export class TriggerTaskService extends WithRunEngine {
});
}
private async callV1(
taskId: string,
environment: AuthenticatedEnvironment,
body: TriggerTaskRequestBody,
options: TriggerTaskServiceOptions = {}
): Promise<TriggerTaskServiceResult | undefined> {
const service = new TriggerTaskServiceV1(this._prisma);
return await service.call(taskId, environment, body, options);
}
private async callV2(
taskId: string,
environment: AuthenticatedEnvironment,
apps/webapp/app/v3/services/triggerTaskV1.server.ts
-762
View File
@@ -1,762 +0,0 @@
import type { IOPacket, TriggerTaskRequestBody } from "@trigger.dev/core/v3";
import {
packetRequiresOffloading,
taskRunErrorEnhancer,
taskRunErrorToString,
} from "@trigger.dev/core/v3";
import {
parseNaturalLanguageDuration,
sanitizeQueueName,
stringifyDuration,
} from "@trigger.dev/core/v3/isomorphic";
import { Prisma } from "@trigger.dev/database";
import { z } from "zod";
import { env } from "~/env.server";
import { MAX_TAGS_PER_RUN } from "~/models/taskRunTag.server";
import type { AuthenticatedEnvironment } from "~/services/apiAuth.server";
import { autoIncrementCounter } from "~/services/autoIncrementCounter.server";
import { logger } from "~/services/logger.server";
import { getEntitlement } from "~/services/platform.v3.server";
import { parseDelay } from "~/utils/delays";
import { resolveIdempotencyKeyTTL } from "~/utils/idempotencyKeys.server";
import { handleMetadataPacket } from "~/utils/packets";
import { marqs } from "~/v3/marqs/index.server";
import { getV3EventRepository } from "../eventRepository/index.server";
import { generateFriendlyId } from "../friendlyIdentifiers";
import { findCurrentWorkerFromEnvironment } from "../models/workerDeployment.server";
import { guardQueueSizeLimitsForEnv } from "../queueSizeLimits.server";
import { uploadPacketToObjectStore } from "../objectStore.server";
import { removeQueueConcurrencyLimits, updateQueueConcurrencyLimits } from "../runQueue.server";
import { isFinalAttemptStatus, isFinalRunStatus } from "../taskStatus";
import { startActiveSpan } from "../tracer.server";
import { clampMaxDuration } from "../utils/maxDuration";
import { BaseService, ServiceValidationError } from "./baseService.server";
import { attemptInEnvironmentWhere, batchRunInEnvironmentWhere } from "./triggerV1Scoping";
import { EnqueueDelayedRunService } from "./enqueueDelayedRun.server";
import { enqueueRun } from "./enqueueRun.server";
import { ExpireEnqueuedRunService } from "./expireEnqueuedRun.server";
import type { TriggerTaskServiceOptions, TriggerTaskServiceResult } from "./triggerTask.server";
import { MAX_ATTEMPTS, OutOfEntitlementError } from "./triggerTask.server";
// This is here for backwords compatibility for v3 users
const QueueOptions = z.object({
name: z.string(),
concurrencyLimit: z.number().int().optional(),
});
/** @deprecated Use TriggerTaskService in `triggerTask.server.ts` instead. */
export class TriggerTaskServiceV1 extends BaseService {
public async call(
taskId: string,
environment: AuthenticatedEnvironment,
body: TriggerTaskRequestBody,
options: TriggerTaskServiceOptions = {},
attempt: number = 0
): Promise<TriggerTaskServiceResult | undefined> {
return await this.traceWithEnv("call()", environment, async (span) => {
span.setAttribute("taskId", taskId);
span.setAttribute("attempt", attempt);
if (attempt > MAX_ATTEMPTS) {
throw new ServiceValidationError(
`Failed to trigger ${taskId} after ${MAX_ATTEMPTS} attempts.`
);
}
const idempotencyKey = options.idempotencyKey ?? body.options?.idempotencyKey;
const idempotencyKeyExpiresAt =
options.idempotencyKeyExpiresAt ??
resolveIdempotencyKeyTTL(body.options?.idempotencyKeyTTL) ??
new Date(Date.now() + 24 * 60 * 60 * 1000 * 30); // 30 days
const delayUntil = await parseDelay(body.options?.delay);
const ttl =
typeof body.options?.ttl === "number"
? stringifyDuration(body.options?.ttl)
: (body.options?.ttl ?? (environment.type === "DEVELOPMENT" ? "10m" : undefined));
const existingRun = idempotencyKey
? await this._prisma.taskRun.findFirst({
where: {
runtimeEnvironmentId: environment.id,
idempotencyKey,
taskIdentifier: taskId,
},
})
: undefined;
if (existingRun) {
if (
existingRun.idempotencyKeyExpiresAt &&
existingRun.idempotencyKeyExpiresAt < new Date()
) {
logger.debug("[TriggerTaskService][call] Idempotency key has expired", {
idempotencyKey: options.idempotencyKey,
run: existingRun,
});
// Update the existing batch to remove the idempotency key
await this._prisma.taskRun.update({
where: { id: existingRun.id },
data: { idempotencyKey: null },
});
} else {
span.setAttribute("runId", existingRun.friendlyId);
return { run: existingRun, isCached: true };
}
}
if (environment.type !== "DEVELOPMENT" && !options.skipChecks) {
const result = await getEntitlement(environment.organizationId);
if (result && result.hasAccess === false) {
throw new OutOfEntitlementError();
}
}
if (!options.skipChecks) {
const queueSizeGuard = await guardQueueSizeLimitsForEnv(environment, marqs);
logger.debug("Queue size guard result", {
queueSizeGuard,
environment: {
id: environment.id,
type: environment.type,
organization: environment.organization,
project: environment.project,
},
});
if (!queueSizeGuard.isWithinLimits) {
throw new ServiceValidationError(
`Cannot trigger ${taskId} as the queue size limit for this environment has been reached. The maximum size is ${queueSizeGuard.maximumSize}`,
undefined,
"warn"
);
}
}
if (
body.options?.tags &&
typeof body.options.tags !== "string" &&
body.options.tags.length > MAX_TAGS_PER_RUN
) {
throw new ServiceValidationError(
`Runs can only have ${MAX_TAGS_PER_RUN} tags, you're trying to set ${body.options.tags.length}.`
);
}
const runFriendlyId = options?.runFriendlyId ?? generateFriendlyId("run");
const payloadPacket = await this.#handlePayloadPacket(
body.payload,
body.options?.payloadType ?? "application/json",
runFriendlyId,
environment
);
const metadataPacket = body.options?.metadata
? handleMetadataPacket(
body.options?.metadata,
body.options?.metadataType ?? "application/json",
env.TASK_RUN_METADATA_MAXIMUM_SIZE
)
: undefined;
const dependentAttempt = body.options?.dependentAttempt
? await this._prisma.taskRunAttempt.findFirst({
where: attemptInEnvironmentWhere(body.options.dependentAttempt, environment.id),
include: {
taskRun: {
select: {
id: true,
status: true,
taskIdentifier: true,
rootTaskRunId: true,
depth: true,
queueTimestamp: true,
queue: true,
taskEventStore: true,
},
},
},
})
: undefined;
if (
dependentAttempt &&
(isFinalAttemptStatus(dependentAttempt.status) ||
isFinalRunStatus(dependentAttempt.taskRun.status))
) {
logger.debug("Dependent attempt or run is in a terminal state", {
dependentAttempt: dependentAttempt,
});
if (isFinalAttemptStatus(dependentAttempt.status)) {
throw new ServiceValidationError(
`Cannot trigger ${taskId} as the parent attempt has a status of ${dependentAttempt.status}`
);
} else {
throw new ServiceValidationError(
`Cannot trigger ${taskId} as the parent run has a status of ${dependentAttempt.taskRun.status}`
);
}
}
const parentAttempt = body.options?.parentAttempt
? await this._prisma.taskRunAttempt.findFirst({
where: attemptInEnvironmentWhere(body.options.parentAttempt, environment.id),
include: {
taskRun: {
select: {
id: true,
status: true,
taskIdentifier: true,
rootTaskRunId: true,
depth: true,
taskEventStore: true,
},
},
},
})
: undefined;
const dependentBatchRun = body.options?.dependentBatch
? await this._prisma.batchTaskRun.findFirst({
where: batchRunInEnvironmentWhere(body.options.dependentBatch, environment.id),
include: {
dependentTaskAttempt: {
include: {
taskRun: {
select: {
id: true,
status: true,
taskIdentifier: true,
rootTaskRunId: true,
depth: true,
queueTimestamp: true,
queue: true,
taskEventStore: true,
},
},
},
},
},
})
: undefined;
if (
dependentBatchRun &&
dependentBatchRun.dependentTaskAttempt &&
(isFinalAttemptStatus(dependentBatchRun.dependentTaskAttempt.status) ||
isFinalRunStatus(dependentBatchRun.dependentTaskAttempt.taskRun.status))
) {
logger.debug("Dependent batch run task attempt or run has been canceled", {
dependentBatchRunId: dependentBatchRun.id,
status: dependentBatchRun.status,
attempt: dependentBatchRun.dependentTaskAttempt,
});
if (isFinalAttemptStatus(dependentBatchRun.dependentTaskAttempt.status)) {
throw new ServiceValidationError(
`Cannot trigger ${taskId} as the parent attempt has a status of ${dependentBatchRun.dependentTaskAttempt.status}`
);
} else {
throw new ServiceValidationError(
`Cannot trigger ${taskId} as the parent run has a status of ${dependentBatchRun.dependentTaskAttempt.taskRun.status}`
);
}
}
const parentBatchRun = body.options?.parentBatch
? await this._prisma.batchTaskRun.findFirst({
where: batchRunInEnvironmentWhere(body.options.parentBatch, environment.id),
include: {
dependentTaskAttempt: {
include: {
taskRun: {
select: {
id: true,
status: true,
taskIdentifier: true,
rootTaskRunId: true,
},
},
},
},
},
})
: undefined;
const { repository, store } = await getV3EventRepository(
environment.organization.id,
dependentAttempt?.taskRun.taskEventStore ??
parentAttempt?.taskRun.taskEventStore ??
dependentBatchRun?.dependentTaskAttempt?.taskRun.taskEventStore
);
try {
const result = await repository.traceEvent(
taskId,
{
context: options.traceContext,
spanParentAsLink: options.spanParentAsLink,
kind: "SERVER",
environment,
taskSlug: taskId,
attributes: {
properties: {},
style: {
icon: options.customIcon ?? "task",
},
},
incomplete: true,
immediate: true,
startTime: options.overrideCreatedAt
? BigInt(options.overrideCreatedAt.getTime()) * BigInt(1000000)
: undefined,
},
async (event, traceContext, traceparent) => {
const run = await autoIncrementCounter.incrementInTransaction(
`v3-run:${environment.id}:${taskId}`,
async (num, tx) => {
const lockedToBackgroundWorker = body.options?.lockToVersion
? await tx.backgroundWorker.findFirst({
where: {
projectId: environment.projectId,
runtimeEnvironmentId: environment.id,
version: body.options?.lockToVersion,
},
})
: undefined;
let queueName = sanitizeQueueName(
await this.#getQueueName(taskId, environment, body.options?.queue?.name)
);
// Check that the queuename is not an empty string
if (!queueName) {
queueName = sanitizeQueueName(`task/${taskId}`);
}
span.setAttribute("queueName", queueName);
const bodyTags =
typeof body.options?.tags === "string" ? [body.options.tags] : body.options?.tags;
const depth = dependentAttempt
? dependentAttempt.taskRun.depth + 1
: parentAttempt
? parentAttempt.taskRun.depth + 1
: dependentBatchRun?.dependentTaskAttempt
? dependentBatchRun.dependentTaskAttempt.taskRun.depth + 1
: 0;
const queueTimestamp =
options.queueTimestamp ??
dependentAttempt?.taskRun.queueTimestamp ??
dependentBatchRun?.dependentTaskAttempt?.taskRun.queueTimestamp ??
delayUntil ??
new Date();
const taskRun = await tx.taskRun.create({
data: {
status: delayUntil ? "DELAYED" : "PENDING",
number: num,
friendlyId: runFriendlyId,
runtimeEnvironmentId: environment.id,
environmentType: environment.type,
organizationId: environment.organizationId,
projectId: environment.projectId,
idempotencyKey,
idempotencyKeyExpiresAt: idempotencyKey ? idempotencyKeyExpiresAt : undefined,
taskIdentifier: taskId,
payload: payloadPacket.data ?? "",
payloadType: payloadPacket.dataType,
context: body.context,
traceContext: traceContext,
traceId: event.traceId,
spanId: event.spanId,
parentSpanId:
options.parentAsLinkType === "replay" ? undefined : traceparent?.spanId,
lockedToVersionId: lockedToBackgroundWorker?.id,
taskVersion: lockedToBackgroundWorker?.version,
sdkVersion: lockedToBackgroundWorker?.sdkVersion,
cliVersion: lockedToBackgroundWorker?.cliVersion,
concurrencyKey: body.options?.concurrencyKey,
queue: queueName,
isTest: body.options?.test ?? false,
delayUntil,
queuedAt: delayUntil ? undefined : new Date(),
queueTimestamp,
maxAttempts: body.options?.maxAttempts,
taskEventStore: store,
ttl,
parentTaskRunId:
dependentAttempt?.taskRun.id ??
parentAttempt?.taskRun.id ??
dependentBatchRun?.dependentTaskAttempt?.taskRun.id,
parentTaskRunAttemptId:
dependentAttempt?.id ??
parentAttempt?.id ??
dependentBatchRun?.dependentTaskAttempt?.id,
rootTaskRunId:
dependentAttempt?.taskRun.rootTaskRunId ??
dependentAttempt?.taskRun.id ??
parentAttempt?.taskRun.rootTaskRunId ??
parentAttempt?.taskRun.id ??
dependentBatchRun?.dependentTaskAttempt?.taskRun.rootTaskRunId ??
dependentBatchRun?.dependentTaskAttempt?.taskRun.id,
replayedFromTaskRunFriendlyId: options.replayedFromTaskRunFriendlyId,
batchId: dependentBatchRun?.id ?? parentBatchRun?.id,
resumeParentOnCompletion: !!(dependentAttempt ?? dependentBatchRun),
depth,
metadata: metadataPacket?.data,
metadataType: metadataPacket?.dataType,
seedMetadata: metadataPacket?.data,
seedMetadataType: metadataPacket?.dataType,
maxDurationInSeconds: body.options?.maxDuration
? clampMaxDuration(body.options.maxDuration)
: undefined,
runTags: bodyTags,
oneTimeUseToken: options.oneTimeUseToken,
machinePreset: body.options?.machine,
scheduleId: options.scheduleId,
scheduleInstanceId: options.scheduleInstanceId,
createdAt: options.overrideCreatedAt,
bulkActionGroupIds: body.options?.bulkActionId
? [body.options.bulkActionId]
: undefined,
},
});
event.setAttribute("runId", taskRun.friendlyId);
span.setAttribute("runId", taskRun.friendlyId);
if (dependentAttempt) {
await tx.taskRunDependency.create({
data: {
taskRunId: taskRun.id,
dependentAttemptId: dependentAttempt.id,
},
});
} else if (dependentBatchRun) {
await tx.taskRunDependency.create({
data: {
taskRunId: taskRun.id,
dependentBatchRunId: dependentBatchRun.id,
},
});
}
if (body.options?.queue) {
const concurrencyLimit =
typeof body.options.queue?.concurrencyLimit === "number"
? Math.max(
Math.min(
body.options.queue.concurrencyLimit,
environment.maximumConcurrencyLimit
),
0
)
: body.options.queue?.concurrencyLimit;
let taskQueue = await tx.taskQueue.findFirst({
where: {
runtimeEnvironmentId: environment.id,
name: queueName,
},
});
if (!taskQueue) {
// handle conflicts with existing queues
taskQueue = await tx.taskQueue.create({
data: {
friendlyId: generateFriendlyId("queue"),
name: queueName,
concurrencyLimit,
runtimeEnvironmentId: environment.id,
projectId: environment.projectId,
type: "NAMED",
},
});
}
if (typeof concurrencyLimit === "number") {
logger.debug("TriggerTaskService: updating concurrency limit", {
runId: taskRun.id,
friendlyId: taskRun.friendlyId,
taskQueue,
orgId: environment.organizationId,
projectId: environment.projectId,
concurrencyLimit,
queueOptions: body.options?.queue,
});
await updateQueueConcurrencyLimits(
environment,
taskQueue.name,
concurrencyLimit
);
} else if (concurrencyLimit === null) {
logger.debug("TriggerTaskService: removing concurrency limit", {
runId: taskRun.id,
friendlyId: taskRun.friendlyId,
taskQueue,
orgId: environment.organizationId,
projectId: environment.projectId,
queueOptions: body.options?.queue,
});
await removeQueueConcurrencyLimits(environment, taskQueue.name);
}
}
if (taskRun.delayUntil) {
await EnqueueDelayedRunService.enqueue(taskRun.id, taskRun.delayUntil);
}
if (!taskRun.delayUntil && taskRun.ttl) {
const expireAt = parseNaturalLanguageDuration(taskRun.ttl);
if (expireAt) {
await ExpireEnqueuedRunService.enqueue(taskRun.id, expireAt);
}
}
return taskRun;
},
async (_, tx) => {
const counter = await tx.taskRunNumberCounter.findUnique({
where: {
taskIdentifier_environmentId: {
taskIdentifier: taskId,
environmentId: environment.id,
},
},
select: { lastNumber: true },
});
return counter?.lastNumber;
},
this._prisma
);
if (!run) {
return;
}
// Now enqueue the run if it's not delayed
if (run.status === "PENDING") {
const enqueueResult = await enqueueRun({
env: environment,
run,
dependentRun:
dependentAttempt?.taskRun ?? dependentBatchRun?.dependentTaskAttempt?.taskRun,
});
if (!enqueueResult.ok) {
// Now we need to fail the run with enqueueResult.error and make sure and
// set the traced event to failed as well
await this._prisma.taskRun.update({
where: { id: run.id },
data: {
status: "SYSTEM_FAILURE",
completedAt: new Date(),
error: enqueueResult.error,
},
});
event.failWithError(enqueueResult.error);
return {
run,
isCached: false,
error: enqueueResult.error,
};
}
}
return { run, isCached: false };
}
);
if (result?.error) {
throw new ServiceValidationError(
taskRunErrorToString(taskRunErrorEnhancer(result.error))
);
}
const run = result?.run;
if (!run) {
return;
}
return {
run,
isCached: result?.isCached,
};
} catch (error) {
// Detect a prisma transaction Unique constraint violation
if (error instanceof Prisma.PrismaClientKnownRequestError) {
logger.debug("TriggerTask: Prisma transaction error", {
code: error.code,
message: error.message,
meta: error.meta,
});
if (error.code === "P2002") {
const target = error.meta?.target;
if (
Array.isArray(target) &&
target.length > 0 &&
typeof target[0] === "string" &&
target[0].includes("oneTimeUseToken")
) {
throw new ServiceValidationError(
`Cannot trigger ${taskId} with a one-time use token as it has already been used.`
);
} else if (
Array.isArray(target) &&
target.length == 2 &&
typeof target[0] === "string" &&
typeof target[1] === "string" &&
target[0] == "runtimeEnvironmentId" &&
target[1] == "name" &&
error.message.includes("prisma.taskQueue.create")
) {
throw new Error(
`Failed to trigger ${taskId} as the queue could not be created do to a unique constraint error, please try again.`
);
} else if (
Array.isArray(target) &&
target.length == 3 &&
typeof target[0] === "string" &&
typeof target[1] === "string" &&
typeof target[2] === "string" &&
target[0] == "runtimeEnvironmentId" &&
target[1] == "taskIdentifier" &&
target[2] == "idempotencyKey"
) {
logger.debug("TriggerTask: Idempotency key violation, retrying...", {
taskId,
environmentId: environment.id,
idempotencyKey,
});
// We need to retry the task run creation as the idempotency key has been used
return await this.call(taskId, environment, body, options, attempt + 1);
} else {
throw new ServiceValidationError(
`Cannot trigger ${taskId} as it has already been triggered with the same idempotency key.`
);
}
}
}
throw error;
}
});
}
async #getQueueName(taskId: string, environment: AuthenticatedEnvironment, queueName?: string) {
if (queueName) {
return queueName;
}
const defaultQueueName = `task/${taskId}`;
const worker = await findCurrentWorkerFromEnvironment(environment);
if (!worker) {
logger.debug("Failed to get queue name: No worker found", {
taskId,
environmentId: environment.id,
});
return defaultQueueName;
}
const task = await this._prisma.backgroundWorkerTask.findFirst({
where: {
workerId: worker.id,
slug: taskId,
},
});
if (!task) {
console.log("Failed to get queue name: No task found", {
taskId,
environmentId: environment.id,
});
return defaultQueueName;
}
const queueConfig = QueueOptions.optional().nullable().safeParse(task.queueConfig);
if (!queueConfig.success) {
console.log("Failed to get queue name: Invalid queue config", {
taskId,
environmentId: environment.id,
queueConfig: task.queueConfig,
});
return defaultQueueName;
}
return queueConfig.data?.name ?? defaultQueueName;
}
async #handlePayloadPacket(
payload: any,
payloadType: string,
pathPrefix: string,
environment: AuthenticatedEnvironment
) {
return await startActiveSpan("handlePayloadPacket()", async (span) => {
const packet = this.#createPayloadPacket(payload, payloadType);
if (!packet.data) {
return packet;
}
const { needsOffloading, size: _size } = packetRequiresOffloading(
packet,
env.TASK_PAYLOAD_OFFLOAD_THRESHOLD
);
if (!needsOffloading) {
return packet;
}
const filename = `${pathPrefix}/payload.json`;
const uploadedFilename = await uploadPacketToObjectStore(
filename,
packet.data,
packet.dataType,
environment
);
return {
data: uploadedFilename,
dataType: "application/store",
};
});
}
#createPayloadPacket(payload: any, payloadType: string): IOPacket {
if (payloadType === "application/json") {
return { data: JSON.stringify(payload), dataType: "application/json" };
}
if (typeof payload === "string") {
return { data: payload, dataType: payloadType };
}
return { dataType: payloadType };
}
}
apps/webapp/app/v3/services/triggerV1Scoping.ts
-20
View File
@@ -1,20 +0,0 @@
import type { Prisma } from "@trigger.dev/database";
// Where-clauses for resolving caller-supplied parent/dependent attempt & batch
// friendlyIds in the V1 trigger path, scoped to the caller's environment so a
// foreign friendlyId can't be wired onto the new run/batch. Standalone builders
// so the scope can be asserted directly in tests.
export function attemptInEnvironmentWhere(
friendlyId: string,
environmentId: string
): Prisma.TaskRunAttemptWhereInput {
return { friendlyId, taskRun: { runtimeEnvironmentId: environmentId } };
}
export function batchRunInEnvironmentWhere(
friendlyId: string,
environmentId: string
): Prisma.BatchTaskRunWhereInput {
return { friendlyId, runtimeEnvironmentId: environmentId };
}
apps/webapp/app/v3/services/updateFatalRunError.server.ts
-54
View File
@@ -1,54 +0,0 @@
import { BaseService } from "./baseService.server";
import { logger } from "~/services/logger.server";
import { isFatalRunStatus } from "../taskStatus";
import type { TaskRunInternalError } from "@trigger.dev/core/v3";
import { TaskRunErrorCodes } from "@trigger.dev/core/v3";
import { FinalizeTaskRunService } from "./finalizeTaskRun.server";
export type UpdateFatalRunErrorServiceOptions = {
reason?: string;
exitCode?: number;
logs?: string;
errorCode?: TaskRunInternalError["code"];
};
export class UpdateFatalRunErrorService extends BaseService {
public async call(runId: string, options?: UpdateFatalRunErrorServiceOptions) {
const opts = {
reason: "Worker crashed",
...options,
};
logger.debug("UpdateFatalRunErrorService.call", { runId, opts });
const taskRun = await this.runStore.findRun({ id: runId }, this._prisma);
if (!taskRun) {
logger.error("[UpdateFatalRunErrorService] Task run not found", { runId });
return;
}
if (!isFatalRunStatus(taskRun.status)) {
logger.warn("[UpdateFatalRunErrorService] Task run is not in a fatal state", {
runId,
status: taskRun.status,
});
return;
}
logger.debug("[UpdateFatalRunErrorService] Updating crash error", { runId, options });
const finalizeService = new FinalizeTaskRunService();
await finalizeService.call({
id: taskRun.id,
status: "CRASHED",
error: {
type: "INTERNAL_ERROR",
code: opts.errorCode ?? TaskRunErrorCodes.TASK_RUN_CRASHED,
message: opts.reason,
stackTrace: opts.logs,
},
});
}
}
apps/webapp/app/v3/sharedSocketConnection.ts
-141
View File
@@ -1,141 +0,0 @@
import { clientWebsocketMessages, serverWebsocketMessages } from "@trigger.dev/core/v3";
import type { StructuredLogger } from "@trigger.dev/core/v3/utils/structuredLogger";
import type { MessageCatalogToSocketIoEvents } from "@trigger.dev/core/v3/zodMessageHandler";
import { ZodMessageHandler, ZodMessageSender } from "@trigger.dev/core/v3/zodMessageHandler";
import { Evt } from "evt";
import { randomUUID } from "node:crypto";
import type { DisconnectReason, Namespace, Socket } from "socket.io";
import { env } from "~/env.server";
import { logger } from "~/services/logger.server";
import { SharedQueueConsumer } from "./marqs/sharedQueueConsumer.server";
interface SharedQueueConsumerPoolOptions {
sender: ZodMessageSender<typeof serverWebsocketMessages>;
poolSize: number;
}
class SharedQueueConsumerPool {
#consumers: SharedQueueConsumer[];
constructor(opts: SharedQueueConsumerPoolOptions) {
this.#consumers = Array(opts.poolSize)
.fill(null)
.map(
() =>
new SharedQueueConsumer(opts.sender, {
interval: env.SHARED_QUEUE_CONSUMER_INTERVAL_MS,
nextTickInterval: env.SHARED_QUEUE_CONSUMER_NEXT_TICK_INTERVAL_MS,
})
);
}
async start() {
await Promise.allSettled(this.#consumers.map((consumer) => consumer.start()));
}
async stop() {
await Promise.allSettled(this.#consumers.map((consumer) => consumer.stop()));
}
}
interface SharedSocketConnectionOptions {
namespace: Namespace<
MessageCatalogToSocketIoEvents<typeof clientWebsocketMessages>,
MessageCatalogToSocketIoEvents<typeof serverWebsocketMessages>
>;
socket: Socket<
MessageCatalogToSocketIoEvents<typeof clientWebsocketMessages>,
MessageCatalogToSocketIoEvents<typeof serverWebsocketMessages>
>;
logger?: StructuredLogger;
poolSize?: number;
}
export class SharedSocketConnection {
public id: string;
public onClose: Evt<DisconnectReason> = new Evt();
private _sender: ZodMessageSender<typeof serverWebsocketMessages>;
private _sharedQueueConsumerPool: SharedQueueConsumerPool;
private _messageHandler: ZodMessageHandler<typeof clientWebsocketMessages>;
private _defaultPoolSize = 10;
constructor(opts: SharedSocketConnectionOptions) {
this.id = randomUUID();
this._sender = new ZodMessageSender({
schema: serverWebsocketMessages,
sender: async (message) => {
return new Promise((resolve, reject) => {
try {
const { type, ...payload } = message;
opts.namespace.emit(type, payload as any);
resolve();
} catch (err) {
reject(err);
}
});
},
canSendMessage() {
// Return true if there is at least 1 connected socket on the namespace
if (opts.namespace.sockets.size === 0) {
return false;
}
return Array.from(opts.namespace.sockets.values()).some((socket) => socket.connected);
},
});
logger.debug("Starting SharedQueueConsumer pool", {
poolSize: opts.poolSize ?? this._defaultPoolSize,
});
this._sharedQueueConsumerPool = new SharedQueueConsumerPool({
poolSize: opts.poolSize ?? this._defaultPoolSize,
sender: this._sender,
});
opts.socket.on("disconnect", this.#handleClose.bind(this));
opts.socket.on("error", this.#handleError.bind(this));
this._messageHandler = new ZodMessageHandler({
schema: clientWebsocketMessages,
logger,
messages: {
READY_FOR_TASKS: async (payload) => {
this._sharedQueueConsumerPool.start();
},
BACKGROUND_WORKER_DEPRECATED: async (payload) => {
// await this._sharedConsumer.deprecateBackgroundWorker(payload.backgroundWorkerId);
},
BACKGROUND_WORKER_MESSAGE: async (payload) => {
switch (payload.data.type) {
case "TASK_RUN_COMPLETED": {
// handled in coordinator namespace
break;
}
case "TASK_HEARTBEAT": {
// handled in coordinator namespace
break;
}
}
},
},
});
this._messageHandler.registerHandlers(opts.socket, opts.logger ?? logger);
}
async initialize() {
this._sender.send("SERVER_READY", { id: this.id });
}
async #handleClose(ev: DisconnectReason) {
await this._sharedQueueConsumerPool.stop();
this.onClose.post(ev);
}
async #handleError(ev: Error) {
logger.error("Websocket error", { ev });
}
}
apps/webapp/app/v3/taskRunHeartbeatFailed.server.ts
-186
View File
@@ -1,186 +0,0 @@
import { logger } from "~/services/logger.server";
import { marqs } from "~/v3/marqs/index.server";
import assertNever from "assert-never";
import { FailedTaskRunService } from "./failedTaskRun.server";
import { BaseService } from "./services/baseService.server";
import type { PrismaClientOrTransaction } from "~/db.server";
import { workerQueue } from "~/services/worker.server";
import { socketIo } from "./handleSocketIo.server";
import { TaskRunErrorCodes } from "@trigger.dev/core/v3";
import { controlPlaneResolver } from "~/v3/runOpsMigration/controlPlaneResolver.server";
import { isV3Disabled } from "./engineDeprecation.server";
export class TaskRunHeartbeatFailedService extends BaseService {
public async call(runId: string) {
const taskRun = await this.runStore.findRun(
{
id: runId,
},
{
select: {
id: true,
engine: true,
friendlyId: true,
status: true,
lockedAt: true,
runtimeEnvironmentId: true,
lockedToVersionId: true,
_count: {
select: {
attempts: true,
},
},
},
},
this._prisma
);
if (!taskRun) {
logger.error("[TaskRunHeartbeatFailedService] Task run not found", {
runId,
});
return;
}
// v3 (engine V1) shutdown: leave abandoned V1 runs as-is instead of doing
// MarQS/DB work to fail or requeue them. v4 (V2) is unaffected.
if (isV3Disabled() && taskRun.engine === "V1") {
logger.debug("[TaskRunHeartbeatFailedService] Skipping heartbeat for shut-down v3 run", {
runId,
});
return;
}
const env = await controlPlaneResolver.resolveEnv(taskRun.runtimeEnvironmentId);
const lockedWorker = await controlPlaneResolver.resolveRunLockedWorker({
lockedToVersionId: taskRun.lockedToVersionId,
});
if (!env) {
logger.debug("TaskRunHeartbeatFailedService: environment not found", { runId });
return;
}
const service = new FailedTaskRunService();
switch (taskRun.status) {
case "PENDING":
case "DEQUEUED":
case "WAITING_TO_RESUME":
case "PAUSED": {
const backInQueue = await marqs?.nackMessage(taskRun.id);
if (backInQueue) {
logger.debug(
`[TaskRunHeartbeatFailedService] ${taskRun.status} run is back in the queue run`,
{
taskRun,
}
);
} else {
logger.debug(
`[TaskRunHeartbeatFailedService] ${taskRun.status} run not back in the queue, failing`,
{ taskRun }
);
await service.call(taskRun.friendlyId, {
ok: false,
id: taskRun.friendlyId,
retry: undefined,
error: {
type: "INTERNAL_ERROR",
code: TaskRunErrorCodes.TASK_RUN_HEARTBEAT_TIMEOUT,
message: "Did not receive a heartbeat from the worker in time",
},
});
}
break;
}
case "EXECUTING":
case "RETRYING_AFTER_FAILURE": {
logger.debug(`[RequeueTaskRunService] ${taskRun.status} failing task run`, { taskRun });
await service.call(taskRun.friendlyId, {
ok: false,
id: taskRun.friendlyId,
retry: undefined,
error: {
type: "INTERNAL_ERROR",
code: TaskRunErrorCodes.TASK_RUN_HEARTBEAT_TIMEOUT,
message: "Did not receive a heartbeat from the worker in time",
},
});
break;
}
case "DELAYED":
case "PENDING_VERSION":
case "WAITING_FOR_DEPLOY": {
logger.debug(
`[TaskRunHeartbeatFailedService] ${taskRun.status} Removing task run from queue`,
{ taskRun }
);
await marqs?.acknowledgeMessage(
taskRun.id,
"Run is either DELAYED or WAITING_FOR_DEPLOY so we cannot requeue it in TaskRunHeartbeatFailedService"
);
break;
}
case "SYSTEM_FAILURE":
case "INTERRUPTED":
case "CRASHED":
case "COMPLETED_WITH_ERRORS":
case "COMPLETED_SUCCESSFULLY":
case "EXPIRED":
case "TIMED_OUT":
case "CANCELED": {
logger.debug("[TaskRunHeartbeatFailedService] Task run is completed", { taskRun });
await marqs?.acknowledgeMessage(
taskRun.id,
"Task run is already completed in TaskRunHeartbeatFailedService"
);
try {
if (env.type === "DEVELOPMENT") {
return;
}
// Signal to exit any leftover containers
socketIo.coordinatorNamespace.emit("REQUEST_RUN_CANCELLATION", {
version: "v1",
runId: taskRun.id,
// Give the run a few seconds to exit to complete any flushing etc
delayInMs: lockedWorker?.lockedToVersion?.supportsLazyAttempts ? 5_000 : undefined,
});
} catch (error) {
logger.error("[TaskRunHeartbeatFailedService] Error signaling run cancellation", {
runId: taskRun.id,
error: error instanceof Error ? error.message : error,
});
}
break;
}
default: {
assertNever(taskRun.status);
}
}
}
public static async enqueue(runId: string, runAt?: Date, tx?: PrismaClientOrTransaction) {
return await workerQueue.enqueue(
"v3.requeueTaskRun",
{ runId },
{ runAt, jobKey: `requeueTaskRun:${runId}` }
);
}
public static async dequeue(runId: string, tx?: PrismaClientOrTransaction) {
return await workerQueue.dequeue(`requeueTaskRun:${runId}`, { tx });
}
}
apps/webapp/package.json
-36
View File
@@ -30,15 +30,12 @@
"@ariakit/react-core": "^0.4.6",
"@aws-sdk/client-ecr": "^3.931.0",
"@aws-sdk/client-s3": "^3.936.0",
"@aws-sdk/client-sqs": "^3.445.0",
"@aws-sdk/client-sts": "^3.840.0",
"@aws-sdk/credential-provider-node": "^3.936.0",
"@aws-sdk/s3-presigned-post": "^3.936.0",
"@aws-sdk/s3-request-presigner": "^3.936.0",
"@better-auth/utils": "^0.2.6",
"@codemirror/autocomplete": "6.4.0",
"@codemirror/commands": "6.1.3",
"@codemirror/lang-javascript": "6.1.2",
"@codemirror/lang-json": "6.0.1",
"@codemirror/lang-sql": "6.5.5",
"@codemirror/language": "6.3.2",
@@ -50,7 +47,6 @@
"@conform-to/zod": "^1.2.2",
"@depot/cli": "0.0.1-cli.2.80.0",
"@depot/sdk-node": "^1.0.0",
"@electric-sql/react": "^0.3.5",
"@headlessui/react": "^1.7.8",
"@heroicons/react": "^2.0.12",
"@internal/cache": "workspace:*",
@@ -65,7 +61,6 @@
"@internal/schedule-engine": "workspace:*",
"@internal/tracing": "workspace:*",
"@internal/tsql": "workspace:*",
"@internal/zod-worker": "workspace:*",
"@internationalized/date": "^3.5.1",
"@jsonhero/schema-infer": "^0.1.5",
"@kapaai/react-sdk": "^0.1.3",
@@ -85,7 +80,6 @@
"@opentelemetry/resources": "2.7.1",
"@opentelemetry/sdk-logs": "0.218.0",
"@opentelemetry/sdk-metrics": "2.7.1",
"@opentelemetry/sdk-node": "0.218.0",
"@opentelemetry/sdk-trace-base": "2.7.1",
"@opentelemetry/sdk-trace-node": "2.7.1",
"@opentelemetry/semantic-conventions": "1.41.1",
@@ -94,11 +88,8 @@
"@radix-ui/react-accordion": "^1.2.11",
"@radix-ui/react-alert-dialog": "^1.0.4",
"@radix-ui/react-dialog": "^1.0.3",
"@radix-ui/react-label": "^2.0.1",
"@radix-ui/react-popover": "^1.0.5",
"@radix-ui/react-portal": "^1.1.9",
"@radix-ui/react-radio-group": "^1.1.3",
"@radix-ui/react-select": "^1.2.1",
"@radix-ui/react-slider": "^1.1.2",
"@radix-ui/react-switch": "^1.0.3",
"@radix-ui/react-tabs": "^1.0.3",
@@ -110,9 +101,7 @@
"@remix-run/node": "2.17.5",
"@remix-run/react": "2.17.5",
"@remix-run/router": "^1.23.3",
"@remix-run/serve": "2.17.5",
"@remix-run/server-runtime": "2.17.5",
"@remix-run/v1-meta": "^0.1.3",
"@s2-dev/streamstore": "^0.22.10",
"@sentry/remix": "9.46.0",
"@slack/web-api": "7.16.0",
@@ -133,13 +122,11 @@
"@trigger.dev/redis-worker": "workspace:*",
"@trigger.dev/sdk": "workspace:*",
"@trigger.dev/sso": "workspace:*",
"@types/pg": "8.6.6",
"@uiw/react-codemirror": "^4.19.5",
"@unkey/cache": "^1.5.0",
"@unkey/error": "^0.2.0",
"@upstash/ratelimit": "^1.1.3",
"@vercel/sdk": "^1.19.1",
"@whatwg-node/fetch": "^0.9.14",
"@window-splitter/react": "1.1.3",
"ai": "^6.0.116",
"assert-never": "^1.2.1",
@@ -157,20 +144,15 @@
"dotenv": "^16.4.5",
"effect": "^3.21.2",
"emails": "workspace:*",
"eventsource": "^4.0.0",
"evt": "^2.4.13",
"express": "4.20.0",
"framer-motion": "^10.12.11",
"graphile-worker": "0.16.6",
"humanize-duration": "^3.27.3",
"input-otp": "^1.4.2",
"intl-parse-accept-language": "^1.0.0",
"ioredis": "~5.6.0",
"isbot": "^3.6.5",
"jose": "^5.4.0",
"json-stable-stringify": "^1.3.0",
"jsonpointer": "^5.0.1",
"lodash.omit": "^4.5.0",
"lucide-react": "^0.229.0",
"marked": "^4.0.18",
"match-sorter": "^6.3.4",
@@ -179,7 +161,6 @@
"neverthrow": "^8.2.0",
"non.geist": "^1.0.2",
"octokit": "^3.2.1",
"ohash": "^1.1.3",
"openai": "^4.33.1",
"p-limit": "^6.2.0",
"p-map": "^6.0.0",
@@ -193,8 +174,6 @@
"qrcode.react": "^4.2.0",
"random-words": "^2.0.0",
"react": "^18.2.0",
"react-aria": "^3.31.1",
"react-collapse": "^5.1.1",
"react-day-picker": "^9.13.0",
"react-dom": "^18.2.0",
"react-grid-layout": "^2.2.2",
@@ -202,8 +181,6 @@
"react-markdown": "^10.1.0",
"react-popper": "^2.3.0",
"react-resizable": "^3.1.3",
"react-resizable-panels": "^2.0.9",
"react-stately": "^3.29.1",
"react-use": "17.5.1",
"recharts": "^2.15.2",
"regression": "^2.0.1",
@@ -213,25 +190,18 @@
"remix-auth-google": "^2.0.0",
"remix-typedjson": "0.3.1",
"remix-utils": "^7.7.0",
"seedrandom": "^3.0.5",
"semver": "^7.5.0",
"simple-oauth2": "^5.0.0",
"simplur": "^3.0.1",
"slug": "^6.0.0",
"socket.io": "4.7.4",
"socket.io-adapter": "^2.5.4",
"socket.io-client": "4.7.5",
"sonner": "^1.0.3",
"sql-formatter": "^15.4.10",
"sqs-consumer": "^7.4.0",
"streamdown": "^2.5.0",
"superjson": "^2.2.1",
"tailwind-merge": "^3.6.0",
"tailwind-scrollbar-hide": "^4.0.0",
"tw-animate-css": "^1.4.0",
"tiny-invariant": "^1.2.0",
"ulid": "^2.3.0",
"ulidx": "^2.2.1",
"uuid": "^14.0.0",
"ws": "^8.11.0",
"zod": "3.25.76",
@@ -256,21 +226,15 @@
"@types/compression": "^1.7.2",
"@types/cookie": "^0.6.0",
"@types/express": "^4.17.13",
"@types/humanize-duration": "^3.27.1",
"@types/json-query": "^2.2.3",
"@types/lodash.omit": "^4.5.7",
"@types/marked": "^4.0.3",
"@types/morgan": "^1.9.3",
"@types/node-fetch": "^2.6.2",
"@types/prismjs": "^1.26.0",
"@types/qs": "^6.9.7",
"@types/react": "18.2.69",
"@types/react-collapse": "^5.0.4",
"@types/react-dom": "18.2.7",
"@types/regression": "^2.0.6",
"@types/seedrandom": "^3.0.8",
"@types/semver": "^7.5.0",
"@types/simple-oauth2": "^5.0.4",
"@types/slug": "^5.0.3",
"@types/supertest": "^6.0.2",
"@types/tar": "^6.1.4",
apps/webapp/test/api-auth.e2e.test.ts
+2 -2
View File
@@ -126,13 +126,13 @@ describe("JWT bearer auth — baseline behavior", () => {
// Exercises the RBAC plugin loader end-to-end. The test server boots
// with RBAC_FORCE_FALLBACK=1 (see internal-packages/testcontainers/src/webapp.ts),
// which makes rbac.server.ts use the default fallback regardless of
// whether a plugin is installed in node_modules. /admin/concurrency
// whether a plugin is installed in node_modules. /admin/feature-flags
// uses rbac.authenticateSession internally; an unauthenticated request
// must flow through LazyController → RoleBaseAccessFallback →
// redirect("/login").
describe("RBAC plugin — fallback wiring", () => {
it("unauthenticated dashboard route redirects to /login via the fallback", async () => {
const res = await server.webapp.fetch("/admin/concurrency", { redirect: "manual" });
const res = await server.webapp.fetch("/admin/feature-flags", { redirect: "manual" });
expect(res.status).toBe(302);
const location = res.headers.get("location") ?? "";
expect(new URL(location, "http://placeholder").pathname).toBe("/login");
apps/webapp/test/auth-dashboard.e2e.full.test.ts
+3 -3
View File
@@ -7,9 +7,9 @@ import { getTestServer } from "./helpers/sharedTestServer";
import { seedTestSession, seedTestUser } from "./helpers/seedTestSession";
describe("Dashboard", () => {
it("shared webapp container redirects /admin/concurrency to /login when unauthenticated", async () => {
it("shared webapp container redirects /admin/feature-flags to /login when unauthenticated", async () => {
const server = getTestServer();
const res = await server.webapp.fetch("/admin/concurrency", { redirect: "manual" });
const res = await server.webapp.fetch("/admin/feature-flags", { redirect: "manual" });
expect(res.status).toBe(302);
});
@@ -26,7 +26,7 @@ describe("Dashboard", () => {
// already proves. If the wrapper config drifts per-route in the
// future, add targeted tests for the divergent ones.
describe("Admin pages — requireSuper gate", () => {
const adminRoutes = ["/admin", "/admin/concurrency", "/admin/back-office"];
const adminRoutes = ["/admin", "/admin/feature-flags", "/admin/back-office"];
for (const path of adminRoutes) {
describe(`GET ${path}`, () => {
apps/webapp/test/cancelDevSessionRunsStoreRouting.test.ts
-249
View File
@@ -1,249 +0,0 @@
// Real PG14 (legacy) + PG17 (new) proof for the dev-session-cancel TaskRun read.
// The DB is never mocked: reads hit the two real containers. Only the pure
// splitEnabled boundary and recording client wrappers are injected.
import { heteroPostgresTest, postgresTest } from "@internal/testcontainers";
import type { PrismaClient } from "@trigger.dev/database";
import { generateRunOpsId } from "@trigger.dev/core/v3/isomorphic";
import { describe, expect, vi } from "vitest";
import type { PrismaReplicaClient } from "~/db.server";
import { CancelDevSessionRunsService } from "~/v3/services/cancelDevSessionRuns.server";
vi.setConfig({ testTimeout: 60_000 });
// 25-char cuid body (no v1 version marker) → LEGACY residency.
function generateLegacyCuid() {
const suffix = Array.from(
{ length: 24 },
() => "0123456789abcdefghijklmnopqrstuvwxyz"[Math.floor(Math.random() * 36)]
).join("");
return `c${suffix}`;
}
async function seedOrgProjectEnv(prisma: PrismaClient, suffix: string) {
const organization = await prisma.organization.create({
data: { title: `test-${suffix}`, slug: `test-${suffix}` },
});
const project = await prisma.project.create({
data: {
name: `test-${suffix}`,
slug: `test-${suffix}`,
organizationId: organization.id,
externalRef: `test-${suffix}`,
},
});
const runtimeEnvironment = await prisma.runtimeEnvironment.create({
data: {
slug: `test-${suffix}`,
type: "DEVELOPMENT",
projectId: project.id,
organizationId: organization.id,
apiKey: `test-${suffix}`,
pkApiKey: `test-${suffix}`,
shortcode: `test-${suffix}`,
},
});
return { organization, project, runtimeEnvironment };
}
async function seedRun(
prisma: PrismaClient,
ids: { id: string; friendlyId: string },
env: { runtimeEnvironmentId: string; projectId: string; organizationId: string }
) {
return prisma.taskRun.create({
data: {
id: ids.id,
friendlyId: ids.friendlyId,
taskIdentifier: "my-task",
payload: JSON.stringify({ foo: "bar" }),
payloadType: "application/json",
traceId: "1234",
spanId: "1234",
queue: "test",
runtimeEnvironmentId: env.runtimeEnvironmentId,
projectId: env.projectId,
organizationId: env.organizationId,
environmentType: "DEVELOPMENT",
// V1 so the (best-effort, error-swallowed) cancel does not require the V2 engine;
// the unit under test is the READ resolution, not the cancel side effect.
engine: "V1",
status: "EXECUTING",
},
});
}
// A read client whose taskRun.findFirst is recorded; throws if used after being marked
// forbidden, so we can prove a store was NEVER read.
function recording(client: PrismaClient, opts: { forbidden?: boolean } = {}) {
const calls: unknown[] = [];
const taskRun = {
findFirst: (args: unknown) => {
calls.push(args);
if (opts.forbidden) {
throw new Error("this store must never be read");
}
return (client as unknown as PrismaReplicaClient).taskRun.findFirst(args as never);
},
};
return { handle: { ...client, taskRun } as unknown as PrismaReplicaClient, calls };
}
describe("CancelDevSessionRunsService store routing (hetero)", () => {
heteroPostgresTest(
"a NEW run (run-ops id) resolves on the new store via read-through, by friendlyId and by id",
async ({ prisma17, prisma14 }) => {
const id = generateRunOpsId();
expect(id.length).toBe(26);
const friendlyId = `run_${id}`;
const { project, organization, runtimeEnvironment } = await seedOrgProjectEnv(
prisma17,
"new"
);
await seedRun(
prisma17,
{ id, friendlyId },
{
runtimeEnvironmentId: runtimeEnvironment.id,
projectId: project.id,
organizationId: organization.id,
}
);
// by friendlyId
{
const newClient = recording(prisma17);
const legacy = recording(prisma14, { forbidden: true });
const service = new CancelDevSessionRunsService({
prisma: prisma17,
readThroughDeps: {
splitEnabled: true,
newClient: newClient.handle,
legacyReplica: legacy.handle,
},
});
await service.call({
runIds: [friendlyId],
cancelledAt: new Date(),
reason: "test",
});
// run-ops id → NEW: new store served the read, legacy never touched.
expect(newClient.calls.length).toBe(1);
expect(legacy.calls.length).toBe(0);
}
// by internal id
{
const newClient = recording(prisma17);
const legacy = recording(prisma14, { forbidden: true });
const service = new CancelDevSessionRunsService({
prisma: prisma17,
readThroughDeps: {
splitEnabled: true,
newClient: newClient.handle,
legacyReplica: legacy.handle,
},
});
await service.call({
runIds: [id],
cancelledAt: new Date(),
reason: "test",
});
expect(newClient.calls.length).toBe(1);
expect(legacy.calls.length).toBe(0);
}
}
);
heteroPostgresTest(
"an OLD in-retention run (cuid) resolves off the LEGACY replica, never a legacy primary",
async ({ prisma17, prisma14 }) => {
const id = generateLegacyCuid();
expect(id.length).toBe(25);
const friendlyId = `run_${id}`;
const { project, organization, runtimeEnvironment } = await seedOrgProjectEnv(
prisma14,
"legacy"
);
await seedRun(
prisma14,
{ id, friendlyId },
{
runtimeEnvironmentId: runtimeEnvironment.id,
projectId: project.id,
organizationId: organization.id,
}
);
const newClient = recording(prisma17);
const legacy = recording(prisma14);
const service = new CancelDevSessionRunsService({
prisma: prisma14,
readThroughDeps: {
splitEnabled: true,
newClient: newClient.handle,
legacyReplica: legacy.handle,
},
});
await service.call({
runIds: [id],
cancelledAt: new Date(),
reason: "test",
});
// NEW first (miss) → resolved off the LEGACY REPLICA handle (no primary handle exists).
expect(newClient.calls.length).toBe(1);
expect(legacy.calls.length).toBe(1);
}
);
});
describe("CancelDevSessionRunsService passthrough (single-DB)", () => {
postgresTest(
"with no read-through deps, the run is read from the single DB and session reads stay on it",
async ({ prisma }) => {
const id = generateRunOpsId();
const friendlyId = `run_${id}`;
const { project, organization, runtimeEnvironment } = await seedOrgProjectEnv(prisma, "pt");
await seedRun(
prisma,
{ id, friendlyId },
{
runtimeEnvironmentId: runtimeEnvironment.id,
projectId: project.id,
organizationId: organization.id,
}
);
const session = await prisma.runtimeEnvironmentSession.create({
data: { environmentId: runtimeEnvironment.id, ipAddress: "127.0.0.1" },
});
// splitEnabled=false → single plain read against the one client; the session
// control-plane read runs on the same prisma.
const service = new CancelDevSessionRunsService({
prisma,
replica: prisma,
readThroughDeps: {
splitEnabled: false,
newClient: prisma as unknown as PrismaReplicaClient,
},
});
await service.call({
runIds: [id],
cancelledAt: new Date(),
reason: "test",
cancelledSessionId: session.id,
});
// Run found + handed to cancel against the single DB; confirm the row is present.
const row = await prisma.taskRun.findFirst({ where: { id } });
expect(row).not.toBeNull();
expect(row?.friendlyId).toBe(friendlyId);
}
);
});
apps/webapp/test/createCheckpoint.batchReplicaLag.test.ts
-65
View File
@@ -1,65 +0,0 @@
// Unit red-green for the checkpoint WAIT_FOR_BATCH replica-lag fix (createCheckpoint.server.ts).
// The service decides whether to suspend a run on `batchRun.resumedAt`; reading it from a lagging
// replica makes a just-resumed batch look unresumed -> it suspends an already-resumed run -> stall.
// The fix threads the primary (`this._prisma`) into `runStore.findBatchTaskRunByFriendlyId`. Here a
// spy runStore records which client the service passed and simulates the lag (only the primary read
// sees the fresh resumedAt): RED = no client -> stale null -> no early return; GREEN = primary -> kept alive.
import { describe, expect, it, vi } from "vitest";
vi.mock("~/services/logger.server", () => ({
logger: { debug: vi.fn(), info: vi.fn(), log: vi.fn(), error: vi.fn(), warn: vi.fn() },
}));
vi.mock("~/v3/marqs/index.server", () => ({
marqs: { replaceMessage: vi.fn(), cancelHeartbeat: vi.fn() },
}));
import { CreateCheckpointService } from "~/v3/services/createCheckpoint.server";
describe("checkpoint WAIT_FOR_BATCH reads the primary, not a lagging replica", () => {
it("threads the primary so an already-resumed batch keeps the run alive", async () => {
// A freezable attempt so control reaches the WAIT_FOR_BATCH arm. This object IS the primary the
// fix must thread into the batch read.
const prisma = {
taskRunAttempt: {
findFirst: async () => ({
id: "attempt_1",
status: "EXECUTING",
taskRunId: "run_1",
taskRun: { id: "run_1", status: "EXECUTING", runtimeEnvironmentId: "env_1" },
backgroundWorker: { id: "bw_1", deployment: { imageReference: "img:1" } },
}),
},
};
let seenClient: unknown = "NOT_CALLED";
const runStore = {
findBatchTaskRunByFriendlyId: async (
_friendlyId: string,
_environmentId: string,
_args: unknown,
client?: unknown
) => {
seenClient = client;
// Lagging replica: only a read handed the primary sees the just-committed resumedAt.
return { resumedAt: client === prisma ? new Date() : null };
},
};
const service = new CreateCheckpointService(prisma as never, {} as never, runStore as never);
let result: unknown;
try {
result = await service.call({
attemptFriendlyId: "attempt_1",
reason: { type: "WAIT_FOR_BATCH", batchFriendlyId: "batch_1" },
} as never);
} catch {
// Buggy path falls through the pre-check into checkpoint creation (unstubbed) and throws; the
// recorded client below is what distinguishes RED from GREEN.
}
expect(seenClient).toBe(prisma); // the fix: primary threaded into the batch read
expect(result).toEqual({ success: false, keepRunAlive: true }); // early-return, run kept alive
});
});
apps/webapp/test/fairDequeuingStrategy.test.ts
-1478
View File
File diff suppressed because it is too large Load Diff
apps/webapp/test/marqsKeyProducer.test.ts
-226
View File
@@ -1,226 +0,0 @@
import { describe, it, expect } from "vitest";
import { MarQSShortKeyProducer } from "../app/v3/marqs/marqsKeyProducer.js";
import type { MarQSKeyProducerEnv } from "~/v3/marqs/types.js";
describe("MarQSShortKeyProducer", () => {
const prefix = "test:";
const producer = new MarQSShortKeyProducer(prefix);
// Sample test data
const sampleEnv: MarQSKeyProducerEnv = {
id: "123456789012345678901234",
organizationId: "987654321098765432109876",
type: "PRODUCTION",
};
const devEnv: MarQSKeyProducerEnv = {
id: "123456789012345678901234",
organizationId: "987654321098765432109876",
type: "DEVELOPMENT",
};
describe("sharedQueueScanPattern", () => {
it("should return correct shared queue scan pattern", () => {
expect(producer.sharedQueueScanPattern()).toBe("test:*sharedQueue");
});
});
describe("queueCurrentConcurrencyScanPattern", () => {
it("should return correct queue current concurrency scan pattern", () => {
expect(producer.queueCurrentConcurrencyScanPattern()).toBe(
"test:org:*:env:*:queue:*:currentConcurrency"
);
});
});
describe("stripKeyPrefix", () => {
it("should strip prefix from key if present", () => {
expect(producer.stripKeyPrefix("test:someKey")).toBe("someKey");
});
it("should return original key if prefix not present", () => {
expect(producer.stripKeyPrefix("someKey")).toBe("someKey");
});
});
describe("queueKey", () => {
it("should generate queue key with environment object", () => {
expect(producer.queueKey(sampleEnv, "testQueue")).toBe(
"org:765432109876:env:345678901234:queue:testQueue"
);
});
it("should generate queue key with separate parameters", () => {
expect(producer.queueKey("org123", "env456", "testQueue")).toBe(
"org:org123:env:env456:queue:testQueue"
);
});
it("should include concurrency key when provided", () => {
expect(producer.queueKey(sampleEnv, "testQueue", "concKey")).toBe(
"org:765432109876:env:345678901234:queue:testQueue:ck:concKey"
);
});
});
describe("queueKeyFromQueue", () => {
it("should generate queue key", () => {
expect(producer.queueKeyFromQueue("org:765432109876:env:345678901234:queue:testQueue")).toBe(
"org:765432109876:env:345678901234:queue:testQueue"
);
});
it("should include concurrency key when provided", () => {
expect(
producer.queueKeyFromQueue("org:765432109876:env:345678901234:queue:testQueue:ck:concKey")
).toBe("org:765432109876:env:345678901234:queue:testQueue:ck:concKey");
});
});
describe("envSharedQueueKey", () => {
it("should return organization-specific shared queue for development environment", () => {
expect(producer.envSharedQueueKey(devEnv)).toBe(
"org:765432109876:env:345678901234:sharedQueue"
);
});
it("should return global shared queue for production environment", () => {
expect(producer.envSharedQueueKey(sampleEnv)).toBe("sharedQueue");
});
});
describe("queueDescriptorFromQueue", () => {
it("should parse queue string into descriptor", () => {
const queueString = "org:123:env:456:queue:testQueue:ck:concKey";
const descriptor = producer.queueDescriptorFromQueue(queueString);
expect(descriptor).toEqual({
name: "testQueue",
environment: "456",
organization: "123",
concurrencyKey: "concKey",
});
});
it("should parse queue string without optional parameters", () => {
const queueString = "org:123:env:456:queue:testQueue";
const descriptor = producer.queueDescriptorFromQueue(queueString);
expect(descriptor).toEqual({
name: "testQueue",
environment: "456",
organization: "123",
concurrencyKey: undefined,
});
});
it("should throw error for invalid queue string", () => {
const invalidQueue = "invalid:queue:string";
expect(() => producer.queueDescriptorFromQueue(invalidQueue)).toThrow("Invalid queue");
});
});
describe("messageKey", () => {
it("should generate correct message key", () => {
expect(producer.messageKey("msg123")).toBe("message:msg123");
});
});
describe("nackCounterKey", () => {
it("should generate correct nack counter key", () => {
expect(producer.nackCounterKey("msg123")).toBe("message:msg123:nacks");
});
});
describe("currentConcurrencyKey", () => {
it("should generate correct current concurrency key", () => {
expect(producer.queueCurrentConcurrencyKey(sampleEnv, "testQueue")).toBe(
"org:765432109876:env:345678901234:queue:testQueue:currentConcurrency"
);
});
it("should include concurrency key when provided", () => {
expect(producer.queueCurrentConcurrencyKey(sampleEnv, "testQueue", "concKey")).toBe(
"org:765432109876:env:345678901234:queue:testQueue:ck:concKey:currentConcurrency"
);
});
});
describe("currentConcurrencyKeyFromQueue", () => {
it("should generate correct current concurrency key", () => {
expect(
producer.queueCurrentConcurrencyKeyFromQueue(
"org:765432109876:env:345678901234:queue:testQueue"
)
).toBe("org:765432109876:env:345678901234:queue:testQueue:currentConcurrency");
});
it("should include concurrency key when provided", () => {
expect(
producer.queueCurrentConcurrencyKeyFromQueue(
"org:765432109876:env:345678901234:queue:testQueue:ck:concKey"
)
).toBe("org:765432109876:env:345678901234:queue:testQueue:ck:concKey:currentConcurrency");
});
});
describe("queueReserveConcurrencyKeyFromQueue", () => {
it("should generate correct queue reserve concurrency key", () => {
expect(
producer.queueReserveConcurrencyKeyFromQueue(
"org:765432109876:env:345678901234:queue:testQueue"
)
).toBe("org:765432109876:env:345678901234:queue:testQueue:reserveConcurrency");
});
it("should NOT include the concurrency key when provided", () => {
expect(
producer.queueReserveConcurrencyKeyFromQueue(
"org:765432109876:env:345678901234:queue:testQueue:ck:concKey"
)
).toBe("org:765432109876:env:345678901234:queue:testQueue:reserveConcurrency");
});
});
describe("queueConcurrencyLimitKeyFromQueue", () => {
it("should generate correct queue concurrency limit key", () => {
expect(
producer.queueConcurrencyLimitKeyFromQueue(
"org:765432109876:env:345678901234:queue:testQueue"
)
).toBe("org:765432109876:env:345678901234:queue:testQueue:concurrency");
});
it("should NOT include the concurrency key when provided", () => {
expect(
producer.queueConcurrencyLimitKeyFromQueue(
"org:765432109876:env:345678901234:queue:testQueue:ck:concKey"
)
).toBe("org:765432109876:env:345678901234:queue:testQueue:concurrency");
});
});
describe("envCurrentConcurrencyKey", () => {
it("should generate correct env current concurrency key with environment object", () => {
expect(producer.envCurrentConcurrencyKey(sampleEnv)).toBe(
"env:345678901234:currentConcurrency"
);
});
it("should generate correct env current concurrency key with env id", () => {
expect(producer.envCurrentConcurrencyKey("env456")).toBe("env:env456:currentConcurrency");
});
});
describe("orgIdFromQueue and envIdFromQueue", () => {
it("should extract org id from queue string", () => {
const queue = "org:123:env:456:queue:testQueue";
expect(producer.orgIdFromQueue(queue)).toBe("123");
});
it("should extract env id from queue string", () => {
const queue = "org:123:env:456:queue:testQueue";
expect(producer.envIdFromQueue(queue)).toBe("456");
});
});
});
apps/webapp/test/setup.ts
+2 -7
View File
@@ -33,13 +33,10 @@ function createWorkerStub() {
vi.mock("~/v3/commonWorker.server", () => ({ commonWorker: createWorkerStub() }));
vi.mock("~/v3/batchTriggerWorker.server", () => ({ batchTriggerWorker: createWorkerStub() }));
vi.mock("~/v3/legacyRunEngineWorker.server", () => ({
legacyRunEngineWorker: createWorkerStub(),
}));
vi.mock("~/v3/alertsWorker.server", () => ({ alertsWorker: createWorkerStub() }));
// RunEngine, MarQS, devPubSub and the socket.io server are further singletons
// that open eager ioredis connections at import via the same pattern. No test
// RunEngine and the socket.io server are further singletons that open eager
// ioredis connections at import via the same pattern. No test
// uses these app-level singletons directly (store-routing tests build their own
// engine and run store), so stub them to no-op proxies.
// Recursive no-op proxy: property access at any depth returns another callable
@@ -157,8 +154,6 @@ vi.mock("~/services/dataStores/organizationDataStoresRegistryInstance.server", (
}));
vi.mock("~/v3/runEngine.server", () => ({ engine: noopProxy() }));
vi.mock("~/v3/marqs/index.server", () => ({ marqs: noopProxy(), MarQS: class {} }));
vi.mock("~/v3/marqs/devPubSub.server", () => ({ devPubSub: noopProxy() }));
vi.mock("~/v3/handleSocketIo.server", () => ({
socketIo: noopProxy(),
roomFromFriendlyRunId: (id: string) => `room:${id}`,
apps/webapp/test/triggerV1Scoping.test.ts
-22
View File
@@ -1,22 +0,0 @@
import { describe, expect, it } from "vitest";
import {
attemptInEnvironmentWhere,
batchRunInEnvironmentWhere,
} from "../app/v3/services/triggerV1Scoping.js";
// Caller-supplied parent/dependent attempt & batch friendlyIds must be resolved
// scoped to the caller's environment — a where clause missing that constraint
// lets a foreign id resolve (the cross-tenant bug). Pins the scope on each query.
describe("triggerV1 scoping where-clauses", () => {
it("scopes attempt lookups to the env via the related run", () => {
const where = attemptInEnvironmentWhere("attempt_x", "env_caller");
expect(where.friendlyId).toBe("attempt_x");
expect(where.taskRun).toEqual({ runtimeEnvironmentId: "env_caller" });
});
it("scopes batch-run lookups to the env directly", () => {
const where = batchRunInEnvironmentWhere("batch_x", "env_caller");
expect(where.friendlyId).toBe("batch_x");
expect(where.runtimeEnvironmentId).toBe("env_caller");
});
});
apps/webapp/test/utils/marqs.ts
-116
View File
@@ -1,116 +0,0 @@
import type { MarQSKeyProducer } from "~/v3/marqs/types";
import { MarQSShortKeyProducer } from "~/v3/marqs/marqsKeyProducer.js";
import type Redis from "ioredis";
export function createKeyProducer(prefix: string): MarQSKeyProducer {
return new MarQSShortKeyProducer(prefix);
}
export type SetupQueueOptions = {
parentQueue: string;
redis: Redis;
score: number;
queueId: string;
orgId: string;
envId: string;
keyProducer: MarQSKeyProducer;
};
export type ConcurrencySetupOptions = {
keyProducer: MarQSKeyProducer;
redis: Redis;
orgId: string;
envId: string;
currentConcurrency?: number;
orgLimit?: number;
envLimit?: number;
isOrgDisabled?: boolean;
};
/**
* Adds a queue to Redis with the given parameters
*/
export async function setupQueue({
redis,
keyProducer,
parentQueue,
score,
queueId,
orgId,
envId,
}: SetupQueueOptions) {
// Add the queue to the parent queue's sorted set
const queue = keyProducer.queueKey(orgId, envId, queueId);
await redis.zadd(parentQueue, score, queue);
}
type SetupConcurrencyOptions = {
redis: Redis;
keyProducer: MarQSKeyProducer;
env: { id: string; currentConcurrency: number; limit?: number; reserveConcurrency?: number };
};
/**
* Sets up concurrency-related Redis keys for orgs and envs
*/
export async function setupConcurrency({ redis, keyProducer, env }: SetupConcurrencyOptions) {
// Set env concurrency limit
if (typeof env.limit === "number") {
await redis.set(keyProducer.envConcurrencyLimitKey(env.id), env.limit.toString());
}
if (env.currentConcurrency > 0) {
// Set current concurrency by adding dummy members to the set
const envCurrentKey = keyProducer.envCurrentConcurrencyKey(env.id);
// Add dummy running job IDs to simulate current concurrency
const dummyJobs = Array.from(
{ length: env.currentConcurrency },
(_, i) => `dummy-job-${i}-${Date.now()}`
);
await redis.sadd(envCurrentKey, ...dummyJobs);
}
if (env.reserveConcurrency && env.reserveConcurrency > 0) {
// Set reserved concurrency by adding dummy members to the set
const envReservedKey = keyProducer.envReserveConcurrencyKey(env.id);
// Add dummy reserved job IDs to simulate reserved concurrency
const dummyJobs = Array.from(
{ length: env.reserveConcurrency },
(_, i) => `dummy-reserved-job-${i}-${Date.now()}`
);
await redis.sadd(envReservedKey, ...dummyJobs);
}
}
/**
* Calculates the standard deviation of a set of numbers.
* Standard deviation measures the amount of variation of a set of values from their mean.
* A low standard deviation indicates that the values tend to be close to the mean.
*
* @param values Array of numbers to calculate standard deviation for
* @returns The standard deviation of the values
*/
export function calculateStandardDeviation(values: number[]): number {
// If there are no values or only one value, the standard deviation is 0
if (values.length <= 1) {
return 0;
}
// Calculate the mean (average) of the values
const mean = values.reduce((sum, value) => sum + value, 0) / values.length;
// Calculate the sum of squared differences from the mean
const squaredDifferences = values.map((value) => Math.pow(value - mean, 2));
const sumOfSquaredDifferences = squaredDifferences.reduce((sum, value) => sum + value, 0);
// Calculate the variance (average of squared differences)
const variance = sumOfSquaredDifferences / (values.length - 1); // Using n-1 for sample standard deviation
// Standard deviation is the square root of the variance
return Math.sqrt(variance);
}
docs/self-hosting/docker.mdx
-2
View File
@@ -390,8 +390,6 @@ See the [webapp environment variables](/self-hosting/env/webapp) for the full li
docker compose logs -f webapp
```
- **Deploy fails with `ERROR: schema "graphile_worker" does not exist`.** This error occurs when Graphile Worker migrations fail to run during webapp startup. Check the webapp logs for certificate-related errors like `self-signed certificate in certificate chain`. This is often caused by PostgreSQL SSL certificate issues when using an external PostgreSQL instance with SSL enabled. Ensure that both the webapp and supervisor containers have access to the same CA certificate used by your PostgreSQL instance. You can configure this by mounting the certificate file and setting the `NODE_EXTRA_CA_CERTS` environment variable to point to the certificate path. Once the certificate issue is resolved, the migrations will complete and create the required `graphile_worker` schema.
- **ClickHouse migrations say "no migrations to run" but schema is missing.** The goose migration tracker is out of sync. Exec into the webapp container, set the GOOSE env vars (from webapp startup logs), and run `goose reset && goose up`.
<Warning>
docs/self-hosting/env/webapp.mdx
Vendored
+2 -7
View File
@@ -52,11 +52,8 @@ mode: "wide"
| `AWS_REGION` | No | — | AWS region for SES. |
| `AWS_ACCESS_KEY_ID` | No | — | AWS access key ID for SES. |
| `AWS_SECRET_ACCESS_KEY` | No | — | AWS secret access key for SES. |
| **Graphile & Redis worker** | | | |
| `WORKER_CONCURRENCY` | No | 10 | Redis worker concurrency. |
| `WORKER_POLL_INTERVAL` | No | 1000 | Redis worker poll interval (ms). |
| `WORKER_SCHEMA` | No | graphile_worker | Graphile worker schema. |
| `GRACEFUL_SHUTDOWN_TIMEOUT` | No | 60000 (1m) | Graphile graceful shutdown timeout (ms). Affects shutdown time. |
| **Worker** | | | |
| `GRACEFUL_SHUTDOWN_TIMEOUT` | No | 60000 (1m) | Graceful shutdown timeout (ms). Affects shutdown time. |
| **Concurrency limits** | | | |
| `DEFAULT_ENV_EXECUTION_CONCURRENCY_LIMIT` | No | 100 | Default env execution concurrency. |
| `DEFAULT_ORG_EXECUTION_CONCURRENCY_LIMIT` | No | 300 | Default org execution concurrency, needs to be 3x env concurrency. |
@@ -168,8 +165,6 @@ mode: "wide"
| `MAXIMUM_DEV_QUEUE_SIZE` | No | — | Maximum queued runs per queue in development environments. |
| `MAXIMUM_DEPLOYED_QUEUE_SIZE` | No | — | Maximum queued runs per queue in deployed (staging/prod) environments. |
| **Misc** | | | |
| `PROVIDER_SECRET` | No | provider-secret | Secret for provider auth. **Must be set to a secure value in self-hosted/production**; the default is insecure. |
| `COORDINATOR_SECRET` | No | coordinator-secret | Secret for coordinator auth. **Must be set to a secure value in self-hosted/production**; the default is insecure. |
| `TRIGGER_TELEMETRY_DISABLED` | No | — | Disable telemetry. |
| `NODE_MAX_OLD_SPACE_SIZE` | No | 8192 | Maximum memory allocation for Node.js heap in MiB (e.g. "4096" for 4GB). |
| `OPENAI_API_KEY` | No | — | OpenAI API key. |
docs/self-hosting/kubernetes.mdx
-1
View File
@@ -602,7 +602,6 @@ kubectl delete namespace trigger
- **Deploy fails**: Verify registry access and authentication
- **Pods stuck pending**: Describe the pod and check the events
- **Worker token issues**: Check webapp and supervisor logs for errors
- **Deploy fails with `ERROR: schema "graphile_worker" does not exist`**: See the [Docker troubleshooting](/self-hosting/docker#troubleshooting) section for details on resolving PostgreSQL SSL certificate issues that prevent Graphile Worker migrations.
See the [Docker troubleshooting](/self-hosting/docker#troubleshooting) section for more information.
internal-packages/database/CLAUDE.md
+1 -1
View File
@@ -10,7 +10,7 @@ Located at `prisma/schema.prisma`. Key models include TaskRun, BackgroundWorker,
```prisma
enum RunEngineVersion {
V1 // Legacy (MarQS + Graphile) - DEPRECATED
V1 // Retired v3 engine - no longer executes; kept for historical rows and rejection
V2 // Current (run-engine + redis-worker)
}
```
internal-packages/testcontainers/src/webapp.ts
-1
View File
@@ -116,7 +116,6 @@ export async function startWebapp(
RUN_ENGINE_WORKER_ENABLED: "0", // disables run engine workers (checked === "0", default "1")
SCHEDULE_WORKER_ENABLED: "0", // disables schedule engine worker (checked === "0")
BATCH_QUEUE_WORKER_ENABLED: "false", // disables batch queue consumers (BoolEnv)
LEGACY_RUN_ENGINE_WORKER_ENABLED: "0", // disables legacy run engine worker
COMMON_WORKER_ENABLED: "0", // disables common worker
RUN_ENGINE_TTL_SYSTEM_DISABLED: "true", // disables TTL expiry system (BoolEnv)
RUN_ENGINE_TTL_CONSUMERS_DISABLED: "true", // disables TTL consumers (BoolEnv)

Some files were not shown because too many files have changed in this diff Show More