Files
triggerdotdev--trigger.dev/apps/webapp/test/engine/triggerFailedTask.test.ts
Daniel Sutton 092b9ef07a fix(run-ops): DNS-safe, sortable base32hex run id (replace base62 KSUID) (#4154)
## Problem

The run-ops split mints NEW-store run ids as **27-char base62 KSUIDs**.
The supervisor writes the run id into the Kubernetes pod name
(`runner-<id>`), and pod names must be DNS-1123 labels (lowercase
`[a-z0-9-]`) — so uppercase base62 ids make k8s reject the pod (422) and
**those runs never launch** (they loop in `PENDING_EXECUTING` until the
heartbeat-stall handler nacks them, forever). `.toLowerCase()` can't fix
it: base62 has both `A`(10) and `a`(36) as distinct symbols, so folding
collides distinct ids and destroys sort order.

## Fix: change the encoding, not the structure

Mint a **26-char lowercase base32hex** run id:

```
run_<24-char base32hex core><region char><version char>
      [ 6-byte ms timestamp ][ 9 CSPRNG bytes ]
```

- **base32hex** (RFC 4648 §7, alphabet `0-9a-v`): lowercase,
order-preserving, DNS-safe; 15 bytes → exactly 24 chars, no padding.
Hand-rolled encode/decode (no new dependency).
- **48-bit ms timestamp** in the leading bytes → plain string sort ==
creation order at millisecond resolution.
- **72 bits CSPRNG** entropy; PK unique constraint is the backstop (no
retry loop).
- **region / version** are raw positional chars (read via one `charAt`
before decoding/routing), version = `"1"`.

DNS-safe from birth and hyphen-free, so **firekeeper is unchanged** —
`runner-<id>-attempt-N` → strip `runner-`, cut at first hyphen still
recovers the exact id incl. region+version.

## Residency discriminator: length → version char

`classifyKind`/`classifyResidency` (`runOpsResidency.ts`) previously
distinguished NEW vs LEGACY by **id length**. That gets ambiguous with a
third format. It now discriminates on the **version char at a fixed
position** (`isRunOpsIdBody`: 26 chars, `[25] === "1"`, base32hex
alphabet) → NEW; everything else → LEGACY. Total, never throws. The
`Residency` (NEW/LEGACY) contract the routing store consumes is
unchanged; the `"ksuid"` `ResidencyKind` label is retained only because
it's the persisted `runOpsMintKsuid` feature-flag value.

## Scope / verification

- Generator + discriminator in `@trigger.dev/core` isomorphic; mint path
+ all id-shape call sites swept (~40 webapp files); changeset added
(`@trigger.dev/core` patch).
- Core unit tests (encode/decode round-trip + property, generator shape,
ms sort-order incl. intra-second, parse partitioned-vs-legacy,
firekeeper round-trip): **24 pass**. `@trigger.dev/core` builds; webapp
typechecks; format/lint clean.

## Open decisions (flagged, not silently chosen)

1. **Backward-compat**: existing 27-char base62 KSUID runs now classify
LEGACY. On test cloud these are the broken/looping runs that never
completed, so this is acceptable — but worth a conscious call before
prod. No transitional length-recognition added (keeps the discriminator
clean).
2. **Storage collation**: the sort guarantee is byte-order — if the
run-ops id column is `TEXT` with default locale collation it's silently
not honored. Confirm whether `COLLATE "C"` / `BYTEA` is needed on the
run-ops schema.
3. **Region sourcing** wiring — see `regionCharForRegion` /
`REGION_CODES`.


---

## ⚠️ Required migration — deploy in lockstep

This PR renames a persisted feature-flag key/value and an env var. These
are **not** changed by the code alone and must be migrated when this
deploys, or affected orgs silently fall back to `cuid` minting (no crash
— `defaultValue: "cuid"`):

1. **Env var** (terraform): `RUN_OPS_MINT_KSUID_ENABLED` →
`RUN_OPS_MINT_ENABLED` (carry the value over).
2. **DB** `organization.featureFlags`: migrate both the key and value
together:
   - key `runOpsMintKsuid` → `runOpsMintKind`
   - value `"ksuid"` → `"runOpsId"`

Until an org's flag row is migrated, its `runOpsMintKind` lookup misses
and it mints `cuid` (legacy) — so no NEW-store ids for that org until
the data lands.

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-05 10:05:54 +01:00

265 lines
9.1 KiB
TypeScript

import { describe, expect } from "vitest";
import { RunEngine } from "@internal/run-engine";
import { setupAuthenticatedEnvironment, setupBackgroundWorker } from "@internal/run-engine/tests";
import { containerTest } from "@internal/testcontainers";
import { trace } from "@opentelemetry/api";
import { RunId, classifyKind, generateRunOpsId } from "@trigger.dev/core/v3/isomorphic";
import { TriggerFailedTaskService } from "../../app/runEngine/services/triggerFailedTask.server";
import { EventRepository } from "../../app/v3/eventRepository/eventRepository.server";
vi.setConfig?.({ testTimeout: 60_000 });
// Bind the service's trace-event writes to the testcontainer DB. Without this,
// call() resolves the repository via getEventRepository → global prisma, which
// points at a database that doesn't exist in CI.
function makeService(prisma: any, engine: RunEngine) {
return new TriggerFailedTaskService({
prisma,
engine,
// Read the parent through the same store the engine wrote it to.
runStore: engine.runStore,
eventRepository: {
repository: new EventRepository(prisma, prisma, {
batchSize: 100,
batchInterval: 1000,
retentionInDays: 30,
partitioningEnabled: false,
}),
store: "taskEvent",
},
});
}
function makeEngine(prisma: any, redisOptions: any) {
return new RunEngine({
prisma,
worker: { redis: redisOptions, workers: 1, tasksPerWorker: 10, pollIntervalMs: 100 },
queue: { redis: redisOptions },
runLock: { redis: redisOptions },
machines: {
defaultMachine: "small-1x",
machines: {
"small-1x": {
name: "small-1x" as const,
cpu: 0.5,
memory: 0.5,
centsPerMs: 0.0001,
},
},
baseCostInCents: 0.0005,
},
tracer: trace.getTracer("test", "0.0.0"),
});
}
describe("TriggerFailedTaskService — failed run residency", () => {
containerTest(
"root failed run mints cuid when split is off (call)",
async ({ prisma, redisOptions }) => {
const engine = makeEngine(prisma, redisOptions);
const environment = await setupAuthenticatedEnvironment(prisma, "PRODUCTION");
const taskIdentifier = "failed-residency-task";
await setupBackgroundWorker(engine, environment, taskIdentifier);
const service = makeService(prisma, engine);
const friendlyId = await service.call({
taskId: taskIdentifier,
environment,
payload: { test: "root" },
errorMessage: "boom",
});
expect(friendlyId).toBeTruthy();
expect(classifyKind(friendlyId!)).toBe("cuid");
// The failed run write must land (persistence) with no parent linkage.
const persisted = await prisma.taskRun.findFirst({ where: { friendlyId: friendlyId! } });
expect(persisted).not.toBeNull();
expect(persisted!.status).toBe("SYSTEM_FAILURE");
expect(persisted!.depth).toBe(0);
expect(persisted!.parentTaskRunId).toBeNull();
await engine.quit();
}
);
containerTest(
"failed child of a NEW (run-ops id) parent mints run-ops id (call)",
async ({ prisma, redisOptions }) => {
const engine = makeEngine(prisma, redisOptions);
const environment = await setupAuthenticatedEnvironment(prisma, "PRODUCTION");
const taskIdentifier = "failed-residency-task";
await setupBackgroundWorker(engine, environment, taskIdentifier);
const parentFriendlyId = RunId.toFriendlyId(generateRunOpsId());
expect(classifyKind(parentFriendlyId)).toBe("runOpsId");
await engine.trigger(
{
friendlyId: parentFriendlyId,
environment,
taskIdentifier,
payload: "{}",
payloadType: "application/json",
traceId: "00000000000000000000000000000000",
spanId: "0000000000000000",
workerQueue: "main",
queue: `task/${taskIdentifier}`,
isTest: false,
tags: [],
} as any,
prisma
);
const service = makeService(prisma, engine);
const friendlyId = await service.call({
taskId: taskIdentifier,
environment,
payload: { test: "child" },
errorMessage: "boom",
parentRunId: parentFriendlyId,
});
expect(classifyKind(friendlyId!)).toBe("runOpsId");
// The failed run write must land (persistence) and link to the resolved parent.
const persisted = await prisma.taskRun.findFirst({ where: { friendlyId: friendlyId! } });
expect(persisted).not.toBeNull();
expect(persisted!.status).toBe("SYSTEM_FAILURE");
const parent = await prisma.taskRun.findFirst({ where: { friendlyId: parentFriendlyId } });
expect(persisted!.parentTaskRunId).toBe(parent!.id);
expect(persisted!.depth).toBe(parent!.depth + 1);
expect(persisted!.rootTaskRunId).toBe(parent!.rootTaskRunId ?? parent!.id);
await engine.quit();
}
);
containerTest(
"failed child of a LEGACY (cuid) parent mints cuid (call)",
async ({ prisma, redisOptions }) => {
const engine = makeEngine(prisma, redisOptions);
const environment = await setupAuthenticatedEnvironment(prisma, "PRODUCTION");
const taskIdentifier = "failed-residency-task";
await setupBackgroundWorker(engine, environment, taskIdentifier);
const parentFriendlyId = RunId.generate().friendlyId; // cuid → LEGACY
expect(classifyKind(parentFriendlyId)).toBe("cuid");
await engine.trigger(
{
friendlyId: parentFriendlyId,
environment,
taskIdentifier,
payload: "{}",
payloadType: "application/json",
traceId: "00000000000000000000000000000000",
spanId: "0000000000000000",
workerQueue: "main",
queue: `task/${taskIdentifier}`,
isTest: false,
tags: [],
} as any,
prisma
);
const service = makeService(prisma, engine);
const friendlyId = await service.call({
taskId: taskIdentifier,
environment,
payload: { test: "child" },
errorMessage: "boom",
parentRunId: parentFriendlyId,
});
expect(classifyKind(friendlyId!)).toBe("cuid");
await engine.quit();
}
);
containerTest(
"failed child of a NEW parent mints run-ops id (callWithoutTraceEvents)",
async ({ prisma, redisOptions }) => {
const engine = makeEngine(prisma, redisOptions);
const environment = await setupAuthenticatedEnvironment(prisma, "PRODUCTION");
const taskIdentifier = "failed-residency-task";
await setupBackgroundWorker(engine, environment, taskIdentifier);
const parentFriendlyId = RunId.toFriendlyId(generateRunOpsId());
await engine.trigger(
{
friendlyId: parentFriendlyId,
environment,
taskIdentifier,
payload: "{}",
payloadType: "application/json",
traceId: "00000000000000000000000000000000",
spanId: "0000000000000000",
workerQueue: "main",
queue: `task/${taskIdentifier}`,
isTest: false,
tags: [],
} as any,
prisma
);
const service = makeService(prisma, engine);
const friendlyId = await service.callWithoutTraceEvents({
environmentId: environment.id,
environmentType: environment.type,
projectId: environment.projectId,
organizationId: environment.organizationId,
taskId: taskIdentifier,
payload: { test: "child" },
errorMessage: "boom",
parentRunId: parentFriendlyId,
});
expect(classifyKind(friendlyId!)).toBe("runOpsId");
await engine.quit();
}
);
containerTest(
"callWithoutTraceEvents returns null (best-effort) when the derived parent row is absent",
async ({ prisma, redisOptions }) => {
const engine = makeEngine(prisma, redisOptions);
const environment = await setupAuthenticatedEnvironment(prisma, "PRODUCTION");
const taskIdentifier = "failed-residency-task";
await setupBackgroundWorker(engine, environment, taskIdentifier);
const service = makeService(prisma, engine);
// A well-formed run-ops parent friendlyId that was NEVER triggered → no row.
// Exercises the missing-parent fallback in callWithoutTraceEvents.
const absentParentFriendlyId = RunId.toFriendlyId(generateRunOpsId());
const friendlyId = await service.callWithoutTraceEvents({
environmentId: environment.id,
environmentType: environment.type,
projectId: environment.projectId,
organizationId: environment.organizationId,
taskId: taskIdentifier,
payload: { test: "absent-parent" },
errorMessage: "boom",
parentRunId: absentParentFriendlyId,
});
// Fallback derives parentTaskRunId from an id with no row; the parentTaskRunId FK rejects the create, so the method returns null instead of throwing.
expect(friendlyId).toBeNull();
const orphan = await prisma.taskRun.findFirst({
where: { parentTaskRunId: RunId.fromFriendlyId(absentParentFriendlyId) },
});
expect(orphan).toBeNull();
await engine.quit();
}
);
});