Files
triggerdotdev--trigger.dev/apps/webapp/test/batchTriggerV3StoreRouting.test.ts
Daniel Sutton 092b9ef07a fix(run-ops): DNS-safe, sortable base32hex run id (replace base62 KSUID) (#4154)
## Problem

The run-ops split mints NEW-store run ids as **27-char base62 KSUIDs**.
The supervisor writes the run id into the Kubernetes pod name
(`runner-<id>`), and pod names must be DNS-1123 labels (lowercase
`[a-z0-9-]`) — so uppercase base62 ids make k8s reject the pod (422) and
**those runs never launch** (they loop in `PENDING_EXECUTING` until the
heartbeat-stall handler nacks them, forever). `.toLowerCase()` can't fix
it: base62 has both `A`(10) and `a`(36) as distinct symbols, so folding
collides distinct ids and destroys sort order.

## Fix: change the encoding, not the structure

Mint a **26-char lowercase base32hex** run id:

```
run_<24-char base32hex core><region char><version char>
      [ 6-byte ms timestamp ][ 9 CSPRNG bytes ]
```

- **base32hex** (RFC 4648 §7, alphabet `0-9a-v`): lowercase,
order-preserving, DNS-safe; 15 bytes → exactly 24 chars, no padding.
Hand-rolled encode/decode (no new dependency).
- **48-bit ms timestamp** in the leading bytes → plain string sort ==
creation order at millisecond resolution.
- **72 bits CSPRNG** entropy; PK unique constraint is the backstop (no
retry loop).
- **region / version** are raw positional chars (read via one `charAt`
before decoding/routing), version = `"1"`.

DNS-safe from birth and hyphen-free, so **firekeeper is unchanged** —
`runner-<id>-attempt-N` → strip `runner-`, cut at first hyphen still
recovers the exact id incl. region+version.

## Residency discriminator: length → version char

`classifyKind`/`classifyResidency` (`runOpsResidency.ts`) previously
distinguished NEW vs LEGACY by **id length**. That gets ambiguous with a
third format. It now discriminates on the **version char at a fixed
position** (`isRunOpsIdBody`: 26 chars, `[25] === "1"`, base32hex
alphabet) → NEW; everything else → LEGACY. Total, never throws. The
`Residency` (NEW/LEGACY) contract the routing store consumes is
unchanged; the `"ksuid"` `ResidencyKind` label is retained only because
it's the persisted `runOpsMintKsuid` feature-flag value.

## Scope / verification

- Generator + discriminator in `@trigger.dev/core` isomorphic; mint path
+ all id-shape call sites swept (~40 webapp files); changeset added
(`@trigger.dev/core` patch).
- Core unit tests (encode/decode round-trip + property, generator shape,
ms sort-order incl. intra-second, parse partitioned-vs-legacy,
firekeeper round-trip): **24 pass**. `@trigger.dev/core` builds; webapp
typechecks; format/lint clean.

## Open decisions (flagged, not silently chosen)

1. **Backward-compat**: existing 27-char base62 KSUID runs now classify
LEGACY. On test cloud these are the broken/looping runs that never
completed, so this is acceptable — but worth a conscious call before
prod. No transitional length-recognition added (keeps the discriminator
clean).
2. **Storage collation**: the sort guarantee is byte-order — if the
run-ops id column is `TEXT` with default locale collation it's silently
not honored. Confirm whether `COLLATE "C"` / `BYTEA` is needed on the
run-ops schema.
3. **Region sourcing** wiring — see `regionCharForRegion` /
`REGION_CODES`.


---

## ⚠️ Required migration — deploy in lockstep

This PR renames a persisted feature-flag key/value and an env var. These
are **not** changed by the code alone and must be migrated when this
deploys, or affected orgs silently fall back to `cuid` minting (no crash
— `defaultValue: "cuid"`):

1. **Env var** (terraform): `RUN_OPS_MINT_KSUID_ENABLED` →
`RUN_OPS_MINT_ENABLED` (carry the value over).
2. **DB** `organization.featureFlags`: migrate both the key and value
together:
   - key `runOpsMintKsuid` → `runOpsMintKind`
   - value `"ksuid"` → `"runOpsId"`

Until an org's flag row is migrated, its `runOpsMintKind` lookup misses
and it mints `cuid` (legacy) — so no NEW-store ids for that org until
the data lands.

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-05 10:05:54 +01:00

253 lines
8.7 KiB
TypeScript

import { heteroPostgresTest } from "@internal/testcontainers";
import { PostgresRunStore } from "@internal/run-store";
import { isUniqueConstraintError, type PrismaClient } from "@trigger.dev/database";
import { generateRunOpsId } from "@trigger.dev/core/v3/isomorphic";
import { describe, expect, vi } from "vitest";
vi.setConfig({ testTimeout: 60_000 });
// Proves BatchTriggerV3's three store seams (cached-run lookup, expired-key clear,
// membership write) route correctly against real PG14 (legacy) + PG17 (run-ops)
// containers, using the service's exact query shapes. The service methods are
// JS #-private, so the seam is driven directly — same approach as the sibling
// legacy-authority test.
async function seedOrgProjectEnv(prisma: PrismaClient, suffix: string) {
const organization = await prisma.organization.create({
data: { title: `test-${suffix}`, slug: `test-${suffix}` },
});
const project = await prisma.project.create({
data: {
name: `test-${suffix}`,
slug: `test-${suffix}`,
organizationId: organization.id,
externalRef: `test-${suffix}`,
},
});
const runtimeEnvironment = await prisma.runtimeEnvironment.create({
data: {
slug: `test-${suffix}`,
type: "DEVELOPMENT",
projectId: project.id,
organizationId: organization.id,
apiKey: `test-${suffix}`,
pkApiKey: `test-${suffix}`,
shortcode: `test-${suffix}`,
},
});
return { organization, project, runtimeEnvironment };
}
async function seedRun(
prisma: PrismaClient,
args: {
runtimeEnvironmentId: string;
projectId: string;
organizationId: string;
taskIdentifier: string;
idempotencyKey?: string;
status?: "PENDING" | "EXECUTING" | "COMPLETED_SUCCESSFULLY" | "COMPLETED_WITH_ERRORS";
idempotencyKeyExpiresAt?: Date;
}
) {
const runId = generateRunOpsId();
return prisma.taskRun.create({
data: {
id: runId,
friendlyId: `run_${runId}`,
taskIdentifier: args.taskIdentifier,
idempotencyKey: args.idempotencyKey ?? null,
idempotencyKeyExpiresAt: args.idempotencyKeyExpiresAt ?? null,
status: args.status ?? "EXECUTING",
payload: JSON.stringify({ foo: "bar" }),
payloadType: "application/json",
traceId: "1234",
spanId: "1234",
queue: "test",
runtimeEnvironmentId: args.runtimeEnvironmentId,
projectId: args.projectId,
organizationId: args.organizationId,
environmentType: "DEVELOPMENT",
engine: "V2",
},
});
}
async function seedBatch(prisma: PrismaClient, runtimeEnvironmentId: string, suffix: string) {
const batchId = generateRunOpsId();
return prisma.batchTaskRun.create({
data: {
id: batchId,
friendlyId: `batch_${suffix}_${batchId}`,
runtimeEnvironmentId,
},
});
}
describe("BatchTriggerV3 · store-seam routing (cross-DB)", () => {
heteroPostgresTest(
"(A) cached-run reuse resolves via the legacy (PG14) authority; a PG17-only key is invisible",
async ({ prisma14, prisma17 }) => {
const { project, organization, runtimeEnvironment } = await seedOrgProjectEnv(
prisma14,
"batch-cached"
);
const newSide = await seedOrgProjectEnv(prisma17, "batch-cached-new");
const legacyStore = new PostgresRunStore({ prisma: prisma14, readOnlyPrisma: prisma14 });
const key1 = "idem-batch-1";
const key2 = "idem-batch-2";
const freshKey = "idem-batch-fresh";
const run1 = await seedRun(prisma14, {
runtimeEnvironmentId: runtimeEnvironment.id,
projectId: project.id,
organizationId: organization.id,
taskIdentifier: "my-task",
idempotencyKey: key1,
});
const run2 = await seedRun(prisma14, {
runtimeEnvironmentId: runtimeEnvironment.id,
projectId: project.id,
organizationId: organization.id,
taskIdentifier: "my-task",
idempotencyKey: key2,
});
// A row with one of the SAME keys lives only on PG17 (run-ops). The
// legacy-pinned read must NOT see it.
await seedRun(prisma17, {
runtimeEnvironmentId: newSide.runtimeEnvironment.id,
projectId: newSide.project.id,
organizationId: newSide.organization.id,
taskIdentifier: "my-task",
idempotencyKey: key1,
});
// The service's exact cached-run query shape, pinned to PG14.
const cachedRuns = await legacyStore.findRuns(
{
where: {
runtimeEnvironmentId: runtimeEnvironment.id,
taskIdentifier: "my-task",
idempotencyKey: { in: [key1, key2, freshKey] },
},
select: {
friendlyId: true,
idempotencyKey: true,
idempotencyKeyExpiresAt: true,
},
},
prisma14
);
// Exactly the 2 seeded rows; the fresh key matches nothing.
expect(cachedRuns).toHaveLength(2);
const friendlyIds = cachedRuns.map((r) => r.friendlyId).sort();
expect(friendlyIds).toEqual([run1.friendlyId, run2.friendlyId].sort());
// Each friendlyId distinct, exactly one row per seeded key.
expect(new Set(friendlyIds).size).toBe(2);
expect(cachedRuns.filter((r) => r.idempotencyKey === key1)).toHaveLength(1);
expect(cachedRuns.filter((r) => r.idempotencyKey === key2)).toHaveLength(1);
}
);
heteroPostgresTest(
"(B) expired-key clear is routed to the legacy (PG14) authority and does not touch PG17",
async ({ prisma14, prisma17 }) => {
const { project, organization, runtimeEnvironment } = await seedOrgProjectEnv(
prisma14,
"batch-expired"
);
const newSide = await seedOrgProjectEnv(prisma17, "batch-expired-new");
const legacyStore = new PostgresRunStore({ prisma: prisma14, readOnlyPrisma: prisma14 });
const expiredKey = "idem-batch-expired";
const legacyRun = await seedRun(prisma14, {
runtimeEnvironmentId: runtimeEnvironment.id,
projectId: project.id,
organizationId: organization.id,
taskIdentifier: "my-task",
idempotencyKey: expiredKey,
idempotencyKeyExpiresAt: new Date(Date.now() - 60_000),
});
// A PG17 row with the same key, to prove the clear does not reach it.
const newRun = await seedRun(prisma17, {
runtimeEnvironmentId: newSide.runtimeEnvironment.id,
projectId: newSide.project.id,
organizationId: newSide.organization.id,
taskIdentifier: "my-task",
idempotencyKey: expiredKey,
});
// The service's exact expired-key clear shape, pinned to PG14.
await legacyStore.clearIdempotencyKey({ byFriendlyIds: [legacyRun.friendlyId] }, prisma14);
const cleared = await prisma14.taskRun.findFirst({ where: { id: legacyRun.id } });
expect(cleared?.idempotencyKey).toBeNull();
// The PG17 row is untouched.
const untouched = await prisma17.taskRun.findFirst({ where: { id: newRun.id } });
expect(untouched?.idempotencyKey).toBe(expiredKey);
}
);
heteroPostgresTest(
"(C) membership write lands on the run-ops (PG17) store; duplicate raises a unique-constraint error",
async ({ prisma17 }) => {
const { project, organization, runtimeEnvironment } = await seedOrgProjectEnv(
prisma17,
"batch-membership"
);
const runOpsStore = new PostgresRunStore({ prisma: prisma17, readOnlyPrisma: prisma17 });
const batch = await seedBatch(prisma17, runtimeEnvironment.id, "membership");
const run = await seedRun(prisma17, {
runtimeEnvironmentId: runtimeEnvironment.id,
projectId: project.id,
organizationId: organization.id,
taskIdentifier: "my-task",
});
await runOpsStore.createBatchTaskRunItem({
batchTaskRunId: batch.id,
taskRunId: run.id,
status: "PENDING",
});
const item = await prisma17.batchTaskRunItem.findFirst({
where: { batchTaskRunId: batch.id, taskRunId: run.id },
});
expect(item).not.toBeNull();
expect(item?.status).toBe("PENDING");
// Re-calling with the SAME pair raises a unique-constraint error at the
// store layer (the service's try/catch is what swallows it).
let caught: unknown;
try {
await runOpsStore.createBatchTaskRunItem({
batchTaskRunId: batch.id,
taskRunId: run.id,
status: "PENDING",
});
} catch (error) {
caught = error;
}
expect(caught).toBeDefined();
expect(isUniqueConstraintError(caught, ["batchTaskRunId", "taskRunId"])).toBe(true);
// Still exactly one row.
const count = await prisma17.batchTaskRunItem.count({
where: { batchTaskRunId: batch.id, taskRunId: run.id },
});
expect(count).toBe(1);
}
);
});