## Summary
Deleting a project triggered an unbounded database cleanup that scanned
the project's entire run history, so deleting a project with many runs
could be very slow. Project deletion is a soft delete again: run data is
retained and the deletion completes quickly.
## Fix
Project deletion ran a cascade hard-delete whose `BulkActionItem` step
filtered through a relation to `TaskRun` scoped by `projectId`. Prisma
compiles that to an `EXISTS`-join over the project's entire `TaskRun`
set (a large, hot table with no `projectId` index), and it ran on every
project deletion unconditionally.
Removing the cascade-cleanup call restores the prior soft-delete
behaviour: queues are removed, the project is marked deleted, and run
data is retained. The cascade-cleanup service (added in
[#4117](https://github.com/triggerdotdev/trigger.dev/pull/4117)) had no
other callers, so it and its test are deleted.
## Problem
The run-ops split mints NEW-store run ids as **27-char base62 KSUIDs**.
The supervisor writes the run id into the Kubernetes pod name
(`runner-<id>`), and pod names must be DNS-1123 labels (lowercase
`[a-z0-9-]`) — so uppercase base62 ids make k8s reject the pod (422) and
**those runs never launch** (they loop in `PENDING_EXECUTING` until the
heartbeat-stall handler nacks them, forever). `.toLowerCase()` can't fix
it: base62 has both `A`(10) and `a`(36) as distinct symbols, so folding
collides distinct ids and destroys sort order.
## Fix: change the encoding, not the structure
Mint a **26-char lowercase base32hex** run id:
```
run_<24-char base32hex core><region char><version char>
[ 6-byte ms timestamp ][ 9 CSPRNG bytes ]
```
- **base32hex** (RFC 4648 §7, alphabet `0-9a-v`): lowercase,
order-preserving, DNS-safe; 15 bytes → exactly 24 chars, no padding.
Hand-rolled encode/decode (no new dependency).
- **48-bit ms timestamp** in the leading bytes → plain string sort ==
creation order at millisecond resolution.
- **72 bits CSPRNG** entropy; PK unique constraint is the backstop (no
retry loop).
- **region / version** are raw positional chars (read via one `charAt`
before decoding/routing), version = `"1"`.
DNS-safe from birth and hyphen-free, so **firekeeper is unchanged** —
`runner-<id>-attempt-N` → strip `runner-`, cut at first hyphen still
recovers the exact id incl. region+version.
## Residency discriminator: length → version char
`classifyKind`/`classifyResidency` (`runOpsResidency.ts`) previously
distinguished NEW vs LEGACY by **id length**. That gets ambiguous with a
third format. It now discriminates on the **version char at a fixed
position** (`isRunOpsIdBody`: 26 chars, `[25] === "1"`, base32hex
alphabet) → NEW; everything else → LEGACY. Total, never throws. The
`Residency` (NEW/LEGACY) contract the routing store consumes is
unchanged; the `"ksuid"` `ResidencyKind` label is retained only because
it's the persisted `runOpsMintKsuid` feature-flag value.
## Scope / verification
- Generator + discriminator in `@trigger.dev/core` isomorphic; mint path
+ all id-shape call sites swept (~40 webapp files); changeset added
(`@trigger.dev/core` patch).
- Core unit tests (encode/decode round-trip + property, generator shape,
ms sort-order incl. intra-second, parse partitioned-vs-legacy,
firekeeper round-trip): **24 pass**. `@trigger.dev/core` builds; webapp
typechecks; format/lint clean.
## Open decisions (flagged, not silently chosen)
1. **Backward-compat**: existing 27-char base62 KSUID runs now classify
LEGACY. On test cloud these are the broken/looping runs that never
completed, so this is acceptable — but worth a conscious call before
prod. No transitional length-recognition added (keeps the discriminator
clean).
2. **Storage collation**: the sort guarantee is byte-order — if the
run-ops id column is `TEXT` with default locale collation it's silently
not honored. Confirm whether `COLLATE "C"` / `BYTEA` is needed on the
run-ops schema.
3. **Region sourcing** wiring — see `regionCharForRegion` /
`REGION_CODES`.
---
## ⚠️ Required migration — deploy in lockstep
This PR renames a persisted feature-flag key/value and an env var. These
are **not** changed by the code alone and must be migrated when this
deploys, or affected orgs silently fall back to `cuid` minting (no crash
— `defaultValue: "cuid"`):
1. **Env var** (terraform): `RUN_OPS_MINT_KSUID_ENABLED` →
`RUN_OPS_MINT_ENABLED` (carry the value over).
2. **DB** `organization.featureFlags`: migrate both the key and value
together:
- key `runOpsMintKsuid` → `runOpsMintKind`
- value `"ksuid"` → `"runOpsId"`
Until an org's flag row is migrated, its `runOpsMintKind` lookup misses
and it mints `cuid` (legacy) — so no NEW-store ids for that org until
the data lands.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## What
Wires the run-ops split into the webapp: database topology, environment
flags, split-mode gating, and the control-plane resolver/cache layer
that the run-store and run-engine seams from the previous PR plug into.
- **DB topology & env** (`apps/webapp/app/db.server.ts`,
`env.server.ts`, `entry.server.tsx`): adds the run-ops database
clients/topology and the environment variables that configure and gate
the split.
- **runOpsMigration module** (new
`apps/webapp/app/v3/runOpsMigration/`): the webapp-side machinery —
`splitMode.server.ts`, `controlPlaneResolver.server.ts` +
`controlPlaneCache.server.ts`, `readThrough.server.ts`,
`crossSeamGuard.server.ts`, `distinctDbSentinel.server.ts`, id-minting
helpers (`mintBatchFriendlyId`, `runOpsMintKind`,
`resolveInheritedMintKind`), `runOpsCascadeCleanup.server.ts`, the split
read gate, and route/unblock catalogs.
- **Store/engine wiring** (`app/v3/runStore.server.ts`,
`runEngine.server.ts`, `runEngineHandlers.server.ts` + new
`runEngineHandlersShared.server.ts`): points the webapp's store/engine
construction at the resolver, and factors shared handler logic out so
both seams use one path.
- **Read-path touch-ups**: `runtimeEnvironment.server.ts`,
`eventRepository/index.server.ts`, `taskRunHeartbeatFailed.server.ts`,
`engineVersion.server.ts` route their run/environment lookups
read-through the resolver.
- `413a94511` — interlocks split mode against the native realtime
backend so the two aren't enabled in an incompatible combination (see
`.server-changes/run-ops-split-realtime-interlock.md`).
- `dc74c57fd` — drops the earlier "known-migrated" read layer; residency
is determined by id-shape only.
## Why
PR5 of the run-ops split stack. This is the webapp foundation layer: it
stands up the DB topology, flags, and resolver/cache the rest of the
stack depends on, and repoints webapp read paths through the resolver.
Additive when the split is not enabled (existing single-DB behavior
preserved behind flags); behavior-changing on the read-through paths and
the realtime interlock.
## Tests
New vitest coverage across `apps/webapp/test/` and colocated
`*.server.test.ts` files: db topology, split mode, split read gate,
cross-seam guard, mint cutover / flip latency, control-plane cache,
control-plane resolver, distinct-db sentinel, read-through loaders
(route loaders, run-detail loaders, `findEnvironmentFromRun`), and the
run-engine handlers. Testcontainers-backed; no mocks. `pnpm-lock.yaml`
synced for the two new webapp deps.
## Notes
Draft, **stacked on #4116** (`runops/pr04-store-engine`). Review that
first; this diff is against it.
Server-change / changeset note to be added at stack-assembly time.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Daniel Sutton