main
8 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Eric Allam
|
90e8bd5c12 |
feat(webapp,database): opt-in per-client Prisma driver adapters (#4539)
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 0s
🚀 Publish Trigger.dev Docker / units (push) Failing after 0s
🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker-v4 (push) Has been skipped
🚀 Publish Trigger.dev Docker / scan-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / scan-supervisor (push) Has been skipped
🦋 Changesets PR / Create Release PR (push) Has been cancelled
🚀 Publish Trigger.dev Docker / 📣 Dispatch main image (push) Has been cancelled
📚 Docs Checks / check-broken-links (push) Has been cancelled
🧭 Helm Chart Prerelease / lint-and-test (push) Has been cancelled
Workflow Checks / Actionlint (push) Has been cancelled
Workflow Checks / Zizmor (push) Has been cancelled
🧭 Helm Chart Prerelease / prerelease (push) Has been cancelled
## What
Adds an opt-in path to run each Prisma client through
**`@prisma/adapter-pg`** (the node-postgres driver) instead of the
built-in engine driver, controlled by a **per-client env var, all off by
default**:
| env var | client |
|---|---|
| `CONTROL_PLANE_DATABASE_WRITER_DRIVER_ADAPTER` | control-plane writer
|
| `CONTROL_PLANE_DATABASE_REPLICA_DRIVER_ADAPTER` | control-plane
replica |
| `RUN_OPS_DATABASE_WRITER_DRIVER_ADAPTER` | new run-ops writer |
| `RUN_OPS_DATABASE_REPLICA_DRIVER_ADAPTER` | new run-ops replica |
| `RUN_OPS_LEGACY_DATABASE_WRITER_DRIVER_ADAPTER` | legacy run-ops
writer |
| `RUN_OPS_LEGACY_DATABASE_REPLICA_DRIVER_ADAPTER` | legacy run-ops
replica |
With every flag unset the construction path is byte-identical to today
(`datasources` URL + Rust engine), so this is inert until a flag is
turned on. Per-client granularity allows enabling the adapter only where
it's wanted.
## How
- Enables the `driverAdapters` preview feature on both schemas
(`@trigger.dev/database` and `@internal/run-ops-database`). This keeps
the **Rust query engine** — it does NOT add `queryCompiler` — so query
behavior, result types, and engine tracing spans are unchanged.
- A shared `buildDriverAdapterPool` builds each client's `pg.Pool` with
an explicit `max`, a bounded `connectionTimeoutMillis` (the
node-postgres pool otherwise waits unbounded on acquire), and an
`onPoolError` handler (an unhandled idle-connection error would
otherwise crash the process). Threaded through all four client builders
via a `useDriverAdapter` flag.
- Adds `@prisma/adapter-pg` + `@types/pg` to the webapp; `pg` is already
pinned at `8.15.6` (adapter-pg 6.x requires `pg < 8.17`).
## Connect-failure handling (the important correctness/security bit)
Under the adapter an unreachable DB no longer surfaces as
`PrismaClientInitializationError` / `P1001`; it becomes a `P2010`
"Database not reachable: <host>" (or a raw
`ECONNREFUSED`/`ENOTFOUND`-class error). Two handlers are updated so a
client on the adapter behaves like today:
- **`isInfrastructureError`** now recognizes those shapes (P2010 with a
connectivity message, and raw connectivity errno codes). Without this,
the DB **hostname would leak into API-client-facing errors** and the
failure would go unlogged. Security-relevant.
- **`isPrismaRetriableError`** treats the adapter's pool-acquire timeout
("timeout exceeded when trying to connect") as retriable, preserving the
`P2024` retry behavior the adapter otherwise drops.
## Evidence
Validated on an isolated stack that mirrors the production DB topology
(chained PgBouncers in front of writer + reader):
- **Behavioral parity:** raw-query results and Prisma error codes/`meta`
are byte-identical between the engine driver and the adapter across the
queried shapes (unique-constraint `meta.target`, record-not-found,
transaction-timeout, serialization-failure, etc.).
- **Feature matrix:** a full 380-project queue-ay pass shows no
adapter-caused regressions — pass/fail parity between adapter-off and
adapter-on, with the residual failures being pre-existing
known-failures/flakes common to both.
## Rollout / rollback
All flags default off; enable per client via env var, roll back by
unsetting and redeploying (no data migration). Recommended first target
is a single writer; enable one client at a time.
## Follow-ups (not in this PR)
- `$metrics`-based pool observability is removed under the adapter (the
Prometheus route + `db.pool.connections.*` instruments); the metrics
replacement (via `pg.Pool` counters) lands in a separate PR.
- Note for operators: on the adapter path, interactive-transaction
`maxWait` does not bound pool acquisition — `connectionTimeoutMillis`
does.
## Note on connection-string parameters
The adapter pool is built from the base DSN, so Prisma-specific DSN
parameters that node-postgres does not understand are not honored when a
client is on the adapter:
- **Prisma TLS spellings** (`sslaccept`, `sslcert`, etc.) —
node-postgres uses `sslmode`/`ssl` instead. Our production DSNs do not
use these Prisma-specific TLS params, but any deployment whose DSN
relies on them must be checked before enabling a flag.
- `pgbouncer=true` and `statement_cache_size` — effectively moot under
the adapter, which uses no persistent named prepared statements.
`connection_limit`, `pool_timeout`, and `schema` are handled explicitly
(passed as `max`/`connectionTimeoutMillis` and PrismaPg's `{schema}`
option).
refs TRI-13039
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
Eric Allam
|
fc576436e2 |
perf(run-ops-database): index BatchTaskRun for the batches list on the dedicated schema (#4396)
## Summary The batches list page orders by `(createdAt DESC, id DESC)`, which is why [#4361](https://github.com/triggerdotdev/trigger.dev/pull/4361) added a matching index on `BatchTaskRun`. That index only landed in `@trigger.dev/database`. The dedicated run-ops database has its own migration history, so it never received the index. `BatchListPresenter` reads both databases and merges, so for environments whose batches live in the dedicated database the page kept falling back to a scan and in-memory sort, which is the exact behaviour #4361 set out to fix. ## Fix Adds the index to the run-ops schema with its own migration. `CREATE INDEX CONCURRENTLY IF NOT EXISTS`, so it is a no-op where the index already exists and still records its ledger row. The second half is the interesting part. Because the two packages own separate migration histories, a run-graph schema change has to be authored twice, and nothing made the miss visible: the run-ops status check truthfully reports "up to date" against its own history, so the apply step just skips. `schemaParity.test.ts` compares the physical shape of every model the run-ops schema declares against its counterpart in `@trigger.dev/database`: scalar fields with their attributes, plus `@@index`, `@@unique`, `@@id` and `@@map`. Relation navigation fields are excluded, since the run-ops schema deliberately drops relations that would cross a database boundary while keeping the scalar FK column. A field counts as a relation when its type resolves to a model name, which keeps enum-typed columns in scope. Two models are listed as run-ops-only: `CompletedWaitpoint` and `WaitpointRunConnection`, both explicit FK-free replacements for a control-plane implicit many-to-many, since an implicit m2m carries a foreign key that cannot resolve across databases. The test also asserts that exception list is exhaustive, so a new unpaired model fails rather than being silently skipped. Confirmed the guard actually fails: reverting the index turns `BatchTaskRun` red with the missing `@@index` named in the diff. |
||
|
Chris Arderne
|
dc87b884e7 |
chore: upgrade to typescript 6 (#4310)
## Summary Upgrades the workspace to TypeScript 6.0.3 and applies the compiler, type, and build configuration changes required to preserve package layouts and existing runtime behavior, apart from correcting the HTTP status field used for deployment connection errors. ## Compatibility - Centralizes TypeScript 6.0.3 through the pnpm workspace catalog. - Replaces compiler options and module resolution modes that TypeScript 6 no longer accepts. - Restores explicit Node types where TypeScript 6 no longer includes them transitively. - Adds explicit declaration build roots that preserve each package's existing output layout. - Patches tsup to stop injecting the removed `baseUrl` option during declaration builds. - Uses type-only assertions for stricter typed-array and stream definitions without changing runtime behavior. - Reads the EventSource v3 HTTP status from `code`, so deployment connection errors include it correctly. - Keeps standalone CLI compatibility fixtures pinned to their existing TypeScript version and lockfiles. `turbo run typecheck` and the complete PR test suite are green. |
||
|
Chris Arderne
|
7faa52597d |
chore: format prisma schemas (#4224)
Creating a Prisma migration now formats its schema first, keeping migration-related schema edits consistently formatted without adding work to the repository-wide format command. Run `pnpm run format:prisma` to format either schema on demand. |
||
|
Daniel Sutton
|
d59743bd35 |
fix(webapp,run-ops-database): keep run-ops batch items co-resident with their batch (#4178)
## Summary Three fixes to the run-ops database split (the Cloud-only mode where run-lifecycle rows live on a dedicated Postgres). All are inert in the default single-database deployment. The main fix: on the batch trigger paths, a parentless batch's item runs chose their physical store from a fresh per-org mint-flag read at processing time, so flipping an org's flag mid-batch could land an item in a different store than its batch, breaking the `TaskRun.batchId` foreign key (or silently orphaning the item). The other two harden the split's safety nets: the schema-parity test now actually compares columns, and the read fan-out gate now signals when it has been silently disabled. ## Batch item residency `RunEngineBatchTriggerService` (api.v2) and the BatchQueue item callback (api.v3) now anchor each item's id mint on the batch's own friendlyId, mirroring the already-safe `BatchTriggerV3Service`. Residency is a pure id-shape check, so an item can no longer diverge from its batch across a mid-batch flag flip. The pre-failed-run fallback is anchored the same way (it also sets `batchId`), and the shared mint branch is consolidated into one helper so every mint path stays in lockstep. No new database queries; single-database mode is unchanged (a cuid-shaped batch friendlyId yields a cuid item). ## Schema parity test The parity test previously read only the dedicated schema and matched model headers with regexes, so it never compared columns and could not catch a run-subgraph column that diverged between the two physical schemas. It now parses both schemas and asserts bidirectional scalar-column parity (type, nullability, array-ness, default) across the run-subgraph models, and fails on any field line it can't parse. Scoped to the run-subgraph models so unrelated control-plane edits don't break it. ## Read fan-out signal The split read fan-out gate is decided by the object identity of the NEW vs control-plane clients. It now warns when both run-ops URLs are set but the NEW client isn't a distinct instance (fan-out silently off), and a new test exercises the real topology-into-gate wiring so a future refactor that aliases the clients can't disable fan-out unnoticed. ## Verification New unit and glue tests cover all three changes; the DB-backed residency, store-routing, and topology suites pass against real Postgres; `typecheck` is clean for both packages. |
||
|
Daniel Sutton
|
092b9ef07a |
fix(run-ops): DNS-safe, sortable base32hex run id (replace base62 KSUID) (#4154)
## Problem
The run-ops split mints NEW-store run ids as **27-char base62 KSUIDs**.
The supervisor writes the run id into the Kubernetes pod name
(`runner-<id>`), and pod names must be DNS-1123 labels (lowercase
`[a-z0-9-]`) — so uppercase base62 ids make k8s reject the pod (422) and
**those runs never launch** (they loop in `PENDING_EXECUTING` until the
heartbeat-stall handler nacks them, forever). `.toLowerCase()` can't fix
it: base62 has both `A`(10) and `a`(36) as distinct symbols, so folding
collides distinct ids and destroys sort order.
## Fix: change the encoding, not the structure
Mint a **26-char lowercase base32hex** run id:
```
run_<24-char base32hex core><region char><version char>
[ 6-byte ms timestamp ][ 9 CSPRNG bytes ]
```
- **base32hex** (RFC 4648 §7, alphabet `0-9a-v`): lowercase,
order-preserving, DNS-safe; 15 bytes → exactly 24 chars, no padding.
Hand-rolled encode/decode (no new dependency).
- **48-bit ms timestamp** in the leading bytes → plain string sort ==
creation order at millisecond resolution.
- **72 bits CSPRNG** entropy; PK unique constraint is the backstop (no
retry loop).
- **region / version** are raw positional chars (read via one `charAt`
before decoding/routing), version = `"1"`.
DNS-safe from birth and hyphen-free, so **firekeeper is unchanged** —
`runner-<id>-attempt-N` → strip `runner-`, cut at first hyphen still
recovers the exact id incl. region+version.
## Residency discriminator: length → version char
`classifyKind`/`classifyResidency` (`runOpsResidency.ts`) previously
distinguished NEW vs LEGACY by **id length**. That gets ambiguous with a
third format. It now discriminates on the **version char at a fixed
position** (`isRunOpsIdBody`: 26 chars, `[25] === "1"`, base32hex
alphabet) → NEW; everything else → LEGACY. Total, never throws. The
`Residency` (NEW/LEGACY) contract the routing store consumes is
unchanged; the `"ksuid"` `ResidencyKind` label is retained only because
it's the persisted `runOpsMintKsuid` feature-flag value.
## Scope / verification
- Generator + discriminator in `@trigger.dev/core` isomorphic; mint path
+ all id-shape call sites swept (~40 webapp files); changeset added
(`@trigger.dev/core` patch).
- Core unit tests (encode/decode round-trip + property, generator shape,
ms sort-order incl. intra-second, parse partitioned-vs-legacy,
firekeeper round-trip): **24 pass**. `@trigger.dev/core` builds; webapp
typechecks; format/lint clean.
## Open decisions (flagged, not silently chosen)
1. **Backward-compat**: existing 27-char base62 KSUID runs now classify
LEGACY. On test cloud these are the broken/looping runs that never
completed, so this is acceptable — but worth a conscious call before
prod. No transitional length-recognition added (keeps the discriminator
clean).
2. **Storage collation**: the sort guarantee is byte-order — if the
run-ops id column is `TEXT` with default locale collation it's silently
not honored. Confirm whether `COLLATE "C"` / `BYTEA` is needed on the
run-ops schema.
3. **Region sourcing** wiring — see `regionCharForRegion` /
`REGION_CODES`.
---
## ⚠️ Required migration — deploy in lockstep
This PR renames a persisted feature-flag key/value and an env var. These
are **not** changed by the code alone and must be migrated when this
deploys, or affected orgs silently fall back to `cuid` minting (no crash
— `defaultValue: "cuid"`):
1. **Env var** (terraform): `RUN_OPS_MINT_KSUID_ENABLED` →
`RUN_OPS_MINT_ENABLED` (carry the value over).
2. **DB** `organization.featureFlags`: migrate both the key and value
together:
- key `runOpsMintKsuid` → `runOpsMintKind`
- value `"ksuid"` → `"runOpsId"`
Until an org's flag row is migrated, its `runOpsMintKind` lookup misses
and it mints `cuid` (legacy) — so no NEW-store ids for that org until
the data lands.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
Daniel Sutton
|
962bc48738 |
feat(run-ops): automatically migrate the dedicated run-ops database (#4150)
## What
Adds the ability to **automatically migrate the dedicated run-ops
database** (the NEW DB in the run-ops split), matching how every other
database in the system is migrated. Follow-up to the run-ops split
activation.
## Changes
- **Migrate runner** — new
`internal-packages/run-ops-database/scripts/migrate.mjs`, exposed as
`db:migrate:deploy` / `db:migrate:status`. Connects via
`RUN_OPS_DATABASE_URL` (the same var the app uses) and expands `${VAR}`
refs like Prisma's dotenv.
- **Self-host** — `docker/scripts/entrypoint.sh` runs the run-ops
migration on boot when the DB is configured, gated by
`SKIP_RUN_OPS_MIGRATIONS`. Single-DB installs never set the URL, so it's
a clean no-op.
- **Single env-var family** — the run-ops DB is now addressed by one
canonical `RUN_OPS_*` family, connect path and migrations resolving the
identical URL:
- `RUN_OPS_DATABASE_URL` (writer) — replaces `TASK_RUN_DATABASE_URL`
- `RUN_OPS_LEGACY_DATABASE_URL` — replaces
`TASK_RUN_LEGACY_DATABASE_URL`
- `RUN_OPS_DATABASE_READ_REPLICA_URL` — replaces
`TASK_RUN_DATABASE_READ_REPLICA_URL`
- the old `TASK_RUN_*` aliases, the `??` coalesce, the
`runOpsNewDatabaseUrl` indirection, and the migrate-only `directUrl` are
all removed (consumers read `env.RUN_OPS_DATABASE_URL` directly).
`directUrl` was dropped because it was only ever used by `prisma
migrate` (never the app runtime) to bypass a pooler for advisory locks —
premature here since the run-ops connection isn't wired to the app yet.
If a pooler is later introduced for the app, a direct URL can be
reintroduced then.
## Safety
- **Pure rename** — nothing deployed sets any `TASK_RUN_*` var (the
split isn't activated anywhere yet; `.env.example`, docker-compose, and
cloud already use `RUN_OPS_*`), so there is no config migration.
- **Single-DB / self-host** — no new required env var; entrypoint and
migrate are no-ops when `RUN_OPS_DATABASE_URL` is unset.
- **Cloud** — runs migrations as pre-deploy ECS tasks (companion cloud
PR), calling these same `db:migrate:deploy` / `db:migrate:status`
commands.
## Verification
- Live migration against a fresh scratch DB with only
`RUN_OPS_DATABASE_URL` set: both migrations applied, no `P1012`/`P1013`;
`${VAR}` expansion, idempotent re-run, `status`, and no-op skip all
pass.
- Schema parity 4/4; `typecheck --filter webapp` 18/18; affected
split/replication tests 34/34.
## Scope
This delivers automatic migrations only. Enabling the app to *use* the
new DB (setting `RUN_OPS_DATABASE_URL` + `RUN_OPS_SPLIT_ENABLED` on the
service) is a separate activation step.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
Daniel Sutton
|
84588bd236 |
feat(run-ops): dedicated run-ops database package + docker service + migration runner (#4113)
## What The **dedicated run-ops database** foundation for the split: a standalone Prisma package plus the infra to run and migrate it. - **`internal-packages/run-ops-database`** — a new Prisma package (`@internal/run-ops-database`) whose schema mirrors the run-execution tables that will live on the dedicated DB, with its own generated client, migrations, and migration runner. - **`prisma/schema.parity.test.ts`** — a parity test that guards the run-ops schema against drift from the control-plane schema for the mirrored tables. - **Docker** — a Postgres 17 service (`docker/Dockerfile.postgres17`, `docker/docker-compose.yml`) so the dedicated DB is available locally under the run-ops compose profile. - **Testcontainers** — hetero fixtures (PG14 legacy + PG17 dedicated) so later PRs can exercise cross-database behaviour with real containers rather than mocks. ## Why This is the **second PR in the run-ops split stack**, stacked on the core primitives. It stands up the dedicated database and its tooling. There is **no runtime wiring** into the webapp here — the app does not read or write this DB yet; that arrives in later PRs. On its own this PR only adds a package, a docker service, and test fixtures. ## Tests Schema-parity test for the run-ops schema; hetero testcontainer fixture smoke test. ## Notes - Draft, **stacked on #4112** (`runops/pr01-core-residency`). Review that one first; this diff is against it. - Server-change / changeset note to be added at stack-assembly time. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |