11 Commits

Author	SHA1	Message	Date
Eric Allam	c0b84595a3	feat(webapp): hosted webhook ingress, delivery pipeline, and dashboard (#4344) ... ## Summary The server half of hosted webhooks: the public ingress endpoint, signature verification, the delivery pipeline (Postgres partitioned storage + ClickHouse for ordering), the in-app partition manager, the HTTP API, and the dashboard (Deliveries, Endpoints, and the in-app test console). The public SDK and docs half is #4537. That PR carries the user-facing API (`webhook()`, `chat.event` / `chat.channels`, the `@trigger.dev/slack` connector) and builds on the shared `@trigger.dev/core` schemas that ship here. ## Shipping behind a flag A `WEBHOOK_ENABLED` env var (default off) gates the public ingress route and the engine worker plus partition cron, so merging and deploying this changes nothing in production until it is flipped on per environment. The dashboard is separately gated per org by the `hasWebhooksAccess` feature flag. ## Note on packages This PR includes the `@trigger.dev/core` schema additions the server compiles against, but carries no changeset. Core is not consumed independently of the SDK, so it is released together with the SDK via #4537. Keeping its changeset off `main` means no release cut from `main` publishes it early.	2026-08-16 14:33:42 +01:00
Eric Allam	b98dd79fe4	feat(webapp,run-store,database): env-configurable transaction resilience (maxWait + tx-start retry) (#4623) ... ## What Makes two transaction-resilience behaviors real and env-var configurable, defaults set to the good values, so we can tune during and after the Aug 15 database patch window without a redeploy: - **maxWait 2s → 10s** (TRI-12982): how long Prisma waits to borrow a connection before it can `BEGIN`. A restart freeze holds the pool full, and the only thing that errored was transaction starts giving up at 2s. - **Retry transaction-start P2028-at-acquisition** (TRI-12984): when Prisma can't borrow a connection within `maxWait` it raises P2028 (`Unable to start a transaction in the given time`) and **no SQL ran**, so retrying is safe. Scoped narrowly: only that error (never P2024 pool-exhaustion), 2 attempts, jittered backoff, and a token-bucket budget so a mass freeze can't amplify into a retry storm. ## Env vars (`DATABASE_*` convention) Generic defaults: | var | default | |---|---| | `DATABASE_TRANSACTION_MAX_WAIT_MS` | `10000` | | `DATABASE_TRANSACTION_START_RETRY_ENABLED` | `true` (kill switch) | | `DATABASE_TRANSACTION_START_RETRY_MAX_ATTEMPTS` | `2` | | `DATABASE_TRANSACTION_START_RETRY_BACKOFF_MIN_MS` | `50` | | `DATABASE_TRANSACTION_START_RETRY_BACKOFF_MAX_MS` | `250` | | `DATABASE_TRANSACTION_START_RETRY_BUDGET_PER_SEC` | `50` | | `DATABASE_TRANSACTION_START_RETRY_BUDGET_BURST` | `100` | Per-writer-pool overrides, each falling back to the generic when unset (same pattern as the per-client pool/connect-timeout work): `RUN_OPS_DATABASE_TRANSACTION_*` and `RUN_OPS_LEGACY_DATABASE_TRANSACTION_*` (all 7 knobs each). Transactions only open on writer pools, so those are the only pools with their own knobs. Each pool gets its **own** token bucket, so a storm on one pool can't drain another's retry budget. ## Design - The retry primitives live in `internal-packages/database` and never read `process.env` (IoC): a P2028-at-acquisition classifier, a `TokenBucketRetryBudget`, and `withTransactionStartRetry`, folded into the `$transaction` helper via a new `startRetry` option. Config is resolved at the app boundary and threaded in. - The `$transaction` helper is the chokepoint (wraps the whole transaction), not the per-statement `$allOperations` extension. - The run engine's writes go through `PostgresRunStore`'s own `.$transaction(...)`, not the webapp helper, so both the helper and the two `PostgresRunStore` sites apply maxWait + retry (sharing the per-pool config). Builds on the `options?: { timeout, maxWait }` seam added in #4514. - Webapp `$transaction` call sites get the default `maxWait` + retry injected at one merge point, so no call site needed editing. ## Evidence - Unit red/green in `internal-packages/database`: reverting the helper wiring turned the acquisition-retry test red (`Unable to start a transaction in the given time`), re-applying it green. Full package suite 25/25. Covers: classifier (P2028-acq yes, P2024 no, in-tx P2028 no), retry (retry-then-succeed, no-retry P2024, stop at maxAttempts, disabled, budget-exhausted, jitter bounds), token bucket, and `$transaction` wiring. - Typecheck clean: webapp, run-store, run-engine. - Full-stack run: bounded queue-ay pass (15 projects, real dev runs through the run-engine `PostgresRunStore` transaction path). 13 pass; the 2 failures are one documented known-failure and one stale-worker-state flake that passes 2/2 with this change active on a fresh app. - Boots cleanly with per-pool overrides set. ## Configuration & rollout Ship **inert** first (zero behavior change), then flip to the good values **live via env** — no redeploy needed for either. ### Inert — behaves exactly as today ``` DATABASE_TRANSACTION_MAX_WAIT_MS=2000 # Prisma's built-in default (change defaults to 10000) DATABASE_TRANSACTION_START_RETRY_ENABLED=false # disable the new retry entirely ``` `maxWait=2000` is what every path used before (Prisma's default; the run-store sites and the helper passed no maxWait). `retry=false` short-circuits `withTransactionStartRetry` to a single run and makes the serialization-retry exclusion a no-op. Verified on the pooler-freeze rig: identical fail-fast P2028 at ~2003ms with zero retries — byte-for-byte current behavior, across all pools. ### Production ("good") — the baked defaults Rely on defaults (nothing to set) or set explicitly: ``` DATABASE_TRANSACTION_MAX_WAIT_MS=10000 DATABASE_TRANSACTION_START_RETRY_ENABLED=true DATABASE_TRANSACTION_START_RETRY_MAX_ATTEMPTS=3 # 3 attempts (2 retries); ~30s acquisition tolerance covers a ~20-25s freeze DATABASE_TRANSACTION_START_RETRY_BACKOFF_MIN_MS=50 DATABASE_TRANSACTION_START_RETRY_BACKOFF_MAX_MS=250 DATABASE_TRANSACTION_START_RETRY_BUDGET_PER_SEC=50 DATABASE_TRANSACTION_START_RETRY_BUDGET_BURST=100 ``` Per-pool overrides `RUN_OPS_DATABASE_TRANSACTION_*` and `RUN_OPS_LEGACY_DATABASE_TRANSACTION_*` (all seven knobs each) are optional and fall back to the generic set — not needed for v1; the generic set covers the control-plane, run-ops, and run-ops-legacy writer pools. Readers open no transactions and take nothing. **Guardrail:** the retry only engages when a pool's `pool_timeout` > `maxWait`. Prod is fine (`DATABASE_POOL_TIMEOUT=60` >> 10). Do not set any writer pool's `pool_timeout` at or under `maxWait`, or saturation failures flip from retryable P2028 to non-retryable P2024 and the retry silently stops helping. ### Rollback Env flip (set inert) or revert. Retry only fires where no SQL ran, and the per-pool token bucket caps a storm. No migration. refs TRI-13295, TRI-12982, TRI-12984	2026-08-15 09:03:10 +01:00
Eric Allam	90e8bd5c12	feat(webapp,database): opt-in per-client Prisma driver adapters (#4539) ... ## What Adds an opt-in path to run each Prisma client through **`@prisma/adapter-pg`** (the node-postgres driver) instead of the built-in engine driver, controlled by a **per-client env var, all off by default**: | env var | client | |---|---| | `CONTROL_PLANE_DATABASE_WRITER_DRIVER_ADAPTER` | control-plane writer | | `CONTROL_PLANE_DATABASE_REPLICA_DRIVER_ADAPTER` | control-plane replica | | `RUN_OPS_DATABASE_WRITER_DRIVER_ADAPTER` | new run-ops writer | | `RUN_OPS_DATABASE_REPLICA_DRIVER_ADAPTER` | new run-ops replica | | `RUN_OPS_LEGACY_DATABASE_WRITER_DRIVER_ADAPTER` | legacy run-ops writer | | `RUN_OPS_LEGACY_DATABASE_REPLICA_DRIVER_ADAPTER` | legacy run-ops replica | With every flag unset the construction path is byte-identical to today (`datasources` URL + Rust engine), so this is inert until a flag is turned on. Per-client granularity allows enabling the adapter only where it's wanted. ## How - Enables the `driverAdapters` preview feature on both schemas (`@trigger.dev/database` and `@internal/run-ops-database`). This keeps the **Rust query engine** — it does NOT add `queryCompiler` — so query behavior, result types, and engine tracing spans are unchanged. - A shared `buildDriverAdapterPool` builds each client's `pg.Pool` with an explicit `max`, a bounded `connectionTimeoutMillis` (the node-postgres pool otherwise waits unbounded on acquire), and an `onPoolError` handler (an unhandled idle-connection error would otherwise crash the process). Threaded through all four client builders via a `useDriverAdapter` flag. - Adds `@prisma/adapter-pg` + `@types/pg` to the webapp; `pg` is already pinned at `8.15.6` (adapter-pg 6.x requires `pg < 8.17`). ## Connect-failure handling (the important correctness/security bit) Under the adapter an unreachable DB no longer surfaces as `PrismaClientInitializationError` / `P1001`; it becomes a `P2010` "Database not reachable: <host>" (or a raw `ECONNREFUSED`/`ENOTFOUND`-class error). Two handlers are updated so a client on the adapter behaves like today: - **`isInfrastructureError`** now recognizes those shapes (P2010 with a connectivity message, and raw connectivity errno codes). Without this, the DB **hostname would leak into API-client-facing errors** and the failure would go unlogged. Security-relevant. - **`isPrismaRetriableError`** treats the adapter's pool-acquire timeout ("timeout exceeded when trying to connect") as retriable, preserving the `P2024` retry behavior the adapter otherwise drops. ## Evidence Validated on an isolated stack that mirrors the production DB topology (chained PgBouncers in front of writer + reader): - **Behavioral parity:** raw-query results and Prisma error codes/`meta` are byte-identical between the engine driver and the adapter across the queried shapes (unique-constraint `meta.target`, record-not-found, transaction-timeout, serialization-failure, etc.). - **Feature matrix:** a full 380-project queue-ay pass shows no adapter-caused regressions — pass/fail parity between adapter-off and adapter-on, with the residual failures being pre-existing known-failures/flakes common to both. ## Rollout / rollback All flags default off; enable per client via env var, roll back by unsetting and redeploying (no data migration). Recommended first target is a single writer; enable one client at a time. ## Follow-ups (not in this PR) - `$metrics`-based pool observability is removed under the adapter (the Prometheus route + `db.pool.connections.*` instruments); the metrics replacement (via `pg.Pool` counters) lands in a separate PR. - Note for operators: on the adapter path, interactive-transaction `maxWait` does not bound pool acquisition — `connectionTimeoutMillis` does. ## Note on connection-string parameters The adapter pool is built from the base DSN, so Prisma-specific DSN parameters that node-postgres does not understand are not honored when a client is on the adapter: - **Prisma TLS spellings** (`sslaccept`, `sslcert`, etc.) — node-postgres uses `sslmode`/`ssl` instead. Our production DSNs do not use these Prisma-specific TLS params, but any deployment whose DSN relies on them must be checked before enabling a flag. - `pgbouncer=true` and `statement_cache_size` — effectively moot under the adapter, which uses no persistent named prepared statements. `connection_limit`, `pool_timeout`, and `schema` are handled explicitly (passed as `max`/`connectionTimeoutMillis` and PrismaPg's `{schema}` option). refs TRI-13039 --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-08-08 21:27:20 +01:00
Eric Allam	c526528d8f	feat(webapp,database): bound Prisma list filter arity (#4480) ... ## Summary Prisma expands `in` / `notIn` into one bind parameter per element, so every distinct list length is a separate prepared statement. Where the length tracks data volume (a batch size, a run-graph fan-out, a prior query's id set) one call site can mint hundreds of them. Each is used about once, but inserting it evicts an entry that was being reused, so the cost lands on unrelated queries sharing the pooler's statement cache. An unbounded list also risks the 65535 bind-parameter ceiling. `boundedIn()` pads a filter list to the next power of two by repeating its last element. `IN` and `NOT IN` ignore duplicates, so results are unchanged, and a call site drops from one statement per length to at most `log2(cap)`. Applied to all existing sites. ## Enforcement Two oxlint rules require the helper: a list filter must be an inline array literal or a `boundedIn()` call. - The first covers filters reached through `where` / `having` / `cursor`, and deliberately never descends into `data`, `create`, `update`, `set` or `equals`. A key named `in` in those positions is user data, not a predicate, and rewriting it would corrupt what gets stored or compared. - The second covers bare filter objects passed to where-building helpers, which the first cannot see. It found five sites in the run-graph batch loaders that were otherwise invisible. Both rules follow filters through the shapes they are actually written in: conditional expressions, logical-and objects, spread-conditional properties, computed keys, and call arguments. An array literal only counts as fixed-arity when nothing spreads into it, since `[...new Set(ids)]` has a runtime length. Twelve sites were hidden behind those shapes until the rules handled them. Scoped to `in` and `notIn`. The scalar-list filters `hasSome` and `hasEvery` compile to `&& $1` and `@> $1`, passing the whole array as a single bind parameter, so their arity never reaches the statement text and there is nothing to bound. Both rules are `error`, so new call sites fail CI. That ratchet has already caught four sites added by other PRs while this one was in review. ## Notes `boundedIn` pads by repeating rather than with null: `x NOT IN (a, b, NULL)` is never true, so null-padding a `notIn` filter would silently return no rows. Lists above 32768 are returned unchanged so padding can never push a query past the parameter limit. Route modules reach the helper through `~/db.server` rather than importing the database barrel directly, since a value import of that barrel into a module that also exports a React component is only safe while dead-code elimination prunes it. Measured on a local rig: 300 distinct list lengths produce 300 prepared statements unpadded, 10 padded. Verified end-to-end against a local stack with the full task-suite sweep, which surfaced no regressions.	2026-08-07 16:39:58 +01:00
Chris Arderne	c7861be520	chore: activate no-unused-vars and import linters (#4096) ... Once this is merged, oxlint is at a pretty sensible baseline. **Enable `no-unused-vars`, `typescript/consistent-type-imports`, and `import/no-duplicates` lint rules** Turns on three previously-disabled oxlint rules across the monorepo and fixes all violations: - **`no-unused-vars`** – enabled as an error with standard ignore patterns: unused function arguments are ignored by default (`args: "none"`), variables/caught errors/destructured array elements prefixed with `_` are allowed, and rest siblings are permitted. - **`typescript/consistent-type-imports`** – enforced as an error; all type-only imports now use the `import type` syntax. - **`import/no-duplicates`** – enforced as an error; duplicate import statements from the same module have been merged. The remaining commits clean up the violations found across the codebase: removing unused variables/imports/type aliases, adding `_` prefixes to intentionally unused bindings, fixing duplicate imports, and converting value imports to `import type` where appropriate.	2026-07-02 11:37:05 +01:00
Matt Aitken	24a915133e	Prisma 6.14.0 upgrade (#2444) ... * Initial work on upgrading to 6.14.0 Set the output to node_modules still to make it easier * Use ./generated Prisma folder, update types to fix issues * Docker compose restart Clickhouse * Prisma instrumentation update * Docker * Removed database dockerignore file, add generated prisma client to the top-level one * Delete v3-catalog package.json * Resolved pnpm lock file * Log errors for very slow queries	2025-08-27 16:52:58 +01:00
Eric Allam	8d5c86fea0	v4: simplified release concurrency system and status changes (#2284) ... * WIP * Make release concurrency system extremely simple, everything just releases all the time * update the deadlock detection to use the new lockedQueueReleaseConcurrencyOnWaitpoint column * WIP new release concurrency system * Remove releaseConcurrency and releaseConcurrencyOnWaitpoint Also removed deadlock detection, and added environment burst concurrency * Added new DEQUEUED status Cleaned up the API run statuses, including now detecting new clients and not breaking older clients by adding an API version header to all requests * Introduce the new "current dequeued concurrency set" * Remove QUEUED_EXECUTING because we no longer "eagerly" release before checkpointing * Remove waitpoint test for QUEUED_EXECUTING * Add isWaiting * Add changeset * Use createdAt for ordering realtime runs instead of number * Clarify the envCurrentDequeuedKey usage * mock the db.server file to fix the tests * Updated changset "EXECUTED" -> "EXECUTING" --------- Co-authored-by: Matt Aitken <matt@mattaitken.com>	2025-07-21 15:45:27 +01:00
Matt Aitken	c49af774ba	Retry batch item completion (#1675) ... * Added isPrismaRetriableError() * Retry completeBatchTaskRunItem if they fail because of a retriable Prisma error * Retry using Redis worker * Handle more retriable errors. Add special condition in for race condition error * Added Postgres connection_timeout with default 20s * Added a simple batchTriggerAndWait example	2025-02-06 19:16:30 +00:00
Eric Allam	e61573600b	Reduce contention on batchTaskRun when setting expected count (#1662)	2025-02-04 13:43:50 +00:00
Eric Allam	b67ea9a2f5	improve batch completion system for run engine v1 (#1656) ... * Automatically retry TriggerTaskService when hitting a unique constraint error on idempotency key * improve batch completion system for run engine v1 * Rename batch stuff to v3 so it's not confusing * Handle unique constraint error on BatchTaskRunItem creation and allow different limits for batchTrigger and batchTriggerAndWait	2025-02-03 11:29:51 +00:00
Matt Aitken	f2babbf637	Internal packages (testcontainers, redis-worker and zod-worker) (#1392) ... * Some notes on the new run engine * lockfile with setup for the run engine * Documenting where TaskRun is currently mutated, to try figure out the shape of the new system * Added notes about how triggering currently works * Details about when triggering happens * Lots of notes about waitpoints * Started scaffolding the RunEngine * Sketch of Prisma waitpoint schema while it’s fresh in my mind * Got Prisma working with testcontainers * Use beforeEach/afterEach * Simple Prisma and Redis test * Return Redis options instead of a client * Simplified things * A very simple FIFO pull-based queue to check the tests working properly * Use vitest extend * Separate redis, postgres and combined tests for faster testing * Some fixes and test improvements * Pass a logger into the queue * A queue processor that processes items from the given queue as fast as it can * Test for retrying an item that wasn’t processed * First draft of waitpoints in the Prisma schema * Remove the custom logger from the test * Added a completedAt to Waitpoint * Notes on the flow for an execution starting * Added redlock, moved some files around * Starting point for the TaskRunExecutionSnapshot table * Added relationships to TaskRunExecutionSnapshot * Change some tsconfig * Moved some things around * Added some packages * WIP on the RunQueue * Fix for some imports * Key producer with some tests * Removed the nv type from the keys… it’s not useful to do global queries * Passing unit tests for all the public key producer functions * Some basic tests passing for the RunQueue * Simple enqueue test working * Enqueue and dequeue for dev is working * Don’t log everything during the tests * Enqueuing/dequeuing from the shared queue is working * Tests for getting a shared queue * The key producer sharedQueue can now be named, to allow multiple separate queues * The key producer uses the name of the queue as the input * Extra info in the Prisma schema * Dequeuing a message gets the payload and sets the task concurrency all in one Lua script * Adding more keys so we can read the concurrency from the queue * Setting the concurrency with dequeue and enquque is working * Improved the tests and fixed some bugs * Acking is resetting the concurrencies * Check the key has been removed after acking * Nacking is working * Changed the package to CommonJS + Node10 so it works with Redlock * Moved the database, otel and emails packages to be in internal-packages * Moved some Prisma code to the database package * Started using the RunEngine for triggering * Progress on run engine triggering, first waitpoint code * Create a delay waitpoint * Moved ZodWorker to an internal package so it can be used in the run engine as well as the webapp * Web app now uses the zod worker package * Added parseNaturalLanguageDuration to core/apps * internal-packages/zod-worker in the lockfile * Pass in the master queue, remove old rebalance workers code * Add masterQueue to TaskRun * Fixed the tests * Moved waitpoint code into the run engine, also the zod worker * Completing waitpoints * An experiment to create a new test container with environment * More changes to triggering * Started testing triggering * Test for a run getting triggered and being enqueued * Removed dequeueMessageInEnv * Update dev queue tests to use the shared queue function * Schema changes for TaskRunExecutionSnapshot * First execution snapshot when the run is created. Dequeue run function added to the engine * Separate internal package for testcontainers so they can be used elsewhere * Remove the simple queue and testcontainers from the run-engine. They’re going to be separate * Fix for the wrong path to the Prisma schem,a * Added the testcontainers package to the run-engine * redis-worker package, just a copy of the simple queue for now * The queue now uses Lua to enqueue dequeue * The queue now has a catalog and an invisible period after dequeuing * Added a visibility timeout and acking, with tests * Added more Redis connection logging, deleted todos * Visibility timeouts are now defined on the catalog and can be overridden when enqueuing * Dequeue multiple items at once * Test for dequeuing multiple items * Export some types to be used elsewhere * Partial refactor of the processor * First stab at a worker with concurrency and NodeWorkers * Don’t have a default visibility timeout in the queue * Worker setup and processing items in a simple test * Process jobs in parallel with retrying * Get the attempt when dequeuing * Workers do exponential backoff * Moved todos * DLQ functionality * DLQ tests * Same cluster for all keys in the same queue * Added DLQ tests * Whitespace * Redis pubsub to redrive from the worker * Fixed database paths * Fix for path to zod-worker * Fixes for typecheck errors, mostly with TS versions and module resolution * Redlock required a patch * Moved the new DB migrations to the new database package folder * Remove the run-engine package * Remove the RunEngine prisma schema changes * Delete triggerTaskV2 * Remove zodworker test script (no tests) * Update test-containers readme * Generate the client first * Use a specific version of the prisma package * Generate the prisma client before running the unit tests	2024-10-08 17:41:22 +01:00