Soft-deleted chats are hard-deleted with their children after a 30d
window by the maintenance sweep, and organization deletion enqueues a
job that soft-deletes the org's chats so the same sweep removes them.
The FK-free cascade lives in a single reusable deleteChatsByIds helper.
Stamp the source PAT id on a minted user-actor token and recheck it is still live at every verify site, so revoking the PAT invalidates any token created from it. Tokens with no source PAT (e.g. the dashboard agent's) skip the recheck.
Model-authored markdown could emit a remote image whose URL the browser fetches on render — a zero-click data beacon. Constrain the shared markdown renderer with a urlTransform that strips the src of any non-local image and limits links to safe schemes.
A user-actor token could be exchanged for an env JWT that outlived it by
requesting a longer expirationTime. Surface the token's exp as expiresAt
and clamp the minted JWT to it. Non-UAT exchanges are unchanged.
A delegated user-actor token with no scope cap could exchange for an env
JWT carrying any requested scopes (admin included). Ceiling every UAT
exchange by its cap, or read-only when it has none, projecting requested
scopes through the scope grammar so read:all still admits reads.
Resolve conflicts in report-health: keep our richer fallback/trust model
(telemetry_absent trust, snapshot fallback, finished/completed throughput)
and adopt main's start-latency-unknown behavior so an absent measurement
reports as unknown instead of a confident 0ms. renderMarkdown keeps our
layout rewrite; main's compactFact change is covered by report-layout's
metricValue. Adapt main's two render assertions to our current output.
## Why this change
`EnvironmentVariableValue.valueReference` is an `onDelete: SetNull`
foreign key. Deleting a `SecretReference` (the env var edit/delete path
for secret values) fires the cascade `UPDATE ONLY
"EnvironmentVariableValue" SET "valueReferenceId" = NULL WHERE $1 =
"valueReferenceId"`. That cascade is scan-shaped: with no index on
`valueReferenceId`, it reads the entire table to find the rows
referencing the deleted secret. The parent `SecretReference` delete does
almost no work itself; its latency is dominated by this cascade.
## Diagnosis
`EnvironmentVariableValue` was indexed on `environmentId` and
`(variableId, environmentId)`, but not on `valueReferenceId`. The SET
NULL cascade therefore did a full sequential scan of the whole table.
Two sibling SET NULL cascades on the same delete
(`OrganizationIntegration.tokenReferenceId`,
`User.mfaSecretReferenceId`) are index-backed and stay fast, which
isolates the missing index as the cause.
## Change
Add `@@index([valueReferenceId])` on `EnvironmentVariableValue`, created
with `CREATE INDEX CONCURRENTLY IF NOT EXISTS` so `prisma migrate
deploy` stays safe on a live table.
## Benchmark (local, seeded)
Local Postgres seeded with 1,000,000 `EnvironmentVariableValue` rows,
`EXPLAIN (ANALYZE, BUFFERS)` on the SET NULL cascade with zero matching
rows (the worst case: reads the whole table, affects nothing):
| | before | after |
|---|---|---|
| plan | Seq Scan (1M rows) | Bitmap Index Scan |
| execution | 183 ms | 2.8 ms |
In a variant where the secret matched several thousand rows, the parent
`SecretReference` delete's
`EnvironmentVariableValue_valueReferenceId_fkey` trigger dropped from
216 ms to 88 ms (the residual is the heap work of nulling those rows).
## Expected impact
The cascade drops from a full-table sequential scan to a targeted index
lookup. The win grows with the table, so the benefit is larger than the
seeded numbers above.
## Risks
- One extra btree to maintain on `EnvironmentVariableValue` writes;
small, single-column, and it should be pre-created before the migration
deploys (per the repo index rules).
- No behavior change: same rows nulled, no ordering or result-set
change, read paths untouched.
Companion to the same fix on `ProjectAlert.channelId`.
## Why this change
Deleting a `ProjectAlertChannel` fires the FK cascade `DELETE FROM ONLY
"ProjectAlert" WHERE $1 = "channelId"`. That cascade is scan-shaped:
with no index on `channelId`, it reads the entire `ProjectAlert` table
to find the few child rows belonging to the deleted channel. The parent
`DELETE ProjectAlertChannel` does almost no work itself; its latency is
dominated by this cascade. `ProjectAlert` is append-heavy and grows over
time, so the scan cost only increases.
## Diagnosis
`ProjectAlert` had no index on `channelId` (only `pkey` + a `friendlyId`
unique). The cascade therefore did a full sequential scan of the whole
table. The sibling `ProjectAlertStorage` cascade on the same delete is
index-backed and stays fast, which isolates the missing index as the
cause.
## Change
Add `@@index([channelId])` on `ProjectAlert`, created with `CREATE INDEX
CONCURRENTLY IF NOT EXISTS` so `prisma migrate deploy` stays safe on a
live table.
## Benchmark (local, seeded)
Local Postgres seeded with 1,000,000 `ProjectAlert` rows across 50
channels (~20k rows per channel), `EXPLAIN (ANALYZE, BUFFERS)` on the
cascade delete:
| | before | after |
|---|---|---|
| plan | Seq Scan (1M rows) | Bitmap Index Scan |
| direct child delete | 740 ms | 22 ms |
| parent delete `ProjectAlert_channelId_fkey` trigger | 77.7 ms | 23.8
ms |
## Expected impact
The cascade drops from a full-table sequential scan to a targeted index
lookup. The win grows with the table: the more rows in `ProjectAlert`,
the more a scan costs and the more the index saves, so the benefit is
larger than the seeded numbers above.
## Risks
- One extra btree to maintain on every `ProjectAlert` insert; acceptable
for a single-column index on a high-insert table, and it should be
pre-created before the migration deploys (per the repo index rules).
- No behavior change: no rows orphaned, no ordering or result-set
change, read paths untouched.
## Follow-up
`ProjectAlert`'s other cascade FK columns (`projectId`, `environmentId`,
`workerDeploymentId`) are also unindexed, but their parents are
soft-deleted rather than physically removed, so those cascades do not
currently fire. Lower priority unless a hard-delete path is introduced.
## What
Follow-up to #4539. The driver-adapter work is inert until a client
flips to the pg driver adapter, but the moment one does, our database
observability degrades: the OTel metrics pipeline reads pool stats from
Prisma's `$metrics`, which is owned by the Rust engine's `quaint` pool.
Under the adapter, `pg.Pool` owns the pool, so those gauges read zero.
The pipeline also only ever scraped a single client (the control-plane
writer singleton).
This PR makes database metrics driver-agnostic and per-client:
- Every configured client registers a metrics source: control-plane
writer/replica, run-ops writer/replica, legacy writer/replica.
Previously only the control-plane writer singleton was scraped.
- Each OTel instrument is observed per client with `db_client` and
`db_driver` (`quaint` | `pg-adapter`) attributes. `db_client` uses our
canonical datasource-role labels (`control-plane-writer`,
`control-plane-replica`, `run-ops-writer`, `run-ops-replica`,
`legacy-run-ops-writer`, `legacy-run-ops-replica`) — the same strings
used for the `db.datasource` span attribute, so a metric and a trace
point at the same pool.
- Pool figures come from the authoritative source per driver:
- **pg-adapter**: `pg.Pool` (`totalCount`/`idleCount`/`waitingCount`,
plus cumulative opened/closed from `connect`/`remove` events).
- **quaint**: the Rust engine's `$metrics` pool gauges/counters, exactly
as before.
- Query counters and duration histograms still come from `$metrics` for
both drivers (the Rust engine executes queries in both cases).
- New `db.pool.connections.waiting` gauge (pg.Pool exposes this; quaint
reports 0).
- Stops exporting Prisma metrics from the Prometheus `/metrics` route.
Pool observability now lives entirely in the OTel pipeline, per driver,
per client.
## Why
So we can flip any client (including the control-plane writer, the
primary desync-fix target) to the driver adapter without losing pool
visibility. Existing dashboards keyed on the same metric names keep
working; they gain a per-client dimension.
## Testing
Unit (`apps/webapp/app/utils/databaseMetrics.server.test.ts`): the pure
normalizer — quaint reads pool from `$metrics`; adapter reads pool from
`pg.Pool` and keeps engine query metrics; `busy` never goes negative;
graceful zeroing when `$metrics` is unavailable (adapter still reports
live pool figures).
Live smoke test against a prod-shaped local stack: three
physically-distinct Postgres DBs (control-plane, run-ops, legacy) behind
dual PgBouncers, split mode on, with a mix of adapter and quaint
clients. Reading the actual emitted OTel metrics, every pool shows up as
its own series:
```
db.pool.connections.total{db_client="control-plane-writer", db_driver="pg-adapter"} = 1
db.pool.connections.total{db_client="control-plane-replica", db_driver="quaint"} = 1
db.pool.connections.total{db_client="run-ops-writer", db_driver="pg-adapter"} = 1
db.pool.connections.total{db_client="run-ops-replica", db_driver="quaint"} = 1
db.pool.connections.total{db_client="legacy-run-ops-writer", db_driver="quaint"} = 1
db.pool.connections.total{db_client="legacy-run-ops-replica",db_driver="quaint"} = 1
db.client.queries.total{db_client="control-plane-writer",db_driver="pg-adapter"} = incrementing
db.client.queries.duration.count{db_client="control-plane-writer",db_driver="pg-adapter"} = incrementing
```
Confirms: metrics are attributed per pool with the correct driver;
adapter pools' figures come from `pg.Pool`; and query counters/duration
histograms keep incrementing under the pg adapter. Also verified
`/metrics` (Prometheus) now returns zero `prisma_*` series while still
serving the app's own metrics.
`pnpm run typecheck --filter webapp` passes.
## Notes
- `/metrics` (Prometheus) no longer includes `prisma_*` series. Anything
scraping that endpoint for Prisma metrics should read the equivalent
`db.*` metrics from the OTel exporter instead.
- **PgBouncer + `?schema=` gotcha (separate from this PR, worth flagging
for rollout):** since #4539 parses `?schema=` from the DSN and passes `{
schema }` to the adapter, node-postgres sends `search_path` as a startup
parameter. A transaction-mode PgBouncer rejects that with `FATAL:
unsupported startup parameter: search_path`. Our prod control-plane DSNs
use the default `public` schema with no `?schema=` param, so this is
latent, but any client we flip to the adapter must not carry `?schema=`
in its DSN (or the pooler needs `ignore_startup_parameters =
search_path`).
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
When the health report had no start-latency measurement for the window,
it printed a confident "p95 0ms" and graded it healthy. It now shows
"unknown" for that metric and skips grading it, so an absent measurement
can't read as a green signal.
A genuinely measured 0ms is still shown as 0ms: the loader keeps "no
measurement" distinct from a measured zero instead of coercing both to
0.
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 0s
🚀 Publish Trigger.dev Docker / units (push) Failing after 0s
🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker-v4 (push) Has been skipped
🚀 Publish Trigger.dev Docker / scan-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / scan-supervisor (push) Has been skipped
🦋 Changesets PR / Create Release PR (push) Has been cancelled
🚀 Publish Trigger.dev Docker / 📣 Dispatch main image (push) Has been cancelled
📚 Docs Checks / check-broken-links (push) Has been cancelled
🧭 Helm Chart Prerelease / lint-and-test (push) Has been cancelled
Workflow Checks / Actionlint (push) Has been cancelled
Workflow Checks / Zizmor (push) Has been cancelled
🧭 Helm Chart Prerelease / prerelease (push) Has been cancelled
## What
Adds an opt-in path to run each Prisma client through
**`@prisma/adapter-pg`** (the node-postgres driver) instead of the
built-in engine driver, controlled by a **per-client env var, all off by
default**:
| env var | client |
|---|---|
| `CONTROL_PLANE_DATABASE_WRITER_DRIVER_ADAPTER` | control-plane writer
|
| `CONTROL_PLANE_DATABASE_REPLICA_DRIVER_ADAPTER` | control-plane
replica |
| `RUN_OPS_DATABASE_WRITER_DRIVER_ADAPTER` | new run-ops writer |
| `RUN_OPS_DATABASE_REPLICA_DRIVER_ADAPTER` | new run-ops replica |
| `RUN_OPS_LEGACY_DATABASE_WRITER_DRIVER_ADAPTER` | legacy run-ops
writer |
| `RUN_OPS_LEGACY_DATABASE_REPLICA_DRIVER_ADAPTER` | legacy run-ops
replica |
With every flag unset the construction path is byte-identical to today
(`datasources` URL + Rust engine), so this is inert until a flag is
turned on. Per-client granularity allows enabling the adapter only where
it's wanted.
## How
- Enables the `driverAdapters` preview feature on both schemas
(`@trigger.dev/database` and `@internal/run-ops-database`). This keeps
the **Rust query engine** — it does NOT add `queryCompiler` — so query
behavior, result types, and engine tracing spans are unchanged.
- A shared `buildDriverAdapterPool` builds each client's `pg.Pool` with
an explicit `max`, a bounded `connectionTimeoutMillis` (the
node-postgres pool otherwise waits unbounded on acquire), and an
`onPoolError` handler (an unhandled idle-connection error would
otherwise crash the process). Threaded through all four client builders
via a `useDriverAdapter` flag.
- Adds `@prisma/adapter-pg` + `@types/pg` to the webapp; `pg` is already
pinned at `8.15.6` (adapter-pg 6.x requires `pg < 8.17`).
## Connect-failure handling (the important correctness/security bit)
Under the adapter an unreachable DB no longer surfaces as
`PrismaClientInitializationError` / `P1001`; it becomes a `P2010`
"Database not reachable: <host>" (or a raw
`ECONNREFUSED`/`ENOTFOUND`-class error). Two handlers are updated so a
client on the adapter behaves like today:
- **`isInfrastructureError`** now recognizes those shapes (P2010 with a
connectivity message, and raw connectivity errno codes). Without this,
the DB **hostname would leak into API-client-facing errors** and the
failure would go unlogged. Security-relevant.
- **`isPrismaRetriableError`** treats the adapter's pool-acquire timeout
("timeout exceeded when trying to connect") as retriable, preserving the
`P2024` retry behavior the adapter otherwise drops.
## Evidence
Validated on an isolated stack that mirrors the production DB topology
(chained PgBouncers in front of writer + reader):
- **Behavioral parity:** raw-query results and Prisma error codes/`meta`
are byte-identical between the engine driver and the adapter across the
queried shapes (unique-constraint `meta.target`, record-not-found,
transaction-timeout, serialization-failure, etc.).
- **Feature matrix:** a full 380-project queue-ay pass shows no
adapter-caused regressions — pass/fail parity between adapter-off and
adapter-on, with the residual failures being pre-existing
known-failures/flakes common to both.
## Rollout / rollback
All flags default off; enable per client via env var, roll back by
unsetting and redeploying (no data migration). Recommended first target
is a single writer; enable one client at a time.
## Follow-ups (not in this PR)
- `$metrics`-based pool observability is removed under the adapter (the
Prometheus route + `db.pool.connections.*` instruments); the metrics
replacement (via `pg.Pool` counters) lands in a separate PR.
- Note for operators: on the adapter path, interactive-transaction
`maxWait` does not bound pool acquisition — `connectionTimeoutMillis`
does.
## Note on connection-string parameters
The adapter pool is built from the base DSN, so Prisma-specific DSN
parameters that node-postgres does not understand are not honored when a
client is on the adapter:
- **Prisma TLS spellings** (`sslaccept`, `sslcert`, etc.) —
node-postgres uses `sslmode`/`ssl` instead. Our production DSNs do not
use these Prisma-specific TLS params, but any deployment whose DSN
relies on them must be checked before enabling a flag.
- `pgbouncer=true` and `statement_cache_size` — effectively moot under
the adapter, which uses no persistent named prepared statements.
`connection_limit`, `pool_timeout`, and `schema` are handled explicitly
(passed as `max`/`connectionTimeoutMillis` and PrismaPg's `{schema}`
option).
refs TRI-13039
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
⚒️ Publish Worker (v4) / build (supervisor) (push) Has been cancelled
## Summary
Prisma expands `in` / `notIn` into one bind parameter per element, so
every distinct list
length is a separate prepared statement. Where the length tracks data
volume (a batch size,
a run-graph fan-out, a prior query's id set) one call site can mint
hundreds of them. Each
is used about once, but inserting it evicts an entry that was being
reused, so the cost
lands on unrelated queries sharing the pooler's statement cache. An
unbounded list also
risks the 65535 bind-parameter ceiling.
`boundedIn()` pads a filter list to the next power of two by repeating
its last element.
`IN` and `NOT IN` ignore duplicates, so results are unchanged, and a
call site drops from
one statement per length to at most `log2(cap)`. Applied to all existing
sites.
## Enforcement
Two oxlint rules require the helper: a list filter must be an inline
array literal or a
`boundedIn()` call.
- The first covers filters reached through `where` / `having` /
`cursor`, and deliberately
never descends into `data`, `create`, `update`, `set` or `equals`. A key
named `in` in
those positions is user data, not a predicate, and rewriting it would
corrupt what gets
stored or compared.
- The second covers bare filter objects passed to where-building
helpers, which the first
cannot see. It found five sites in the run-graph batch loaders that were
otherwise
invisible.
Both rules follow filters through the shapes they are actually written
in: conditional
expressions, logical-and objects, spread-conditional properties,
computed keys, and call
arguments. An array literal only counts as fixed-arity when nothing
spreads into it, since
`[...new Set(ids)]` has a runtime length. Twelve sites were hidden
behind those shapes
until the rules handled them.
Scoped to `in` and `notIn`. The scalar-list filters `hasSome` and
`hasEvery` compile to
`&& $1` and `@> $1`, passing the whole array as a single bind parameter,
so their arity never
reaches the statement text and there is nothing to bound.
Both rules are `error`, so new call sites fail CI. That ratchet has
already caught four
sites added by other PRs while this one was in review.
## Notes
`boundedIn` pads by repeating rather than with null: `x NOT IN (a, b,
NULL)` is never true,
so null-padding a `notIn` filter would silently return no rows. Lists
above 32768 are
returned unchanged so padding can never push a query past the parameter
limit.
Route modules reach the helper through `~/db.server` rather than
importing the database
barrel directly, since a value import of that barrel into a module that
also exports a React
component is only safe while dead-code elimination prunes it.
Measured on a local rig: 300 distinct list lengths produce 300 prepared
statements
unpadded, 10 padded. Verified end-to-end against a local stack with the
full task-suite
sweep, which surfaced no regressions.
## What
The internal tracing `ParentBasedSampler` in `tracer.server.ts` left
`remoteParentSampled` at its default of `AlwaysOn`. Any request arriving
with a `traceparent` whose sampled flag was set got recorded in full,
bypassing `INTERNAL_OTEL_TRACE_SAMPLING_RATE` entirely. Because the SDK
propagates its (always-sampled) trace context on calls back to the
platform from inside running tasks, the large majority of API server
spans inherited a sampled parent and ignored the divisor. The sampling
knob was effectively inert on the busiest service.
This registers a custom propagator
(`NonInheritingTraceContextPropagator`) that stops adopting the inbound
trace as the parent:
- `inject` still delegates to the standard W3C trace + baggage
propagators, so outbound propagation is unchanged.
- `extract` drops the parent span (`trace.deleteSpan`) while preserving
baggage, so every incoming request roots its own trace and the ratio
sampler applies uniformly.
`remoteParentSampled` is also set to the ratio sampler as a
belt-and-suspenders fallback, in case an inbound sampled parent ever
reaches the sampler another way.
Two effects: the divisor becomes effective on the API server, and the
API no longer stitches onto (and inflates) the propagated task-run
traces, which is where the very large, un-thinnable trace chains came
from. Rooting each request removes those chains rather than only
diluting them.
Only the internal APM trace pipeline
(`INTERNAL_OTEL_TRACE_EXPORTER_URL`) is affected. The user-facing
run-trace pipeline (`otel.v1.traces` -> ClickHouse) is a separate path
and is untouched. The only consumer of the global propagator's `extract`
is the OTel HTTP/Express auto-instrumentation, so the blast radius is
inbound-request trace shape.
## Evidence (local full-stack red/green, divisor 10)
A local OTLP/JSON sink counting spans; a driver fires N requests at a
real endpoint, each carrying a distinct sampled `traceparent`, then
counts how many spans/traces carry that run's marker.
| run | code | sent | kept traces | kept fraction |
| --- | --- | --- | --- | --- |
| before | unmodified | 500 | 500 | 1.00 |
| after | this PR | 500 | 67 | 0.134 |
| after | this PR | 2000 | 213 | 0.1065 |
Before: 100% of inherited-sampled requests kept, divisor ignored. After:
~10% kept (the divisor), converging on it at larger N. In every
after-run each kept request is a single self-rooted trace (kept spans ==
kept distinct traces), confirming the inherited chains are gone, not
just thinned. `typecheck` passes.
## Rollout / rollback
No flag. Behavior stays governed by the existing
`INTERNAL_OTEL_TRACE_SAMPLING_RATE`. Rollback is a straight revert with
no data migration.
## Notes
Internal dashboards that count raw span or request volume from this
pipeline will read lower once this ships. That is expected: those counts
were inflated by the bypass, not a real drop in traffic.
Latency/percentile monitors retain plenty of samples at the current
divisor.
refs TRI-13031
## What
A trigger request carrying a Unicode NUL (`U+0000`) in the **idempotency
key** or **debounce key** reached `prisma.taskRun.create()` and failed
the insert, so the caller got an opaque 500 and the run was never
created.
These two keys are stored in `jsonb` columns (`idempotencyKeyOptions`,
`debounce`), and Postgres rejects a NUL inside a `jsonb` value with
`SQLSTATE 22P05` ("unsupported Unicode escape sequence ... cannot be
converted to text"). This fix strips the NUL from both keys at the
single trigger-input chokepoint (`#buildEngineTriggerInput`), which
every trigger path flows through (single, batch item, mollified, and
drainer replay).
Stripping matches the existing precedent for run errors and task events.
It does not change dedup behaviour: the idempotency **dedup identity**
is the hashed key (a clean 64-char digest), computed independently of
the raw key we clean, so dedup keeps working exactly as before. For
debounce the key is used directly, so the cleaned key also becomes the
grouping key, an acceptable change for input that is already malformed.
## Why not payload / metadata / tags
Those are `text` columns fed by `JSON.stringify`, which escapes a NUL to
a safe escape sequence, so they do not hit this failure on the normal
JSON path. (A raw NUL in a `text` column throws a different code,
`22021`, and is not what triggers this issue.) The observed failures are
the `jsonb` `22P05` variant, which is only reachable via the two key
fields.
## Evidence
Red then green (containerTest, real Postgres): with the fix reverted,
triggering through the real service with a NUL in
`idempotencyKeyOptions.key` / `debounce.key` fails with the exact
`22P05` signature; with the fix, the run is created and the stored key
has the NUL removed.
Full-stack e2e (isolated stack, real HTTP): `POST
/api/v1/tasks/:taskId/trigger` with a NUL inside
`idempotencyKeyOptions.key` (`"acme<NUL>inc"`) and, separately,
`debounce.key` (`"grp<NUL>1"`):
- both returned `HTTP 200` with a created run (previously `500`)
- stored `idempotencyKeyOptions` = `{ "key": "acmeinc", "scope": "run"
}` (7 chars, NUL removed)
- stored `debounce.key` = `"grp1"` (4 chars, NUL removed)
- both runs render in the dashboard
Unit tests cover the helper (strip, no-op fast path, object-reference
reuse, null/undefined pass-through).
## Rollout / rollback
Server-only webapp change, no flag. Zero behaviour change for clean
input; only affects inputs that previously 500'd. Rollback is a straight
revert, no data migration.
## Known limitation
A raw NUL in a plain-string idempotency key (not created via
`idempotencyKeys.create()`) lands in a `text` column and throws `22021`
instead. That variant is not addressed here because stripping it would
change the dedup identity, so it warrants a separate decision. Not
observed in practice.
refs TRI-13030
📦 Preview packages (pkg.pr.new) / Build and publish previews (push) Has been cancelled
📚 Publish docs / publish (push) Has been cancelled
Implementing PlanetScale Insights improvement.
## Summary
Validating a schedule (creating or updating one through the API or the
dashboard, and deploying a project that declares schedules) looks up the
newest version of a task by slug. That lookup reads *every* version of
the task and sorts them to return one. A project gains a row per task on
every deploy, so the work grows with the project's age: the oldest
projects pay the most, and dev-mode redeploys make it worse. This was
picked because it was the largest single consumer of database time on
the schedules path, and the fix is a sort key with no index behind it.
## Fix
`BackgroundWorkerTask` is indexed on `(projectId, slug)`, which serves
the equality but not the `ORDER BY createdAt DESC`. Postgres seeks the
index, then bitmap-scans and top-N sorts the whole group to produce a
single row. Adding `createdAt` to the index lets it scan backward and
stop at the first row.
The same call site also selected all 21 columns, including five JSON
blobs, to read one field (`triggerSource`), so it now selects that field
alone.
## Benchmark
Local Postgres 17, 997,000 seeded rows / 748 MB, group sizes chosen to
match the distribution seen in production.
| Group size | Before | After |
| --- | --- | --- |
| 15,000 versions of one task | 11.118 ms, 1,510 buffers, 15,000 rows
scanned | 0.027 ms, 4 buffers, 1 row |
| 2,000 versions of one task | 2.081 ms, 1,455 buffers, 2,000 rows
scanned | 0.022 ms, 4 buffers, 1 row |
```
before: Limit -> Sort (top-N heapsort) -> Bitmap Heap Scan
after: Limit -> Index Scan Backward using BackgroundWorkerTask_projectId_slug_createdAt_idx
```
An ascending index scanned backward is enough here, so no descending
index is needed.
## Impact and risk
Real-world gain lands between the two rows above and scales with how
many deploys a project has accumulated. Projects with few deploys will
see little change, since there is barely anything to sort.
The new index costs noticeably more than the existing two-column one: 43
MB against 7.3 MB on the benchmark rig. Adding `createdAt` makes every
key unique, which defeats btree deduplication, so this is a real disk
and write cost rather than a rounding error. Writes to this table happen
at deploy time, not on the run path, so the write amplification is
acceptable. The existing `(projectId, slug)` index is now a redundant
prefix and could be dropped, but this PR keeps it so index usage can be
observed before removing it.
Behavior is unchanged: same predicate, same ordering, same row returned.
The narrowed select is the only code change, and the field it keeps is
the only one the caller read.
Deploy note: the migration is
`20260806100000_add_background_worker_task_project_id_slug_created_at_index`
and uses `CREATE INDEX CONCURRENTLY IF NOT EXISTS`, so it can be
pre-applied by hand before the deploy.
## Summary
`syncDeclarativeSchedules` runs on every background-worker creation
(every deploy, and every file save during `trigger dev`). It issued one
instance-delete per declarative schedule the current worker no longer
declares, in a loop, and the overwhelming majority of those deletes
matched zero rows. This collapses the loop into at most two set-based
statements and skips the instance delete entirely when the current
environment owns no instance of the schedule.
## Why so many, and mostly no-op
The loop runs once per entry in `missingSchedules`, which starts as
every DECLARATIVE schedule for the whole project across all its
environments (the query filters only by `projectId`). A schedule leaves
that set only when a declared task matches it by `taskIdentifier`
**and** the schedule already has an instance in the current environment.
That last clause is the amplifier. When a task's schedule has no
instance in the current environment, the create branch inserts a
brand-new `TaskSchedule` row with an instance for this environment
rather than adding an instance to the existing row. So the same
scheduled task, once it has run in dev and been deployed to prod, exists
as two separate schedule rows: one carrying a dev instance, one carrying
a prod instance.
On a dev worker sync of that project:
- the dev-instance row matches the declared task and is removed from the
set
- the prod-instance row has the same `taskIdentifier` but no dev
instance, so it stays in the set and gets `deleteMany(taskScheduleId =
prodRow, environmentId = dev)`, which matches zero rows
So every declarative task that has been synced in another environment
contributes one guaranteed no-op delete per sync, and the count scales
with (declarative tasks x environments), plus any leftover rows from
renamed or removed tasks. A project does not need to have dropped a
schedule to generate these; it just needs the same declarative tasks
present in more than one environment, which is the normal
develop-in-dev, deploy-to-prod case.
## Fix
The candidate schedules are already loaded with their instances, so the
branch is decided in memory:
- schedules with no instances (or only current-environment instances)
are removed in a single `taskSchedule.deleteMany`
- schedules that still have another environment's instance have only the
current environment's instance detached, in a single
`taskScheduleInstance.deleteMany`, and only when such an instance
actually exists
Behavior is unchanged (cascade delete still removes the instances of a
deleted schedule); the difference is statement count. A zero-row delete
writes no WAL and creates no dead tuples, so the removed work was pure
query and commit overhead.
Verified with a testcontainer test (red before, green after) counting
the emitted deletes across the no-op, batched-detach, and
schedule-delete cases, and end to end through `trigger dev`: three
declarative schedules created, surviving a re-sync, then two removed in
a single batched delete with the third preserved.
Debouncing with a `delay` longer than an hour did nothing at all.
The engine applied a server-side ceiling on how long a debounced run
could be pushed back, measured from the run's `createdAt` and defaulting
to one hour. A run is only pushed back while its new execution time
stays inside that ceiling, so a `delay` at or above it could never push
anything: the waiting run was released, the trigger started its own run,
and the next trigger repeated it. A `delay: "12h"` produced one run per
trigger, each correctly delayed by 12h, with no error raised and nothing
on the run to show the debounce key had been ignored.
The ceiling is now unset by default. A debounce key with no `maxDelay`
keeps collapsing triggers for as long as they keep arriving, which is
what the docs have always described. Self-hosters who want a bound can
still set `RUN_ENGINE_MAXIMUM_DEBOUNCE_DURATION_MS`.
That has a consequence worth stating plainly, so the docs now carry a
warning for it: with no `maxDelay`, a continuously triggered key never
executes. Set `maxDelay` when the work has to happen eventually.
**Failing fast on an unusable `maxDelay`.** A caller who sets `maxDelay`
no longer than their `delay` hits exactly the dead end described above,
so that pair is now rejected at trigger time instead of silently
behaving as if no debounce were set:
```
debounce.maxDelay (1h) must be longer than debounce.delay (12h). A debounced run is only
pushed back while it stays inside maxDelay, so with these values every trigger would create
its own run.
```
An unparseable `maxDelay` is rejected too, rather than quietly falling
back to no bound at all, and so is a `delay` given as a date rather than
a duration, which could never work because the value is re-applied on
every push.
The same check runs against a configured server ceiling, so a
self-hosted deployment that sets
`RUN_ENGINE_MAXIMUM_DEBOUNCE_DURATION_MS` gets the error rather than the
silent failure this PR is about. With no `maxDelay` and no configured
ceiling, which is the default, there is nothing to conflict with and
nothing is rejected.
The docs, the `TriggerOptions` JSDoc and the engine option all now state
that the room available to push is the gap between `delay` and
`maxDelay`. The run engine suite gains the case that motivated this:
four triggers on one key with a 12h delay now collapse to a single run.
The agent ships Chat and Investigate here; Watch — telling the user later —
follows in its own PR. The whole user-facing feature leaves: the watch card,
chips, wake banner and toast, the unread-wake badge and its poll, the watch
routes, checks, batches and sweeps, the watch alert email and its channel type,
and the watchMaintenance cron.
The agent no longer promises it either: schedule_watch and the alert tools are
gone from the tool set, and the Watches section is out of the system prompt.
Leaving that text in would have had the agent refuse to poll for something it
could no longer offer.
The datastore's watch tables stay. The migrations ship in this PR, so the drizzle
schema that describes them has to ship too — a schema that no longer matched the
migrated tables would make the next generate emit a drop.
## What
Rate-limit the API by **environment** rather than per API key.
Previously the limiter keyed its bucket on the hash of the full
`Authorization` header — one bucket per key. With additional environment
API keys (`tr_*_sk_*`), an environment can mint many keys and each got
its own full bucket, so more keys = higher effective rate limit. This
collapses all of an environment's keys onto a single shared
per-environment bucket, so the ceiling is exactly the configured limit
regardless of key mix.
## How
- `authorizationRateLimitMiddleware` now lets the override return `{
config?, identifier? }`. `identifier`, when present, is the rate limit
bucket key; otherwise it falls back to the hashed `Authorization` header
(unchanged legacy behavior, still used by `engineRateLimiter` and any
unauthenticated fallthrough).
- `apiRateLimiter`'s override resolves the environment id and uses it as
the identifier:
- **Additional keys** (`isAdditionalApiKey`) resolve via a new
`resolveAdditionalApiKeyRateLimitScope()` — a **scope-agnostic** keyHash
→ (environmentId, org limiter config) lookup. It is deliberately
permissive (restricted keys resolve too) because it's used **only for
bucketing, never as an auth decision** — request auth still goes through
the RBAC bearer controller, which enforces scopes. Revoked/expired keys
are excluded so they can't hold a bucket warm.
- **Root/legacy keys** reuse the environment already resolved by
`authenticateAuthorizationHeader` and key on `environment.id` too.
- The identifier is always the stable environment id, never the secret
key (which can rotate and would split the bucket).
- The whole override result is cached per key by the existing SWR cache,
so **no extra per-request lookup and no separate Redis mapping** is
added.
## Behavior notes
- Root + additional keys of the same environment now share one bucket
(ceiling = configured limit, not a multiple of it). Restricted
additional keys are included — they were the biggest gap, since they
authenticate via the RBAC controller and previously fell back to per-key
buckets.
- **Public JWTs** keep their existing fixed-window, per-token bucketing.
- One-time bucket reset on deploy (bucket keys change); harmless.
## Tests
- New: two tokens resolving to the same identifier share one bucket.
- New: with no identifier, bucketing stays per-key (legacy behavior
preserved).
- Updated existing override tests to the new `{ config }` return shape.
Base: `feat/multi-keys-surface`. Closes TRI-12888.
## Summary
Projects can create, inspect, expire, and revoke multiple API keys for
each environment. Plaintext values are shown only at creation; stored
credentials are hashed and the API keys page displays only an obfuscated
suffix afterward.
Self-hosted installations support full-access additional keys by
default. Authorization extensions can provide additional access presets
and optional task selection. Additional keys can also mint scoped public
access tokens through the Trigger.dev API without receiving the
environment signing key.
## Feature notes
- Only admin+ can create API keys (Developer can make in Development
branch).
- JWT self-signing will be a server call when used with new `_ak_` keys.
- JWTs with long expiry can keep working even with api key deleted (gets
priveleges from api key, signed with root key)
- Unfiltered session listings intentionally preserve the existing broad
task-read behavior. Filtered listings enforce task-level scopes for
every requested task.
- Buffered runs without a task identifier are not safely authorizable,
so cancel/replay requests fail closed rather than resolving an unscoped
run.
- Batch and waitpoint endpoints intentionally return server-minted,
narrowly scoped public tokens to all callers. These tokens have bounded
lifetimes and may remain valid until expiry after API-key revocation.
## Deployment notes
Deploy the management UI and public-token endpoint with new key creation
disabled. Enable creation for selected organizations after the
authentication path and released SDK have been verified, then expand
availability gradually.
Revoking an API key prevents new bearer requests and new token minting.
Public tokens already minted by that key remain valid until their own
expiration because they are signed by the environment signing key.
## TODO
- [x] Add "Created by" to the key table
- [x] Document that streamed batch ingestion is non-atomic and may
partially accept items before a validation or authorization error.
## Follow-ups
- [x] Add an organization-level feature flag for the API key management
UI and creation action.
- [x] Document rollout ordering: enable additional-key lookup before
enabling issuance.
- [x] Add a system-wide gate that can stop new key issuance without
disabling authentication for existing keys.
- [x] Replace the generic SDK compatibility warning with the first
published compatible version. Old SDK will mint an unusable token if
given an `_ak_` key.
- [x] Add public documentation covering creation, storage, expiration,
revocation, SDK compatibility, and public-token lifetime behavior.
- [x] Add observability for key creation, revocation, policy preparation
failures, and public-token mint failures.
- [ ] Exercise create, copy-once display, authenticate, mint, expire,
and revoke flows end to end before broad enablement.
Route conflicts were the tab-title work meeting the agent page-context handle: both
sides kept, duplicate meta exports resolved to pageMeta, duplicate imports merged
with unused bindings dropped. Lockfile regenerated from the merged manifests.
Adds a shared `pageMeta()` helper and 74 route declarations, so a title
reads `run_abc | Runs | Trigger.dev` — the specific thing first, then
the page. Org pages also carry the organization: `Team | Acme |
Trigger.dev`. Inside a project no scope is added, because the dashboard
switches projects in every tab at once.
Page names are unchanged; what's new is that a page says which one it is
at all. Three wording changes on purpose: the queue page now names the
queue, the model page names the model, and entity pages carry their
section.
A retried card submit was only repairable while the first attempt's watch was
still active. Once it had fired, expired, or answered in one shot, the retry
re-evaluated the condition and created a second operation.
A watch_submissions ledger, keyed (chat_id, client_request_id), is now written
before the condition is read and carries the outcome once there is one. A retry
looks it up first: a recorded outcome is replayed, a different draft under the
same id conflicts, and only a pending row proceeds - converging on the watch id
reserved up front rather than creating another.
The page load reported unread wakes only, so a fresh browser whose watch was
created elsewhere and hasn't fired yet never started polling: the wake landed
without a toast or a dot until a reload. The loader now returns the active-watch
presence too, in one read per page load.
Katia Bulatova