## What
The one-off worker container boot is billed to whichever test resolves
the fixture first. This moves it into a `beforeAll` with its own
timeout.
## Why
vitest runs the fixture chain *inside* the test timer:
```js
// @vitest/runner 4.1.7
setFn(task, withTimeout(...withFixtures(handler)..., timeout, ...))
```
There is no `fixtureTimeout`. So booting Postgres (plus `CREATE
DATABASE`, schema push, ClickHouse and Redis) lands on the first test
and consumes a budget sized for test work.
That is why losing the image pre-pull on fork PRs was fatal rather than
merely slower: the extra ~10s crossed the 60s cap. Since fork time is
roughly internal + 10s and forks exceed 60s, internal runs were already
clearing that cap by under 10s — a latent flake regardless of forks.
## How
`withWarmup` wraps each fixture family and lazily registers a
`beforeAll` on first touch, with its own generous timeout. Registration
is lazy so only files that actually use a family pay for it —
`@internal/testcontainers` is imported by hundreds of test files, many
of which only need Redis. It registers once per file, since `isolate`
gives each file a fresh module registry.
Eight families are wrapped. `isolatedRedisTest`,
`replicationContainerTest` and `postgresAndRedisTest` are deliberately
untouched: they use per-test containers by design, so there is no
one-off boot to hoist.
No test file or CI changes, and it applies to every package using these
fixtures.
## Verification
Proven by mutation. `src/warmup.test.ts` runs container tests under a
deliberately tight cap:
| | Result |
| --- | --- |
| with the warm-up | passes |
| warm-up neutered | fails, `Test timed out` |
It is kept as a regression test — without it, unwrapping a fixture would
break nothing visibly.
`triggerFailedTask.call.test.ts`, one of the five shard casualties,
passes locally in 20.4s.
## Also here
`@internal/testcontainers` had no `test` script, so `turbo run test
--filter "@internal/*"` skipped the package and its existing
`heteroDedicated.test.ts` never ran in CI. Adding the script (matching
the sibling packages') runs both files; verified green through turbo
exactly as CI invokes it.
## What
Three corrections to the pre-pull lists, each verified against what the
suites actually use.
## Changes
**`ryuk:0.11.0` -> `0.14.0`** in `e2e-webapp.yml` and
`e2e-webapp-auth-full.yml`. The installed testcontainers hardcodes the
image it starts:
```js
// testcontainers@11.14.0 build/reaper/reaper.js
: ImageName.fromString("testcontainers/ryuk:0.14.0").string;
```
So those two lines were pre-pulling an image nothing starts, and the one
actually used was never pre-pulled. The other three workflows already
say 0.14.0.
**`postgres:17` added** to `unit-tests-webapp.yml`. The webapp suite
references `docker.io/postgres:17` across 10 files but only
`postgres:14` was pre-pulled. `unit-tests-internal.yml` already pulls
both.
**Electric pinned to its digest** in `unit-tests-webapp.yml`. The tests
run `electricsql/electric:1.2.4@sha256:20da...` while the pre-pull asked
for the bare tag, so the pre-pull did not necessarily populate the
manifest the tests then request.
## Not changed
The otel collector and s2 images are pulled by other workflows but are
not used by the webapp suite, so they are deliberately not added here.
`postgresAndRedisTest` uses per-test containers by design and needs
nothing pre-pulled.
## What
The `Pre-pull testcontainer images` step is gated on
`env.DOCKERHUB_USERNAME`. Fork PRs receive no repository secrets, so
that variable is empty and the step is skipped along with the DockerHub
login it was grouped with.
## Why
With the pre-pull skipped, testcontainers pulls images lazily — inside
the first test that resolves the fixture, against that test's
`testTimeout`. On PR #4534 that pushed five webapp shards past their 60s
cap across three runs, each failing as `Test timed out in 60000ms` while
42 of 43 files in the shard passed.
Measured cost of the missing pre-pull, comparing the delta from vitest
start to the first container fixture on the same runner class:
| Run | Delta |
| --- | --- |
| internal x2 | +139.9s, +139.4s |
| fork x2 | +149.7s, +149.4s |
A 10.0s penalty, bimodal to within 0.3s.
Note the pulls themselves succeed anonymously — there are no rate-limit
errors in any of the failing logs. Only the login needs credentials, so
the pre-pull can run unconditionally.
## Scope
Removes the `if:` from the pre-pull step in all five workflows that have
one. The DockerHub login stays gated, since it genuinely needs secrets.
## Summary
Memoize shared context values so provider renders do not unnecessarily
rerender every consumer. Oxlint now enforces this pattern for the rest
of the dashboard.
Base: [#4677](https://github.com/triggerdotdev/trigger.dev/pull/4677)
## Summary
Enable lint rules that prefer direct iteration and concise function
callback types.
The existing code now uses direct iteration where no index is needed,
and callback contracts use function types consistently.
Base: [#4675](https://github.com/triggerdotdev/trigger.dev/pull/4675)
## Summary
Enable JSX cleanup rules for shorthand fragments and self-closing
components.
The existing JSX is automatically simplified, and future components will
follow the same concise form.
Base: [#4673](https://github.com/triggerdotdev/trigger.dev/pull/4673)
## Summary
Enable small cleanup rules for redundant boolean expressions, object
ownership checks, assignments, and object construction.
The existing call sites now use the simpler equivalent forms, keeping
future code consistent without changing behavior.
Base: [#4672](https://github.com/triggerdotdev/trigger.dev/pull/4672)
## Summary
Enable additional lint rules that catch unsafe optional-chain
assertions, inherited-property iteration, anonymous symbols, and unsafe
external links.
The existing violations now use explicit values and own-property checks,
so the rules can prevent those patterns from returning.
Adds an optional priority class for run pods.
```
KUBERNETES_RUN_POD_PRIORITY_CLASS_NAME
```
When set, the value is applied as `priorityClassName` on the run pod
spec. When unset, pods are created exactly as before.
Off by default, and inert unless set. It sits beside the existing
`KUBERNETES_SCHEDULER_NAME` option and follows the same conditional
shape:
```ts
...(env.KUBERNETES_RUN_POD_PRIORITY_CLASS_NAME
? { priorityClassName: env.KUBERNETES_RUN_POD_PRIORITY_CLASS_NAME }
: {}),
```
## Verification
`typecheck --filter supervisor`, `format` and `lint` clean. No changeset
or `.server-changes/` note: off by default, no user-visible behaviour
change.
## What
Runs that carry no external trace context (schedules, task-to-task
triggers) fall back to a trace id generated once in the [`TracingSDK`
constructor](https://github.com/triggerdotdev/trigger.dev/blob/main/packages/core/src/v3/otel/tracingSDK.ts#L165).
With `experimental_processKeepAlive` the SDK outlives the run, so every
run on a warm process is exported to the external OTLP endpoint under
that one id.
Across our production traces, 80.3% contained spans from more than one
run, worst case 25. Per-run cost and latency attribution is unusable as
a result. This is the same warm-start hazard c043c4a6a fixed for the
external-context path, which left the fallback captured at construction.
## How
`FallbackExternalTraceIds` hands out one id per internal trace, shared
by the span and log wrappers so a run's spans and logs agree.
The id is keyed off the record's own internal trace id rather than
ambient state at export time, because batch processors drain
asynchronously and a run's records routinely export after the next run
has started. The map is bounded and evicts least-recently-used, so a run
that is still exporting can't lose its id.
Granularity follows the internal trace, so a run and the runs it
triggers stay on one trace.
**Risk:** the wrappers only exist when `exporters` / `logExporters` are
configured, so deployments that don't export externally are untouched.
Nothing outside `tracingSDK.ts` changes.
**Known gap (pre-existing):** sampling and id selection still branch on
ambient `getExternalTraceContext()`, so records draining across a run
boundary in mixed mode are misplaced in both directions. It can't use
the approach here — the external id comes from the run's incoming
`traceparent`, which isn't carried on the record — so closing it means
capturing `internalTraceId -> external context` in a span processor.
Happy to follow up separately.
---
## Testing
`packages/core` suite passes. `pnpm run format` and `pnpm run lint:fix`
produce no diff.
Six cases in `externalSpanExporterWrapper.test.ts`, each
mutation-checked rather than just observed passing: one id per run,
stability within a run, correct id when records drain after the next run
started (spans and logs together), external export stays off when
unconfigured, retention of a run still exporting while the map churns,
and the bound itself.
**CI:** the five failing `webapp` shards are the ones containing
`containerTest` suites. Fork PRs receive no repository secrets, so
`unit-tests-webapp.yml` skips the DockerHub login and the image pre-pull
(both gated on `env.DOCKERHUB_USERNAME`) and the container tests time
out at 60s. Same five shards across five runs, every failure a 60s
timeout, and those shards pass on internal PRs. Happy to be corrected if
you can run them with secrets available.
---
## Changelog
Unrelated runs are no longer merged into a single trace in your external
observability tool when they happen to execute on the same warm worker
process. A run and the runs it triggers still share one trace, so a run
tree stays together.
---
## Screenshots
_n/a_
---
_Supersedes #4526 (auto-closed before I was vouched) and #4533 (opened
ready rather than as a draft). GitHub won't reopen either._
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Iss <74388823+isshaddad@users.noreply.github.com>
Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
<!-- ccr-slack-attribution -->
_Requested by **Matt Aitken** · [Slack
thread](https://triggerdotdev.slack.com/archives/C0BKB98B84W/p1787045331358929)_
Copy-only reword of the banner shown to org admins who have not set a
billing limit yet.
**Before** — the banner read "Protect your organization from unexpected
usage spikes." with a button labelled "Configure billing limit".
**After** — it reads "Add a billing limit to your account to prevent
overspending" with a button labelled "Billing limit settings".
The new wording names the action up front and matches the destination it
sends you to, so the banner reads as a settings link rather than a
one-off setup step.
## ✅ Checklist
- [x] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [x] The PR title follows the convention.
- [x] I ran and tested the code works
---
## Testing
Formatting and linting pass (`oxfmt --check`, `oxlint`). No tests or
snapshots assert this copy. The change is two string literals in one
component, with no behaviour attached.
---
## Changelog
Reworded the billing-limit banner for organizations without a limit
configured, and relabelled its button to "Billing limit settings".
---
## How
Both strings live in `NoLimitConfiguredBanner` in
`apps/webapp/app/components/billing/OrgBanner.tsx`: the heading is the
`canManageBillingLimits` branch of the banner's children, and the label
is the `<span>` inside the `LinkButton`. Only those two literals
changed. The button still points at `v3BillingLimitsPath(organization)`
(`/orgs/{slug}/settings/billing-limits`), so routing, permissions and
the non-admin variant of the message are untouched.
Co-authored-by: Claude <noreply@anthropic.com>
Adds KUBERNETES_RUNNER_SECURITY_CONTEXT (off | baseline | restricted), selecting how constrained the run container is.
baseline drops the capability bounding set and blocks privilege escalation. restricted additionally pins the container to a non-root uid, chosen by runtime so bun images get their own.
Default is off, so this is inert on merge.
## Summary
A run triggered with a `ttl` could get permanently stuck showing as
queued. If the run started executing and was then requeued after a
failure (a stalled heartbeat, a worker dying mid-run) once its TTL had
already elapsed, the next dequeue pass silently dropped it from every
queue structure. The run stayed QUEUED in the database forever, and
nothing (dequeue, the TTL consumer, queue repair) could ever see it
again.
## Root cause
Enqueue registers a TTL entry for the TTL consumer, and the first
dequeue removes it ("the run is executing, not expired"). A nack rewrote
the message preserving the original `ttlExpiresAt` without
re-registering that entry. The next dequeue pass then took the
expired-TTL branch: remove the run from the queue sorted sets and leave
the message for the TTL consumer to finalize. But the consumer's entry
was gone, so nothing ever finalized the run.
The fix has two halves:
- `nackMessage` drops `ttlExpiresAt` from the rewritten message. TTL
only applies to runs that have never been dequeued (the same contract as
`includeTtl` on re-enqueues), so a requeued run stays dequeuable and is
never expired by its original deadline.
- The dequeue expired-TTL branches now (re-)register the TTL entry
instead of assuming it exists, so any message still carrying a lapsed
`ttlExpiresAt` with no TTL entry (including ones written before this
fix) finalizes as EXPIRED instead of orphaning.
## Verification
New engine test suite `ttlNackRequeue.test.ts` (testcontainers, real
Redis and Postgres). All four tests fail before the fix and pass after:
- a heartbeat-stalled EXECUTING run with a lapsed TTL is requeued and
dequeued again instead of orphaned (the full production failure chain)
- requeue-after-failure strips `ttlExpiresAt` so later dequeues do not
treat the run as expired
- a lapsed-TTL message whose TTL entry is missing is re-registered by
dequeue and finalized as EXPIRED, for both plain and concurrency-key
queues
Also ran the existing ttl, heartbeats, dequeuing and attemptFailures
engine suites plus the full run-queue suite (149 tests) against the
change.
Adds two optional env vars that rewrite the registry host of run pod images at pod creation, so a supervisor can pull from a registry in its own region. Off by default and inert unless both are set. Exact host-prefix matching, so look-alike hosts pass through untouched.
Replaces the hardcoded runner seccomp profile path with KUBERNETES_RUNNER_SECCOMP_PROFILE_PATH, and the node-24-only condition with KUBERNETES_RUNNER_SECCOMP_PROFILE_RUNTIMES (none | node-24-plus | all).
Both defaults reproduce current behaviour, so this is inert on merge. Widening the scope or turning attachment off becomes a config change rather than a deploy.
The supervisor now supports routing an organization's runs to specific
nodes. `KUBERNETES_ORG_PLACEMENT_OVERRIDES` takes JSON keyed by the
internal org ID, adding node selector entries and tolerations to that
org's run pods, e.g. to route an org onto a dedicated, tainted node
pool:
```json
{"<orgId>": {"nodeSelector": {"pool": "dedicated"}, "tolerations": "dedicated=runs:NoSchedule"}}
```
The node selector merges over the defaults (the override wins on key
collision, with a warning logged). Tolerations append to the existing
runner and scheduled-run sets. Overrides are validated at startup
similar to `KUBERNETES_RUNNER_TOLERATIONS`.
Exposed in the Helm chart as
`supervisor.config.kubernetes.orgPlacementOverrides`, where tolerations
can also be given as a list.
## What & why
The dashboard agent can now run its model calls through AWS Bedrock
instead of the direct Anthropic API, chosen by a single env switch. It's
**off by default** (`DASHBOARD_AGENT_MODEL_PROVIDER` unset ⇒
`anthropic`), so merging changes nothing at runtime — the Bedrock path
is a dormant branch until an operator sets the switch and AWS config.
The default Anthropic path is byte-for-byte unchanged.
This also carries a related tenant-isolation hardening for the agent's
delegated token (kept together deliberately — both land the agent on
Bedrock for HIPAA readiness). Refs: TRI-13251, TRI-11032.
## What's inside
**Provider seam** —
`internal-packages/dashboard-agent/src/model-provider.ts`: the registry
now holds both `anthropic` and `bedrock`; `resolveDashboardAgentModel()`
maps the canonical `"anthropic:<id>"` strings the managed prompts carry
to the active provider, and the cache-breakpoint helpers emit the active
provider's shape — Anthropic `cacheControl` vs Bedrock `cachePoint`.
Managed prompt strings stay canonical, so stored prompts don't change
meaning. Unmapped model ids throw rather than shipping a guaranteed-404
profile. All agent, watch, compaction and title callsites route through
the resolver; the `dashboardAgentModelKey` locals override (test mock
injection) is preserved.
**Cache telemetry** — `step-cache.ts`: cache token usage is read from
the active provider (Anthropic reports it on provider metadata; Bedrock
reports the write on metadata and the read via standard usage), so
`gen_ai.usage.cache_*` is populated on both. This also fixes a latent
ordering bug where step attributes could null-overwrite the prompt-cache
read count.
**Webapp callsites** — `dashboardAgentHeadStart.server.ts` and the
head-start route resolve the model and the cache breakpoint through the
same seam, so the warm-up prefix and the following turn share one
provider. The head-start firing gate is provider-aware: on Bedrock it
gates on `AWS_REGION` and lets the SDK resolve credentials (IAM role /
static keys / session token / bearer), so a role-based deploy still
warms; on Anthropic it stays `Boolean(ANTHROPIC_API_KEY)`.
`app/env.server.ts` gains the optional AWS vars and validates
`DASHBOARD_AGENT_MODEL_PROVIDER`. `ANTHROPIC_API_KEY` is untouched and
not required on a Bedrock deploy.
**Tenant-isolation hardening** —
`internal-packages/rbac/src/fallback.ts`: for a **scoped** context, the
OSS `authenticateUserActor` now applies the same membership floor as the
session path — a delegated user-actor token whose user is not a member
of the scoped org/project is denied (403). Unscoped tokens keep their
prior behavior (no tenant claim, no lookup). The user lookup falls back
replica→primary so replication lag can't spuriously 401 a just-joined
member. Members and admins are unaffected. Previously this invariant
held only through per-route discipline; this makes it structural.
## Enabling Bedrock (later, ops)
- Set `DASHBOARD_AGENT_MODEL_PROVIDER=bedrock` **identically** in both
the webapp and the agent task container — the webapp warms the cache
prefix and the task reads it, so a split would silently miss the cache.
- Set `AWS_REGION` and provide credentials the Bedrock SDK can resolve
(IAM role preferred). For v1 this runs **without** an Anthropic API key.
Note: with no Anthropic key set, rollback is "turn the agent off", not
"unset the switch" (unsetting falls back to the Anthropic provider,
which then has no key).
- Two things to confirm before rollout: the Sonnet inference-profile id
is validated against the SDK's own model-id union but still warrants a
live smoke test; and Bedrock prompt caching for Sonnet is a 5-minute
window (not Anthropic's 1h), so input-token cost rises when flipped.
## Testing
Unit tests cover both provider paths: the provider switch and
per-provider cache shapes, a structural regex asserting Bedrock ids are
real inference profiles (not an echo of the table), the split-metadata
cache telemetry, and real-Postgres RBAC tests — member allowed, scoped
non-member denied (org-only and project-only), missing user → 401, admin
non-member exempt, unscoped success. `typecheck --filter webapp` and the
dashboard-agent + rbac suites pass.
## Summary
When an AI SDK tool call failed inside a run, the span showed up under
the "Errors only" filter but the span inspector gave no hint of what
went wrong. The exception was recorded on the span all along; the
`ai.toolCall` and `ai.embed` inspector views just never rendered span
events. Failed tool call and embedding spans now show the standard error
block (message plus stack trace) below the Input section.
## Root cause
Generic spans render exception span events via the `SpanEvents`
component, but the AI-specific span entities replace the whole panel
with their own layout and dropped the events entirely. The span's events
are now passed into `AIToolCallSpanDetails` and `AIEmbedSpanDetails` and
rendered with the same `SpanEvents` component the generic view uses.
Errored generation spans (`ai.generateText` and friends) use a tabbed
view and still don't surface errors; that needs its own design pass and
is left for a follow-up.
## The bug
A concurrency key is an unrestricted client string
(`ConcurrencyKeySchema` is `z.union([z.string(),
z.number()]).transform(String)`), and `concurrencyKeySection` does no
escaping, so `*` reaches the queue raw. `queueKey` then renders it as
`...:queue:<q>:ck:*`, which is byte-identical to the wildcard member the
CK scripts keep in the master queue to mean "this base queue has
concurrency-key work".
Every CK script ends with the same pair:
```lua
-- Rebalance master queue with ck:* member
redis.call('ZADD', masterQueueKey, earliestIdx[2], ckWildcardName)
-- Remove old-format entry from master queue (transition cleanup)
redis.call('ZREM', masterQueueKey, queueName)
```
`ckWildcardName` is `toCkWildcard(message.queue)`, and for a `*`-keyed
run that returns the identical string, so the cleanup on the second line
deletes what the rebalance on the first line just wrote.
The master queue then has no entry for that base queue, while `ckIndex`
and the variant queues still hold the work. **Every concurrency key on
the queue stops being dequeued**, not just the `*` one. It is silent,
and it only recovers if some later write happens to re-add the member.
Reproduced before the fix:
```
master queue AFTER normal ck enqueue: ["{org:...}:queue:task/my-task:ck:*"]
master queue AFTER ck='*' enqueue: []
ckIndex members (work still queued): [":ck:user-1", ":ck:*"]
dequeued: []
```
Blast radius is bounded to the environment that triggers it, so it is
self-inflicted rather than cross-tenant, but a single trigger stalls the
queue for everything on it.
## The fix
Guard the cleanup so it never removes the wildcard member:
```lua
if queueName ~= ckWildcardName then
redis.call('ZREM', masterQueueKey, queueName)
end
```
Applied to all 10 CK scripts (4 enqueue, 6 ack/nack/dead-letter). No
key-format change and no migration: a queue already stranded in Redis is
repaired by its next write.
I considered rejecting `*` at the API boundary instead and rejected it.
Existing Redis state and `TaskRun.concurrencyKey` rows already hold raw
`:`-bearing and `*` keys, so changing key construction would orphan
in-flight messages and split concurrency accounting mid-deploy. Boundary
validation would still be reasonable as belt-and-braces later, but the
Lua guard alone fixes it including for state already out there.
## Testing
`ckWildcardKey.test.ts` covers the enqueue, ack and nack paths. All
three pass with the guard and **all three fail without it**, verified by
reverting. Full `src/run-queue/` suite is green (166 tests).
## Note for #4367
The virtual-time branch adds three more CK scripts with the same pattern
(`enqueueMessageCkVtimeTracked`, `enqueueMessageWithTtlCkVtimeTracked`,
`nackMessageCkVtimeTracked`). They do not exist on main so they are not
in this PR; the same guard needs applying there, and I will do that on
that branch.
## Summary
`lefthook.yml` has been in the repo since #4147, but nothing installs
lefthook and nothing runs `lefthook install`, so the pre-push hook it
describes has never fired for anyone. #3977 had removed the `lefthook`
devDependency a week before #4147 landed, and #4147 only added the
config file.
This supplies the missing half:
```diff
+ "prepare": "lefthook install",
+ "lefthook": "^2.1.10",
"onlyBuiltDependencies": [
+ "lefthook",
```
With those in place, `pnpm install` wires the hook up on clone, and the
format and lint checks actually run before a push instead of first
failing in CI.
Also here: the pre-push jobs run in parallel rather than in sequence,
and `CONTRIBUTING.md` documents the hook, including how to skip it and
the fact that GitButler only runs hooks when "Run hooks" is enabled in
its settings.
`lefthook@2.1.10` is the current release.
<!-- ccr-slack-attribution -->
_Requested by **Matt Aitken** · [Slack
thread](https://triggerdotdev.slack.com/archives/C045W9WM3E1/p1786948064703889?thread_ts=1786948064.703889&cid=C045W9WM3E1)_
Adds `gtremper` to `.github/VOUCHED.td`.
Before: PRs from `gtremper` are auto-closed by the vouch check.
After: `gtremper` is vouched, so their PRs stay open and run CI.
Done as a direct file edit rather than the issue-comment flow because
there is no open Vouch Request issue for this user, matching the
precedent in #3804.
Co-authored-by: Claude <noreply@anthropic.com>
## Summary
`packages/core` declared `socket.io`, the server package, but never
imported it. Its only Socket.IO usage is the client:
```
packages/core/src/v3/zodSocket.ts
packages/core/src/v3/runEngineWorker/supervisor/session.ts
import { io } from "socket.io-client";
```
The only occurrence of `socket.io` outside those client imports was the
`package.json` line itself. Since `@trigger.dev/core` is published, that
line meant every consumer installed a server package nothing in the tree
imports.
`socket.io-client` is untouched. `apps/webapp` and `apps/supervisor`
keep their own `socket.io` dependencies, so the server side is
unaffected.
Found with `pnpm run knip:deps`, which the repo already ships.
`pnpm run typecheck` passes across all 57 workspaces, and
`@trigger.dev/core` builds clean.
Stacked on #4639.
## Summary
`find-my-way` was resolving `9.3.0` even though its only parent,
`fastify@5.8.5`, declares `^9.0.0` and so already permitted a newer
release. The lockfile had not re-resolved since. This adds a floor so it
lands on a current 9.x:
```json
"find-my-way@>=9 <9.7.0": "^9.7.0"
```
It resolves to `9.7.0`. Nothing outside the 9.x line is touched, and no
parent is asked to accept anything its declared range did not already
allow.
The whole path is development only: `find-my-way` arrives through
`fastify`, which arrives through `evalite`, a devDependency of
`apps/webapp` used by the `eval:dev` harness.
Stacked on #4638.
## Summary
The existing `ip-address` override is scoped to a single parent,
`@jsonhero/json-infer-types>ip-address`. A second parent reaches
`ip-address` independently: `express-rate-limit@8.6.0`, which is itself
pinned by our `@modelcontextprotocol/sdk@>=1.26.0>express-rate-limit`
override. That path was resolving `10.2.0` while the scoped path
resolved `10.5.0`, so the tree carried two copies.
This adds a matching scoped override for the second parent:
```json
"express-rate-limit>ip-address": "^10.3.1"
```
`express-rate-limit` declares `^10.2.0`, so this asks nothing of it that
its own range did not already allow. The tree now resolves a single
`ip-address@10.5.0`.
The existing `@jsonhero/json-infer-types` override stays: that package
declares `ip-address: ^8.1.0`, so removing it brings an 8.x copy back.
Stacked on #4637.
## Summary
`nanoid` was pinned at exactly `3.3.8` in five manifests. Two of those
five never imported it: in `internal-packages/schedule-engine` and
`internal-packages/webhook-engine` the only occurrence of the string
`nanoid` in the entire package was the `package.json` line itself. Both
are removed rather than bumped.
The three that genuinely use it move to `3.3.18`, a version already
present in the tree via `postcss`, so this pulls in nothing new.
| Package | Uses it | Change |
| --- | --- | --- |
| `internal-packages/schedule-engine` | no | removed |
| `internal-packages/webhook-engine` | no | removed |
| `apps/webapp` | yes | `3.3.8` to `3.3.18` |
| `packages/core` | yes | `3.3.8` to `3.3.18` |
| `internal-packages/run-engine` | yes | `3.3.8` to `3.3.18` |
| `packages/redis-worker` | yes | `^5.0.7` to `^5.1.16` |
`redis-worker` is on the 5.x line and is included because its declared
range already permitted a newer release; the lockfile had simply not
re-resolved, leaving it on `5.1.2`.
The unused declarations were found with `pnpm run knip:deps`, which the
repo already ships.
`pnpm run typecheck` passes across all 57 workspaces.
## Summary
`socket.io` was pinned at exactly `4.7.4` in three manifests
(`apps/webapp`, `apps/supervisor`, `packages/core`). That pin capped
`engine.io` at 6.5.4, because 4.7.4 declares `engine.io: ~6.5.2`.
Moving all three pins to `4.8.3` lifts that cap: 4.8.3 declares
`engine.io: ~6.6.0`. The webapp's direct `engine.io` devDependency moves
from `^6.5.4` to `^6.6.7` to match.
These are direct dependencies, so they are bumped in place rather than
forced with an override.
## Result
The tree previously carried two `engine.io` copies. It now carries one:
```
engine.io@6.6.8
└─┬ socket.io@4.8.3
├── @trigger.dev/core (dependencies)
├─┬ react-email
│ └── emails (devDependencies)
├── supervisor (dependencies)
└── webapp (dependencies)
```
`react-email` was already resolving `socket.io@4.8.3` in this same tree,
so that combination was already running here before this change.
## Servers move, clients do not
This bumps `socket.io` (the server) only. `socket.io-client` stays at
`4.7.5` in `packages/core` and `packages/cli-v3`, deliberately: the fix
is server-side, and clients ship inside user deployments, so leaving
them alone keeps the blast radius small. That means a 4.8.3 server will
be talking to 4.7.5 clients indefinitely, which is worth being explicit
about.
That pairing is safe because neither wire protocol changed. Both
versions report the same protocol numbers:
| | 4.7.4 | 4.8.3 |
| --- | --- | --- |
| Socket.IO protocol (`socket.io-parser`) | 5 | 5 |
| Engine.IO protocol (`engine.io-parser`) | 4 | 4 |
The version bump moves `socket.io-parser` 4.2.6 to 4.2.7 and `engine.io`
6.5.4 to 6.6.8, but the protocol constants each exports are unchanged.
The 4.8.0 changes are additive on the client (custom transport
implementations, a `tryAllTransports` option) and bug fixes on the
server.
Verified rather than assumed, with a cross-version matrix covering both
transports and both directions:
```
PASS server 4.8.3 <- client 4.7.5 websocket / polling
PASS server 4.8.3 <- client 4.8.3 websocket / polling
PASS server 4.7.4 <- client 4.7.5 websocket / polling
PASS server 4.7.4 <- client 4.8.3 websocket / polling
```
Each case exercised connect, a server-initiated emit, `emitWithAck`,
room join, room broadcast, and a binary payload. Compatibility holds in
both directions, so there is no upgrade-ordering requirement between
server and client.
`pnpm run typecheck` passes across all 57 workspaces.
Stacked on #4634.
## Summary
`axios` was resolving to 1.16.1 through `@slack/web-api`, which declares
`^1.16.0`. The lockfile had simply not re-resolved since, so the tree
sat on an older 1.x release than the range allows.
This adds a scoped override so the 1.x line picks up a current release:
```json
"axios@>=1.15.2 <1.18.0": "^1.18.0"
```
It resolves to 1.19.0. No parent bump is needed, since `^1.16.0` already
permits it, and `@slack/web-api` is the only consumer.
Stacked on #4633 so the two lockfile changes do not collide.
## Summary
`npm-run-all` has had no release since 4.1.5 in 2018, and pnpm now
covers the one thing we used it for. The webapp's `build` script was its
only consumer anywhere in the repo, so the dependency goes away
entirely.
`run-s build:**` becomes an explicit chain:
```
pnpm run build:remix && pnpm run build:server && pnpm run build:otlpworker && pnpm run build:sentry && pnpm run upload:sourcemaps
```
## Why this shape
I compared both forms side by side against the real `run-s` before
swapping:
| Behaviour | `run-s build:**` | explicit chain |
| --- | --- | --- |
| Scripts selected | remix, server, otlpworker, sentry | identical |
| Order | declaration order | identical |
| `upload:sourcemaps` matched by the glob | no | no |
| Second script fails | aborts, third never runs | identical |
| Exit code on failure | `1` | `1` |
`pnpm run --sequential "/^build:/"` was the closer-looking option, but
it keeps running scripts after one fails, so it is not a faithful
replacement.
The one thing given up is that `build:**` automatically picked up any
new `build:*` script, where the chain has to be edited. With four
entries that felt like the better trade.
`pnpm run build --filter webapp` passes end to end locally, all five
steps in order.
## Summary
Following a link straight into an organization's settings (for example
the usage limit link in a billing email) and then clicking "Back to app"
took you to `/`, which resolves to whichever organization you last had
selected, not the one whose settings you were looking at. The button now
links to the organization in the URL, so you land back in the org you
came from.
The org index route already redirects to the best project in that org,
so the destination is unchanged apart from being the right org.
Account settings still links to `/`, since that page is not org scoped
and has no org to return to.
## Summary
A number of `pnpm.overrides` entries had drifted behind the releases
they were written against. An override fixes the resolved version
outright, so in every one of these cases the tree was pinned to the
floor value rather than picking up later releases in the same line. This
raises each floor to a current release, and widens the selectors that
were scoped to an exact upper bound so they keep matching.
| Override | Before | After |
| --- | --- | --- |
| `body-parser` (under `express@^4`) | `1.20.3` | `^1.20.6` |
| `tar` | `7.5.19` | `7.5.21` |
| `hono` | `4.12.25` | `4.12.34` |
| `undici` (6.x) | `6.27.0` | `6.28.0` |
| `undici` (7.x) | `7.28.0` | `7.29.0` |
| `js-yaml` (3.x) | `3.14.2` | `3.15.1` |
| `js-yaml` (4.x) | `4.1.1` | `4.3.1` |
| `dompurify` | `^3.4.1` | `^3.4.13` |
| `vite` | `^6.4.2` | `^6.4.3` |
| `protobufjs` | `^7.5.6` | `^7.6.5` |
| `socket.io-parser` | `^4.2.6` | `^4.2.7` |
| `postcss` | `^8.5.10` | `^8.5.23` |
| `fast-uri` | `^3.1.2` | `^3.1.5` |
| `brace-expansion` (1.x) | `1.1.13` | `1.1.18` |
| `brace-expansion` (2.x) | `2.0.3` | `2.1.4` |
| `brace-expansion` (5.x) | `5.0.6` | `5.0.9` |
| `ip-address` (under `@jsonhero/json-infer-types`) | `^10.2.0` |
`^10.3.1` |
Every parent's declared range still accepts the new resolution, so
nothing is forced outside its stated bounds by this change.
Two of these changed a default rather than just moving version.
`js-yaml` 4.2.0 stopped resolving underscore-separated scalars such as
`1_000` as numbers, which is the YAML 1.2 behaviour, and there are none
in any YAML in this repo. `brace-expansion` 2.1.x now caps expansion
size by default, well above anything a real glob produces, and
`minimatch` calls it with no options. Neither is reachable from how we
use them.
`undici@5.29.0` and `vite@4.4.9` are left alone: their parents cap below
the newer lines, so moving either would mean taking the parent across a
major.
Verified with a clean install, and `pnpm run typecheck` passes.
## Summary
The server half of hosted webhooks: the public ingress endpoint,
signature verification, the delivery pipeline (Postgres partitioned
storage + ClickHouse for ordering), the in-app partition manager, the
HTTP API, and the dashboard (Deliveries, Endpoints, and the in-app test
console).
The public SDK and docs half is #4537. That PR carries the user-facing
API (`webhook()`, `chat.event` / `chat.channels`, the
`@trigger.dev/slack` connector) and builds on the shared
`@trigger.dev/core` schemas that ship here.
## Shipping behind a flag
A `WEBHOOK_ENABLED` env var (default off) gates the public ingress route
and the engine worker plus partition cron, so merging and deploying this
changes nothing in production until it is flipped on per environment.
The dashboard is separately gated per org by the `hasWebhooksAccess`
feature flag.
## Note on packages
This PR includes the `@trigger.dev/core` schema additions the server
compiles against, but carries no changeset. Core is not consumed
independently of the SDK, so it is released together with the SDK via
#4537. Keeping its changeset off `main` means no release cut from `main`
publishes it early.
## What
Makes two transaction-resilience behaviors real and env-var
configurable, defaults set to the good values, so we can tune during and
after the Aug 15 database patch window without a redeploy:
- **maxWait 2s → 10s** (TRI-12982): how long Prisma waits to borrow a
connection before it can `BEGIN`. A restart freeze holds the pool full,
and the only thing that errored was transaction starts giving up at 2s.
- **Retry transaction-start P2028-at-acquisition** (TRI-12984): when
Prisma can't borrow a connection within `maxWait` it raises P2028
(`Unable to start a transaction in the given time`) and **no SQL ran**,
so retrying is safe. Scoped narrowly: only that error (never P2024
pool-exhaustion), 2 attempts, jittered backoff, and a token-bucket
budget so a mass freeze can't amplify into a retry storm.
## Env vars (`DATABASE_*` convention)
Generic defaults:
| var | default |
|---|---|
| `DATABASE_TRANSACTION_MAX_WAIT_MS` | `10000` |
| `DATABASE_TRANSACTION_START_RETRY_ENABLED` | `true` (kill switch) |
| `DATABASE_TRANSACTION_START_RETRY_MAX_ATTEMPTS` | `2` |
| `DATABASE_TRANSACTION_START_RETRY_BACKOFF_MIN_MS` | `50` |
| `DATABASE_TRANSACTION_START_RETRY_BACKOFF_MAX_MS` | `250` |
| `DATABASE_TRANSACTION_START_RETRY_BUDGET_PER_SEC` | `50` |
| `DATABASE_TRANSACTION_START_RETRY_BUDGET_BURST` | `100` |
Per-writer-pool overrides, each falling back to the generic when unset
(same pattern as the per-client pool/connect-timeout work):
`RUN_OPS_DATABASE_TRANSACTION_*` and
`RUN_OPS_LEGACY_DATABASE_TRANSACTION_*` (all 7 knobs each). Transactions
only open on writer pools, so those are the only pools with their own
knobs. Each pool gets its **own** token bucket, so a storm on one pool
can't drain another's retry budget.
## Design
- The retry primitives live in `internal-packages/database` and never
read `process.env` (IoC): a P2028-at-acquisition classifier, a
`TokenBucketRetryBudget`, and `withTransactionStartRetry`, folded into
the `$transaction` helper via a new `startRetry` option. Config is
resolved at the app boundary and threaded in.
- The `$transaction` helper is the chokepoint (wraps the whole
transaction), not the per-statement `$allOperations` extension.
- The run engine's writes go through `PostgresRunStore`'s own
`.$transaction(...)`, not the webapp helper, so both the helper and the
two `PostgresRunStore` sites apply maxWait + retry (sharing the per-pool
config). Builds on the `options?: { timeout, maxWait }` seam added in
#4514.
- Webapp `$transaction` call sites get the default `maxWait` + retry
injected at one merge point, so no call site needed editing.
## Evidence
- Unit red/green in `internal-packages/database`: reverting the helper
wiring turned the acquisition-retry test red (`Unable to start a
transaction in the given time`), re-applying it green. Full package
suite 25/25. Covers: classifier (P2028-acq yes, P2024 no, in-tx P2028
no), retry (retry-then-succeed, no-retry P2024, stop at maxAttempts,
disabled, budget-exhausted, jitter bounds), token bucket, and
`$transaction` wiring.
- Typecheck clean: webapp, run-store, run-engine.
- Full-stack run: bounded queue-ay pass (15 projects, real dev runs
through the run-engine `PostgresRunStore` transaction path). 13 pass;
the 2 failures are one documented known-failure and one
stale-worker-state flake that passes 2/2 with this change active on a
fresh app.
- Boots cleanly with per-pool overrides set.
## Configuration & rollout
Ship **inert** first (zero behavior change), then flip to the good
values **live via env** — no redeploy needed for either.
### Inert — behaves exactly as today
```
DATABASE_TRANSACTION_MAX_WAIT_MS=2000 # Prisma's built-in default (change defaults to 10000)
DATABASE_TRANSACTION_START_RETRY_ENABLED=false # disable the new retry entirely
```
`maxWait=2000` is what every path used before (Prisma's default; the
run-store sites and the helper passed no maxWait). `retry=false`
short-circuits `withTransactionStartRetry` to a single run and makes the
serialization-retry exclusion a no-op. Verified on the pooler-freeze
rig: identical fail-fast P2028 at ~2003ms with zero retries —
byte-for-byte current behavior, across all pools.
### Production ("good") — the baked defaults
Rely on defaults (nothing to set) or set explicitly:
```
DATABASE_TRANSACTION_MAX_WAIT_MS=10000
DATABASE_TRANSACTION_START_RETRY_ENABLED=true
DATABASE_TRANSACTION_START_RETRY_MAX_ATTEMPTS=3 # 3 attempts (2 retries); ~30s acquisition tolerance covers a ~20-25s freeze
DATABASE_TRANSACTION_START_RETRY_BACKOFF_MIN_MS=50
DATABASE_TRANSACTION_START_RETRY_BACKOFF_MAX_MS=250
DATABASE_TRANSACTION_START_RETRY_BUDGET_PER_SEC=50
DATABASE_TRANSACTION_START_RETRY_BUDGET_BURST=100
```
Per-pool overrides `RUN_OPS_DATABASE_TRANSACTION_*` and
`RUN_OPS_LEGACY_DATABASE_TRANSACTION_*` (all seven knobs each) are
optional and fall back to the generic set — not needed for v1; the
generic set covers the control-plane, run-ops, and run-ops-legacy writer
pools. Readers open no transactions and take nothing.
**Guardrail:** the retry only engages when a pool's `pool_timeout` >
`maxWait`. Prod is fine (`DATABASE_POOL_TIMEOUT=60` >> 10). Do not set
any writer pool's `pool_timeout` at or under `maxWait`, or saturation
failures flip from retryable P2028 to non-retryable P2024 and the retry
silently stops helping.
### Rollback
Env flip (set inert) or revert. Retry only fires where no SQL ran, and
the per-pool token bucket caps a storm. No migration.
refs TRI-13295, TRI-12982, TRI-12984
<!-- ccr-slack-attribution -->
_Requested by **Matt Aitken** · [Slack
thread](https://triggerdotdev.slack.com/archives/C045W9WM3E1/p1786732623292829?thread_ts=1786732623.292829&cid=C045W9WM3E1)_
**Before:** you pause an environment, then a deploy lands (or a
background worker is created, or an admin changes the
concurrency/burst-factor). The environment starts picking up runs again
even though the dashboard still shows it as paused.
**After:** a paused environment stays paused until it is resumed, no
matter what else pushes its concurrency limit.
Pausing an environment sets `paused` in the database and writes a `0`
env concurrency limit into the run queue — the `0` is the only thing
that actually stops dequeueing. Any caller that pushed the limit without
an explicit value (`finalizeDeployment`, `createBackgroundWorker`, the
two admin environment routes) rewrote the real limit and silently
un-paused the environment.
## ✅ Checklist
- [x] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [x] The PR title follows the convention.
- [x] I ran and tested the code works
---
## Testing
`apps/webapp/test/pauseEnvironment.server.test.ts` gains two
`containerTest` cases that wire a real `RunEngine` (real Redis) in place
of the stubbed app singleton and assert the actual run-queue env limit:
- pause a PRODUCTION env → limit is `0` → run the real
`FinalizeDeploymentService` → limit is still `0`, plus a control on a
running env in the same test proving that deploy path really does push
the limit (so the `0` can't just mean "nothing happened").
- pause → resume → the real limit is restored, so the clamp can't
regress resuming.
Both cases fail on `main` (`expected 17 to be +0` and `expected +0 to be
17`) and pass with this change. `pnpm run typecheck --filter webapp` is
clean.
---
## Changelog
Fix paused environments starting to run work again after a deploy.
---
## How
The clamp lives in the shared `updateEnvConcurrencyLimits` helper in
`apps/webapp/app/v3/runQueue.server.ts`, so every present and future
caller is covered: when no explicit limit is passed and the environment
is paused, `0` is written instead of the stored maximum. An
explicitly-passed limit still wins, which is what pausing itself relies
on. The resume path now passes the post-update environment state (its
in-memory copy was read before the un-pause and would otherwise be
clamped back to `0`), and the helper no longer mutates the caller's
environment object — that aliasing made a pause followed by a resume on
the same object write `0` twice. The existing `!paused` guards in
`allocateConcurrency` and the queue-level guard in
`createBackgroundWorker` are left in place as defence in depth, and
queue-level `TaskQueue.paused` behaviour is untouched.
---------
Co-authored-by: Claude <noreply@anthropic.com>
## Summary
After a deployment promotion or rollback, newly triggered runs could
keep dispatching onto the previously deployed version for up to 30
seconds. Runs now resolve the current version fresh on every dequeue, so
a promotion or rollback takes effect immediately.
## Fix
The dequeue path resolved the worker version through a 30s in-process
cache that nothing invalidated on promotion, and it loaded the worker's
entire task and queue set only to keep the single row matching the run.
Both go away: the resolve now fetches just the matched task and queue by
unique index and reads them fresh, so there is no cache left to serve a
stale version.
```
- cache.get(env:current) # 30s TTL, never invalidated -> stale
- worker + ALL tasks + ALL queues
+ worker + one task WHERE slug=... + one queue WHERE id/name=... # fresh
```
A kill-switch env var (`RUN_OPS_WORKER_VERSION_FRESH_READ_ENABLED`,
default on) falls back to the old cached path without a code deploy.
Verified end-to-end on an isolated stack: a run triggered after a
mid-stream promotion now dequeues onto the new version, with the
previous stale behavior reproduced first.
## Summary
When resolving the current worker for a development environment,
`findCurrentWorkerFromEnvironment` loaded the entire `BackgroundWorker`
row,
including the large `metadata` JSON, even though it only ever returns a
handful
of small fields. It is a frequently-run query, so the wasted payload
adds up:
every call pulled data it immediately threw away.
## Fix
Add a `select` to the development-environment lookup listing exactly the
fields
the function returns (`id`, `friendlyId`, `version`, `sdkVersion`,
`cliVersion`,
`supportsLazyAttempts`, `engine`). The query plan is unchanged, still a
single-row indexed lookup; only the row width shrinks. No behavior
change: the
dropped columns were never read.
## Summary
The worker-version resolve path fetched every column of every
`BackgroundWorkerTask` for a worker (`include: { tasks: true }`), plus
full `WorkerDeployment` and `TaskQueue` rows, just to match one task at
dequeue. That pulls large JSON columns none of this path reads (task
`payloadSchema`/`config`/`queueConfig`/`description`, deployment
`externalBuildData`/`buildServerMetadata`/`errorData`/`git`, queue
`rateLimit`), so each resolve transfers and deserializes far more than
it uses.
## Fix
Replace the includes with explicit `select`s of only the columns dequeue
reads, in both the passthrough resolver and the app resolver:
- task: `id`, `slug`, `machineConfig`, `retryConfig`,
`maxDurationInSeconds`
- deployment: `id`, `friendlyId`, `imageReference`, `imagePlatform`
- queue: `id`, `name` (the queue matcher keys on both)
The shared `ResolvedWorkerVersion` element types narrow to match
(mirrored in the cache), which also shrinks each cached worker-version
entry.
## Impact
The `tasks` read fetches every task of a worker to match one, so its
cost scales with task count and payload-schema size. For a worker with
~70 registered tasks, dropping the unread columns cuts the per-query
transfer roughly:
| Task shape | Before | After | Reduction |
|---|---|---|---|
| Light (no payload schema, small config) | ~28 KB | ~14 KB | ~54% |
| Typical (mixed schemas / config) | ~62 KB | ~14 KB | ~77% |
| Schema-heavy (large `payloadSchema`) | ~200 KB | ~14 KB | ~93% |
The `after` size is roughly fixed because the kept columns are small;
the win grows with how heavy the dropped JSON is. Narrowing `deployment`
(four JSON columns off a single row) and `queues` saves further on top.
No behavior change: pure read-shape narrowing, no flag and no schema
change, so rollback is a plain revert. Verified with a red/green
run-engine test that asserts the resolved task, deployment, and queue
carry only the used columns, plus the queue feature-matrix runs (batch,
retry-policy, machine-preset, plain trigger) that exercise the kept
columns.
nicktrn