re2-test-supervisor-main-cffaa05
7845 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 nicktrn
|
cffaa05517 |
feat(supervisor): optional priority class for run pods (#4671)
Adds an optional priority class for run pods.
```
KUBERNETES_RUN_POD_PRIORITY_CLASS_NAME
```
When set, the value is applied as `priorityClassName` on the run pod
spec. When unset, pods are created exactly as before.
Off by default, and inert unless set. It sits beside the existing
`KUBERNETES_SCHEDULER_NAME` option and follows the same conditional
shape:
```ts
...(env.KUBERNETES_RUN_POD_PRIORITY_CLASS_NAME
? { priorityClassName: env.KUBERNETES_RUN_POD_PRIORITY_CLASS_NAME }
: {}),
```
## Verification
`typecheck --filter supervisor`, `format` and `lint` clean. No changeset
or `.server-changes/` note: off by default, no user-visible behaviour
change.
re2-prod-supervisor-cffaa05
re2-test-supervisor-main-cffaa05
|
||
|
Chris Arderne
|
12ec4667cb | feat(webapp): enable development branches for all organizations (#4670) | ||
|
Marcus Nerløe
|
b83cf671de |
fix(core): mint the fallback external trace id per run (#4534)
## What
Runs that carry no external trace context (schedules, task-to-task
triggers) fall back to a trace id generated once in the [`TracingSDK`
constructor](https://github.com/triggerdotdev/trigger.dev/blob/main/packages/core/src/v3/otel/tracingSDK.ts#L165).
With `experimental_processKeepAlive` the SDK outlives the run, so every
run on a warm process is exported to the external OTLP endpoint under
that one id.
Across our production traces, 80.3% contained spans from more than one
run, worst case 25. Per-run cost and latency attribution is unusable as
a result. This is the same warm-start hazard
|
||
|
claude[bot]
|
d7056a9c67 |
chore(webapp): reword the no-billing-limit banner copy (#4656)
<!-- ccr-slack-attribution --> _Requested by **Matt Aitken** · [Slack thread](https://triggerdotdev.slack.com/archives/C0BKB98B84W/p1787045331358929)_ Copy-only reword of the banner shown to org admins who have not set a billing limit yet. **Before** — the banner read "Protect your organization from unexpected usage spikes." with a button labelled "Configure billing limit". **After** — it reads "Add a billing limit to your account to prevent overspending" with a button labelled "Billing limit settings". The new wording names the action up front and matches the destination it sends you to, so the banner reads as a settings link rather than a one-off setup step. ## ✅ Checklist - [x] I have followed every step in the [contributing guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md) - [x] The PR title follows the convention. - [x] I ran and tested the code works --- ## Testing Formatting and linting pass (`oxfmt --check`, `oxlint`). No tests or snapshots assert this copy. The change is two string literals in one component, with no behaviour attached. --- ## Changelog Reworded the billing-limit banner for organizations without a limit configured, and relabelled its button to "Billing limit settings". --- ## How Both strings live in `NoLimitConfiguredBanner` in `apps/webapp/app/components/billing/OrgBanner.tsx`: the heading is the `canManageBillingLimits` branch of the banner's children, and the label is the `<span>` inside the `LinkButton`. Only those two literals changed. The button still points at `v3BillingLimitsPath(organization)` (`/orgs/{slug}/settings/billing-limits`), so routing, permissions and the non-admin variant of the message are untouched. Co-authored-by: Claude <noreply@anthropic.com> |
||
|
nicktrn
|
e91fb746f7 |
feat(supervisor): configurable security context for run pods
Adds KUBERNETES_RUNNER_SECURITY_CONTEXT (off | baseline | restricted), selecting how constrained the run container is. baseline drops the capability bounding set and blocks privilege escalation. restricted additionally pins the container to a non-root uid, chosen by runtime so bun images get their own. Default is off, so this is inert on merge.re2-test-supervisor-runtime-uid |
||
|
Matt Aitken
|
444c2215ca |
fix(run-engine): stop requeued runs with a lapsed ttl being orphaned in the queue (#4669)
## Summary
A run triggered with a `ttl` could get permanently stuck showing as
queued. If the run started executing and was then requeued after a
failure (a stalled heartbeat, a worker dying mid-run) once its TTL had
already elapsed, the next dequeue pass silently dropped it from every
queue structure. The run stayed QUEUED in the database forever, and
nothing (dequeue, the TTL consumer, queue repair) could ever see it
again.
## Root cause
Enqueue registers a TTL entry for the TTL consumer, and the first
dequeue removes it ("the run is executing, not expired"). A nack rewrote
the message preserving the original `ttlExpiresAt` without
re-registering that entry. The next dequeue pass then took the
expired-TTL branch: remove the run from the queue sorted sets and leave
the message for the TTL consumer to finalize. But the consumer's entry
was gone, so nothing ever finalized the run.
The fix has two halves:
- `nackMessage` drops `ttlExpiresAt` from the rewritten message. TTL
only applies to runs that have never been dequeued (the same contract as
`includeTtl` on re-enqueues), so a requeued run stays dequeuable and is
never expired by its original deadline.
- The dequeue expired-TTL branches now (re-)register the TTL entry
instead of assuming it exists, so any message still carrying a lapsed
`ttlExpiresAt` with no TTL entry (including ones written before this
fix) finalizes as EXPIRED instead of orphaning.
## Verification
New engine test suite `ttlNackRequeue.test.ts` (testcontainers, real
Redis and Postgres). All four tests fail before the fix and pass after:
- a heartbeat-stalled EXECUTING run with a lapsed TTL is requeued and
dequeued again instead of orphaned (the full production failure chain)
- requeue-after-failure strips `ttlExpiresAt` so later dequeues do not
treat the run as expired
- a lapsed-TTL message whose TTL entry is missing is re-registered by
dequeue and finalized as EXPIRED, for both plain and concurrency-key
queues
Also ran the existing ttl, heartbeats, dequeuing and attemptFailures
engine suites plus the full run-queue suite (149 tests) against the
change.
|
||
|
nicktrn
|
2496a8a863 |
feat(supervisor): optional image registry rewrite for run pods
Adds two optional env vars that rewrite the registry host of run pod images at pod creation, so a supervisor can pull from a registry in its own region. Off by default and inert unless both are set. Exact host-prefix matching, so look-alike hosts pass through untouched.re2-prod-supervisor-2026-08-18 re2-test-supervisor-2026-08-18 re2-test-supervisor-main-2496a8a8 |
||
|
Chris Arderne
|
b4313c8199 | feat: logs search v2 (#4615) | ||
|
nicktrn
|
158f6957e4 |
feat(supervisor): make the runner seccomp profile configurable
Replaces the hardcoded runner seccomp profile path with KUBERNETES_RUNNER_SECCOMP_PROFILE_PATH, and the node-24-only condition with KUBERNETES_RUNNER_SECCOMP_PROFILE_RUNTIMES (none | node-24-plus | all). Both defaults reproduce current behaviour, so this is inert on merge. Widening the scope or turning attachment off becomes a config change rather than a deploy. |
||
|
Chris Arderne
|
74db5a3f58 | docs: document cron schedule windows (#4657) | ||
|
Saadi Myftija
|
7e677008ed |
feat(supervisor): per-org placement overrides for run pods (#4655)
The supervisor now supports routing an organization's runs to specific
nodes. `KUBERNETES_ORG_PLACEMENT_OVERRIDES` takes JSON keyed by the
internal org ID, adding node selector entries and tolerations to that
org's run pods, e.g. to route an org onto a dedicated, tainted node
pool:
```json
{"<orgId>": {"nodeSelector": {"pool": "dedicated"}, "tolerations": "dedicated=runs:NoSchedule"}}
```
The node selector merges over the defaults (the override wins on key
collision, with a warning logged). Tolerations append to the existing
runner and scheduled-run sets. Overrides are validated at startup
similar to `KUBERNETES_RUNNER_TOLERATIONS`.
Exposed in the Helm chart as
`supervisor.config.kubernetes.orgPlacementOverrides`, where tolerations
can also be given as a list.
|
||
|
Chris Arderne
|
53ca44dd2d | chore: cache and clean up Knip analysis (#4658) | ||
|
Katia Bulatova
|
e768d0a724 |
feat(webapp): run the dashboard agent through AWS Bedrock behind an env switch (#4609)
## What & why The dashboard agent can now run its model calls through AWS Bedrock instead of the direct Anthropic API, chosen by a single env switch. It's **off by default** (`DASHBOARD_AGENT_MODEL_PROVIDER` unset ⇒ `anthropic`), so merging changes nothing at runtime — the Bedrock path is a dormant branch until an operator sets the switch and AWS config. The default Anthropic path is byte-for-byte unchanged. This also carries a related tenant-isolation hardening for the agent's delegated token (kept together deliberately — both land the agent on Bedrock for HIPAA readiness). Refs: TRI-13251, TRI-11032. ## What's inside **Provider seam** — `internal-packages/dashboard-agent/src/model-provider.ts`: the registry now holds both `anthropic` and `bedrock`; `resolveDashboardAgentModel()` maps the canonical `"anthropic:<id>"` strings the managed prompts carry to the active provider, and the cache-breakpoint helpers emit the active provider's shape — Anthropic `cacheControl` vs Bedrock `cachePoint`. Managed prompt strings stay canonical, so stored prompts don't change meaning. Unmapped model ids throw rather than shipping a guaranteed-404 profile. All agent, watch, compaction and title callsites route through the resolver; the `dashboardAgentModelKey` locals override (test mock injection) is preserved. **Cache telemetry** — `step-cache.ts`: cache token usage is read from the active provider (Anthropic reports it on provider metadata; Bedrock reports the write on metadata and the read via standard usage), so `gen_ai.usage.cache_*` is populated on both. This also fixes a latent ordering bug where step attributes could null-overwrite the prompt-cache read count. **Webapp callsites** — `dashboardAgentHeadStart.server.ts` and the head-start route resolve the model and the cache breakpoint through the same seam, so the warm-up prefix and the following turn share one provider. The head-start firing gate is provider-aware: on Bedrock it gates on `AWS_REGION` and lets the SDK resolve credentials (IAM role / static keys / session token / bearer), so a role-based deploy still warms; on Anthropic it stays `Boolean(ANTHROPIC_API_KEY)`. `app/env.server.ts` gains the optional AWS vars and validates `DASHBOARD_AGENT_MODEL_PROVIDER`. `ANTHROPIC_API_KEY` is untouched and not required on a Bedrock deploy. **Tenant-isolation hardening** — `internal-packages/rbac/src/fallback.ts`: for a **scoped** context, the OSS `authenticateUserActor` now applies the same membership floor as the session path — a delegated user-actor token whose user is not a member of the scoped org/project is denied (403). Unscoped tokens keep their prior behavior (no tenant claim, no lookup). The user lookup falls back replica→primary so replication lag can't spuriously 401 a just-joined member. Members and admins are unaffected. Previously this invariant held only through per-route discipline; this makes it structural. ## Enabling Bedrock (later, ops) - Set `DASHBOARD_AGENT_MODEL_PROVIDER=bedrock` **identically** in both the webapp and the agent task container — the webapp warms the cache prefix and the task reads it, so a split would silently miss the cache. - Set `AWS_REGION` and provide credentials the Bedrock SDK can resolve (IAM role preferred). For v1 this runs **without** an Anthropic API key. Note: with no Anthropic key set, rollback is "turn the agent off", not "unset the switch" (unsetting falls back to the Anthropic provider, which then has no key). - Two things to confirm before rollout: the Sonnet inference-profile id is validated against the SDK's own model-id union but still warrants a live smoke test; and Bedrock prompt caching for Sonnet is a 5-minute window (not Anthropic's 1h), so input-token cost rises when flipped. ## Testing Unit tests cover both provider paths: the provider switch and per-provider cache shapes, a structural regex asserting Bedrock ids are real inference profiles (not an echo of the table), the split-metadata cache telemetry, and real-Postgres RBAC tests — member allowed, scoped non-member denied (org-only and project-only), missing user → 401, admin non-member exempt, unscoped success. `typecheck --filter webapp` and the dashboard-agent + rbac suites pass. |
||
|
Chris Arderne
|
b33197691b | chore: enforce no unused deps or code in ci (#4654) | ||
|
Matt Aitken
|
40c4064f96 |
fix(webapp): show errors on AI tool call and embed spans in the run inspector (#4653)
## Summary When an AI SDK tool call failed inside a run, the span showed up under the "Errors only" filter but the span inspector gave no hint of what went wrong. The exception was recorded on the span all along; the `ai.toolCall` and `ai.embed` inspector views just never rendered span events. Failed tool call and embedding spans now show the standard error block (message plus stack trace) below the Input section. ## Root cause Generic spans render exception span events via the `SpanEvents` component, but the AI-specific span entities replace the whole panel with their own layout and dropped the events entirely. The span's events are now passed into `AIToolCallSpanDetails` and `AIEmbedSpanDetails` and rendered with the same `SpanEvents` component the generic view uses. Errored generation spans (`ai.generateText` and friends) use a tabbed view and still don't surface errors; that needs its own design pass and is left for a follow-up. |
||
|
Wes Mason
|
a55f7cdf4d |
fix(run-engine): stop a '*' concurrency key stranding its whole base queue (#4628)
## The bug
A concurrency key is an unrestricted client string
(`ConcurrencyKeySchema` is `z.union([z.string(),
z.number()]).transform(String)`), and `concurrencyKeySection` does no
escaping, so `*` reaches the queue raw. `queueKey` then renders it as
`...:queue:<q>:ck:*`, which is byte-identical to the wildcard member the
CK scripts keep in the master queue to mean "this base queue has
concurrency-key work".
Every CK script ends with the same pair:
```lua
-- Rebalance master queue with ck:* member
redis.call('ZADD', masterQueueKey, earliestIdx[2], ckWildcardName)
-- Remove old-format entry from master queue (transition cleanup)
redis.call('ZREM', masterQueueKey, queueName)
```
`ckWildcardName` is `toCkWildcard(message.queue)`, and for a `*`-keyed
run that returns the identical string, so the cleanup on the second line
deletes what the rebalance on the first line just wrote.
The master queue then has no entry for that base queue, while `ckIndex`
and the variant queues still hold the work. **Every concurrency key on
the queue stops being dequeued**, not just the `*` one. It is silent,
and it only recovers if some later write happens to re-add the member.
Reproduced before the fix:
```
master queue AFTER normal ck enqueue: ["{org:...}:queue:task/my-task:ck:*"]
master queue AFTER ck='*' enqueue: []
ckIndex members (work still queued): [":ck:user-1", ":ck:*"]
dequeued: []
```
Blast radius is bounded to the environment that triggers it, so it is
self-inflicted rather than cross-tenant, but a single trigger stalls the
queue for everything on it.
## The fix
Guard the cleanup so it never removes the wildcard member:
```lua
if queueName ~= ckWildcardName then
redis.call('ZREM', masterQueueKey, queueName)
end
```
Applied to all 10 CK scripts (4 enqueue, 6 ack/nack/dead-letter). No
key-format change and no migration: a queue already stranded in Redis is
repaired by its next write.
I considered rejecting `*` at the API boundary instead and rejected it.
Existing Redis state and `TaskRun.concurrencyKey` rows already hold raw
`:`-bearing and `*` keys, so changing key construction would orphan
in-flight messages and split concurrency accounting mid-deploy. Boundary
validation would still be reasonable as belt-and-braces later, but the
Lua guard alone fixes it including for state already out there.
## Testing
`ckWildcardKey.test.ts` covers the enqueue, ack and nack paths. All
three pass with the guard and **all three fail without it**, verified by
reverting. Full `src/run-queue/` suite is green (166 tests).
## Note for #4367
The virtual-time branch adds three more CK scripts with the same pattern
(`enqueueMessageCkVtimeTracked`, `enqueueMessageWithTtlCkVtimeTracked`,
`nackMessageCkVtimeTracked`). They do not exist on main so they are not
in this PR; the same guard needs applying there, and I will do that on
that branch.
|
||
|
Chris Arderne
|
99f0787148 | feat(cli,webapp): default new projects to node-24 (#4649) | ||
|
Chris Arderne
|
7d9f1a3268 | docs: document additional environment API keys (#4406) docs-release-2026-08-17 | ||
|
nicktrn
|
6e7710282c |
ci: make the lefthook pre-push hook actually install (#4642)
## Summary `lefthook.yml` has been in the repo since #4147, but nothing installs lefthook and nothing runs `lefthook install`, so the pre-push hook it describes has never fired for anyone. #3977 had removed the `lefthook` devDependency a week before #4147 landed, and #4147 only added the config file. This supplies the missing half: ```diff + "prepare": "lefthook install", + "lefthook": "^2.1.10", "onlyBuiltDependencies": [ + "lefthook", ``` With those in place, `pnpm install` wires the hook up on clone, and the format and lint checks actually run before a push instead of first failing in CI. Also here: the pre-push jobs run in parallel rather than in sequence, and `CONTRIBUTING.md` documents the hook, including how to skip it and the fact that GitButler only runs hooks when "Run hooks" is enabled in its settings. `lefthook@2.1.10` is the current release. |
||
|
claude[bot]
|
3d0b46fee5 |
chore: vouch gtremper (#4648)
<!-- ccr-slack-attribution --> _Requested by **Matt Aitken** · [Slack thread](https://triggerdotdev.slack.com/archives/C045W9WM3E1/p1786948064703889?thread_ts=1786948064.703889&cid=C045W9WM3E1)_ Adds `gtremper` to `.github/VOUCHED.td`. Before: PRs from `gtremper` are auto-closed by the vouch check. After: `gtremper` is vouched, so their PRs stay open and run CI. Done as a direct file edit rather than the issue-comment flow because there is no open Vouch Request issue for this user, matching the precedent in #3804. Co-authored-by: Claude <noreply@anthropic.com> |
||
|
nicktrn
|
d62dd0dc30 |
chore(core): drop the unused socket.io dependency (#4640)
## Summary
`packages/core` declared `socket.io`, the server package, but never
imported it. Its only Socket.IO usage is the client:
```
packages/core/src/v3/zodSocket.ts
packages/core/src/v3/runEngineWorker/supervisor/session.ts
import { io } from "socket.io-client";
```
The only occurrence of `socket.io` outside those client imports was the
`package.json` line itself. Since `@trigger.dev/core` is published, that
line meant every consumer installed a server package nothing in the tree
imports.
`socket.io-client` is untouched. `apps/webapp` and `apps/supervisor`
keep their own `socket.io` dependencies, so the server side is
unaffected.
Found with `pnpm run knip:deps`, which the repo already ships.
`pnpm run typecheck` passes across all 57 workspaces, and
`@trigger.dev/core` builds clean.
Stacked on #4639.
|
||
|
nicktrn
|
362479d7b2 |
chore(deps): raise the find-my-way floor (#4639)
## Summary `find-my-way` was resolving `9.3.0` even though its only parent, `fastify@5.8.5`, declares `^9.0.0` and so already permitted a newer release. The lockfile had not re-resolved since. This adds a floor so it lands on a current 9.x: ```json "find-my-way@>=9 <9.7.0": "^9.7.0" ``` It resolves to `9.7.0`. Nothing outside the 9.x line is touched, and no parent is asked to accept anything its declared range did not already allow. The whole path is development only: `find-my-way` arrives through `fastify`, which arrives through `evalite`, a devDependency of `apps/webapp` used by the `eval:dev` harness. Stacked on #4638. |
||
|
nicktrn
|
b4f7800ff1 |
chore(deps): cover the second ip-address parent (#4638)
## Summary The existing `ip-address` override is scoped to a single parent, `@jsonhero/json-infer-types>ip-address`. A second parent reaches `ip-address` independently: `express-rate-limit@8.6.0`, which is itself pinned by our `@modelcontextprotocol/sdk@>=1.26.0>express-rate-limit` override. That path was resolving `10.2.0` while the scoped path resolved `10.5.0`, so the tree carried two copies. This adds a matching scoped override for the second parent: ```json "express-rate-limit>ip-address": "^10.3.1" ``` `express-rate-limit` declares `^10.2.0`, so this asks nothing of it that its own range did not already allow. The tree now resolves a single `ip-address@10.5.0`. The existing `@jsonhero/json-infer-types` override stays: that package declares `ip-address: ^8.1.0`, so removing it brings an 8.x copy back. Stacked on #4637. |
||
|
nicktrn
|
f3c46f140e |
chore(deps): raise nanoid floors, drop unused declarations (#4637)
## Summary `nanoid` was pinned at exactly `3.3.8` in five manifests. Two of those five never imported it: in `internal-packages/schedule-engine` and `internal-packages/webhook-engine` the only occurrence of the string `nanoid` in the entire package was the `package.json` line itself. Both are removed rather than bumped. The three that genuinely use it move to `3.3.18`, a version already present in the tree via `postcss`, so this pulls in nothing new. | Package | Uses it | Change | | --- | --- | --- | | `internal-packages/schedule-engine` | no | removed | | `internal-packages/webhook-engine` | no | removed | | `apps/webapp` | yes | `3.3.8` to `3.3.18` | | `packages/core` | yes | `3.3.8` to `3.3.18` | | `internal-packages/run-engine` | yes | `3.3.8` to `3.3.18` | | `packages/redis-worker` | yes | `^5.0.7` to `^5.1.16` | `redis-worker` is on the 5.x line and is included because its declared range already permitted a newer release; the lockfile had simply not re-resolved, leaving it on `5.1.2`. The unused declarations were found with `pnpm run knip:deps`, which the repo already ships. `pnpm run typecheck` passes across all 57 workspaces. |
||
|
nicktrn
|
148615b526 |
chore(webapp,supervisor,core): move socket.io to 4.8.3 (#4635)
## Summary `socket.io` was pinned at exactly `4.7.4` in three manifests (`apps/webapp`, `apps/supervisor`, `packages/core`). That pin capped `engine.io` at 6.5.4, because 4.7.4 declares `engine.io: ~6.5.2`. Moving all three pins to `4.8.3` lifts that cap: 4.8.3 declares `engine.io: ~6.6.0`. The webapp's direct `engine.io` devDependency moves from `^6.5.4` to `^6.6.7` to match. These are direct dependencies, so they are bumped in place rather than forced with an override. ## Result The tree previously carried two `engine.io` copies. It now carries one: ``` engine.io@6.6.8 └─┬ socket.io@4.8.3 ├── @trigger.dev/core (dependencies) ├─┬ react-email │ └── emails (devDependencies) ├── supervisor (dependencies) └── webapp (dependencies) ``` `react-email` was already resolving `socket.io@4.8.3` in this same tree, so that combination was already running here before this change. ## Servers move, clients do not This bumps `socket.io` (the server) only. `socket.io-client` stays at `4.7.5` in `packages/core` and `packages/cli-v3`, deliberately: the fix is server-side, and clients ship inside user deployments, so leaving them alone keeps the blast radius small. That means a 4.8.3 server will be talking to 4.7.5 clients indefinitely, which is worth being explicit about. That pairing is safe because neither wire protocol changed. Both versions report the same protocol numbers: | | 4.7.4 | 4.8.3 | | --- | --- | --- | | Socket.IO protocol (`socket.io-parser`) | 5 | 5 | | Engine.IO protocol (`engine.io-parser`) | 4 | 4 | The version bump moves `socket.io-parser` 4.2.6 to 4.2.7 and `engine.io` 6.5.4 to 6.6.8, but the protocol constants each exports are unchanged. The 4.8.0 changes are additive on the client (custom transport implementations, a `tryAllTransports` option) and bug fixes on the server. Verified rather than assumed, with a cross-version matrix covering both transports and both directions: ``` PASS server 4.8.3 <- client 4.7.5 websocket / polling PASS server 4.8.3 <- client 4.8.3 websocket / polling PASS server 4.7.4 <- client 4.7.5 websocket / polling PASS server 4.7.4 <- client 4.8.3 websocket / polling ``` Each case exercised connect, a server-initiated emit, `emitWithAck`, room join, room broadcast, and a binary payload. Compatibility holds in both directions, so there is no upgrade-ordering requirement between server and client. `pnpm run typecheck` passes across all 57 workspaces. Stacked on #4634. |
||
|
nicktrn
|
869156e3b8 |
chore(deps): raise the axios floor (#4634)
## Summary `axios` was resolving to 1.16.1 through `@slack/web-api`, which declares `^1.16.0`. The lockfile had simply not re-resolved since, so the tree sat on an older 1.x release than the range allows. This adds a scoped override so the 1.x line picks up a current release: ```json "axios@>=1.15.2 <1.18.0": "^1.18.0" ``` It resolves to 1.19.0. No parent bump is needed, since `^1.16.0` already permits it, and `@slack/web-api` is the only consumer. Stacked on #4633 so the two lockfile changes do not collide. |
||
|
nicktrn
|
a34d23973e |
chore(webapp): replace npm-run-all with an explicit build chain (#4633)
## Summary `npm-run-all` has had no release since 4.1.5 in 2018, and pnpm now covers the one thing we used it for. The webapp's `build` script was its only consumer anywhere in the repo, so the dependency goes away entirely. `run-s build:**` becomes an explicit chain: ``` pnpm run build:remix && pnpm run build:server && pnpm run build:otlpworker && pnpm run build:sentry && pnpm run upload:sourcemaps ``` ## Why this shape I compared both forms side by side against the real `run-s` before swapping: | Behaviour | `run-s build:**` | explicit chain | | --- | --- | --- | | Scripts selected | remix, server, otlpworker, sentry | identical | | Order | declaration order | identical | | `upload:sourcemaps` matched by the glob | no | no | | Second script fails | aborts, third never runs | identical | | Exit code on failure | `1` | `1` | `pnpm run --sequential "/^build:/"` was the closer-looking option, but it keeps running scripts after one fails, so it is not a faithful replacement. The one thing given up is that `build:**` automatically picked up any new `build:*` script, where the chain has to be edited. With four entries that felt like the better trade. `pnpm run build --filter webapp` passes end to end locally, all five steps in order. |
||
|
nicktrn
|
512a619ea8 |
fix(webapp): back to app returns to the current org (#4632)
## Summary Following a link straight into an organization's settings (for example the usage limit link in a billing email) and then clicking "Back to app" took you to `/`, which resolves to whichever organization you last had selected, not the one whose settings you were looking at. The button now links to the organization in the URL, so you land back in the org you came from. The org index route already redirects to the best project in that org, so the destination is unchanged apart from being the right org. Account settings still links to `/`, since that page is not org scoped and has no org to return to. |
||
|
nicktrn
|
7ba81e983d |
chore(deps): raise stale transitive dependency floors (#4629)
## Summary A number of `pnpm.overrides` entries had drifted behind the releases they were written against. An override fixes the resolved version outright, so in every one of these cases the tree was pinned to the floor value rather than picking up later releases in the same line. This raises each floor to a current release, and widens the selectors that were scoped to an exact upper bound so they keep matching. | Override | Before | After | | --- | --- | --- | | `body-parser` (under `express@^4`) | `1.20.3` | `^1.20.6` | | `tar` | `7.5.19` | `7.5.21` | | `hono` | `4.12.25` | `4.12.34` | | `undici` (6.x) | `6.27.0` | `6.28.0` | | `undici` (7.x) | `7.28.0` | `7.29.0` | | `js-yaml` (3.x) | `3.14.2` | `3.15.1` | | `js-yaml` (4.x) | `4.1.1` | `4.3.1` | | `dompurify` | `^3.4.1` | `^3.4.13` | | `vite` | `^6.4.2` | `^6.4.3` | | `protobufjs` | `^7.5.6` | `^7.6.5` | | `socket.io-parser` | `^4.2.6` | `^4.2.7` | | `postcss` | `^8.5.10` | `^8.5.23` | | `fast-uri` | `^3.1.2` | `^3.1.5` | | `brace-expansion` (1.x) | `1.1.13` | `1.1.18` | | `brace-expansion` (2.x) | `2.0.3` | `2.1.4` | | `brace-expansion` (5.x) | `5.0.6` | `5.0.9` | | `ip-address` (under `@jsonhero/json-infer-types`) | `^10.2.0` | `^10.3.1` | Every parent's declared range still accepts the new resolution, so nothing is forced outside its stated bounds by this change. Two of these changed a default rather than just moving version. `js-yaml` 4.2.0 stopped resolving underscore-separated scalars such as `1_000` as numbers, which is the YAML 1.2 behaviour, and there are none in any YAML in this repo. `brace-expansion` 2.1.x now caps expansion size by default, well above anything a real glob produces, and `minimatch` calls it with no options. Neither is reachable from how we use them. `undici@5.29.0` and `vite@4.4.9` are left alone: their parents cap below the newer lines, so moving either would mean taking the parent across a major. Verified with a clean install, and `pnpm run typecheck` passes. |
||
|
Eric Allam
|
c0b84595a3 |
feat(webapp): hosted webhook ingress, delivery pipeline, and dashboard (#4344)
## Summary The server half of hosted webhooks: the public ingress endpoint, signature verification, the delivery pipeline (Postgres partitioned storage + ClickHouse for ordering), the in-app partition manager, the HTTP API, and the dashboard (Deliveries, Endpoints, and the in-app test console). The public SDK and docs half is #4537. That PR carries the user-facing API (`webhook()`, `chat.event` / `chat.channels`, the `@trigger.dev/slack` connector) and builds on the shared `@trigger.dev/core` schemas that ship here. ## Shipping behind a flag A `WEBHOOK_ENABLED` env var (default off) gates the public ingress route and the engine worker plus partition cron, so merging and deploying this changes nothing in production until it is flipped on per environment. The dashboard is separately gated per org by the `hasWebhooksAccess` feature flag. ## Note on packages This PR includes the `@trigger.dev/core` schema additions the server compiles against, but carries no changeset. Core is not consumed independently of the SDK, so it is released together with the SDK via #4537. Keeping its changeset off `main` means no release cut from `main` publishes it early. |
||
|
Eric Allam
|
b98dd79fe4 |
feat(webapp,run-store,database): env-configurable transaction resilience (maxWait + tx-start retry) (#4623)
## What
Makes two transaction-resilience behaviors real and env-var
configurable, defaults set to the good values, so we can tune during and
after the Aug 15 database patch window without a redeploy:
- **maxWait 2s → 10s** (TRI-12982): how long Prisma waits to borrow a
connection before it can `BEGIN`. A restart freeze holds the pool full,
and the only thing that errored was transaction starts giving up at 2s.
- **Retry transaction-start P2028-at-acquisition** (TRI-12984): when
Prisma can't borrow a connection within `maxWait` it raises P2028
(`Unable to start a transaction in the given time`) and **no SQL ran**,
so retrying is safe. Scoped narrowly: only that error (never P2024
pool-exhaustion), 2 attempts, jittered backoff, and a token-bucket
budget so a mass freeze can't amplify into a retry storm.
## Env vars (`DATABASE_*` convention)
Generic defaults:
| var | default |
|---|---|
| `DATABASE_TRANSACTION_MAX_WAIT_MS` | `10000` |
| `DATABASE_TRANSACTION_START_RETRY_ENABLED` | `true` (kill switch) |
| `DATABASE_TRANSACTION_START_RETRY_MAX_ATTEMPTS` | `2` |
| `DATABASE_TRANSACTION_START_RETRY_BACKOFF_MIN_MS` | `50` |
| `DATABASE_TRANSACTION_START_RETRY_BACKOFF_MAX_MS` | `250` |
| `DATABASE_TRANSACTION_START_RETRY_BUDGET_PER_SEC` | `50` |
| `DATABASE_TRANSACTION_START_RETRY_BUDGET_BURST` | `100` |
Per-writer-pool overrides, each falling back to the generic when unset
(same pattern as the per-client pool/connect-timeout work):
`RUN_OPS_DATABASE_TRANSACTION_*` and
`RUN_OPS_LEGACY_DATABASE_TRANSACTION_*` (all 7 knobs each). Transactions
only open on writer pools, so those are the only pools with their own
knobs. Each pool gets its **own** token bucket, so a storm on one pool
can't drain another's retry budget.
## Design
- The retry primitives live in `internal-packages/database` and never
read `process.env` (IoC): a P2028-at-acquisition classifier, a
`TokenBucketRetryBudget`, and `withTransactionStartRetry`, folded into
the `$transaction` helper via a new `startRetry` option. Config is
resolved at the app boundary and threaded in.
- The `$transaction` helper is the chokepoint (wraps the whole
transaction), not the per-statement `$allOperations` extension.
- The run engine's writes go through `PostgresRunStore`'s own
`.$transaction(...)`, not the webapp helper, so both the helper and the
two `PostgresRunStore` sites apply maxWait + retry (sharing the per-pool
config). Builds on the `options?: { timeout, maxWait }` seam added in
#4514.
- Webapp `$transaction` call sites get the default `maxWait` + retry
injected at one merge point, so no call site needed editing.
## Evidence
- Unit red/green in `internal-packages/database`: reverting the helper
wiring turned the acquisition-retry test red (`Unable to start a
transaction in the given time`), re-applying it green. Full package
suite 25/25. Covers: classifier (P2028-acq yes, P2024 no, in-tx P2028
no), retry (retry-then-succeed, no-retry P2024, stop at maxAttempts,
disabled, budget-exhausted, jitter bounds), token bucket, and
`$transaction` wiring.
- Typecheck clean: webapp, run-store, run-engine.
- Full-stack run: bounded queue-ay pass (15 projects, real dev runs
through the run-engine `PostgresRunStore` transaction path). 13 pass;
the 2 failures are one documented known-failure and one
stale-worker-state flake that passes 2/2 with this change active on a
fresh app.
- Boots cleanly with per-pool overrides set.
## Configuration & rollout
Ship **inert** first (zero behavior change), then flip to the good
values **live via env** — no redeploy needed for either.
### Inert — behaves exactly as today
```
DATABASE_TRANSACTION_MAX_WAIT_MS=2000 # Prisma's built-in default (change defaults to 10000)
DATABASE_TRANSACTION_START_RETRY_ENABLED=false # disable the new retry entirely
```
`maxWait=2000` is what every path used before (Prisma's default; the
run-store sites and the helper passed no maxWait). `retry=false`
short-circuits `withTransactionStartRetry` to a single run and makes the
serialization-retry exclusion a no-op. Verified on the pooler-freeze
rig: identical fail-fast P2028 at ~2003ms with zero retries —
byte-for-byte current behavior, across all pools.
### Production ("good") — the baked defaults
Rely on defaults (nothing to set) or set explicitly:
```
DATABASE_TRANSACTION_MAX_WAIT_MS=10000
DATABASE_TRANSACTION_START_RETRY_ENABLED=true
DATABASE_TRANSACTION_START_RETRY_MAX_ATTEMPTS=3 # 3 attempts (2 retries); ~30s acquisition tolerance covers a ~20-25s freeze
DATABASE_TRANSACTION_START_RETRY_BACKOFF_MIN_MS=50
DATABASE_TRANSACTION_START_RETRY_BACKOFF_MAX_MS=250
DATABASE_TRANSACTION_START_RETRY_BUDGET_PER_SEC=50
DATABASE_TRANSACTION_START_RETRY_BUDGET_BURST=100
```
Per-pool overrides `RUN_OPS_DATABASE_TRANSACTION_*` and
`RUN_OPS_LEGACY_DATABASE_TRANSACTION_*` (all seven knobs each) are
optional and fall back to the generic set — not needed for v1; the
generic set covers the control-plane, run-ops, and run-ops-legacy writer
pools. Readers open no transactions and take nothing.
**Guardrail:** the retry only engages when a pool's `pool_timeout` >
`maxWait`. Prod is fine (`DATABASE_POOL_TIMEOUT=60` >> 10). Do not set
any writer pool's `pool_timeout` at or under `maxWait`, or saturation
failures flip from retryable P2028 to non-retryable P2024 and the retry
silently stops helping.
### Rollback
Env flip (set inert) or revert. Retry only fires where no SQL ran, and
the per-pool token bucket caps a storm. No migration.
refs TRI-13295, TRI-12982, TRI-12984
|
||
|
claude[bot]
|
69f396fbef |
fix(webapp): keep paused environments paused when concurrency limits are pushed (#4625)
<!-- ccr-slack-attribution --> _Requested by **Matt Aitken** · [Slack thread](https://triggerdotdev.slack.com/archives/C045W9WM3E1/p1786732623292829?thread_ts=1786732623.292829&cid=C045W9WM3E1)_ **Before:** you pause an environment, then a deploy lands (or a background worker is created, or an admin changes the concurrency/burst-factor). The environment starts picking up runs again even though the dashboard still shows it as paused. **After:** a paused environment stays paused until it is resumed, no matter what else pushes its concurrency limit. Pausing an environment sets `paused` in the database and writes a `0` env concurrency limit into the run queue — the `0` is the only thing that actually stops dequeueing. Any caller that pushed the limit without an explicit value (`finalizeDeployment`, `createBackgroundWorker`, the two admin environment routes) rewrote the real limit and silently un-paused the environment. ## ✅ Checklist - [x] I have followed every step in the [contributing guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md) - [x] The PR title follows the convention. - [x] I ran and tested the code works --- ## Testing `apps/webapp/test/pauseEnvironment.server.test.ts` gains two `containerTest` cases that wire a real `RunEngine` (real Redis) in place of the stubbed app singleton and assert the actual run-queue env limit: - pause a PRODUCTION env → limit is `0` → run the real `FinalizeDeploymentService` → limit is still `0`, plus a control on a running env in the same test proving that deploy path really does push the limit (so the `0` can't just mean "nothing happened"). - pause → resume → the real limit is restored, so the clamp can't regress resuming. Both cases fail on `main` (`expected 17 to be +0` and `expected +0 to be 17`) and pass with this change. `pnpm run typecheck --filter webapp` is clean. --- ## Changelog Fix paused environments starting to run work again after a deploy. --- ## How The clamp lives in the shared `updateEnvConcurrencyLimits` helper in `apps/webapp/app/v3/runQueue.server.ts`, so every present and future caller is covered: when no explicit limit is passed and the environment is paused, `0` is written instead of the stored maximum. An explicitly-passed limit still wins, which is what pausing itself relies on. The resume path now passes the post-update environment state (its in-memory copy was read before the un-pause and would otherwise be clamped back to `0`), and the helper no longer mutates the caller's environment object — that aliasing made a pause followed by a resume on the same object write `0` twice. The existing `!paused` guards in `allocateConcurrency` and the queue-level guard in `createBackgroundWorker` are left in place as defence in depth, and queue-level `TaskQueue.paused` behaviour is untouched. --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Eric Allam
|
dc8f90e66e |
fix(run-engine,webapp): resolve dequeue worker version fresh per task (#4622)
## Summary After a deployment promotion or rollback, newly triggered runs could keep dispatching onto the previously deployed version for up to 30 seconds. Runs now resolve the current version fresh on every dequeue, so a promotion or rollback takes effect immediately. ## Fix The dequeue path resolved the worker version through a 30s in-process cache that nothing invalidated on promotion, and it loaded the worker's entire task and queue set only to keep the single row matching the run. Both go away: the resolve now fetches just the matched task and queue by unique index and reads them fresh, so there is no cache left to serve a stale version. ``` - cache.get(env:current) # 30s TTL, never invalidated -> stale - worker + ALL tasks + ALL queues + worker + one task WHERE slug=... + one queue WHERE id/name=... # fresh ``` A kill-switch env var (`RUN_OPS_WORKER_VERSION_FRESH_READ_ENABLED`, default on) falls back to the old cached path without a code deploy. Verified end-to-end on an isolated stack: a run triggered after a mid-stream promotion now dequeues onto the new version, with the previous stale behavior reproduced first. |
||
|
Eric Allam
|
dd78dd92ee |
perf(webapp): select only needed columns in dev current-worker lookup (#4621)
## Summary When resolving the current worker for a development environment, `findCurrentWorkerFromEnvironment` loaded the entire `BackgroundWorker` row, including the large `metadata` JSON, even though it only ever returns a handful of small fields. It is a frequently-run query, so the wasted payload adds up: every call pulled data it immediately threw away. ## Fix Add a `select` to the development-environment lookup listing exactly the fields the function returns (`id`, `friendlyId`, `version`, `sdkVersion`, `cliVersion`, `supportsLazyAttempts`, `engine`). The query plan is unchanged, still a single-row indexed lookup; only the row width shrinks. No behavior change: the dropped columns were never read. |
||
|
Eric Allam
|
8dc8e1b58b |
perf(run-engine,webapp): narrow the control-plane worker-version read to the columns dequeue uses (#4619)
## Summary
The worker-version resolve path fetched every column of every
`BackgroundWorkerTask` for a worker (`include: { tasks: true }`), plus
full `WorkerDeployment` and `TaskQueue` rows, just to match one task at
dequeue. That pulls large JSON columns none of this path reads (task
`payloadSchema`/`config`/`queueConfig`/`description`, deployment
`externalBuildData`/`buildServerMetadata`/`errorData`/`git`, queue
`rateLimit`), so each resolve transfers and deserializes far more than
it uses.
## Fix
Replace the includes with explicit `select`s of only the columns dequeue
reads, in both the passthrough resolver and the app resolver:
- task: `id`, `slug`, `machineConfig`, `retryConfig`,
`maxDurationInSeconds`
- deployment: `id`, `friendlyId`, `imageReference`, `imagePlatform`
- queue: `id`, `name` (the queue matcher keys on both)
The shared `ResolvedWorkerVersion` element types narrow to match
(mirrored in the cache), which also shrinks each cached worker-version
entry.
## Impact
The `tasks` read fetches every task of a worker to match one, so its
cost scales with task count and payload-schema size. For a worker with
~70 registered tasks, dropping the unread columns cuts the per-query
transfer roughly:
| Task shape | Before | After | Reduction |
|---|---|---|---|
| Light (no payload schema, small config) | ~28 KB | ~14 KB | ~54% |
| Typical (mixed schemas / config) | ~62 KB | ~14 KB | ~77% |
| Schema-heavy (large `payloadSchema`) | ~200 KB | ~14 KB | ~93% |
The `after` size is roughly fixed because the kept columns are small;
the win grows with how heavy the dropped JSON is. Narrowing `deployment`
(four JSON columns off a single row) and `queues` saves further on top.
No behavior change: pure read-shape narrowing, no flag and no schema
change, so rollback is a plain revert. Verified with a red/green
run-engine test that asserts the resolved task, deployment, and queue
carry only the used columns, plus the queue feature-matrix runs (batch,
retry-policy, machine-preset, plain trigger) that exercise the kept
columns.
|
||
|
Chris Arderne
|
1240d91e43 | perf(clickhouse): add task_events_v2 inserted_at minmax index (#4620) | ||
|
Eric Allam
|
4c21af8669 |
feat(webapp): CI guard for unindexed onDelete cascade FK columns (#4618)
## What A relation with `onDelete: Cascade | SetNull` whose child FK column has no index makes every parent delete fire a cascade that sequentially scans the whole child table. That has shipped three times recently and had to be fixed after the fact (#4554 `ProjectAlert.channelId`, #4555 `EnvironmentVariableValue.valueReferenceId`, #4588 `PersonalAccessToken.userId`). This adds a schema-aware CI guard that catches the next one before it merges. ## How `apps/webapp/scripts/fkCascadeIndexGuard.ts` parses both Prisma schemas (`@trigger.dev/database`, `@internal/run-ops-database`) and flags any `onDelete: Cascade | SetNull` relation whose leading FK scalar is not the leading column of some index (`@@index` / `@@unique` / `@@id` / field-level `@id`/`@unique`) on the child model. A leading FK column lets the cascade's `WHERE fk = $1` use the index instead of a seq scan. It is modeled on the existing `runOpsLegacyGuard` (same `--check` gate, same baseline-regenerate pattern), and it is lighter: it only reads `schema.prisma` as text, so its CI job needs no Prisma client generation and no raised heap. ## Why a baseline, not a hard rule Not every unindexed cascade FK is a live bug. When the parent is only ever soft-deleted, the cascade never fires, so the missing index is harmless. Hard vs soft delete lives in application code (`parent.delete()` vs `parent.update({ deletedAt })`), not in the schema, and a `deletedAt` column proves neither direction. So the guard makes no such judgment: it flags every unindexed cascade FK uniformly and carries a baseline of the 72 currently-accepted cases. Only violations **not** in the baseline fail `--check`. The value is the forcing function: a newly added cascade FK stops CI and makes the author answer "is the parent ever hard-deleted?" Add the index if yes; regenerate the baseline with a reason if no. ## Wiring - `apps/webapp/package.json`: `guard:fk-cascade-index` script (regenerate with no args, gate with `-- --check`). - `.github/workflows/fk-cascade-guard.yml`: the reusable workflow. - `.github/workflows/pr_checks.yml`: runs on webapp-affecting changes, aggregated into `all-checks`. ## Verification - The three already-fixed columns are correctly seen as indexed (absent from the baseline). - `--check` passes on the current schemas (72 baselined, 0 new). - A synthetic new unindexed cascade FK fails with exit 1 and an actionable message. - Adding `@@index([fk])`, or a composite leading with the FK, clears it. No false positives. - `oxfmt` and `oxlint` clean on the new script. ## Rollback Pure tooling addition, no runtime code, no schema or data change. Revert to remove. |
||
|
Eric Allam
|
603c278687 |
docs: clarify when changesets and server-changes files are needed (#4617)
## Summary Clarifies when to add a changeset or a `.server-changes/` file. The friction that keeps coming up is treating these as "I touched a public package or a server app, so I owe a note." They are user-facing release notes that go straight into the changelog customers read, not a catalog of every change. The guidance now leads with the real test: would a user or customer care about this change? Add a note when the change is something they would notice, act on, or want to hear about. Skip it otherwise, even when a public package or server app is touched, for example: - internal-only or admin-only changes, refactors, test-only changes, chores - performance or query tuning with no user-visible behavior change - public packages that are not consumed independently (e.g. `@trigger.dev/redis-worker`), where a version bump means nothing to a user Anyone who wants the exact history reads the commits. Updates every place that encoded the old "touched a package or app, so add a note" rule so they agree: `AGENTS.md`, `.server-changes/README.md`, `CONTRIBUTING.md`, `CHANGESETS.md`, `.claude/rules/server-apps.md`, and `.claude/REVIEW.md` (the last drives automated review flagging, so it stops flagging exactly the changes the new guidance says to skip). Also handles the mixed-PR case where the package change needs no changeset but the server change is user-facing. |
||
|
Eric Allam
|
fe199f7f92 |
perf(webapp): aggregate admin notification interaction counts in the database (#4616)
## Summary The notifications admin list loaded every interaction row for the notifications on the current page just to show three per-notification counters (seen, clicked, dismissed), then counted them in memory. On notifications with many interactions this made the page slow to load and heavy on memory, even though only 20 notifications are shown. ## Fix Compute the counters in a single grouped aggregate in the database instead, returning one row per notification rather than one row per interaction: ```sql SELECT "notificationId", COUNT(*) AS seen, COUNT(*) FILTER (WHERE "webappClickedAt" IS NOT NULL) AS clicked, COUNT(*) FILTER (WHERE "webappDismissedAt" IS NOT NULL OR "cliDismissedAt" IS NOT NULL) AS dismissed FROM "PlatformNotificationInteraction" WHERE "notificationId" IN (...) GROUP BY "notificationId" ``` Behavior is unchanged; notifications with no interactions report zero. |
||
|
Saadi Myftija
|
c4b5e27258 |
feat(cli): build deployment images on prebuilt base images (#4602)
The generated deploy Containerfile now starts from the prebuilt base images published by base-images/ (`triggerdotdev/node` and `triggerdotdev/bun` on DockerHub, pinned by digest) instead of installing system packages during every project's build. Uncustomized projects run no apt at all and their base layers are identical across every project, so worker nodes cache one copy fleet-wide. The build stage uses the -build toolchain variant for uncustomized and package-only projects; projects with image instructions build FROM base so instructions and their downloads run exactly once. ### Notes - User packages install in their own sorted RUN with --allow-downgrades (a pin of a preinstalled package is a downgrade against the prebuilt base), preceded by a dpkg repair whenever instructions came first, since apt-get install refuses to run on state a dpkg -i instruction left broken. - Deployed runtime images inherit newer package versions than today's live-archive installs (the published bases upgrade everything to their snapshot), plus the base images' OCI labels. Runtime env, user, workdir, and entrypoint are unchanged. |
||
|
Matt Aitken
|
949e9cf1ec |
fix(webapp): show the real app version instead of v0.0.0 in organization settings (#4611)
## Summary Since the move from the Remix compiler to Vite ([#4188](https://github.com/triggerdotdev/trigger.dev/pull/4188)), the "App version" on the organization settings page shows `v0.0.0` unless the image was built from a semver release tag (which bakes in `BUILD_APP_VERSION`). Self-hosted builds and any image built from `main` are affected. This restores the real version. ## Root cause The Vite SSR bundle resolves workspace packages to TS source via the `@triggerdotdev/source` condition, so `@trigger.dev/core`'s `VERSION` constant is bundled as its raw `"0.0.0"` placeholder. `scripts/updateVersion.ts` still stamps the real version at build time, but only into the packages' dist output, which the bundle no longer reads. The old Remix compiler bundled the stamped dist, which is why this used to work. The fix is a small Vite plugin that applies the same substitution to the source version modules of `@trigger.dev/core` and `@trigger.dev/sdk` during bundling. Beyond the settings page, this also restores real values in the `trigger-version` request header and the version attributes the bundled packages emit. Verified by building the server bundle and confirming the VERSION constants carry the package versions, with no `"0.0.0"` occurrences left in the build output. |
||
|
Chris Arderne
|
3e7964e7fa |
feat: surface cron windows in webapp, cli, sdk (#4572)
## Summary Adds execution-window product surfaces for both declarative and imperative schedules. - Declarative schedules can set `window` through `schedules.task()`, with support for whole-minute, hour, and percentage values. - Imperative schedules can create, update, clear, and inspect windows through the API and dashboard. - Schedule API responses preserve `nextRun` as the nominal CRON time and expose `nextRunEffectiveAt` as the stable assigned time. - The dashboard displays configured windows alongside assigned upcoming-run times. - Deploy output summarizes declarative schedules and suggests adding a wider window when the default 60-second placement range is used. ## Design Window validation remains authoritative on the server and ensures each window is compatible with the schedule cadence. Omitting a window uses the default 60-second range, while explicit zero-duration windows remain supported. Deployment summaries are derived from the deployment's stored task metadata, so they reflect the declarations associated with that deployment. |
||
|
Chris Arderne
|
d98f64bb00 | fix(webapp): hide misleading root API key creation dates (#4612) | ||
|
nicktrn
|
fa7eea39d8 |
fix(core): stop custom metric exporters breaking the metrics export (#4613)
## Summary Projects that configure their own `metricExporters` or `metricReaders` in `trigger.config.ts` were losing task metrics on nearly every run, and seeing an unexplained `Failed to flush tracingSDK` alongside `OTLPExporterError: Bad Request` in their run logs. Spans and logs kept working, so the runs otherwise looked healthy. ## Root cause and fix Every configured exporter gets its own `PeriodicExportingMetricReader`, and `meterProvider.forceFlush()` fans out across all readers with `Promise.all`, so two collections can land on the same millisecond. `@opentelemetry/host-metrics` divides by the elapsed interval to compute `process.cpu.utilization` ([common.ts](https://github.com/open-telemetry/opentelemetry-js-contrib/blob/main/packages/host-metrics/src/stats/common.ts)), so a zero interval yields `0/0`. `JSON.stringify(NaN)` is `null`, and a collector rejects `"asDouble": null` with a 400 that drops the **entire** request, not just the offending point. `flush()` and `shutdown()` now walk the metric readers one at a time, so collections can no longer share a timestamp. Each reader is isolated, so one failing reader cannot skip the readers behind it, and every failure is logged with the reader that produced it. The first error is still rethrown, so callers see failures exactly as before. As a second layer, non-finite data points are dropped just before our own export, so a metric that divides by zero cannot take the rest of the batch with it. Exporters and readers supplied through `trigger.config.ts` are untouched by that filter and still receive raw data. The trade-off is that configured exporters now flush after the built-in one rather than alongside it, so flush latency is the sum rather than the max. An internal test package's dependency on core was replaced with a local helper, because core now needs that package in `devDependencies` and the two together formed a workspace cycle. ## Verification Tested against a real collector in a container: a batch containing a `NaN` reading is rejected with a 400 without the fix and accepted with it, and a single flush is asserted to collect from one reader at a time. |
||
|
Matt Aitken
|
1114d9d6f9 |
fix(redis-worker): stop fair queue leaking concurrency slots (#4540)
## Summary Fair queue consumers could leak the per-tenant concurrency slots that gate admission. Slots were freed on some paths and skipped on others, and once enough leaked slots accumulated for a tenant, every queue that tenant owned stopped being served until someone cleared the set by hand. This PR frees slots on every path and, more importantly, makes the remaining failure modes self-healing. ## Design The fix applies one rule uniformly: releasing a concurrency slot is best-effort cleanup and must never block the message's primary state transition. Blocking completion re-delivers the message, which duplicates customer work; blocking a retry loses the attempt increment, so the message can circle forever; blocking a reclaim strands the message in flight. A leaked slot is the better failure in every one of those trades because it is the only one that is recoverable. A failed release is therefore logged and the transition proceeds. Leaked slots then heal through two mechanisms: - `reserve` re-admits a message that is already a member of its own concurrency set, since re-admitting it does not increase concurrency. A message whose earlier release failed can no longer be blocked by its own leftover slot. - A reconcile loop periodically removes any set member with no in-flight record (interval configurable via `reconcileIntervalMs`, default 60s). The check-and-remove is atomic, and it is sound because a message is always registered in flight before its slot is reserved, so a member with no in-flight record can only be a leak. This also covers leaks this PR cannot prevent directly, such as a release that resolves the wrong concurrency group from queue metadata. Ordering hardening from earlier revisions stays: slots are released before the in-flight record needed to describe them is discarded, the release Lua scripts write the message back to the queue before removing it from in-flight (Lua does not roll back on error), and dangling in-flight entries with no payload are dropped instead of being rescanned forever. Every guard test was verified to fail without its specific fix, including the duplicate-execution case: completing a message while its slot release fails used to re-deliver and re-execute it. |
||
|
Chris Arderne
|
20a0ac5055 | chore: fix lint warnings (#4605) | ||
|
Saadi Myftija
|
035e71010d |
feat(base-images): immutable per-publish image tags (#4607)
Every publish now also pushes an immutable per-publish tag alongside the mutable one, named after the snapshot date and commit (e.g. `22-bookworm-20260812-45444a7`), so previously published digests stay tag-referenced after republishes. Shipped CLI releases pin those digests, so they must remain resolvable indefinitely. Merging triggers a republish; the fresh tag-protected digests will then be pinned by #4602 before it merges. |
||
|
Eric Allam
|
eefe0a378d |
perf(webapp): bound environment loads in the env layout and batches list (#4606)
## Summary Follow-up to #4595. Dashboard pages under an environment loaded every environment in the project on each page just to resolve the one named in the URL. On projects with many preview branches that meant reading hundreds of (mostly archived) rows on every page load. ## Fix The environment-scoped layout loader now scopes its lookup to the slug in the URL (`where: { slug: envParam }`), resolving the current environment through the `projectId, slug` composite index instead of loading the whole project. Archived branches stay viewable by slug. `BatchListPresenter` is bounded to the current environment, since every batch in that list already belongs to it. Verified on a project seeded with 2,000 archived branch environments: the layout lookup drops from all environments to one, and both a normal environment page and an archived branch page render correctly.docs-release-2026-08-13 |
||
|
github-actions[bot]
|
6685cbd599 |
chore: release v4.5.11 (#4557)
## Summary 4 new features, 24 improvements, 10 bug fixes. ## Highlights - Allow `trigger deploy` to authenticate with an environment API key from `TRIGGER_ACCESS_TOKEN`. ([#4561](https://github.com/triggerdotdev/trigger.dev/pull/4561)) ## Improvements - Chat in the browser now reconnects when the connection drops mid-turn, instead of leaving the reply stuck as if it were still generating. Reports can be fetched as structured data with the `json` format, and the shortest report period is now one minute (`1m`, `30m`, `1h`, `7d`). The `mint-token` command's help is clearer too: a token minted without `--cap` is read-only, and `--ttl` shows the correct maximum lifetime of 7 days. ([#4418](https://github.com/triggerdotdev/trigger.dev/pull/4418)) - The dev environment onboarding now tracks real progress. After you run `init`, the setup checklist marks your project as initialized, and it updates live as your dev server connects and your tasks register. The blank state also adds a "Copy AI agent prompt" button that copies a ready-to-paste setup prompt (pre-filled with your project reference) for Claude Code, Cursor, or any coding agent. ([#4563](https://github.com/triggerdotdev/trigger.dev/pull/4563)) The `init` scaffold now imports from `@trigger.dev/sdk` instead of the deprecated `@trigger.dev/sdk/v3` subpath. - Deployed images now ship dependencies and bundled task code as separate layers. Repeat deploys with unchanged dependencies typically push and pull far less data, making deploys and worker image pulls faster. ([#4551](https://github.com/triggerdotdev/trigger.dev/pull/4551)) - The current-worker API now reports each task's queue, so you can see which tasks write to a given queue. ([#4525](https://github.com/triggerdotdev/trigger.dev/pull/4525)) - Watch-mode chat streams now survive quiet windows and page reloads, and a reply cut off by a lost connection shows an error instead of appearing finished. Aborting a resumed subscription only closes your local stream — call `stopGeneration(chatId)` or pass `stopOnAbort: true` to stop the run. Also fixed a race where quickly restarting a stream could break stop and reconnect, and stopping a chat now hands it back to your other tabs instead of leaving them read-only. ([#4516](https://github.com/triggerdotdev/trigger.dev/pull/4516)) ## Server changes These changes affect the self-hosted Docker image and Trigger.dev Cloud: - The dashboard agent now has a monthly message allowance and plan-based limits on watches. Queries stay read-only with clearer errors when busy, and messages with unusual characters no longer fail to send. ([#4516](https://github.com/triggerdotdev/trigger.dev/pull/4516)) - Meet the dashboard agent: a chat in every environment that answers questions about your runs, queues, errors and health with real data and links, replacing Ask AI everywhere it used to appear. Investigate a failed run, an error, a backed-up queue or a run that hasn't started to get a worked-through answer — what happened, why, and how to fix it, with every claim linked to the runs, errors and deploys behind it. It reads your data read-only, works on preview and dev branches with that branch's own data, and reads the same everywhere — dashboard, terminal, editor. A very long chat keeps working: the agent summarises the earlier part and carries on. **Watch…** on a run, queue, error or the health report tells you when things change: a run finishes, a queue clears or grows past a number you pick, an error comes back, an environment recovers. The answer arrives in the chat and, if you want, by email, Slack or webhook — and the agent can look into bad news on its own. A watch reaches you on any browser you sign in from, without opening the chat first. A sample of conversations is scored automatically so the agent keeps getting better; only the score and a one-line summary are kept, never your messages, data or code, and we can switch it off for your organization on request. Ask the agent instead of the Docs buttons in page headers — they stay there when the agent isn't available to you. Separately, a queue's wait times, peak depth, throughput and throttling can now be read from the API. ([#4418](https://github.com/triggerdotdev/trigger.dev/pull/4418)) - Add backend support for delaying cron schedules within a specified window with a minimum of 60 seconds. ([#4566](https://github.com/triggerdotdev/trigger.dev/pull/4566)) - Reduced recurring background database load from the billing-limit recovery check, so paused environments are reconciled with less overhead. ([#4590](https://github.com/triggerdotdev/trigger.dev/pull/4590)) - Validating a schedule when deploying or updating a schedule now does less work on projects with many preview branches, so those operations stay fast as branches accumulate. ([#4598](https://github.com/triggerdotdev/trigger.dev/pull/4598)) - Project pages now load faster for projects with a large number of preview branches, by no longer loading archived branch environments that aren't shown. ([#4595](https://github.com/triggerdotdev/trigger.dev/pull/4595)) - Database queries that filter on a list of values now reuse cached query plans more consistently, instead of forcing the database to re-plan whenever the list length changes. ([#4480](https://github.com/triggerdotdev/trigger.dev/pull/4480)) - Routine cleanup of old dashboard agent data now runs on its own schedule. ([#4599](https://github.com/triggerdotdev/trigger.dev/pull/4599)) - Database connection metrics are now reported for every configured database connection instead of only the primary one, and stay accurate regardless of connection type. ([#4541](https://github.com/triggerdotdev/trigger.dev/pull/4541)) - Deployment-related API endpoints now draw from their own generous rate limit budget, configurable via the `DEPLOYMENT_RATE_LIMIT_*` environment variables, so runtime API traffic no longer competes with deployments for the same per-environment budget. ([#4565](https://github.com/triggerdotdev/trigger.dev/pull/4565)) - Deleting or editing a secret environment variable is now fast and no longer slows down as a project accumulates variables. ([#4555](https://github.com/triggerdotdev/trigger.dev/pull/4555)) - Speed up personal access token lookups by indexing them on their owner ([#4588](https://github.com/triggerdotdev/trigger.dev/pull/4588)) - Switching project or organization in the sidebar now keeps you on the same page instead of sending you back to Tasks. Pages for a specific run, deploy or other single item open the matching list instead. ([#4585](https://github.com/triggerdotdev/trigger.dev/pull/4585)) - Reduced database load when loading the dashboard by removing an unused organization member count that was being calculated on every page navigation. ([#4587](https://github.com/triggerdotdev/trigger.dev/pull/4587)) - The environment variables page now loads a page at a time, keeping it fast for projects with a large number of variables. Search matches variable names across every page. ([#4597](https://github.com/triggerdotdev/trigger.dev/pull/4597)) - Groundwork for an alternative database connection driver, gated behind configuration and disabled by default, so there is no change to default behavior. ([#4539](https://github.com/triggerdotdev/trigger.dev/pull/4539)) - Deleting an alert channel is now fast and no longer slows down as a project builds up alert history. ([#4554](https://github.com/triggerdotdev/trigger.dev/pull/4554)) - Reduced internal overhead on the API under high load. ([#4532](https://github.com/triggerdotdev/trigger.dev/pull/4532)) - Out-of-date upgrade prompts no longer appear in the dashboard: the "V4" badges and the notices saying preview branches and the queues table need V4 have been removed. The side menu still warns you when a project is on v3, with updated wording and a link to the v4 upgrade guide. ([#4589](https://github.com/triggerdotdev/trigger.dev/pull/4589)) - Make background worker registration cheaper for projects with many scheduled tasks by scoping declarative schedule reconciliation to the current environment and dropping redundant schedule lookups. ([#4577](https://github.com/triggerdotdev/trigger.dev/pull/4577)) - Speed up setting and importing environment variables for projects with many variables. ([#4579](https://github.com/triggerdotdev/trigger.dev/pull/4579)) - Loading the deployments list is now faster, especially when filtering by deployment status on projects with many deployments. ([#4591](https://github.com/triggerdotdev/trigger.dev/pull/4591)) - Fixed the billing limits page timing out for organizations with many preview branches, especially while a spend limit was being enforced. The page now loads quickly, so you can raise or resolve your limit without delay. ([#4594](https://github.com/triggerdotdev/trigger.dev/pull/4594)) - Fix the Concurrency page showing the plan's default concurrency for the dev environment instead of the environment's actual limit. ([#4596](https://github.com/triggerdotdev/trigger.dev/pull/4596)) - Creating an organization sometimes left you back on the creation form even though the organization had already been created, so clicking Create again made a duplicate. Creating an organization now completes and takes you to your new organization. ([#4530](https://github.com/triggerdotdev/trigger.dev/pull/4530)) - Ensure creating a project completes instead of returning to its creation form after a navigation error. ([#4584](https://github.com/triggerdotdev/trigger.dev/pull/4584)) - Renaming a project now keeps you on the project settings page and tells you what happened, instead of silently moving you to the tasks page or clearing the form with no explanation. ([#4601](https://github.com/triggerdotdev/trigger.dev/pull/4601)) - Fixed support threads showing no account details for some customers, so the team can see your plan, organizations and projects when you get in touch. ([#4575](https://github.com/triggerdotdev/trigger.dev/pull/4575)) - In the light theme, the Format, Clear and Copy buttons on the query editor no longer blend into the query text behind them. ([#4592](https://github.com/triggerdotdev/trigger.dev/pull/4592)) - The health report now says start latency is "unknown" when there is no data for it, instead of showing a healthy-looking 0ms ([#4544](https://github.com/triggerdotdev/trigger.dev/pull/4544)) - Realtime streams written inside a chat session run now use the same backend as the session itself, and runs are no longer created against a backend that cannot serve them. ([#4564](https://github.com/triggerdotdev/trigger.dev/pull/4564)) - The grouped "watch updates" notification now shows the total number of results waiting, instead of only the most recent batch's count. ([#4525](https://github.com/triggerdotdev/trigger.dev/pull/4525)) <details> <summary>Raw changeset output</summary> # Releases ## @trigger.dev/build@4.5.11 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.11` ## trigger.dev@4.5.11 ### Patch Changes - Chat in the browser now reconnects when the connection drops mid-turn, instead of leaving the reply stuck as if it were still generating. Reports can be fetched as structured data with the `json` format, and the shortest report period is now one minute (`1m`, `30m`, `1h`, `7d`). The `mint-token` command's help is clearer too: a token minted without `--cap` is read-only, and `--ttl` shows the correct maximum lifetime of 7 days. ([#4418](https://github.com/triggerdotdev/trigger.dev/pull/4418)) - Allow `trigger deploy` to authenticate with an environment API key from `TRIGGER_ACCESS_TOKEN`. ([#4561](https://github.com/triggerdotdev/trigger.dev/pull/4561)) - The dev environment onboarding now tracks real progress. After you run `init`, the setup checklist marks your project as initialized, and it updates live as your dev server connects and your tasks register. The blank state also adds a "Copy AI agent prompt" button that copies a ready-to-paste setup prompt (pre-filled with your project reference) for Claude Code, Cursor, or any coding agent. ([#4563](https://github.com/triggerdotdev/trigger.dev/pull/4563)) The `init` scaffold now imports from `@trigger.dev/sdk` instead of the deprecated `@trigger.dev/sdk/v3` subpath. - Deployed images now ship dependencies and bundled task code as separate layers. Repeat deploys with unchanged dependencies typically push and pull far less data, making deploys and worker image pulls faster. ([#4551](https://github.com/triggerdotdev/trigger.dev/pull/4551)) - Updated dependencies: - `@trigger.dev/core@4.5.11` - `@trigger.dev/build@4.5.11` - `@trigger.dev/schema-to-json@4.5.11` ## @trigger.dev/core@4.5.11 ### Patch Changes - Chat in the browser now reconnects when the connection drops mid-turn, instead of leaving the reply stuck as if it were still generating. Reports can be fetched as structured data with the `json` format, and the shortest report period is now one minute (`1m`, `30m`, `1h`, `7d`). The `mint-token` command's help is clearer too: a token minted without `--cap` is read-only, and `--ttl` shows the correct maximum lifetime of 7 days. ([#4418](https://github.com/triggerdotdev/trigger.dev/pull/4418)) - The current-worker API now reports each task's queue, so you can see which tasks write to a given queue. ([#4525](https://github.com/triggerdotdev/trigger.dev/pull/4525)) ## @trigger.dev/python@4.5.11 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.11` - `@trigger.dev/sdk@4.5.11` - `@trigger.dev/build@4.5.11` ## @trigger.dev/react-hooks@4.5.11 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.11` ## @trigger.dev/redis-worker@4.5.11 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.11` ## @trigger.dev/rsc@4.5.11 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.11` ## @trigger.dev/schema-to-json@4.5.11 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.11` ## @trigger.dev/sdk@4.5.11 ### Patch Changes - Chat in the browser now reconnects when the connection drops mid-turn, instead of leaving the reply stuck as if it were still generating. Reports can be fetched as structured data with the `json` format, and the shortest report period is now one minute (`1m`, `30m`, `1h`, `7d`). The `mint-token` command's help is clearer too: a token minted without `--cap` is read-only, and `--ttl` shows the correct maximum lifetime of 7 days. ([#4418](https://github.com/triggerdotdev/trigger.dev/pull/4418)) - Watch-mode chat streams now survive quiet windows and page reloads, and a reply cut off by a lost connection shows an error instead of appearing finished. Aborting a resumed subscription only closes your local stream — call `stopGeneration(chatId)` or pass `stopOnAbort: true` to stop the run. Also fixed a race where quickly restarting a stream could break stop and reconnect, and stopping a chat now hands it back to your other tabs instead of leaving them read-only. ([#4516](https://github.com/triggerdotdev/trigger.dev/pull/4516)) - Updated dependencies: - `@trigger.dev/core@4.5.11` </details> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>helm-v4.5.11 v.docker.4.5.11 v4.5.11 |
||
|
Matt Aitken
|
6485f37bf2 |
fix(webapp): show the dev environment's actual limit in the concurrency page Total column (#4596)
## Summary On the Concurrency page, the dev environment row's Total always showed the plan's included dev concurrency, even when the environment's limit had been raised. The row's own "Extra concurrency" value was already derived from the real limit, so the two columns could disagree with each other. ## Root cause The Total cell renders `planConcurrencyLimit + allocation`, where `allocation` is the state behind the editable prod/staging inputs. Dev environments are deliberately excluded from that allocation map (dev concurrency is not purchasable), so the dev row's allocation always resolved to 0 and the Total fell back to the plan value. The dev row now renders the environment's actual `maximumConcurrencyLimit` instead. |