1984 Commits

Author SHA1 Message Date
Daniel Sutton c5c2ea92ca feat(core): add shard-routable run-ops id format and resolveShard (#4750)
## Summary

Adds a second generation of run-ops id, plus the resolver that reads a
store key straight out of an id. A gen-2 id keeps the existing
26-character layout, but the character at index 24 becomes a routing
shard key instead of a region code, and the version character at index
25 becomes `"2"`. Nothing mints gen-2 ids yet, so this is inert on
merge.

## Design

The version character is a single character, so the gen-1 and gen-2
shape checks can never both match. That is what makes the two
generations provably disjoint rather than disjoint by convention.

```ts
resolveShard(id) // gen-2 body    -> its shard key, [a-z0-9]
                 // gen-1 v1 body -> "new"
                 // anything else -> "legacy"
```

`resolveShard` is total: it returns a key for any input string,
including an empty or malformed one, and never throws.
`classifyResidency` keeps its signature and its two values, and now
reports gen-2 ids as part of the dedicated family, so existing consumers
of that boolean are unaffected.

The body stays 26 characters rather than 27 deliberately. The older
27-character format is still in the wild and has to keep resolving to
legacy, and a longer gen-2 shape would need probabilistic disambiguation
against it. A rare misroute is not an acceptable property for a routing
key.

The one behavior change is that a 26-character body ending in `"2"` now
routes by its shard key instead of falling back to legacy. Two test
assertions pinned the old result and are updated here. A repository-wide
search confirms they are the only two of their kind.

Verified against the full run-store corpus (68 files, 370 tests) with no
test-file changes there, plus the run-engine residency and waitpoint
suites. No changeset: the new surface has no caller, so a version bump
would tell a user nothing.
2026-08-21 13:04:25 +01:00
Chris Arderne 4392e79ce2 chore: adopt stable React Compiler lint rules (#4737) 2026-08-20 14:17:40 +02:00
github-actions[bot] ce40d0259f chore: release v4.5.12 (#4610) 2026-08-20 12:47:22 +01:00
claude[bot] 518978bc52 fix(core): don't assume a 64-character idempotency key is pre-hashed on reset (#4626)
<!-- ccr-slack-attribution -->
_Requested by **Matt Aitken** · [Slack
thread](https://triggerdotdev.slack.com/archives/C045W9WM3E1/p1786741966214949?thread_ts=1786741966.214949&cid=C045W9WM3E1)_

`idempotencyKeys.reset()` now honours an explicitly passed `scope` even
when the key material happens to be 64 characters long.

**Before:** `resetIdempotencyKey` treated *any* 64-character string as
an already-computed hash and sent it to the API verbatim. That
short-circuit ran before the scope logic, so if your key material is
itself a 64-character digest (a common pattern when you hash your own
dedup identity) the `scope` you passed was silently discarded and the
un-hashed material went on the wire. The server stores the hash, so the
reset matched no run and returned 404 every single time. Key material of
any other length worked fine, which made this look arbitrary.

**After:** a 64-character key with an explicit `scope` is sent verbatim
first and, only when that attempt comes back a definitive not-found,
retried as the derived scope hash. Every call that worked before behaves
identically, and the previously impossible case now resolves on the
fallback.

## How

A 64-character string is forwarded unchanged, exactly as before, when:

- the idempotency key catalog recognises it (it came from
`idempotencyKeys.create()` in this process), or
- no `scope` was passed, so there is nothing to derive a hash from, or
- the scope hash cannot be derived (e.g. `scope: "run"` outside a task
context with no `parentRunId`).

Otherwise the key is ambiguous: it may be raw material the caller wants
hashed with the scope, or it may already be the stored hash. Reset sends
the verbatim value first because that is what every previous version
sent, so anything that resolved before still resolves with the same
single request, the same target run, and the same errors. The derived
hash is the new behaviour, so it only runs once the verbatim attempt has
failed with a 404, a definitive "no run under this key". Any other error
(a 503, a connection error) leaves the verbatim key's state unknown, and
resetting a different key on unknown state would be an untargeted write
the caller never asked for, so those errors surface unchanged. That has
an honest cost: when the endpoint answers 503 for a miss it cannot
confirm, the caller sees the 503 and retries rather than silently
falling through to the derived key. When both attempts miss, the
verbatim attempt's 404 is surfaced, again matching what previous
versions threw.

A side benefit of this order: a key from `idempotencyKeys.create()`
reset with a `scope` from a cold process resolves in a single request,
because the created key is itself the stored value.

`isIdempotencyKey` is deliberately left alone: it applies the same
length rule on the trigger path, but it is self-consistent there, and
changing it would invalidate already-stored keys.

The `attachedOptions?.key` / `attachedOptions?.scope` fallbacks below
the old guard were unreachable (every catalog entry is a 64-character
digest, so it always hit the short-circuit first) and re-deriving from
them produces the identical hash anyway. They are removed rather than
left as dead code.

---

##  Checklist

- [x] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [x] The PR title follows the convention.
- [x] I ran and tested the code works

---

## Testing

Tests in `packages/core/src/v3/idempotencyKeys.test.ts` drive the real
`resetIdempotencyKey` against a local HTTP server and assert on the
exact values that reach the wire, in order. Nothing is mocked. They
cover:

- 64-character material + explicit `scope` derives the global- and
run-scoped hash once the verbatim key misses (fails without this change)
- the verbatim key wins when runs exist under both the verbatim value
and the derived hash, so the pre-existing target is preserved
- keys from `idempotencyKeys.create()` are forwarded unchanged: catalog
hit, no scope, and scope with a cold catalog (the last now a single
request)
- a transient failure of the verbatim attempt surfaces its error without
ever touching the derived key
- error surfacing: a double miss reports the key the caller passed, and
a non-404 from the fallback is not swallowed
- ordinary short material is still hashed, and underivable run/attempt
scopes still send a 64-character key verbatim while still throwing for
shorter material

```
pnpm run test ./src/v3/idempotencyKeys.test.ts --run   # 18 passed
pnpm run build --filter @trigger.dev/core              # clean
pnpm run format && pnpm run lint                       # clean
```

---

## Changelog

`idempotencyKeys.reset()` now works when your idempotency key is itself
64 characters long. Previously any 64-character key was assumed to be
already hashed, so passing one along with a `scope` silently ignored the
scope and the reset never found a matching run.

---

## Follow-ups (not in this PR)

- `docs/idempotency.mdx` describes the `idempotencyKey` parameter of
`reset()` as "the 64-character hash string" in one place while showing
raw material plus `{ scope: "global" }` a few lines later. Worth
reconciling.
- No surface currently exposes the stored hash that the reset endpoint
matches on: `ctx.run.idempotencyKey`, the run page and the
`idempotency_key` query column all show the user-provided key. That is
what leads people to send a value reset cannot match.

---------

Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Matt Aitken <matt@mattaitken.com>
2026-08-20 12:14:51 +01:00
Chris Arderne 9dca03f682 chore: enforce exhaustive React hook dependencies (#4712)
## Summary

Enables exhaustive React Hook dependency checking and resolves the
existing violations across the dashboard and React hooks package.
Effects and callbacks now track current values without introducing
request, subscription, or render loops.

## Design

Dependencies are included directly when the hook lifecycle should follow
them. Timers, Remix fetchers, and realtime subscriptions use stable
callbacks or latest-value refs where restarting work would change
behavior.

Unnecessary memoization was removed where ordinary derivation is
clearer. Full lint and typechecks for the webapp and React hooks package
pass.
2026-08-20 09:59:38 +01:00
Chris Arderne 19908436b8 perf(ci): speed up webapp test execution (#4709)
## Summary

Speeds up webapp test jobs by balancing measured work across runners,
reducing repeated container setup, and ensuring test workers release
shutdown resources promptly. Unit tests run across 24 duration-aware
shards, while E2E tests run across two balanced shards.

## Design

`RunEngine` shutdown now closes processing resources before support
resources, continues cleanup if one close fails, and reuses one shutdown
promise for concurrent callers. Redis workers clear completed shutdown
deadlines so finished tests no longer wait on idle timers.

Container-heavy suites are split only where it improves parallelism, and
repeated replication and engine fixtures are consolidated where one
end-to-end case provides coverage. Timing weights are refreshed for all
affected files.

Dependency installation overlaps container pulls, and both workflows use
WarpBuild's Node setup action.
2026-08-20 07:08:22 +01:00
Oskar Otwinowski 8b0385c429 feat(run-engine): trigger tasks pinned to an external deployment id (#4664)
The SDK discovers an external deployment id at runtime (explicit
TRIGGER_EXTERNAL_DEPLOYMENT_ID always; platform commit-SHA variables and
generic fallbacks when TRIGGER_AUTOMATIC_SKEW_VERSION_PROTECTION=1) and
sends it alongside lockToVersion; the server resolves precedence
(version > external id > current). An id held by a deployed deployment
pins the run to that worker; an in-flight or unknown id parks the run in
PENDING_VERSION with the id in TaskRun.annotations, wakes it pinned when
a deployment carrying the id finalizes (ClickHouse candidates, Postgres
authoritative), and expires it after a deadline that re-checks Postgres
before acting. Parking outranks delaying and preserves delayUntil. The
id is projected to ClickHouse task_runs_v2.external_deployment_id during
replication. Redis cache for id-to-worker resolution, guarded
version-aware writes.

Ids are not unique. Several deployments can hold one id - a --force
rebuild is the ordinary way to get there - so resolution always picks
the highest version among the candidates, never the newest by timestamp.
The rule is applied identically on both paths that can bind a run to a
worker: resolveExternalDeployment at trigger time, and
PendingVersionSystem when a landing deployment wakes a parked run.
Version comparison is numeric on the counter half, so 20260807.10
outranks 20260807.9.

A run whose id never lands expires at the deadline with
EXTERNAL_DEPLOYMENT_NOT_FOUND and an error naming the id it waited for,
which is what a failed build or a typo looks like from the caller.
Default deadline is one hour (EXTERNAL_DEPLOYMENT_PARK_DEADLINE_MS).

Debounce registration happens in both the parked and the delayed branch
through one helper, so a debounced run that parks still binds its
debounce key; without it every later trigger for the same key created
another parked run, and all of them executed when the deployment landed.
The two DELAYED-only status checks in DebounceSystem also accept
PENDING_VERSION, without which the lock-contention fallback would
rethrow a 5xx the SDK retries and amplifies, and the fast path would
push every trigger on a parked key through the redlock.

Resolution is skipped in development. A dev environment cannot hold a
WorkerDeployment - trigger dev registers a BackgroundWorker with nothing
behind it, and deploy --env refuses dev - so an external deployment id
there could only ever park, and the parked run then expired against the
dev TTL while a connected dev worker sat idle. The id is still annotated
so the dashboard shows what the app sent (TRI-13000).
2026-08-19 17:43:53 +02:00
Oskar Otwinowski 6bfce6387d feat(deploy): --external-id and --force for deploy idempotency (#4663)
A deploy can carry an opaque external id (commit SHA, CI run id, release
tag). Repeating an id that already deployed returns the existing version
as a no-op instead of rebuilding; an id with a build in flight is
rejected with 409 naming that version; a failed id rebuilds freely.
--force is non-destructive to deployments that already succeeded - both
persist and the higher version wins - but cancels a build still in
flight, so one id never has two live builds racing to define it.
Cancelling writes a terminal status and appends a finalized event, which
aborts a build the platform drives; a build it does not drive keeps
running but can never land, and the CLI says so. Ids are deliberately
not unique - reuse is resolved in application code by highest version,
never timestamps. The no-op path mints no build credentials and no event
stream (TRI-12923).

What that means for callers: a --force rebuild leaves two deployments
holding one id, and runs triggered with it go to the higher version once
the rebuild lands, so the takeover needs no separate promotion. Until a
successful build exists for an id, runs triggered with it park and then
expire rather than falling back to current - a failed build is therefore
visible to the caller as expired runs, not as runs on the wrong release.
2026-08-19 17:43:51 +02:00
Oskar Otwinowski 689538d327 feat(core): external deployment id wire contract (#4662)
An external deployment id is an opaque, caller-chosen name for a release
- a commit SHA, a CI run id, a release tag. This adds the shared
contract that both halves of the feature read, and nothing else: no
deploy writes one yet and no trigger sends one.

ExternalDeploymentId is defined once and reused by
InitializeDeploymentRequestBody.externalId and
TriggerTaskRequestBody.options.externalDeploymentId, so a value accepted
by one half can never be rejected by the other. A value that is blank
once trimmed is treated as absent rather than rejected, so an unset CI
variable expanding to an empty string is not a 400. The 128 character
limit fits a SHA-256 commit hash with room for composite ids, and
EXTERNAL_DEPLOYMENT_ID_MAX_LENGTH is the single source of truth that the
request schemas and the CLI both read.

RunAnnotations.externalDeploymentId records the request, not the
outcome: lockedToVersionId and taskVersion are overwritten when a run
locks, whereas this stays true forever, and it can carry the pin for a
run parked before its deployment exists.

Also lands the runtime discovery helpers as pure functions over an
environment reader: the explicit TRIGGER_EXTERNAL_DEPLOYMENT_ID
variable, the platform and CI commit-SHA table, and the
TRIGGER_AUTOMATIC_SKEW_VERSION_PROTECTION gate. Nothing calls them yet.

refs TRI-13000
2026-08-19 17:43:51 +02:00
Chris Arderne 108f43ee9b fix(webapp,react-hooks): enforce stable hook ordering (#4688)
## Summary

Enforce stable React hook ordering in the dashboard and React hooks
package.

Conditional hook calls now keep a consistent order, and overloaded
realtime stream arguments are resolved before entering the shared hook
implementation.

Base: `main`
2026-08-19 16:35:41 +01:00
Chris Arderne 4dabfca1d5 feat(webapp,cli,core): list production project runtime updates (#4659) 2026-08-19 13:44:55 +01:00
Chris Arderne f4320937c5 chore: prefer direct iteration and function callback types (#4677)
## Summary

Enable lint rules that prefer direct iteration and concise function
callback types.

The existing code now uses direct iteration where no index is needed,
and callback contracts use function types consistently.

Base: [#4675](https://github.com/triggerdotdev/trigger.dev/pull/4675)
2026-08-19 08:28:58 +01:00
Chris Arderne 8572e8edbf chore: reject redundant standalone blocks (#4675)
## Summary

Enable the rule that rejects unnecessary standalone blocks.

The existing empty branches are removed so future control flow remains
purposeful.

Base: [#4674](https://github.com/triggerdotdev/trigger.dev/pull/4674)
2026-08-19 08:28:58 +01:00
Chris Arderne 0f725cf2ba chore: enable lint cleanup rules (#4673)
## Summary

Enable small cleanup rules for redundant boolean expressions, object
ownership checks, assignments, and object construction.

The existing call sites now use the simpler equivalent forms, keeping
future code consistent without changing behavior.

Base: [#4672](https://github.com/triggerdotdev/trigger.dev/pull/4672)
2026-08-19 08:28:57 +01:00
Chris Arderne fe1d5f6961 chore: enable additional correctness lint rules (#4672)
## Summary

Enable additional lint rules that catch unsafe optional-chain
assertions, inherited-property iteration, anonymous symbols, and unsafe
external links.

The existing violations now use explicit values and own-property checks,
so the rules can prevent those patterns from returning.
2026-08-19 08:28:56 +01:00
Marcus Nerløe b83cf671de fix(core): mint the fallback external trace id per run (#4534)
## What

Runs that carry no external trace context (schedules, task-to-task
triggers) fall back to a trace id generated once in the [`TracingSDK`
constructor](https://github.com/triggerdotdev/trigger.dev/blob/main/packages/core/src/v3/otel/tracingSDK.ts#L165).
With `experimental_processKeepAlive` the SDK outlives the run, so every
run on a warm process is exported to the external OTLP endpoint under
that one id.

Across our production traces, 80.3% contained spans from more than one
run, worst case 25. Per-run cost and latency attribution is unusable as
a result. This is the same warm-start hazard c043c4a6a fixed for the
external-context path, which left the fallback captured at construction.

## How

`FallbackExternalTraceIds` hands out one id per internal trace, shared
by the span and log wrappers so a run's spans and logs agree.

The id is keyed off the record's own internal trace id rather than
ambient state at export time, because batch processors drain
asynchronously and a run's records routinely export after the next run
has started. The map is bounded and evicts least-recently-used, so a run
that is still exporting can't lose its id.

Granularity follows the internal trace, so a run and the runs it
triggers stay on one trace.

**Risk:** the wrappers only exist when `exporters` / `logExporters` are
configured, so deployments that don't export externally are untouched.
Nothing outside `tracingSDK.ts` changes.

**Known gap (pre-existing):** sampling and id selection still branch on
ambient `getExternalTraceContext()`, so records draining across a run
boundary in mixed mode are misplaced in both directions. It can't use
the approach here — the external id comes from the run's incoming
`traceparent`, which isn't carried on the record — so closing it means
capturing `internalTraceId -> external context` in a span processor.
Happy to follow up separately.

---

## Testing

`packages/core` suite passes. `pnpm run format` and `pnpm run lint:fix`
produce no diff.

Six cases in `externalSpanExporterWrapper.test.ts`, each
mutation-checked rather than just observed passing: one id per run,
stability within a run, correct id when records drain after the next run
started (spans and logs together), external export stays off when
unconfigured, retention of a run still exporting while the map churns,
and the bound itself.

**CI:** the five failing `webapp` shards are the ones containing
`containerTest` suites. Fork PRs receive no repository secrets, so
`unit-tests-webapp.yml` skips the DockerHub login and the image pre-pull
(both gated on `env.DOCKERHUB_USERNAME`) and the container tests time
out at 60s. Same five shards across five runs, every failure a 60s
timeout, and those shards pass on internal PRs. Happy to be corrected if
you can run them with secrets available.

---

## Changelog

Unrelated runs are no longer merged into a single trace in your external
observability tool when they happen to execute on the same warm worker
process. A run and the runs it triggers still share one trace, so a run
tree stays together.

---

## Screenshots

_n/a_

---

_Supersedes #4526 (auto-closed before I was vouched) and #4533 (opened
ready rather than as a draft). GitHub won't reopen either._

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Iss <74388823+isshaddad@users.noreply.github.com>
Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
2026-08-18 18:59:00 +01:00
Chris Arderne b33197691b chore: enforce no unused deps or code in ci (#4654) 2026-08-18 11:35:51 +01:00
Chris Arderne 99f0787148 feat(cli,webapp): default new projects to node-24 (#4649) 2026-08-18 07:23:52 +01:00
nicktrn d62dd0dc30 chore(core): drop the unused socket.io dependency (#4640)
## Summary

`packages/core` declared `socket.io`, the server package, but never
imported it. Its only Socket.IO usage is the client:

```
packages/core/src/v3/zodSocket.ts
packages/core/src/v3/runEngineWorker/supervisor/session.ts
  import { io } from "socket.io-client";
```

The only occurrence of `socket.io` outside those client imports was the
`package.json` line itself. Since `@trigger.dev/core` is published, that
line meant every consumer installed a server package nothing in the tree
imports.

`socket.io-client` is untouched. `apps/webapp` and `apps/supervisor`
keep their own `socket.io` dependencies, so the server side is
unaffected.

Found with `pnpm run knip:deps`, which the repo already ships.

`pnpm run typecheck` passes across all 57 workspaces, and
`@trigger.dev/core` builds clean.

Stacked on #4639.
2026-08-16 22:29:39 +01:00
nicktrn f3c46f140e chore(deps): raise nanoid floors, drop unused declarations (#4637)
## Summary

`nanoid` was pinned at exactly `3.3.8` in five manifests. Two of those
five never imported it: in `internal-packages/schedule-engine` and
`internal-packages/webhook-engine` the only occurrence of the string
`nanoid` in the entire package was the `package.json` line itself. Both
are removed rather than bumped.

The three that genuinely use it move to `3.3.18`, a version already
present in the tree via `postcss`, so this pulls in nothing new.

| Package | Uses it | Change |
| --- | --- | --- |
| `internal-packages/schedule-engine` | no | removed |
| `internal-packages/webhook-engine` | no | removed |
| `apps/webapp` | yes | `3.3.8` to `3.3.18` |
| `packages/core` | yes | `3.3.8` to `3.3.18` |
| `internal-packages/run-engine` | yes | `3.3.8` to `3.3.18` |
| `packages/redis-worker` | yes | `^5.0.7` to `^5.1.16` |

`redis-worker` is on the 5.x line and is included because its declared
range already permitted a newer release; the lockfile had simply not
re-resolved, leaving it on `5.1.2`.

The unused declarations were found with `pnpm run knip:deps`, which the
repo already ships.

`pnpm run typecheck` passes across all 57 workspaces.
2026-08-16 22:12:18 +01:00
nicktrn 148615b526 chore(webapp,supervisor,core): move socket.io to 4.8.3 (#4635)
## Summary

`socket.io` was pinned at exactly `4.7.4` in three manifests
(`apps/webapp`, `apps/supervisor`, `packages/core`). That pin capped
`engine.io` at 6.5.4, because 4.7.4 declares `engine.io: ~6.5.2`.

Moving all three pins to `4.8.3` lifts that cap: 4.8.3 declares
`engine.io: ~6.6.0`. The webapp's direct `engine.io` devDependency moves
from `^6.5.4` to `^6.6.7` to match.

These are direct dependencies, so they are bumped in place rather than
forced with an override.

## Result

The tree previously carried two `engine.io` copies. It now carries one:

```
engine.io@6.6.8
└─┬ socket.io@4.8.3
  ├── @trigger.dev/core (dependencies)
  ├─┬ react-email
  │ └── emails (devDependencies)
  ├── supervisor (dependencies)
  └── webapp (dependencies)
```

`react-email` was already resolving `socket.io@4.8.3` in this same tree,
so that combination was already running here before this change.

## Servers move, clients do not

This bumps `socket.io` (the server) only. `socket.io-client` stays at
`4.7.5` in `packages/core` and `packages/cli-v3`, deliberately: the fix
is server-side, and clients ship inside user deployments, so leaving
them alone keeps the blast radius small. That means a 4.8.3 server will
be talking to 4.7.5 clients indefinitely, which is worth being explicit
about.

That pairing is safe because neither wire protocol changed. Both
versions report the same protocol numbers:

| | 4.7.4 | 4.8.3 |
| --- | --- | --- |
| Socket.IO protocol (`socket.io-parser`) | 5 | 5 |
| Engine.IO protocol (`engine.io-parser`) | 4 | 4 |

The version bump moves `socket.io-parser` 4.2.6 to 4.2.7 and `engine.io`
6.5.4 to 6.6.8, but the protocol constants each exports are unchanged.
The 4.8.0 changes are additive on the client (custom transport
implementations, a `tryAllTransports` option) and bug fixes on the
server.

Verified rather than assumed, with a cross-version matrix covering both
transports and both directions:

```
PASS  server 4.8.3 <- client 4.7.5   websocket / polling
PASS  server 4.8.3 <- client 4.8.3   websocket / polling
PASS  server 4.7.4 <- client 4.7.5   websocket / polling
PASS  server 4.7.4 <- client 4.8.3   websocket / polling
```

Each case exercised connect, a server-initiated emit, `emitWithAck`,
room join, room broadcast, and a binary payload. Compatibility holds in
both directions, so there is no upgrade-ordering requirement between
server and client.

`pnpm run typecheck` passes across all 57 workspaces.

Stacked on #4634.
2026-08-16 21:11:34 +01:00
Eric Allam c0b84595a3 feat(webapp): hosted webhook ingress, delivery pipeline, and dashboard (#4344)
## Summary

The server half of hosted webhooks: the public ingress endpoint,
signature verification, the delivery pipeline (Postgres partitioned
storage + ClickHouse for ordering), the in-app partition manager, the
HTTP API, and the dashboard (Deliveries, Endpoints, and the in-app test
console).

The public SDK and docs half is #4537. That PR carries the user-facing
API (`webhook()`, `chat.event` / `chat.channels`, the
`@trigger.dev/slack` connector) and builds on the shared
`@trigger.dev/core` schemas that ship here.

## Shipping behind a flag

A `WEBHOOK_ENABLED` env var (default off) gates the public ingress route
and the engine worker plus partition cron, so merging and deploying this
changes nothing in production until it is flipped on per environment.
The dashboard is separately gated per org by the `hasWebhooksAccess`
feature flag.

## Note on packages

This PR includes the `@trigger.dev/core` schema additions the server
compiles against, but carries no changeset. Core is not consumed
independently of the SDK, so it is released together with the SDK via
#4537. Keeping its changeset off `main` means no release cut from `main`
publishes it early.
2026-08-16 14:33:42 +01:00
Saadi Myftija c4b5e27258 feat(cli): build deployment images on prebuilt base images (#4602)
The generated deploy Containerfile now starts from the prebuilt base
images published by base-images/ (`triggerdotdev/node` and
`triggerdotdev/bun` on DockerHub, pinned by digest) instead of
installing system packages during every project's build. Uncustomized
projects run no apt at all and their base layers are identical across
every project, so worker nodes cache one copy fleet-wide. The build
stage uses the -build toolchain variant for uncustomized and
package-only projects; projects with image instructions build FROM base
so instructions and their downloads run exactly once.

### Notes

- User packages install in their own sorted RUN with --allow-downgrades
(a pin of a preinstalled package is a downgrade against the prebuilt
base), preceded by a dpkg repair whenever instructions came first, since
apt-get install refuses to run on state a dpkg -i instruction left
broken.
- Deployed runtime images inherit newer package versions than today's
live-archive installs (the published bases upgrade everything to their
snapshot), plus the base images' OCI labels. Runtime env, user, workdir,
and entrypoint are unchanged.
2026-08-14 12:27:06 +02:00
Chris Arderne 3e7964e7fa feat: surface cron windows in webapp, cli, sdk (#4572)
## Summary

Adds execution-window product surfaces for both declarative and
imperative schedules.

- Declarative schedules can set `window` through `schedules.task()`,
with support for whole-minute, hour, and percentage values.
- Imperative schedules can create, update, clear, and inspect windows
through the API and dashboard.
- Schedule API responses preserve `nextRun` as the nominal CRON time and
expose `nextRunEffectiveAt` as the stable assigned time.
- The dashboard displays configured windows alongside assigned
upcoming-run times.
- Deploy output summarizes declarative schedules and suggests adding a
wider window when the default 60-second placement range is used.

## Design

Window validation remains authoritative on the server and ensures each
window is compatible with the schedule cadence. Omitting a window uses
the default 60-second range, while explicit zero-duration windows remain
supported.

Deployment summaries are derived from the deployment's stored task
metadata, so they reflect the declarations associated with that
deployment.
2026-08-14 10:07:14 +01:00
nicktrn fa7eea39d8 fix(core): stop custom metric exporters breaking the metrics export (#4613)
## Summary

Projects that configure their own `metricExporters` or `metricReaders`
in `trigger.config.ts` were losing task metrics on nearly every run, and
seeing an unexplained `Failed to flush tracingSDK` alongside
`OTLPExporterError: Bad Request` in their run logs. Spans and logs kept
working, so the runs otherwise looked healthy.

## Root cause and fix

Every configured exporter gets its own `PeriodicExportingMetricReader`,
and `meterProvider.forceFlush()` fans out across all readers with
`Promise.all`, so two collections can land on the same millisecond.
`@opentelemetry/host-metrics` divides by the elapsed interval to compute
`process.cpu.utilization`
([common.ts](https://github.com/open-telemetry/opentelemetry-js-contrib/blob/main/packages/host-metrics/src/stats/common.ts)),
so a zero interval yields `0/0`. `JSON.stringify(NaN)` is `null`, and a
collector rejects `"asDouble": null` with a 400 that drops the
**entire** request, not just the offending point.

`flush()` and `shutdown()` now walk the metric readers one at a time, so
collections can no longer share a timestamp. Each reader is isolated, so
one failing reader cannot skip the readers behind it, and every failure
is logged with the reader that produced it. The first error is still
rethrown, so callers see failures exactly as before.

As a second layer, non-finite data points are dropped just before our
own export, so a metric that divides by zero cannot take the rest of the
batch with it. Exporters and readers supplied through
`trigger.config.ts` are untouched by that filter and still receive raw
data.

The trade-off is that configured exporters now flush after the built-in
one rather than alongside it, so flush latency is the sum rather than
the max.

An internal test package's dependency on core was replaced with a local
helper, because core now needs that package in `devDependencies` and the
two together formed a workspace cycle.

## Verification

Tested against a real collector in a container: a batch containing a
`NaN` reading is rejected with a 400 without the fix and accepted with
it, and a single flush is asserted to collect from one reader at a time.
2026-08-14 08:40:07 +01:00
Matt Aitken 1114d9d6f9 fix(redis-worker): stop fair queue leaking concurrency slots (#4540)
## Summary

Fair queue consumers could leak the per-tenant concurrency slots that
gate admission. Slots were freed on some paths and skipped on others,
and once enough leaked slots accumulated for a tenant, every queue that
tenant owned stopped being served until someone cleared the set by hand.
This PR frees slots on every path and, more importantly, makes the
remaining failure modes self-healing.

## Design

The fix applies one rule uniformly: releasing a concurrency slot is
best-effort cleanup and must never block the message's primary state
transition. Blocking completion re-delivers the message, which
duplicates customer work; blocking a retry loses the attempt increment,
so the message can circle forever; blocking a reclaim strands the
message in flight. A leaked slot is the better failure in every one of
those trades because it is the only one that is recoverable. A failed
release is therefore logged and the transition proceeds.

Leaked slots then heal through two mechanisms:

- `reserve` re-admits a message that is already a member of its own
concurrency set, since re-admitting it does not increase concurrency. A
message whose earlier release failed can no longer be blocked by its own
leftover slot.
- A reconcile loop periodically removes any set member with no in-flight
record (interval configurable via `reconcileIntervalMs`, default 60s).
The check-and-remove is atomic, and it is sound because a message is
always registered in flight before its slot is reserved, so a member
with no in-flight record can only be a leak. This also covers leaks this
PR cannot prevent directly, such as a release that resolves the wrong
concurrency group from queue metadata.

Ordering hardening from earlier revisions stays: slots are released
before the in-flight record needed to describe them is discarded, the
release Lua scripts write the message back to the queue before removing
it from in-flight (Lua does not roll back on error), and dangling
in-flight entries with no payload are dropped instead of being rescanned
forever.

Every guard test was verified to fail without its specific fix,
including the duplicate-execution case: completing a message while its
slot release fails used to re-deliver and re-execute it.
2026-08-13 19:36:20 +01:00
github-actions[bot] 6685cbd599 chore: release v4.5.11 (#4557)
## Summary
4 new features, 24 improvements, 10 bug fixes.

## Highlights

- Allow `trigger deploy` to authenticate with an environment API key
from `TRIGGER_ACCESS_TOKEN`.
([#4561](https://github.com/triggerdotdev/trigger.dev/pull/4561))

## Improvements
- Chat in the browser now reconnects when the connection drops mid-turn,
instead of leaving the reply stuck as if it were still generating.
Reports can be fetched as structured data with the `json` format, and
the shortest report period is now one minute (`1m`, `30m`, `1h`, `7d`).
The `mint-token` command's help is clearer too: a token minted without
`--cap` is read-only, and `--ttl` shows the correct maximum lifetime of
7 days.
([#4418](https://github.com/triggerdotdev/trigger.dev/pull/4418))
- The dev environment onboarding now tracks real progress. After you run
`init`, the setup checklist marks your project as initialized, and it
updates live as your dev server connects and your tasks register. The
blank state also adds a "Copy AI agent prompt" button that copies a
ready-to-paste setup prompt (pre-filled with your project reference) for
Claude Code, Cursor, or any coding agent.
([#4563](https://github.com/triggerdotdev/trigger.dev/pull/4563))
  
The `init` scaffold now imports from `@trigger.dev/sdk` instead of the
deprecated `@trigger.dev/sdk/v3` subpath.
- Deployed images now ship dependencies and bundled task code as
separate layers. Repeat deploys with unchanged dependencies typically
push and pull far less data, making deploys and worker image pulls
faster.
([#4551](https://github.com/triggerdotdev/trigger.dev/pull/4551))
- The current-worker API now reports each task's queue, so you can see
which tasks write to a given queue.
([#4525](https://github.com/triggerdotdev/trigger.dev/pull/4525))
- Watch-mode chat streams now survive quiet windows and page reloads,
and a reply cut off by a lost connection shows an error instead of
appearing finished. Aborting a resumed subscription only closes your
local stream — call `stopGeneration(chatId)` or pass `stopOnAbort: true`
to stop the run. Also fixed a race where quickly restarting a stream
could break stop and reconnect, and stopping a chat now hands it back to
your other tabs instead of leaving them read-only.
([#4516](https://github.com/triggerdotdev/trigger.dev/pull/4516))

## Server changes

These changes affect the self-hosted Docker image and Trigger.dev Cloud:

- The dashboard agent now has a monthly message allowance and plan-based
limits on watches. Queries stay read-only with clearer errors when busy,
and messages with unusual characters no longer fail to send.
([#4516](https://github.com/triggerdotdev/trigger.dev/pull/4516))
- Meet the dashboard agent: a chat in every environment that answers
questions about your runs, queues, errors and health with real data and
links, replacing Ask AI everywhere it used to appear. Investigate a
failed run, an error, a backed-up queue or a run that hasn't started to
get a worked-through answer — what happened, why, and how to fix it,
with every claim linked to the runs, errors and deploys behind it. It
reads your data read-only, works on preview and dev branches with that
branch's own data, and reads the same everywhere — dashboard, terminal,
editor. A very long chat keeps working: the agent summarises the earlier
part and carries on.
  
**Watch…** on a run, queue, error or the health report tells you when
things change: a run finishes, a queue clears or grows past a number you
pick, an error comes back, an environment recovers. The answer arrives
in the chat and, if you want, by email, Slack or webhook — and the agent
can look into bad news on its own. A watch reaches you on any browser
you sign in from, without opening the chat first.
  
A sample of conversations is scored automatically so the agent keeps
getting better; only the score and a one-line summary are kept, never
your messages, data or code, and we can switch it off for your
organization on request. Ask the agent instead of the Docs buttons in
page headers — they stay there when the agent isn't available to you.
Separately, a queue's wait times, peak depth, throughput and throttling
can now be read from the API.
([#4418](https://github.com/triggerdotdev/trigger.dev/pull/4418))
- Add backend support for delaying cron schedules within a specified
window with a minimum of 60 seconds.
([#4566](https://github.com/triggerdotdev/trigger.dev/pull/4566))
- Reduced recurring background database load from the billing-limit
recovery check, so paused environments are reconciled with less
overhead.
([#4590](https://github.com/triggerdotdev/trigger.dev/pull/4590))
- Validating a schedule when deploying or updating a schedule now does
less work on projects with many preview branches, so those operations
stay fast as branches accumulate.
([#4598](https://github.com/triggerdotdev/trigger.dev/pull/4598))
- Project pages now load faster for projects with a large number of
preview branches, by no longer loading archived branch environments that
aren't shown.
([#4595](https://github.com/triggerdotdev/trigger.dev/pull/4595))
- Database queries that filter on a list of values now reuse cached
query plans more consistently, instead of forcing the database to
re-plan whenever the list length changes.
([#4480](https://github.com/triggerdotdev/trigger.dev/pull/4480))
- Routine cleanup of old dashboard agent data now runs on its own
schedule.
([#4599](https://github.com/triggerdotdev/trigger.dev/pull/4599))
- Database connection metrics are now reported for every configured
database connection instead of only the primary one, and stay accurate
regardless of connection type.
([#4541](https://github.com/triggerdotdev/trigger.dev/pull/4541))
- Deployment-related API endpoints now draw from their own generous rate
limit budget, configurable via the `DEPLOYMENT_RATE_LIMIT_*` environment
variables, so runtime API traffic no longer competes with deployments
for the same per-environment budget.
([#4565](https://github.com/triggerdotdev/trigger.dev/pull/4565))
- Deleting or editing a secret environment variable is now fast and no
longer slows down as a project accumulates variables.
([#4555](https://github.com/triggerdotdev/trigger.dev/pull/4555))
- Speed up personal access token lookups by indexing them on their owner
([#4588](https://github.com/triggerdotdev/trigger.dev/pull/4588))
- Switching project or organization in the sidebar now keeps you on the
same page instead of sending you back to Tasks. Pages for a specific
run, deploy or other single item open the matching list instead.
([#4585](https://github.com/triggerdotdev/trigger.dev/pull/4585))
- Reduced database load when loading the dashboard by removing an unused
organization member count that was being calculated on every page
navigation.
([#4587](https://github.com/triggerdotdev/trigger.dev/pull/4587))
- The environment variables page now loads a page at a time, keeping it
fast for projects with a large number of variables. Search matches
variable names across every page.
([#4597](https://github.com/triggerdotdev/trigger.dev/pull/4597))
- Groundwork for an alternative database connection driver, gated behind
configuration and disabled by default, so there is no change to default
behavior.
([#4539](https://github.com/triggerdotdev/trigger.dev/pull/4539))
- Deleting an alert channel is now fast and no longer slows down as a
project builds up alert history.
([#4554](https://github.com/triggerdotdev/trigger.dev/pull/4554))
- Reduced internal overhead on the API under high load.
([#4532](https://github.com/triggerdotdev/trigger.dev/pull/4532))
- Out-of-date upgrade prompts no longer appear in the dashboard: the
"V4" badges and the notices saying preview branches and the queues table
need V4 have been removed. The side menu still warns you when a project
is on v3, with updated wording and a link to the v4 upgrade guide.
([#4589](https://github.com/triggerdotdev/trigger.dev/pull/4589))
- Make background worker registration cheaper for projects with many
scheduled tasks by scoping declarative schedule reconciliation to the
current environment and dropping redundant schedule lookups.
([#4577](https://github.com/triggerdotdev/trigger.dev/pull/4577))
- Speed up setting and importing environment variables for projects with
many variables.
([#4579](https://github.com/triggerdotdev/trigger.dev/pull/4579))
- Loading the deployments list is now faster, especially when filtering
by deployment status on projects with many deployments.
([#4591](https://github.com/triggerdotdev/trigger.dev/pull/4591))
- Fixed the billing limits page timing out for organizations with many
preview branches, especially while a spend limit was being enforced. The
page now loads quickly, so you can raise or resolve your limit without
delay. ([#4594](https://github.com/triggerdotdev/trigger.dev/pull/4594))
- Fix the Concurrency page showing the plan's default concurrency for
the dev environment instead of the environment's actual limit.
([#4596](https://github.com/triggerdotdev/trigger.dev/pull/4596))
- Creating an organization sometimes left you back on the creation form
even though the organization had already been created, so clicking
Create again made a duplicate. Creating an organization now completes
and takes you to your new organization.
([#4530](https://github.com/triggerdotdev/trigger.dev/pull/4530))
- Ensure creating a project completes instead of returning to its
creation form after a navigation error.
([#4584](https://github.com/triggerdotdev/trigger.dev/pull/4584))
- Renaming a project now keeps you on the project settings page and
tells you what happened, instead of silently moving you to the tasks
page or clearing the form with no explanation.
([#4601](https://github.com/triggerdotdev/trigger.dev/pull/4601))
- Fixed support threads showing no account details for some customers,
so the team can see your plan, organizations and projects when you get
in touch.
([#4575](https://github.com/triggerdotdev/trigger.dev/pull/4575))
- In the light theme, the Format, Clear and Copy buttons on the query
editor no longer blend into the query text behind them.
([#4592](https://github.com/triggerdotdev/trigger.dev/pull/4592))
- The health report now says start latency is "unknown" when there is no
data for it, instead of showing a healthy-looking 0ms
([#4544](https://github.com/triggerdotdev/trigger.dev/pull/4544))
- Realtime streams written inside a chat session run now use the same
backend as the session itself, and runs are no longer created against a
backend that cannot serve them.
([#4564](https://github.com/triggerdotdev/trigger.dev/pull/4564))
- The grouped "watch updates" notification now shows the total number of
results waiting, instead of only the most recent batch's count.
([#4525](https://github.com/triggerdotdev/trigger.dev/pull/4525))

<details>
<summary>Raw changeset output</summary>

# Releases
## @trigger.dev/build@4.5.11

### Patch Changes

- Updated dependencies:
  - `@trigger.dev/core@4.5.11`
## trigger.dev@4.5.11

### Patch Changes

- Chat in the browser now reconnects when the connection drops mid-turn,
instead of leaving the reply stuck as if it were still generating.
Reports can be fetched as structured data with the `json` format, and
the shortest report period is now one minute (`1m`, `30m`, `1h`, `7d`).
The `mint-token` command's help is clearer too: a token minted without
`--cap` is read-only, and `--ttl` shows the correct maximum lifetime of
7 days.
([#4418](https://github.com/triggerdotdev/trigger.dev/pull/4418))
- Allow `trigger deploy` to authenticate with an environment API key
from `TRIGGER_ACCESS_TOKEN`.
([#4561](https://github.com/triggerdotdev/trigger.dev/pull/4561))
- The dev environment onboarding now tracks real progress. After you run
`init`, the setup checklist marks your project as initialized, and it
updates live as your dev server connects and your tasks register. The
blank state also adds a "Copy AI agent prompt" button that copies a
ready-to-paste setup prompt (pre-filled with your project reference) for
Claude Code, Cursor, or any coding agent.
([#4563](https://github.com/triggerdotdev/trigger.dev/pull/4563))

The `init` scaffold now imports from `@trigger.dev/sdk` instead of the
deprecated `@trigger.dev/sdk/v3` subpath.

- Deployed images now ship dependencies and bundled task code as
separate layers. Repeat deploys with unchanged dependencies typically
push and pull far less data, making deploys and worker image pulls
faster.
([#4551](https://github.com/triggerdotdev/trigger.dev/pull/4551))
- Updated dependencies:
  - `@trigger.dev/core@4.5.11`
  - `@trigger.dev/build@4.5.11`
  - `@trigger.dev/schema-to-json@4.5.11`
## @trigger.dev/core@4.5.11

### Patch Changes

- Chat in the browser now reconnects when the connection drops mid-turn,
instead of leaving the reply stuck as if it were still generating.
Reports can be fetched as structured data with the `json` format, and
the shortest report period is now one minute (`1m`, `30m`, `1h`, `7d`).
The `mint-token` command's help is clearer too: a token minted without
`--cap` is read-only, and `--ttl` shows the correct maximum lifetime of
7 days.
([#4418](https://github.com/triggerdotdev/trigger.dev/pull/4418))
- The current-worker API now reports each task's queue, so you can see
which tasks write to a given queue.
([#4525](https://github.com/triggerdotdev/trigger.dev/pull/4525))
## @trigger.dev/python@4.5.11

### Patch Changes

- Updated dependencies:
  - `@trigger.dev/core@4.5.11`
  - `@trigger.dev/sdk@4.5.11`
  - `@trigger.dev/build@4.5.11`
## @trigger.dev/react-hooks@4.5.11

### Patch Changes

- Updated dependencies:
  - `@trigger.dev/core@4.5.11`
## @trigger.dev/redis-worker@4.5.11

### Patch Changes

- Updated dependencies:
  - `@trigger.dev/core@4.5.11`
## @trigger.dev/rsc@4.5.11

### Patch Changes

- Updated dependencies:
  - `@trigger.dev/core@4.5.11`
## @trigger.dev/schema-to-json@4.5.11

### Patch Changes

- Updated dependencies:
  - `@trigger.dev/core@4.5.11`
## @trigger.dev/sdk@4.5.11

### Patch Changes

- Chat in the browser now reconnects when the connection drops mid-turn,
instead of leaving the reply stuck as if it were still generating.
Reports can be fetched as structured data with the `json` format, and
the shortest report period is now one minute (`1m`, `30m`, `1h`, `7d`).
The `mint-token` command's help is clearer too: a token minted without
`--cap` is read-only, and `--ttl` shows the correct maximum lifetime of
7 days.
([#4418](https://github.com/triggerdotdev/trigger.dev/pull/4418))
- Watch-mode chat streams now survive quiet windows and page reloads,
and a reply cut off by a lost connection shows an error instead of
appearing finished. Aborting a resumed subscription only closes your
local stream — call `stopGeneration(chatId)` or pass `stopOnAbort: true`
to stop the run. Also fixed a race where quickly restarting a stream
could break stop and reconnect, and stopping a chat now hands it back to
your other tabs instead of leaving them read-only.
([#4516](https://github.com/triggerdotdev/trigger.dev/pull/4516))
- Updated dependencies:
  - `@trigger.dev/core@4.5.11`

</details>

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2026-08-13 15:48:29 +01:00
Katia Bulatova 480bede0ad feat(webapp,sdk): dashboard agent plan enforcement, component gallery — and fixes (#4516)
Plan enforcement for the dashboard agent — message quota and watch
limits — plus the component gallery, fixes and test hardening from the
same stack (#4548, #4549, #4550, #4552, #4556 merged here).

## Plan enforcement
([TRI-12863](https://linear.app/triggerdotdev/issue/TRI-12863))

**Agent message quota.** The Free-plan allowance becomes a real
server-side limit with a durable counter. New `agent_message_usage`
table keyed `(organization_id, period)` — deliberately not joined to
chats, so deleting a chat can't free quota within the period. Both send
paths count one user message (wakes never count) and refuse at the cap
with `403 message_quota_reached`, which the client renders as an upgrade
panel, never a silent drop. The refusal code is a single shared constant
on both sides.

**Watch limits.** A watch whose window exceeds the plan's
`agentWatchMaxHours`, or that would push the org past its
`agentWatchers` count, is refused with `watch_limit_reached` (409 on the
API, an upgrade hint on the card). Plan limits only tighten the existing
code ceilings (`min(plan, 24h)`, per-chat cap of 3 still applies). A
plan limit of zero means zero, not unlimited. Questions answerable
instantly are answered before any plan refusal — a one-shot consumes no
slot and never sees an upgrade nag.

**Fails open by design.** Cloud ships the actual per-plan numbers
separately (TRI-12863 P0). Until then absent limits resolve to the
unlimited sentinel and the upgrade UI is gated on billing presence —
self-hosted sees no cap, no upsell, with tests proving the fallback.
Both quotas are nudges, not security boundaries: a failing limit read
never blocks a send.

## Component gallery

An admin-only gallery of every agent card state: five
`storybook.agent-*` pages (chat UI, view blocks, report, investigation,
watch) with their shared shell and manifest, demo fixtures, two
demo-only cards, toast examples, and the screenshot script. No LLM and
no data — every state renders from fixtures under
`dashboard-agent/demo/`, never reachable from a production path.
Designers and reviewers can look at every state, including the report
states, without seeding anything.

## And fixes

**SDK: watch-mode chat subscriptions survive quiet windows** (TRI-13065,
TRI-13070) — watch mode keeps reconnecting across empty long-poll
windows and only stops on abort or a settled session; a passive
subscriber can no longer stop a turn it doesn't own (`stopOnAbort` is
explicit, default off). Review findings fixed alongside: a superseded
stream's async teardown no longer removes the live successor's abort
controller or multi-tab claim, and stopping a generation hands the chat
back to the user's other tabs.

**Query boundary pinned end-to-end**
([TRI-11165](https://linear.app/triggerdotdev/issue/TRI-11165)) — a
route-level test drives `api.v1.query` with a real signed environment
JWT (writes refused before ClickHouse, a read passes); `readonly=1` made
non-overridable; a per-turn cap stops the model burning a turn rewriting
a query it can't fix (deterministic SQL errors only — busy/transport
rejections don't count).

**chat.agent durability regression suite**
([TRI-11166](https://linear.app/triggerdotdev/issue/TRI-11166)) —
testcontainers-backed coverage of the two audit criticals (cross-tenant
isolation, no duplicate mid-stream turn, both control-broken) plus
crash-resume, cursor-based refresh, clean rollback of a mid-write turn
failure (torn by a real constraint violation), and OOM-restart replay.

**Investigation sweep backoff** — stale investigations get an attempt
counter and backoff so a poison row can't pin the sweep queue head
(migration `0005`: `sweep_attempts`, `last_sweep_attempt_at`).

## Screenshots

<img width="1440" height="791" alt="Screenshot 2026-08-06 at 00 36 19"
src="https://github.com/user-attachments/assets/6a68cd42-8580-469d-afe7-e28d1eef18e1"
/>

🤖 Generated with [Claude Code](https://claude.com/claude-code)
2026-08-12 13:59:35 +02:00
Chris Arderne ed1bb72fb8 feat: implement cron window spread backend (#4566)
- New DB fields on Schedule and ScheduleInstance
- Use `queueTimestamp` for the "effectiveAt" delayed start time,
propagate it to Clickhouse TaskRun table
- Disable fastpath for delayed jobs
- Add schedule timing logic, API endpoints with windows, persistence
- Calculate phase for every schedule, only persist when window is
non-null
- Additional o11y for phased rollout
2026-08-12 12:24:32 +01:00
Chris Arderne 7b390e5984 feat(cli,webapp): allow deploys with environment API keys (#4561) 2026-08-12 10:11:31 +01:00
Katia Bulatova 0b750d00dd feat(webapp): dashboard agent — Watch (#4525)
Watch is the agent noticing something later: you ask it to tell you when
a condition holds, and it answers when it does — or when it can't any
more.

A watch is a **durable one-shot promise**. The condition is checked on a
schedule by deterministic code (no LLM in the checks), the answer lands
in the chat once, and then the watch is over. Ten kinds: three on a run,
five on a queue, error recurrence, health recovery.

## Stack

Stacked on **#4529** (UI), which is stacked on **#4418** (chat, reports,
investigate). Merge those first. **#4516** (storybook gallery) sits on
top of this branch.

## How to review


[**GUIDEBOOK.md**](https://github.com/triggerdotdev/trigger.dev/blob/feat/dashboard-agent-flows-watch/internal-packages/dashboard-agent/GUIDEBOOK.md)
on this branch is the behaviour reference — it states the conditions
rather than the code, so you can predict what happens without running
anything. "The ten watch kinds, and what makes each fire" and "Creating
a watch" describe exactly this PR, and the tables there are the spec the
code is written against.

## What's inside

- **Ten watch kinds**, one deterministic check each
(`dashboardAgentWatch*Checks.ts`), with the spec union in
`dashboard-agent-contracts/src/watch.ts`.
- **Scheduling** — each watch schedules its own next check; due watches
of one `(environment, cadence)` group can be checked together in one
batch pass, with a sweep as the backstop for expiry, redelivery and
retention.
- **Delivery** — the in-chat wake and card, an optional email alert (new
`DASHBOARD_AGENT_WATCH` alert channel, so it shows on the project's
Alerts page with one-click unsubscribe), and an optional investigation
when the outcome needs attention.
- **Submission ledger** — `watch_submissions`, keyed `(chat_id,
client_request_id)`, so a retried card submission replays the recorded
outcome instead of creating a second watch.
- **Watch token** — a delayed-execution credential accepted only by the
watch endpoints, re-checked against the user's live access on every
tick.
- **Unread work** — the panel polls for wakes that landed while it was
closed, so a chat can go unread and light the launcher dot.

## Key decisions

**A check result is a 4-way, and only two of them are verdicts.**
`satisfied` / `terminal_unsatisfied` are answers; `pending` and
`unavailable` are not. Any exception inside any check is caught in one
place and becomes `unavailable` with an unverified observation — a check
that failed is never evidence.

**A completed window is an answer, and whether it is good or bad news is
declared per kind, never inferred.** There is a table for that in the
guidebook: `run_failed` completing its window is *good* news ("hasn't
failed"), `backlog_drain` completing it is not. One rule overrides the
table: a window that completed on an unverified observation is neutral
and says only that the watch ended without a confirmed answer. **An
unreadable source is never a negative answer** — and, because
investigations only open on `attention`, it never starts one either.

**Identity is `(chat, project, environment)` plus the condition,**
enforced by a partial unique index over active rows
(`watches_chat_active_identity_key`), not by the read-then-insert check.
Cadence, window, note and `ticks` are deliberately not part of it. Two
different chats may watch the same thing — a watch is a promise to a
chat.

**The server resolves the target's name, whatever the model calls it.**
The model can't tell a task queue (`task/<id>`) from a custom queue, so
both spellings are tried and the stored one wins — and the rewrite
happens **before** identity and before the row is written, so the
identity, the checks, the link and the wording all see one spelling.

**Freshness fences.** Depth falls back from the live counter to the
newest 60 s ClickHouse bucket, which only counts as current within 60 s
of now. A non-current reading at or below the *quiet line* is refused as
`unavailable` rather than believed, so a stale empty bucket is never
read as "drained". The stall streak is the one piece of carried state:
it lives in the previous check's facts and *freezes* on an unreadable
reading rather than breaking.

**Chain reliability.** There is no shared cron — each watch (or batch
group) schedules its own next tick, so the failure mode to review is the
chain dying. A failed batch check is caught, the next tick is scheduled
anyway and the run resolves rather than failing, so the chain survives a
check that couldn't run; the sweep re-arms groups and finalizes anything
still active past its deadline, even when delivery isn't configured.
Wake redelivery is id-deduped rather than conditional, because the sweep
can't know whether the user was already told. Access is re-authorized on
**every** check against the primary — replica lag would extend access
the user has already lost.

**Wording lives in one place.** `watch-wording.ts` is read by the card,
banner, toast, email and the agent's own narration, and the numbers come
from the frozen observation rather than a fresh read, so a retry
produces the same sentence. Replay reproduces the **recorded** decision
instead of deciding again — the transcript is append-once, so a second
decision would contradict it forever.

**Cancellation is the ending without an answer** — no resolution, no
wake. One exception, decided during testing: a watch the *user*
cancelled leaves a single neutral transcript line ("Stopped watching
…"), keyed off the watch id so a retry can't repeat it. The other four
reasons stay silent.

**Email is opt-in and only a fired watch emails.** An expiry is narrated
in the chat and nowhere else. Both gates (agent access, a configured
email transport) are checked at subscribe time *and* again at delivery,
and the subscription outcome is frozen on the ledger row so a retry
replays it. Neither gate is a plan check.

**One watch offer per turn.** The prompt and the renderer guard this
independently — if the turn already proposed a watch card, the action
button is dropped, because the card is the better affordance. Two eval
cases pin the prompt side: exactly one offer with the line last and the
button after it, and zero offers when the rendered card already carries
one — deterministic assertions, over a real-model run.

## Testing

Unit tests (vitest, testcontainers, no mocks) under
`apps/webapp/test/dashboardAgentWatch*.test.ts` and
`internal-packages/dashboard-agent/src/watch-*.test.ts` cover the
invariants above: the 4-way check results and the freshness fences,
identity/dedup and the submission ledger, queue-name resolution, the
batch chain surviving a failed check, sweep boundaries and alert-once,
tenancy and the watch token's scope, and the wording snapshot. The
load-bearing ones were verified by control-breaking the guard first and
checking the test goes red.

Live-tested end to end against a local stack, following the guidebook:
all ten watch kinds firing and expiring, cancellation, the email pair (a
fired watch mails, an expired one does not), and watch recovery from a
health report.
2026-08-12 09:51:40 +02:00
Katia Bulatova 4569657923 feat(webapp): dashboard agent — chat, reports, investigate (#4418)
## What & why

This is the system behind the Dashboard Agent — an assistant that
answers questions about a project's runs, errors, queues, deploys and
health, and can investigate failures end to end.

The agent runs as a chat.agent task in its own Trigger project. It has
no access to the main database or ClickHouse; all platform data is read
through the public API using a delegated, read-only user token.

Everything here is behind `canAccessDashboardAgent` and inert with the
flag off. The UI that mounts the panel lands in #4529.

## Stack

`#4418` (this, base) ← `#4529` UI ← `#4525` Watch ← `#4516` storybook
gallery. The scenario/contract reference for the whole stack is
`internal-packages/dashboard-agent/GUIDEBOOK.md` (it lands on the Watch
branch): it states, per feature, what makes each thing happen and where
that is decided.

## What's inside

**Agent runtime and tools** — `internal-packages/dashboard-agent`:
prompt, tool set (API reads, TRQL query, docs, navigation,
evidence/investigations, repo source), conversation compaction, a
prompt-prefix token budget pinned by snapshot test, and sampled
LLM-judged turn evals. The package cannot import webapp server code,
which is what makes the "no DB access" claim structural rather than a
convention.

**Contracts** — `internal-packages/dashboard-agent-contracts`:
`trigger://` URIs, intents, and the block envelope every rendered card
travels in.

**Conversation store** — `internal-packages/dashboard-agent-db`: drizzle
over postgres-js in its own `trigger_dashboard_agent` Postgres schema,
plus one additive migration.

**Auth boundary** — the user-actor token gains an optional environment
claim; one guard (`userActorEnvironment.server.ts`) enforces it so
routes don't each re-derive the rule. Token minting, cap ceiling, and
the RBAC fallback path for self-hosted.

**Transport** — webapp resource routes that mint the token and proxy
each turn, and SDK-side mid-turn reconnect.

**Public API the agent reads through** — orgs, projects, environments,
runs, queue metrics, workers, a run's commit metadata, repo snapshot,
reports, and `POST /api/v1/query`.

**Reports** — the health report's layout is declared once and shared by
the card, the markdown surface and the JSON/MCP surface, so the same
report reads the same in the dashboard, the terminal and an editor.

**Block renderers** — the report and investigation cards the flows above
already emit (`app/components/dashboard-agent/`). The panel that hosts
them, and the rest of the chat UI, is #4529.

**Query safety and CSP** — see below.

## Key decisions

- **The agent is a separate Trigger project, not webapp code.** It reads
platform data over the public API with a delegated user-actor token
whose `cap` ceilings it to read scopes. No Prisma, no ClickHouse, no
webapp imports.
- **The PAT-only auth helper now refuses user-actor tokens.** This is an
intentional behavioral change: its callers consume only a bare userId
and do not enforce delegated-token capabilities. Actor-aware routes
continue through the scoped route builders instead.
- **RBAC fallback builds a delegated token's ability from its own cap**,
never the blanket ability a PAT gets (read-only when the token declares
none). Without this, the agent's read-only cap would buy a write JWT on
self-hosted.
- **Org creation checks RBAC only for user-actor tokens, and only after
the env gate**, so an install with `ORG_CREATION_API_ENABLED` off
returns 404 rather than 403, and an ordinary PAT never consults an
ability the route has no org to scope. Both orderings are pinned by
test.
- **The query path is read-only in depth.** TRQL rejects write
statements at the grammar level (they don't parse, rather than being
filtered), ClickHouse runs with `readonly=1`, and the org/project/env
filters are injected server-side from the credential — the request body
cannot widen scope. An unparseable query denies instead of falling
through to the permissive resource.
- **Document-wide img-src CSP.** Remote images are an
outbound-request/exfiltration surface, so the policy permits only
own-origin/data/blob, the required SSO avatar hosts, and the favicon
endpoint. Operators can add exact origins through CSP_IMG_SRC_ALLOWLIST;
wildcard hosts and bare schemes are intentionally not allowed.
- **The chat transport reconnects on a mid-turn EOF**
(`@trigger.dev/sdk`). A body that ends without a turn-complete is
terminal only when the server says `X-Session-Settled: true`; otherwise
the transport resubscribes from `lastEventId` with bounded backoff, and
any record re-earns the budget. Previously a closed long-poll window or
a proxy restart left the reply stuck as if still generating.
- **Conversations live in their own datastore**, schema-scoped and
foreign-key-free (it references `organizationId`/`userId` by id, because
in cloud it is a different database). It is a display read-model for the
History tab and transport resume; `chat.agent`'s object-store snapshot
remains the model's source of truth.
- **Deterministic first.** Reports and health checks contain no LLM —
they are computed from the same data the dashboard shows, and the model
only narrates and links them. That is what makes a number in an answer
auditable.

## Testing

- 63 new test files, run with `pnpm run test --filter webapp` and
per-package vitest. Heaviest coverage on the auth boundary
(`userActorPatOnlyBoundary`, `userActorTokenClaimsAndScopes`,
`contextlessPatRoutes`, `rbacFallbackBranch`), TRQL read-only, the
report layout, and the SDK reconnect.
- The agent package has a separate eval lane (`pnpm run test:evals`,
`vitest.eval.config.ts`) that hits the real model, so it never runs in
`pnpm test`.
- Live-tested against a local stack scenario by scenario; the GUIDEBOOK
lists the condition each behaviour is expected under, which is what
those runs were checked against.

## Changelog

`.server-changes/dashboard-agent.md`, plus changesets for
`@trigger.dev/core` (report schemas), `@trigger.dev/sdk` (chat
reconnect) and the CLI's `mint-token` help text.
2026-08-11 18:56:14 +02:00
Eric Allam 6449a644b9 feat(webapp,cli,database): track real dev onboarding progress (#4563)
## Summary

The dev environment "Get set up" panel used to be a static list of CLI
commands that only disappeared once your tasks registered, so nothing
ever changed after you ran `init` and people assumed it was stuck. It
now tracks real progress: `trigger init` records the project as
initialized, so step 1 checks off, and the panel updates live as the dev
server connects and your tasks register.

It also adds a prominent "Copy AI agent prompt" button, presented as a
clear alternative ("or") to the manual CLI steps, that copies a
ready-to-paste setup prompt pre-filled with your project reference for
Claude Code, Cursor, or any coding agent.

## Notes

- Adds a `Project.initializedAt` column (migration
`20260811065646_add_project_initialized_at`); the CLI `init` command
calls a new project-scoped `POST /api/v1/projects/:ref/init` best-effort
at the end of setup.
- The `init` scaffold now imports from `@trigger.dev/sdk` instead of the
deprecated `/v3` subpath.

## Screenshots

<img width="2400" height="1794" alt="v7-redesigned-card"
src="https://github.com/user-attachments/assets/c2fb4fa1-9484-4700-8bd3-110d66f5a44e"
/>
2026-08-11 11:43:33 +01:00
Saadi Myftija bd8ce4a50f feat(deployments): split project dependencies and code into separate layers (#4551)
Deploy images previously shipped node_modules and the bundled task code
in a single layer, so every deploy re-pushed and re-pulled the full
dependency tree even when nothing in it changed. The generated
Containerfile now copies `/app/node_modules` as its own layer and the
app files separately. With unchanged dependencies the dependency layer
is identical across deploys, so registries and workers already have it
and only the code layer moves.
2026-08-10 14:44:11 +02:00
github-actions[bot] 72f50c2dad chore: release v4.5.10 (#4440)
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 0s
🚀 Publish Trigger.dev Docker / units (push) Failing after 0s
🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker-v4 (push) Has been skipped
🚀 Publish Trigger.dev Docker / scan-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / scan-supervisor (push) Has been skipped
🧭 Helm Chart Release / lint-and-test (push) Has been cancelled
🚀 Publish Trigger.dev Docker / 📣 Dispatch main image (push) Has been cancelled
🧭 Helm Chart Release / release (push) Has been cancelled
2026-08-07 14:06:43 +01:00
Matt Aitken 04f9c4e1a5 fix(webapp,run-engine,core): drop the hidden debounce ceiling, fail fast on an unusable maxDelay (#4521)
Debouncing with a `delay` longer than an hour did nothing at all.

The engine applied a server-side ceiling on how long a debounced run
could be pushed back, measured from the run's `createdAt` and defaulting
to one hour. A run is only pushed back while its new execution time
stays inside that ceiling, so a `delay` at or above it could never push
anything: the waiting run was released, the trigger started its own run,
and the next trigger repeated it. A `delay: "12h"` produced one run per
trigger, each correctly delayed by 12h, with no error raised and nothing
on the run to show the debounce key had been ignored.

The ceiling is now unset by default. A debounce key with no `maxDelay`
keeps collapsing triggers for as long as they keep arriving, which is
what the docs have always described. Self-hosters who want a bound can
still set `RUN_ENGINE_MAXIMUM_DEBOUNCE_DURATION_MS`.

That has a consequence worth stating plainly, so the docs now carry a
warning for it: with no `maxDelay`, a continuously triggered key never
executes. Set `maxDelay` when the work has to happen eventually.

**Failing fast on an unusable `maxDelay`.** A caller who sets `maxDelay`
no longer than their `delay` hits exactly the dead end described above,
so that pair is now rejected at trigger time instead of silently
behaving as if no debounce were set:

```
debounce.maxDelay (1h) must be longer than debounce.delay (12h). A debounced run is only
pushed back while it stays inside maxDelay, so with these values every trigger would create
its own run.
```

An unparseable `maxDelay` is rejected too, rather than quietly falling
back to no bound at all, and so is a `delay` given as a date rather than
a duration, which could never work because the value is re-applied on
every push.

The same check runs against a configured server ceiling, so a
self-hosted deployment that sets
`RUN_ENGINE_MAXIMUM_DEBOUNCE_DURATION_MS` gets the error rather than the
silent failure this PR is about. With no `maxDelay` and no configured
ceiling, which is the default, there is nothing to conflict with and
nothing is rejected.

The docs, the `TriggerOptions` JSDoc and the engine option all now state
that the room available to push is the gap between `delay` and
`maxDelay`. The run engine suite gains the case that motivated this:
four triggers on one key with a 12h delay now collapse to a single run.
2026-08-07 07:55:35 +00:00
Matt Aitken c084fa6e29 fix(sdk,react-hooks): forward debounce when batch triggering with an array (#4520)
Passing `debounce` in the per-item options of a batch trigger did
nothing when the items were an array. The option was accepted by the
types and by the API, then dropped before the request went out, so every
item created its own run instead of collapsing onto the debounce key.

Four public entry points were affected: `task.batchTrigger`,
`task.batchTriggerAndWait`, `tasks.batchTrigger`, and
`tasks.batchTriggerAndWait`. The streaming (async iterable) forms of the
same calls were already correct, as were `batch.trigger`,
`batch.triggerAndWait`, `batch.triggerByTask`, and
`batch.triggerByTaskAndWait`.

`useTaskTrigger` in `@trigger.dev/react-hooks` had the same silent drop
on the single-trigger path, so that is fixed here too. It also drops
`machine`, `priority`, `region`, `idempotencyKeyTTL`, and
`idempotencyKeyOptions`; those are left alone, since forwarding them is
a behaviour change beyond this bug.

Each batch item builder constructs its options field by field, which is
why one of them could fall behind without anything catching it.
TypeScript did not help: the literal is returned from a `.map` callback
inside `Promise.all`, so excess-property checking never fired against
the `BatchItemNDJSON[]` annotation, and the server's schema silently
strips unknown keys. A misspelled option name therefore reproduced this
bug with no compile error and no server error. Every builder now ends in
`satisfies BatchItemNDJSON`, which does catch it:

```
error TS2561: Object literal may only specify known properties, but 'debounceTYPO'
does not exist in type '{ ... debounce?: {...} | undefined; }'.
Did you mean to write 'debounce'?
```

The new test drives all six public batch surfaces in both array and
async-iterable form and asserts on the NDJSON that actually reaches the
wire. Each item carries a distinct debounce key so the test catches a
wrong item-to-option pairing, not just a wholesale drop.

Fixes #3304
2026-08-06 16:54:24 +01:00
Chris Arderne 9409ddf9bc feat(webapp): add multiple environment API key management (#4390)
## Summary

Projects can create, inspect, expire, and revoke multiple API keys for
each environment. Plaintext values are shown only at creation; stored
credentials are hashed and the API keys page displays only an obfuscated
suffix afterward.

Self-hosted installations support full-access additional keys by
default. Authorization extensions can provide additional access presets
and optional task selection. Additional keys can also mint scoped public
access tokens through the Trigger.dev API without receiving the
environment signing key.

## Feature notes
- Only admin+ can create API keys (Developer can make in Development
branch).
- JWT self-signing will be a server call when used with new `_ak_` keys.
- JWTs with long expiry can keep working even with api key deleted (gets
priveleges from api key, signed with root key)
- Unfiltered session listings intentionally preserve the existing broad
task-read behavior. Filtered listings enforce task-level scopes for
every requested task.
- Buffered runs without a task identifier are not safely authorizable,
so cancel/replay requests fail closed rather than resolving an unscoped
run.
- Batch and waitpoint endpoints intentionally return server-minted,
narrowly scoped public tokens to all callers. These tokens have bounded
lifetimes and may remain valid until expiry after API-key revocation.

## Deployment notes

Deploy the management UI and public-token endpoint with new key creation
disabled. Enable creation for selected organizations after the
authentication path and released SDK have been verified, then expand
availability gradually.

Revoking an API key prevents new bearer requests and new token minting.
Public tokens already minted by that key remain valid until their own
expiration because they are signed by the environment signing key.

## TODO
- [x] Add "Created by" to the key table
- [x] Document that streamed batch ingestion is non-atomic and may
 partially accept items before a validation or authorization error.

## Follow-ups

- [x] Add an organization-level feature flag for the API key management
UI and creation action.
- [x] Document rollout ordering: enable additional-key lookup before
enabling issuance.
- [x] Add a system-wide gate that can stop new key issuance without
disabling authentication for existing keys.
- [x] Replace the generic SDK compatibility warning with the first
published compatible version. Old SDK will mint an unusable token if
given an `_ak_` key.
- [x] Add public documentation covering creation, storage, expiration,
revocation, SDK compatibility, and public-token lifetime behavior.
- [x] Add observability for key creation, revocation, policy preparation
failures, and public-token mint failures.
- [ ] Exercise create, copy-once display, authenticate, mint, expire,
and revoke flows end to end before broad enablement.
2026-08-06 15:27:10 +01:00
Chris Arderne 1a16d61a37 fix(build): support decorator metadata with TypeScript 7 (#4505)
## Summary

Allow projects using TypeScript 7 to enable `emitDecoratorMetadata()`
without adding the TypeScript 6 compiler to every Trigger.dev CLI
installation. Addresses #4500.

## Fix

The extension now resolves TypeScript from the project and
feature-detects the legacy compiler API. TypeScript 5 and 6 continue
using the project's compiler, while TypeScript 7 projects can install
Microsoft's optional `@typescript/typescript6` compatibility package
alongside TypeScript 7.

When no compatible compiler API is available, the build reports an
actionable installation error. The extension documentation includes
setup commands for npm, pnpm, and Bun.

Verified with TypeScript 5, TypeScript 6, TypeScript 7 with and without
the compatibility package, emitted decorator metadata, packed ESM and
CommonJS consumers, package export checks, and typechecking.
2026-08-05 16:33:32 +01:00
Chris Arderne 85f5b37c68 chore: upgrade to TypeScript 7 (#4318)
## Summary

Upgrade the monorepo to TypeScript 7.0.2 and update package build
tooling for compatibility with the native compiler.

## Design

Package builds now use `tshy` 4, while the packages still using `tsup`
move to `tsdown`. The few scripts that depend on the legacy TypeScript
compiler API use an explicit TypeScript 6 alias; declaration portability
coverage invokes the TypeScript 7 CLI directly.

Turbo is updated so workspace tasks can read the regenerated pnpm
lockfile.

---------

Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-08-05 15:49:44 +01:00
nicktrn c01a4f18f4 feat(supervisor): cancel a resumed run's in-flight checkpoint (#4502)
A run controller must call the continue route to resume, so the
supervisor already knows synchronously that any checkpoint still running
for that run is pointless. It only acted on that for the compute path.

The continue route now cancels it for the Kubernetes path too, matching
what completion already does since #4493. Called after the reply so the
runner is never delayed, and skipped when there is no checkpoint client
or when the compute path owns the run. The request is bounded by a 5s
timeout so a hung call cannot leave the handler pending.

`checkpoint_cancel_requests_total{result}` records the outcome, using
the same label names as the delete path where they overlap: `sent`,
`no_client`, `not_applicable`, `http_error`.

No changeset: `CheckpointClient` is a server-only internal API, same as
#4493.

refs TRI-12915
2026-08-05 12:01:14 +01:00
nicktrn 4f69c43e6b feat(supervisor): reclaim a run's checkpoint storage when it finishes (#4493)
When a run reaches a terminal state, ask the checkpoint service to
reclaim the storage its checkpoints occupied. Storage for finished runs
is not otherwise reclaimed, so nothing frees it today.

**Off by default** behind `DELETE_CHECKPOINTS_ON_COMPLETION`, and the
service-side handler ships separately, so merging this changes no
behaviour.

## Where the tenancy comes from

Addressing a run's checkpoints needs org, project, environment,
deployment version and run id. All five are already in hand at
`attempt.complete`, and three are **signed** by the deployment token:

| Value | Source | Trust |
| -- | -- | -- |
| org | claim `org_id` | signed |
| environment | claim `environment_id` | signed |
| deployment version | claim `deployment_version` | signed |
| project ref | `x-trigger-workload-project-ref` header |
runner-supplied |
| run | route param | runner-supplied |

`authorizeWorkloadRequest` previously returned only `environment_id`,
and only in enforce mode, so it now also returns the verified `claims`.
That difference is deliberate and documented on the method: claims are
used to address a run's **own** resources locally, never to scope the
platform, which is why `environmentId` stays enforce-only.

The two runner-supplied values are safe because the signed ones are
outermost - a runner lying about either can only name something inside
its own org and environment, and a project ref that doesn't pair with
its signed environment matches nothing. The run id is read from
`params.runFriendlyId`, the same value the platform just validated,
rather than from the body or a header. Where both a claim and a header
exist (`deployment_version`), the claim wins.

## Placement

The call sits after `reply.json(...)`, so the runner sees no added
latency - the same shape the suspend route already uses. The service
enqueues and returns 202, so it is one fast local hop.

Terminal means `RUN_FINISHED` **or `RUN_PENDING_CANCEL`** - a run
cancelled mid-execution never restores, and skipping it would leave its
storage behind. Retries are excluded deliberately: reclamation is
per-run, so a retry is covered by the final completion.

Also gated on `!snapshotService`, so it stays inert where checkpoints
aren't the kind this reclaims.

## Observability

`checkpoint_delete_requests_total{result}` counts `sent` **and every
reason we decide not to send**: `disabled`, `not_terminal`, `no_claims`,
`no_project_ref`, `http_error`.

The negative labels are the point - without them, "no requests are
happening" looks identical to the feature being switched off.
`no_claims` is reachable even under enforcement, since enforce only
rejects a *present-but-invalid* token; an absent or legacy id still
passes with no claims attached.

## Notes for review

- **No changeset**: `CheckpointClient` is `core/v3/serverOnly`, an
internal service-to-service API rather than customer-facing surface.
- **No `.server-changes/` note**: there is nothing a dashboard user
would notice here. Happy to add one if you disagree.
- `pnpm run typecheck` can't complete in my checkout -
`@trigger.dev/database` fails to build on a missing `tsc` in the pnpm
store, unrelated to this diff. Verified with `tsc --noEmit` against the
supervisor project instead: **zero errors in `apps/supervisor/src`**.
Worth noting it caught a real bug here - the completion response is
wrapped, so the status is `data.result.attemptStatus`.

refs TRI-12789
2026-08-04 15:01:29 +01:00
Chris Arderne 763b5dc582 feat(webapp): enforce scopes for environment API keys (#4389)
## Summary

Environment API keys backed by the additional-key table can authenticate
API requests using their stored effective scopes. Revoked and expired
keys are rejected, branch environments retain their existing routing
behavior, and last-used timestamps are updated on a throttled
best-effort basis.

## Design

API route builders receive the resolved ability and reject restricted
keys on routes without an authorization declaration. Existing
deployment, environment variable, queue, run, task, batch, session, and
waitpoint routes declare the resources they access.

Trigger and batch responses return server-signed public access tokens,
so additional keys never need access to the environment signing secret.
Root-key rotation also keeps public tokens valid for the existing grace
window.

## Feature notes
- Root environment keys remain unrestricted for backward compatibility.
Additional keys enforce their persisted scopes and fail closed on routes
   without an authorization declaration.
- Machine-key requests never exchange one credential for another.
Additional keys cannot retrieve the root key, and rotated root keys are
not upgraded
   during their grace window.
- Public JWT validation remains host-owned, while installed RBAC plugins
continue to supply root-key abilities.
- Unfiltered session and run listings preserve existing broad task-read
behavior. Filtered requests enforce the supplied task identifiers.
- Related-run summaries remain embedded in run retrieval for API
compatibility. Retrieving or mutating a related run independently still
requires
   permission for that run.
- Queue management authorizes at collection scope, matching the queue
permissions currently issued.
- Batch responses deliberately include server-signed public access
tokens for all clients. Selected-task credentials continue using their
original
   credential for per-item authorization.
- Two-phase batches authorize declared task identifiers before creation
and authorize every streamed item. Streaming paths that cannot declare
the
   complete task set remain fail closed.
- Authentication telemetry records successful credential resolution
separately from subsequent resource-authorization failures.
- API keys are high-entropy random tokens. SHA-256 is intentionally used
for deterministic indexed lookup, not password hashing.

## Deployment notes

The schema migration must be present before this code is deployed.
Because bearer resolution runs on every authenticated request, deploy
the resolver with additional-key lookup disabled, verify root-key and
public-token parity, then enable lookup before any additional keys can
be issued.

The multi-task authorization tightening changes the result for narrowly
scoped tokens that request tasks outside their grants. Observe
would-deny results before enforcing that check. Request-idempotency keys
are also newly isolated by environment and task, so a retry crossing the
deployment boundary may execute once more before old cache entries
expire.

## Follow-ups

- [x] Add a system-wide kill switch for additional-key lookup, defaulted
off for the initial deployment.
- [x] Add authentication observability by credential kind, result,
latency, and lookup path without recording credential values.
- [ ] ~Add would-deny observability and an independent enforcement
switch for multi-task authorization.~
- [ ] ~Add an independent switch for server-issued batch tokens while
root-key parity is verified.~
- [ ] Confirm every API route reachable by a restricted key has an
explicit authorization declaration or intentionally fails closed.
- [x] Verify root-key rotation, revoked-key grace, and public-token
validation through each bearer resolver path.
2026-08-03 14:00:29 +01:00
Eric Allam db6228dd1e chore(webapp,core,sdk): upgrade @s2-dev/streamstore to 0.25 and migrate S2 hosts (#4349) 2026-08-01 11:33:34 +01:00
nicktrn a91c08c731 fix(core): retry run start-attempt on transient connection errors (#4441)
## What
`startRunAttempt` — the run controller's first call when a run starts —
had no retry on transient connection errors. A brief connection blip on
that call would abandon the start and send the run back through the
queue, delaying its first attempt.

This adds a jittered backoff retry, matching the existing
`continueRunExecution` path with a shorter budget, so a transient blip
is ridden out in place instead of bouncing the run.

## Why a shorter budget
The continue path retries generously. Start-attempt keeps a tighter
budget (6 attempts, ~25-40s jittered) so it rides out a transient blip
but never keeps retrying past the point the run would already have been
requeued.

## Safety
Retrying is safe: start-attempt is guarded server-side by the snapshot
id — a retry after a start has already committed is rejected, so it can
never double-start an attempt. A pure connection error (the common case)
never reached the server.

## Scope
One retry-options object on `startRunAttempt`; no other behavior change.
Warm starts share this path and get the same resilience.
2026-07-31 13:57:45 +00:00
claude[bot] 17d849b2d6 feat(cli): expose region option on the MCP trigger_task tool (#4439)
<!-- ccr-slack-attribution -->
_Requested by **Eric Allam** · [Slack
thread](https://triggerdotdev.slack.com/archives/C0BEM9Z73TM/p1785491472104199)_

## Checklist

- [ ] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [x] The PR title follows the convention.
- [ ] I ran and tested the code works

---

## Testing

Static checks only, all clean:

- `pnpm run typecheck --filter trigger.dev`
- `pnpm run format`
- `pnpm run lint`

No live task was triggered against a running project, so the "ran and
tested" box above is left unchecked.

---

## Changelog

**Before:** triggering a task through the MCP server always ran it in
the project's default region. There was no way to pick one.

**After:** the `trigger_task` tool accepts an optional `region` option,
so you can choose the region a run executes in.

**How:** `region: z.string().optional()` was added to
`TriggerTaskInput.options` in `packages/cli-v3/src/mcp/schemas.ts`. No
call-site change was needed — `tools/tasks.ts` passes `options` through
verbatim, and `TriggerTaskRequestBody.options.region` already existed.
The tool description in `docs/mcp-tools.mdx` gained a matching line, and
a patch changeset is included.

There is no batch-trigger MCP tool, so there is no sibling tool to
mirror this change on.

---

## Screenshots

N/A — no UI changes.

Co-authored-by: Claude <noreply@anthropic.com>
2026-07-31 11:54:42 +01:00
github-actions[bot] 86b948b47a chore: release v4.5.9 (#4408)
🚀 Publish Trigger.dev Docker / units (push) Failing after 0s
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 1s
🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker-v4 (push) Has been skipped
🚀 Publish Trigger.dev Docker / scan-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / scan-supervisor (push) Has been skipped
🧭 Helm Chart Release / lint-and-test (push) Has been cancelled
🚀 Publish Trigger.dev Docker / 📣 Dispatch main image (push) Has been cancelled
🧭 Helm Chart Release / release (push) Has been cancelled
2026-07-30 10:12:14 +01:00
Chris Arderne 2f1734c858 fix(core,webapp): redact sensitive fields in logs by default and cap their size (#4401) 2026-07-29 17:59:47 +01:00
Chris Arderne 8ebc8a41af fix(webapp,redis-worker): stop logging raw metadata, alert payloads, and job items (#4403) 2026-07-29 17:59:36 +01:00
Chris Arderne 878c15811a fix(cli): redact environment values from build debug logs (#4420) 2026-07-29 16:36:25 +00:00