github-featured/triggerdotdev--trigger.dev
2
0
Fork 0
Code Issues Pull Requests Deployments Models Agent Notes Packages Projects Releases Wiki Activity
7,847 Commits 300 Branches 4,167 Tags
feat/hosted-webhooks-api
Commit Graph

798 Commits

Author SHA1 Message Date
Eric Allam 494d8e9964 fix(cli,core): fail indexing on duplicate webhook ids 
Two webhook() declarations sharing an id used to silently overwrite each other in the worker manifest. Indexing now fails with the colliding ids and their file paths, matching how duplicate task ids are already handled.
 2026-08-16 14:33:45 +01:00
Eric Allam c0b84595a3 feat(webapp): hosted webhook ingress, delivery pipeline, and dashboard (#4344) 
## Summary

The server half of hosted webhooks: the public ingress endpoint,
signature verification, the delivery pipeline (Postgres partitioned
storage + ClickHouse for ordering), the in-app partition manager, the
HTTP API, and the dashboard (Deliveries, Endpoints, and the in-app test
console).

The public SDK and docs half is #4537. That PR carries the user-facing
API (`webhook()`, `chat.event` / `chat.channels`, the
`@trigger.dev/slack` connector) and builds on the shared
`@trigger.dev/core` schemas that ship here.

## Shipping behind a flag

A `WEBHOOK_ENABLED` env var (default off) gates the public ingress route
and the engine worker plus partition cron, so merging and deploying this
changes nothing in production until it is flipped on per environment.
The dashboard is separately gated per org by the `hasWebhooksAccess`
feature flag.

## Note on packages

This PR includes the `@trigger.dev/core` schema additions the server
compiles against, but carries no changeset. Core is not consumed
independently of the SDK, so it is released together with the SDK via
#4537. Keeping its changeset off `main` means no release cut from `main`
publishes it early.
 2026-08-16 14:33:42 +01:00
Chris Arderne 3e7964e7fa feat: surface cron windows in webapp, cli, sdk (#4572) 
## Summary

Adds execution-window product surfaces for both declarative and
imperative schedules.

- Declarative schedules can set `window` through `schedules.task()`,
with support for whole-minute, hour, and percentage values.
- Imperative schedules can create, update, clear, and inspect windows
through the API and dashboard.
- Schedule API responses preserve `nextRun` as the nominal CRON time and
expose `nextRunEffectiveAt` as the stable assigned time.
- The dashboard displays configured windows alongside assigned
upcoming-run times.
- Deploy output summarizes declarative schedules and suggests adding a
wider window when the default 60-second placement range is used.

## Design

Window validation remains authoritative on the server and ensures each
window is compatible with the schedule cadence. Omitting a window uses
the default 60-second range, while explicit zero-duration windows remain
supported.

Deployment summaries are derived from the deployment's stored task
metadata, so they reflect the declarations associated with that
deployment.
 2026-08-14 10:07:14 +01:00
nicktrn fa7eea39d8 fix(core): stop custom metric exporters breaking the metrics export (#4613) 
## Summary

Projects that configure their own `metricExporters` or `metricReaders`
in `trigger.config.ts` were losing task metrics on nearly every run, and
seeing an unexplained `Failed to flush tracingSDK` alongside
`OTLPExporterError: Bad Request` in their run logs. Spans and logs kept
working, so the runs otherwise looked healthy.

## Root cause and fix

Every configured exporter gets its own `PeriodicExportingMetricReader`,
and `meterProvider.forceFlush()` fans out across all readers with
`Promise.all`, so two collections can land on the same millisecond.
`@opentelemetry/host-metrics` divides by the elapsed interval to compute
`process.cpu.utilization`
([common.ts](https://github.com/open-telemetry/opentelemetry-js-contrib/blob/main/packages/host-metrics/src/stats/common.ts)),
so a zero interval yields `0/0`. `JSON.stringify(NaN)` is `null`, and a
collector rejects `"asDouble": null` with a 400 that drops the
**entire** request, not just the offending point.

`flush()` and `shutdown()` now walk the metric readers one at a time, so
collections can no longer share a timestamp. Each reader is isolated, so
one failing reader cannot skip the readers behind it, and every failure
is logged with the reader that produced it. The first error is still
rethrown, so callers see failures exactly as before.

As a second layer, non-finite data points are dropped just before our
own export, so a metric that divides by zero cannot take the rest of the
batch with it. Exporters and readers supplied through
`trigger.config.ts` are untouched by that filter and still receive raw
data.

The trade-off is that configured exporters now flush after the built-in
one rather than alongside it, so flush latency is the sum rather than
the max.

An internal test package's dependency on core was replaced with a local
helper, because core now needs that package in `devDependencies` and the
two together formed a workspace cycle.

## Verification

Tested against a real collector in a container: a batch containing a
`NaN` reading is rejected with a 400 without the fix and accepted with
it, and a single flush is asserted to collect from one reader at a time.
 2026-08-14 08:40:07 +01:00
github-actions[bot] 6685cbd599 chore: release v4.5.11 (#4557) 
## Summary
4 new features, 24 improvements, 10 bug fixes.

## Highlights

- Allow `trigger deploy` to authenticate with an environment API key
from `TRIGGER_ACCESS_TOKEN`.
([#4561](https://github.com/triggerdotdev/trigger.dev/pull/4561))

## Improvements
- Chat in the browser now reconnects when the connection drops mid-turn,
instead of leaving the reply stuck as if it were still generating.
Reports can be fetched as structured data with the `json` format, and
the shortest report period is now one minute (`1m`, `30m`, `1h`, `7d`).
The `mint-token` command's help is clearer too: a token minted without
`--cap` is read-only, and `--ttl` shows the correct maximum lifetime of
7 days.
([#4418](https://github.com/triggerdotdev/trigger.dev/pull/4418))
- The dev environment onboarding now tracks real progress. After you run
`init`, the setup checklist marks your project as initialized, and it
updates live as your dev server connects and your tasks register. The
blank state also adds a "Copy AI agent prompt" button that copies a
ready-to-paste setup prompt (pre-filled with your project reference) for
Claude Code, Cursor, or any coding agent.
([#4563](https://github.com/triggerdotdev/trigger.dev/pull/4563))
  
The `init` scaffold now imports from `@trigger.dev/sdk` instead of the
deprecated `@trigger.dev/sdk/v3` subpath.
- Deployed images now ship dependencies and bundled task code as
separate layers. Repeat deploys with unchanged dependencies typically
push and pull far less data, making deploys and worker image pulls
faster.
([#4551](https://github.com/triggerdotdev/trigger.dev/pull/4551))
- The current-worker API now reports each task's queue, so you can see
which tasks write to a given queue.
([#4525](https://github.com/triggerdotdev/trigger.dev/pull/4525))
- Watch-mode chat streams now survive quiet windows and page reloads,
and a reply cut off by a lost connection shows an error instead of
appearing finished. Aborting a resumed subscription only closes your
local stream — call `stopGeneration(chatId)` or pass `stopOnAbort: true`
to stop the run. Also fixed a race where quickly restarting a stream
could break stop and reconnect, and stopping a chat now hands it back to
your other tabs instead of leaving them read-only.
([#4516](https://github.com/triggerdotdev/trigger.dev/pull/4516))

## Server changes

These changes affect the self-hosted Docker image and Trigger.dev Cloud:

- The dashboard agent now has a monthly message allowance and plan-based
limits on watches. Queries stay read-only with clearer errors when busy,
and messages with unusual characters no longer fail to send.
([#4516](https://github.com/triggerdotdev/trigger.dev/pull/4516))
- Meet the dashboard agent: a chat in every environment that answers
questions about your runs, queues, errors and health with real data and
links, replacing Ask AI everywhere it used to appear. Investigate a
failed run, an error, a backed-up queue or a run that hasn't started to
get a worked-through answer — what happened, why, and how to fix it,
with every claim linked to the runs, errors and deploys behind it. It
reads your data read-only, works on preview and dev branches with that
branch's own data, and reads the same everywhere — dashboard, terminal,
editor. A very long chat keeps working: the agent summarises the earlier
part and carries on.
  
**Watch…** on a run, queue, error or the health report tells you when
things change: a run finishes, a queue clears or grows past a number you
pick, an error comes back, an environment recovers. The answer arrives
in the chat and, if you want, by email, Slack or webhook — and the agent
can look into bad news on its own. A watch reaches you on any browser
you sign in from, without opening the chat first.
  
A sample of conversations is scored automatically so the agent keeps
getting better; only the score and a one-line summary are kept, never
your messages, data or code, and we can switch it off for your
organization on request. Ask the agent instead of the Docs buttons in
page headers — they stay there when the agent isn't available to you.
Separately, a queue's wait times, peak depth, throughput and throttling
can now be read from the API.
([#4418](https://github.com/triggerdotdev/trigger.dev/pull/4418))
- Add backend support for delaying cron schedules within a specified
window with a minimum of 60 seconds.
([#4566](https://github.com/triggerdotdev/trigger.dev/pull/4566))
- Reduced recurring background database load from the billing-limit
recovery check, so paused environments are reconciled with less
overhead.
([#4590](https://github.com/triggerdotdev/trigger.dev/pull/4590))
- Validating a schedule when deploying or updating a schedule now does
less work on projects with many preview branches, so those operations
stay fast as branches accumulate.
([#4598](https://github.com/triggerdotdev/trigger.dev/pull/4598))
- Project pages now load faster for projects with a large number of
preview branches, by no longer loading archived branch environments that
aren't shown.
([#4595](https://github.com/triggerdotdev/trigger.dev/pull/4595))
- Database queries that filter on a list of values now reuse cached
query plans more consistently, instead of forcing the database to
re-plan whenever the list length changes.
([#4480](https://github.com/triggerdotdev/trigger.dev/pull/4480))
- Routine cleanup of old dashboard agent data now runs on its own
schedule.
([#4599](https://github.com/triggerdotdev/trigger.dev/pull/4599))
- Database connection metrics are now reported for every configured
database connection instead of only the primary one, and stay accurate
regardless of connection type.
([#4541](https://github.com/triggerdotdev/trigger.dev/pull/4541))
- Deployment-related API endpoints now draw from their own generous rate
limit budget, configurable via the `DEPLOYMENT_RATE_LIMIT_*` environment
variables, so runtime API traffic no longer competes with deployments
for the same per-environment budget.
([#4565](https://github.com/triggerdotdev/trigger.dev/pull/4565))
- Deleting or editing a secret environment variable is now fast and no
longer slows down as a project accumulates variables.
([#4555](https://github.com/triggerdotdev/trigger.dev/pull/4555))
- Speed up personal access token lookups by indexing them on their owner
([#4588](https://github.com/triggerdotdev/trigger.dev/pull/4588))
- Switching project or organization in the sidebar now keeps you on the
same page instead of sending you back to Tasks. Pages for a specific
run, deploy or other single item open the matching list instead.
([#4585](https://github.com/triggerdotdev/trigger.dev/pull/4585))
- Reduced database load when loading the dashboard by removing an unused
organization member count that was being calculated on every page
navigation.
([#4587](https://github.com/triggerdotdev/trigger.dev/pull/4587))
- The environment variables page now loads a page at a time, keeping it
fast for projects with a large number of variables. Search matches
variable names across every page.
([#4597](https://github.com/triggerdotdev/trigger.dev/pull/4597))
- Groundwork for an alternative database connection driver, gated behind
configuration and disabled by default, so there is no change to default
behavior.
([#4539](https://github.com/triggerdotdev/trigger.dev/pull/4539))
- Deleting an alert channel is now fast and no longer slows down as a
project builds up alert history.
([#4554](https://github.com/triggerdotdev/trigger.dev/pull/4554))
- Reduced internal overhead on the API under high load.
([#4532](https://github.com/triggerdotdev/trigger.dev/pull/4532))
- Out-of-date upgrade prompts no longer appear in the dashboard: the
"V4" badges and the notices saying preview branches and the queues table
need V4 have been removed. The side menu still warns you when a project
is on v3, with updated wording and a link to the v4 upgrade guide.
([#4589](https://github.com/triggerdotdev/trigger.dev/pull/4589))
- Make background worker registration cheaper for projects with many
scheduled tasks by scoping declarative schedule reconciliation to the
current environment and dropping redundant schedule lookups.
([#4577](https://github.com/triggerdotdev/trigger.dev/pull/4577))
- Speed up setting and importing environment variables for projects with
many variables.
([#4579](https://github.com/triggerdotdev/trigger.dev/pull/4579))
- Loading the deployments list is now faster, especially when filtering
by deployment status on projects with many deployments.
([#4591](https://github.com/triggerdotdev/trigger.dev/pull/4591))
- Fixed the billing limits page timing out for organizations with many
preview branches, especially while a spend limit was being enforced. The
page now loads quickly, so you can raise or resolve your limit without
delay. ([#4594](https://github.com/triggerdotdev/trigger.dev/pull/4594))
- Fix the Concurrency page showing the plan's default concurrency for
the dev environment instead of the environment's actual limit.
([#4596](https://github.com/triggerdotdev/trigger.dev/pull/4596))
- Creating an organization sometimes left you back on the creation form
even though the organization had already been created, so clicking
Create again made a duplicate. Creating an organization now completes
and takes you to your new organization.
([#4530](https://github.com/triggerdotdev/trigger.dev/pull/4530))
- Ensure creating a project completes instead of returning to its
creation form after a navigation error.
([#4584](https://github.com/triggerdotdev/trigger.dev/pull/4584))
- Renaming a project now keeps you on the project settings page and
tells you what happened, instead of silently moving you to the tasks
page or clearing the form with no explanation.
([#4601](https://github.com/triggerdotdev/trigger.dev/pull/4601))
- Fixed support threads showing no account details for some customers,
so the team can see your plan, organizations and projects when you get
in touch.
([#4575](https://github.com/triggerdotdev/trigger.dev/pull/4575))
- In the light theme, the Format, Clear and Copy buttons on the query
editor no longer blend into the query text behind them.
([#4592](https://github.com/triggerdotdev/trigger.dev/pull/4592))
- The health report now says start latency is "unknown" when there is no
data for it, instead of showing a healthy-looking 0ms
([#4544](https://github.com/triggerdotdev/trigger.dev/pull/4544))
- Realtime streams written inside a chat session run now use the same
backend as the session itself, and runs are no longer created against a
backend that cannot serve them.
([#4564](https://github.com/triggerdotdev/trigger.dev/pull/4564))
- The grouped "watch updates" notification now shows the total number of
results waiting, instead of only the most recent batch's count.
([#4525](https://github.com/triggerdotdev/trigger.dev/pull/4525))

<details>
<summary>Raw changeset output</summary>

# Releases
## @trigger.dev/build@4.5.11

### Patch Changes

- Updated dependencies:
  - `@trigger.dev/core@4.5.11`
## trigger.dev@4.5.11

### Patch Changes

- Chat in the browser now reconnects when the connection drops mid-turn,
instead of leaving the reply stuck as if it were still generating.
Reports can be fetched as structured data with the `json` format, and
the shortest report period is now one minute (`1m`, `30m`, `1h`, `7d`).
The `mint-token` command's help is clearer too: a token minted without
`--cap` is read-only, and `--ttl` shows the correct maximum lifetime of
7 days.
([#4418](https://github.com/triggerdotdev/trigger.dev/pull/4418))
- Allow `trigger deploy` to authenticate with an environment API key
from `TRIGGER_ACCESS_TOKEN`.
([#4561](https://github.com/triggerdotdev/trigger.dev/pull/4561))
- The dev environment onboarding now tracks real progress. After you run
`init`, the setup checklist marks your project as initialized, and it
updates live as your dev server connects and your tasks register. The
blank state also adds a "Copy AI agent prompt" button that copies a
ready-to-paste setup prompt (pre-filled with your project reference) for
Claude Code, Cursor, or any coding agent.
([#4563](https://github.com/triggerdotdev/trigger.dev/pull/4563))

The `init` scaffold now imports from `@trigger.dev/sdk` instead of the
deprecated `@trigger.dev/sdk/v3` subpath.

- Deployed images now ship dependencies and bundled task code as
separate layers. Repeat deploys with unchanged dependencies typically
push and pull far less data, making deploys and worker image pulls
faster.
([#4551](https://github.com/triggerdotdev/trigger.dev/pull/4551))
- Updated dependencies:
  - `@trigger.dev/core@4.5.11`
  - `@trigger.dev/build@4.5.11`
  - `@trigger.dev/schema-to-json@4.5.11`
## @trigger.dev/core@4.5.11

### Patch Changes

- Chat in the browser now reconnects when the connection drops mid-turn,
instead of leaving the reply stuck as if it were still generating.
Reports can be fetched as structured data with the `json` format, and
the shortest report period is now one minute (`1m`, `30m`, `1h`, `7d`).
The `mint-token` command's help is clearer too: a token minted without
`--cap` is read-only, and `--ttl` shows the correct maximum lifetime of
7 days.
([#4418](https://github.com/triggerdotdev/trigger.dev/pull/4418))
- The current-worker API now reports each task's queue, so you can see
which tasks write to a given queue.
([#4525](https://github.com/triggerdotdev/trigger.dev/pull/4525))
## @trigger.dev/python@4.5.11

### Patch Changes

- Updated dependencies:
  - `@trigger.dev/core@4.5.11`
  - `@trigger.dev/sdk@4.5.11`
  - `@trigger.dev/build@4.5.11`
## @trigger.dev/react-hooks@4.5.11

### Patch Changes

- Updated dependencies:
  - `@trigger.dev/core@4.5.11`
## @trigger.dev/redis-worker@4.5.11

### Patch Changes

- Updated dependencies:
  - `@trigger.dev/core@4.5.11`
## @trigger.dev/rsc@4.5.11

### Patch Changes

- Updated dependencies:
  - `@trigger.dev/core@4.5.11`
## @trigger.dev/schema-to-json@4.5.11

### Patch Changes

- Updated dependencies:
  - `@trigger.dev/core@4.5.11`
## @trigger.dev/sdk@4.5.11

### Patch Changes

- Chat in the browser now reconnects when the connection drops mid-turn,
instead of leaving the reply stuck as if it were still generating.
Reports can be fetched as structured data with the `json` format, and
the shortest report period is now one minute (`1m`, `30m`, `1h`, `7d`).
The `mint-token` command's help is clearer too: a token minted without
`--cap` is read-only, and `--ttl` shows the correct maximum lifetime of
7 days.
([#4418](https://github.com/triggerdotdev/trigger.dev/pull/4418))
- Watch-mode chat streams now survive quiet windows and page reloads,
and a reply cut off by a lost connection shows an error instead of
appearing finished. Aborting a resumed subscription only closes your
local stream — call `stopGeneration(chatId)` or pass `stopOnAbort: true`
to stop the run. Also fixed a race where quickly restarting a stream
could break stop and reconnect, and stopping a chat now hands it back to
your other tabs instead of leaving them read-only.
([#4516](https://github.com/triggerdotdev/trigger.dev/pull/4516))
- Updated dependencies:
  - `@trigger.dev/core@4.5.11`

</details>

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
 2026-08-13 15:48:29 +01:00
Chris Arderne ed1bb72fb8 feat: implement cron window spread backend (#4566) 
- New DB fields on Schedule and ScheduleInstance
- Use `queueTimestamp` for the "effectiveAt" delayed start time,
propagate it to Clickhouse TaskRun table
- Disable fastpath for delayed jobs
- Add schedule timing logic, API endpoints with windows, persistence
- Calculate phase for every schedule, only persist when window is
non-null
- Additional o11y for phased rollout
 2026-08-12 12:24:32 +01:00
Katia Bulatova 0b750d00dd feat(webapp): dashboard agent — Watch (#4525) 
Watch is the agent noticing something later: you ask it to tell you when
a condition holds, and it answers when it does — or when it can't any
more.

A watch is a **durable one-shot promise**. The condition is checked on a
schedule by deterministic code (no LLM in the checks), the answer lands
in the chat once, and then the watch is over. Ten kinds: three on a run,
five on a queue, error recurrence, health recovery.

## Stack

Stacked on **#4529** (UI), which is stacked on **#4418** (chat, reports,
investigate). Merge those first. **#4516** (storybook gallery) sits on
top of this branch.

## How to review


[**GUIDEBOOK.md**](https://github.com/triggerdotdev/trigger.dev/blob/feat/dashboard-agent-flows-watch/internal-packages/dashboard-agent/GUIDEBOOK.md)
on this branch is the behaviour reference — it states the conditions
rather than the code, so you can predict what happens without running
anything. "The ten watch kinds, and what makes each fire" and "Creating
a watch" describe exactly this PR, and the tables there are the spec the
code is written against.

## What's inside

- **Ten watch kinds**, one deterministic check each
(`dashboardAgentWatch*Checks.ts`), with the spec union in
`dashboard-agent-contracts/src/watch.ts`.
- **Scheduling** — each watch schedules its own next check; due watches
of one `(environment, cadence)` group can be checked together in one
batch pass, with a sweep as the backstop for expiry, redelivery and
retention.
- **Delivery** — the in-chat wake and card, an optional email alert (new
`DASHBOARD_AGENT_WATCH` alert channel, so it shows on the project's
Alerts page with one-click unsubscribe), and an optional investigation
when the outcome needs attention.
- **Submission ledger** — `watch_submissions`, keyed `(chat_id,
client_request_id)`, so a retried card submission replays the recorded
outcome instead of creating a second watch.
- **Watch token** — a delayed-execution credential accepted only by the
watch endpoints, re-checked against the user's live access on every
tick.
- **Unread work** — the panel polls for wakes that landed while it was
closed, so a chat can go unread and light the launcher dot.

## Key decisions

**A check result is a 4-way, and only two of them are verdicts.**
`satisfied` / `terminal_unsatisfied` are answers; `pending` and
`unavailable` are not. Any exception inside any check is caught in one
place and becomes `unavailable` with an unverified observation — a check
that failed is never evidence.

**A completed window is an answer, and whether it is good or bad news is
declared per kind, never inferred.** There is a table for that in the
guidebook: `run_failed` completing its window is *good* news ("hasn't
failed"), `backlog_drain` completing it is not. One rule overrides the
table: a window that completed on an unverified observation is neutral
and says only that the watch ended without a confirmed answer. **An
unreadable source is never a negative answer** — and, because
investigations only open on `attention`, it never starts one either.

**Identity is `(chat, project, environment)` plus the condition,**
enforced by a partial unique index over active rows
(`watches_chat_active_identity_key`), not by the read-then-insert check.
Cadence, window, note and `ticks` are deliberately not part of it. Two
different chats may watch the same thing — a watch is a promise to a
chat.

**The server resolves the target's name, whatever the model calls it.**
The model can't tell a task queue (`task/<id>`) from a custom queue, so
both spellings are tried and the stored one wins — and the rewrite
happens **before** identity and before the row is written, so the
identity, the checks, the link and the wording all see one spelling.

**Freshness fences.** Depth falls back from the live counter to the
newest 60 s ClickHouse bucket, which only counts as current within 60 s
of now. A non-current reading at or below the *quiet line* is refused as
`unavailable` rather than believed, so a stale empty bucket is never
read as "drained". The stall streak is the one piece of carried state:
it lives in the previous check's facts and *freezes* on an unreadable
reading rather than breaking.

**Chain reliability.** There is no shared cron — each watch (or batch
group) schedules its own next tick, so the failure mode to review is the
chain dying. A failed batch check is caught, the next tick is scheduled
anyway and the run resolves rather than failing, so the chain survives a
check that couldn't run; the sweep re-arms groups and finalizes anything
still active past its deadline, even when delivery isn't configured.
Wake redelivery is id-deduped rather than conditional, because the sweep
can't know whether the user was already told. Access is re-authorized on
**every** check against the primary — replica lag would extend access
the user has already lost.

**Wording lives in one place.** `watch-wording.ts` is read by the card,
banner, toast, email and the agent's own narration, and the numbers come
from the frozen observation rather than a fresh read, so a retry
produces the same sentence. Replay reproduces the **recorded** decision
instead of deciding again — the transcript is append-once, so a second
decision would contradict it forever.

**Cancellation is the ending without an answer** — no resolution, no
wake. One exception, decided during testing: a watch the *user*
cancelled leaves a single neutral transcript line ("Stopped watching
…"), keyed off the watch id so a retry can't repeat it. The other four
reasons stay silent.

**Email is opt-in and only a fired watch emails.** An expiry is narrated
in the chat and nowhere else. Both gates (agent access, a configured
email transport) are checked at subscribe time *and* again at delivery,
and the subscription outcome is frozen on the ledger row so a retry
replays it. Neither gate is a plan check.

**One watch offer per turn.** The prompt and the renderer guard this
independently — if the turn already proposed a watch card, the action
button is dropped, because the card is the better affordance. Two eval
cases pin the prompt side: exactly one offer with the line last and the
button after it, and zero offers when the rendered card already carries
one — deterministic assertions, over a real-model run.

## Testing

Unit tests (vitest, testcontainers, no mocks) under
`apps/webapp/test/dashboardAgentWatch*.test.ts` and
`internal-packages/dashboard-agent/src/watch-*.test.ts` cover the
invariants above: the 4-way check results and the freshness fences,
identity/dedup and the submission ledger, queue-name resolution, the
batch chain surviving a failed check, sweep boundaries and alert-once,
tenancy and the watch token's scope, and the wording snapshot. The
load-bearing ones were verified by control-breaking the guard first and
checking the test goes red.

Live-tested end to end against a local stack, following the guidebook:
all ten watch kinds firing and expiring, cancellation, the email pair (a
fired watch mails, an expired one does not), and watch recovery from a
health report.
 2026-08-12 09:51:40 +02:00
Katia Bulatova 4569657923 feat(webapp): dashboard agent — chat, reports, investigate (#4418) 
## What & why

This is the system behind the Dashboard Agent — an assistant that
answers questions about a project's runs, errors, queues, deploys and
health, and can investigate failures end to end.

The agent runs as a chat.agent task in its own Trigger project. It has
no access to the main database or ClickHouse; all platform data is read
through the public API using a delegated, read-only user token.

Everything here is behind `canAccessDashboardAgent` and inert with the
flag off. The UI that mounts the panel lands in #4529.

## Stack

`#4418` (this, base) ← `#4529` UI ← `#4525` Watch ← `#4516` storybook
gallery. The scenario/contract reference for the whole stack is
`internal-packages/dashboard-agent/GUIDEBOOK.md` (it lands on the Watch
branch): it states, per feature, what makes each thing happen and where
that is decided.

## What's inside

**Agent runtime and tools** — `internal-packages/dashboard-agent`:
prompt, tool set (API reads, TRQL query, docs, navigation,
evidence/investigations, repo source), conversation compaction, a
prompt-prefix token budget pinned by snapshot test, and sampled
LLM-judged turn evals. The package cannot import webapp server code,
which is what makes the "no DB access" claim structural rather than a
convention.

**Contracts** — `internal-packages/dashboard-agent-contracts`:
`trigger://` URIs, intents, and the block envelope every rendered card
travels in.

**Conversation store** — `internal-packages/dashboard-agent-db`: drizzle
over postgres-js in its own `trigger_dashboard_agent` Postgres schema,
plus one additive migration.

**Auth boundary** — the user-actor token gains an optional environment
claim; one guard (`userActorEnvironment.server.ts`) enforces it so
routes don't each re-derive the rule. Token minting, cap ceiling, and
the RBAC fallback path for self-hosted.

**Transport** — webapp resource routes that mint the token and proxy
each turn, and SDK-side mid-turn reconnect.

**Public API the agent reads through** — orgs, projects, environments,
runs, queue metrics, workers, a run's commit metadata, repo snapshot,
reports, and `POST /api/v1/query`.

**Reports** — the health report's layout is declared once and shared by
the card, the markdown surface and the JSON/MCP surface, so the same
report reads the same in the dashboard, the terminal and an editor.

**Block renderers** — the report and investigation cards the flows above
already emit (`app/components/dashboard-agent/`). The panel that hosts
them, and the rest of the chat UI, is #4529.

**Query safety and CSP** — see below.

## Key decisions

- **The agent is a separate Trigger project, not webapp code.** It reads
platform data over the public API with a delegated user-actor token
whose `cap` ceilings it to read scopes. No Prisma, no ClickHouse, no
webapp imports.
- **The PAT-only auth helper now refuses user-actor tokens.** This is an
intentional behavioral change: its callers consume only a bare userId
and do not enforce delegated-token capabilities. Actor-aware routes
continue through the scoped route builders instead.
- **RBAC fallback builds a delegated token's ability from its own cap**,
never the blanket ability a PAT gets (read-only when the token declares
none). Without this, the agent's read-only cap would buy a write JWT on
self-hosted.
- **Org creation checks RBAC only for user-actor tokens, and only after
the env gate**, so an install with `ORG_CREATION_API_ENABLED` off
returns 404 rather than 403, and an ordinary PAT never consults an
ability the route has no org to scope. Both orderings are pinned by
test.
- **The query path is read-only in depth.** TRQL rejects write
statements at the grammar level (they don't parse, rather than being
filtered), ClickHouse runs with `readonly=1`, and the org/project/env
filters are injected server-side from the credential — the request body
cannot widen scope. An unparseable query denies instead of falling
through to the permissive resource.
- **Document-wide img-src CSP.** Remote images are an
outbound-request/exfiltration surface, so the policy permits only
own-origin/data/blob, the required SSO avatar hosts, and the favicon
endpoint. Operators can add exact origins through CSP_IMG_SRC_ALLOWLIST;
wildcard hosts and bare schemes are intentionally not allowed.
- **The chat transport reconnects on a mid-turn EOF**
(`@trigger.dev/sdk`). A body that ends without a turn-complete is
terminal only when the server says `X-Session-Settled: true`; otherwise
the transport resubscribes from `lastEventId` with bounded backoff, and
any record re-earns the budget. Previously a closed long-poll window or
a proxy restart left the reply stuck as if still generating.
- **Conversations live in their own datastore**, schema-scoped and
foreign-key-free (it references `organizationId`/`userId` by id, because
in cloud it is a different database). It is a display read-model for the
History tab and transport resume; `chat.agent`'s object-store snapshot
remains the model's source of truth.
- **Deterministic first.** Reports and health checks contain no LLM —
they are computed from the same data the dashboard shows, and the model
only narrates and links them. That is what makes a number in an answer
auditable.

## Testing

- 63 new test files, run with `pnpm run test --filter webapp` and
per-package vitest. Heaviest coverage on the auth boundary
(`userActorPatOnlyBoundary`, `userActorTokenClaimsAndScopes`,
`contextlessPatRoutes`, `rbacFallbackBranch`), TRQL read-only, the
report layout, and the SDK reconnect.
- The agent package has a separate eval lane (`pnpm run test:evals`,
`vitest.eval.config.ts`) that hits the real model, so it never runs in
`pnpm test`.
- Live-tested against a local stack scenario by scenario; the GUIDEBOOK
lists the condition each behaviour is expected under, which is what
those runs were checked against.

## Changelog

`.server-changes/dashboard-agent.md`, plus changesets for
`@trigger.dev/core` (report schemas), `@trigger.dev/sdk` (chat
reconnect) and the CLI's `mint-token` help text.
 2026-08-11 18:56:14 +02:00
github-actions[bot] 72f50c2dad chore: release v4.5.10 (#4440)
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 0s
🚀 Publish Trigger.dev Docker / units (push) Failing after 0s
🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker-v4 (push) Has been skipped
🚀 Publish Trigger.dev Docker / scan-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / scan-supervisor (push) Has been skipped
🧭 Helm Chart Release / lint-and-test (push) Has been cancelled
🚀 Publish Trigger.dev Docker / 📣 Dispatch main image (push) Has been cancelled
🧭 Helm Chart Release / release (push) Has been cancelled
 2026-08-07 14:06:43 +01:00
Matt Aitken 04f9c4e1a5 fix(webapp,run-engine,core): drop the hidden debounce ceiling, fail fast on an unusable maxDelay (#4521) 
Debouncing with a `delay` longer than an hour did nothing at all.

The engine applied a server-side ceiling on how long a debounced run
could be pushed back, measured from the run's `createdAt` and defaulting
to one hour. A run is only pushed back while its new execution time
stays inside that ceiling, so a `delay` at or above it could never push
anything: the waiting run was released, the trigger started its own run,
and the next trigger repeated it. A `delay: "12h"` produced one run per
trigger, each correctly delayed by 12h, with no error raised and nothing
on the run to show the debounce key had been ignored.

The ceiling is now unset by default. A debounce key with no `maxDelay`
keeps collapsing triggers for as long as they keep arriving, which is
what the docs have always described. Self-hosters who want a bound can
still set `RUN_ENGINE_MAXIMUM_DEBOUNCE_DURATION_MS`.

That has a consequence worth stating plainly, so the docs now carry a
warning for it: with no `maxDelay`, a continuously triggered key never
executes. Set `maxDelay` when the work has to happen eventually.

**Failing fast on an unusable `maxDelay`.** A caller who sets `maxDelay`
no longer than their `delay` hits exactly the dead end described above,
so that pair is now rejected at trigger time instead of silently
behaving as if no debounce were set:

```
debounce.maxDelay (1h) must be longer than debounce.delay (12h). A debounced run is only
pushed back while it stays inside maxDelay, so with these values every trigger would create
its own run.
```

An unparseable `maxDelay` is rejected too, rather than quietly falling
back to no bound at all, and so is a `delay` given as a date rather than
a duration, which could never work because the value is re-applied on
every push.

The same check runs against a configured server ceiling, so a
self-hosted deployment that sets
`RUN_ENGINE_MAXIMUM_DEBOUNCE_DURATION_MS` gets the error rather than the
silent failure this PR is about. With no `maxDelay` and no configured
ceiling, which is the default, there is nothing to conflict with and
nothing is rejected.

The docs, the `TriggerOptions` JSDoc and the engine option all now state
that the room available to push is the gap between `delay` and
`maxDelay`. The run engine suite gains the case that motivated this:
four triggers on one key with a 12h delay now collapse to a single run.
 2026-08-07 07:55:35 +00:00
Chris Arderne 85f5b37c68 chore: upgrade to TypeScript 7 (#4318) 
## Summary

Upgrade the monorepo to TypeScript 7.0.2 and update package build
tooling for compatibility with the native compiler.

## Design

Package builds now use `tshy` 4, while the packages still using `tsup`
move to `tsdown`. The few scripts that depend on the legacy TypeScript
compiler API use an explicit TypeScript 6 alias; declaration portability
coverage invokes the TypeScript 7 CLI directly.

Turbo is updated so workspace tasks can read the regenerated pnpm
lockfile.

---------

Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
 2026-08-05 15:49:44 +01:00
nicktrn c01a4f18f4 feat(supervisor): cancel a resumed run's in-flight checkpoint (#4502) 
A run controller must call the continue route to resume, so the
supervisor already knows synchronously that any checkpoint still running
for that run is pointless. It only acted on that for the compute path.

The continue route now cancels it for the Kubernetes path too, matching
what completion already does since #4493. Called after the reply so the
runner is never delayed, and skipped when there is no checkpoint client
or when the compute path owns the run. The request is bounded by a 5s
timeout so a hung call cannot leave the handler pending.

`checkpoint_cancel_requests_total{result}` records the outcome, using
the same label names as the delete path where they overlap: `sent`,
`no_client`, `not_applicable`, `http_error`.

No changeset: `CheckpointClient` is a server-only internal API, same as
#4493.

refs TRI-12915
 2026-08-05 12:01:14 +01:00
nicktrn 4f69c43e6b feat(supervisor): reclaim a run's checkpoint storage when it finishes (#4493) 
When a run reaches a terminal state, ask the checkpoint service to
reclaim the storage its checkpoints occupied. Storage for finished runs
is not otherwise reclaimed, so nothing frees it today.

**Off by default** behind `DELETE_CHECKPOINTS_ON_COMPLETION`, and the
service-side handler ships separately, so merging this changes no
behaviour.

## Where the tenancy comes from

Addressing a run's checkpoints needs org, project, environment,
deployment version and run id. All five are already in hand at
`attempt.complete`, and three are **signed** by the deployment token:

| Value | Source | Trust |
| -- | -- | -- |
| org | claim `org_id` | signed |
| environment | claim `environment_id` | signed |
| deployment version | claim `deployment_version` | signed |
| project ref | `x-trigger-workload-project-ref` header |
runner-supplied |
| run | route param | runner-supplied |

`authorizeWorkloadRequest` previously returned only `environment_id`,
and only in enforce mode, so it now also returns the verified `claims`.
That difference is deliberate and documented on the method: claims are
used to address a run's **own** resources locally, never to scope the
platform, which is why `environmentId` stays enforce-only.

The two runner-supplied values are safe because the signed ones are
outermost - a runner lying about either can only name something inside
its own org and environment, and a project ref that doesn't pair with
its signed environment matches nothing. The run id is read from
`params.runFriendlyId`, the same value the platform just validated,
rather than from the body or a header. Where both a claim and a header
exist (`deployment_version`), the claim wins.

## Placement

The call sits after `reply.json(...)`, so the runner sees no added
latency - the same shape the suspend route already uses. The service
enqueues and returns 202, so it is one fast local hop.

Terminal means `RUN_FINISHED` **or `RUN_PENDING_CANCEL`** - a run
cancelled mid-execution never restores, and skipping it would leave its
storage behind. Retries are excluded deliberately: reclamation is
per-run, so a retry is covered by the final completion.

Also gated on `!snapshotService`, so it stays inert where checkpoints
aren't the kind this reclaims.

## Observability

`checkpoint_delete_requests_total{result}` counts `sent` **and every
reason we decide not to send**: `disabled`, `not_terminal`, `no_claims`,
`no_project_ref`, `http_error`.

The negative labels are the point - without them, "no requests are
happening" looks identical to the feature being switched off.
`no_claims` is reachable even under enforcement, since enforce only
rejects a *present-but-invalid* token; an absent or legacy id still
passes with no claims attached.

## Notes for review

- **No changeset**: `CheckpointClient` is `core/v3/serverOnly`, an
internal service-to-service API rather than customer-facing surface.
- **No `.server-changes/` note**: there is nothing a dashboard user
would notice here. Happy to add one if you disagree.
- `pnpm run typecheck` can't complete in my checkout -
`@trigger.dev/database` fails to build on a missing `tsc` in the pnpm
store, unrelated to this diff. Verified with `tsc --noEmit` against the
supervisor project instead: **zero errors in `apps/supervisor/src`**.
Worth noting it caught a real bug here - the completion response is
wrapped, so the status is `data.result.attemptStatus`.

refs TRI-12789
 2026-08-04 15:01:29 +01:00
Chris Arderne 763b5dc582 feat(webapp): enforce scopes for environment API keys (#4389) 
## Summary

Environment API keys backed by the additional-key table can authenticate
API requests using their stored effective scopes. Revoked and expired
keys are rejected, branch environments retain their existing routing
behavior, and last-used timestamps are updated on a throttled
best-effort basis.

## Design

API route builders receive the resolved ability and reject restricted
keys on routes without an authorization declaration. Existing
deployment, environment variable, queue, run, task, batch, session, and
waitpoint routes declare the resources they access.

Trigger and batch responses return server-signed public access tokens,
so additional keys never need access to the environment signing secret.
Root-key rotation also keeps public tokens valid for the existing grace
window.

## Feature notes
- Root environment keys remain unrestricted for backward compatibility.
Additional keys enforce their persisted scopes and fail closed on routes
   without an authorization declaration.
- Machine-key requests never exchange one credential for another.
Additional keys cannot retrieve the root key, and rotated root keys are
not upgraded
   during their grace window.
- Public JWT validation remains host-owned, while installed RBAC plugins
continue to supply root-key abilities.
- Unfiltered session and run listings preserve existing broad task-read
behavior. Filtered requests enforce the supplied task identifiers.
- Related-run summaries remain embedded in run retrieval for API
compatibility. Retrieving or mutating a related run independently still
requires
   permission for that run.
- Queue management authorizes at collection scope, matching the queue
permissions currently issued.
- Batch responses deliberately include server-signed public access
tokens for all clients. Selected-task credentials continue using their
original
   credential for per-item authorization.
- Two-phase batches authorize declared task identifiers before creation
and authorize every streamed item. Streaming paths that cannot declare
the
   complete task set remain fail closed.
- Authentication telemetry records successful credential resolution
separately from subsequent resource-authorization failures.
- API keys are high-entropy random tokens. SHA-256 is intentionally used
for deterministic indexed lookup, not password hashing.

## Deployment notes

The schema migration must be present before this code is deployed.
Because bearer resolution runs on every authenticated request, deploy
the resolver with additional-key lookup disabled, verify root-key and
public-token parity, then enable lookup before any additional keys can
be issued.

The multi-task authorization tightening changes the result for narrowly
scoped tokens that request tasks outside their grants. Observe
would-deny results before enforcing that check. Request-idempotency keys
are also newly isolated by environment and task, so a retry crossing the
deployment boundary may execute once more before old cache entries
expire.

## Follow-ups

- [x] Add a system-wide kill switch for additional-key lookup, defaulted
off for the initial deployment.
- [x] Add authentication observability by credential kind, result,
latency, and lookup path without recording credential values.
- [ ] ~Add would-deny observability and an independent enforcement
switch for multi-task authorization.~
- [ ] ~Add an independent switch for server-issued batch tokens while
root-key parity is verified.~
- [ ] Confirm every API route reachable by a restricted key has an
explicit authorization declaration or intentionally fails closed.
- [x] Verify root-key rotation, revoked-key grace, and public-token
validation through each bearer resolver path.
 2026-08-03 14:00:29 +01:00
Eric Allam db6228dd1e chore(webapp,core,sdk): upgrade @s2-dev/streamstore to 0.25 and migrate S2 hosts (#4349) 2026-08-01 11:33:34 +01:00
nicktrn a91c08c731 fix(core): retry run start-attempt on transient connection errors (#4441) 
## What
`startRunAttempt` — the run controller's first call when a run starts —
had no retry on transient connection errors. A brief connection blip on
that call would abandon the start and send the run back through the
queue, delaying its first attempt.

This adds a jittered backoff retry, matching the existing
`continueRunExecution` path with a shorter budget, so a transient blip
is ridden out in place instead of bouncing the run.

## Why a shorter budget
The continue path retries generously. Start-attempt keeps a tighter
budget (6 attempts, ~25-40s jittered) so it rides out a transient blip
but never keeps retrying past the point the run would already have been
requeued.

## Safety
Retrying is safe: start-attempt is guarded server-side by the snapshot
id — a retry after a start has already committed is rejected, so it can
never double-start an attempt. A pure connection error (the common case)
never reached the server.

## Scope
One retry-options object on `startRunAttempt`; no other behavior change.
Warm starts share this path and get the same resilience.
 2026-07-31 13:57:45 +00:00
github-actions[bot] 86b948b47a chore: release v4.5.9 (#4408)
🚀 Publish Trigger.dev Docker / units (push) Failing after 0s
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 1s
🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker-v4 (push) Has been skipped
🚀 Publish Trigger.dev Docker / scan-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / scan-supervisor (push) Has been skipped
🧭 Helm Chart Release / lint-and-test (push) Has been cancelled
🚀 Publish Trigger.dev Docker / 📣 Dispatch main image (push) Has been cancelled
🧭 Helm Chart Release / release (push) Has been cancelled
 2026-07-30 10:12:14 +01:00
Chris Arderne 2f1734c858 fix(core,webapp): redact sensitive fields in logs by default and cap their size (#4401) 2026-07-29 17:59:47 +01:00
Chris Arderne a81ad4949c feat(database,rbac): add multiple environment API key foundations (#4388) 
Adds the storage model and authorization contracts needed for multiple
environment API keys. Credentials are represented by hashed values,
revocation and expiration state, and persisted effective scopes.

The built-in authorization fallback exposes full-access policy
preparation, while optional authorization extensions can supply
additional presets and task-aware scope generation. This change does not
create, display, or authenticate additional keys.
 2026-07-29 16:24:00 +00:00
Eric Allam 4eb9292cbe feat(webapp,run-engine): queue metrics and health dashboard (#4131) 
## Summary

Three related changes, each independently gated:

**Queue metrics and health.** Per-queue depth, throughput (enqueued,
started, completed), concurrency, whether a queue is throttled, and
scheduling delay (how long a run waits between becoming eligible and
actually starting), plus a per concurrency-key breakdown for keyed
queues. Collected from inside the run queue itself, stored in
ClickHouse, and surfaced on the Queues list, a new per-queue detail
page, the task pages, and the run inspector. The question it answers is
"does this queue have enough concurrency to keep up, and if not, which
key or which limit is the constraint".

**Percent-based queue concurrency limits.** A queue's concurrency
override can now be expressed as a percentage of the environment limit,
stored as the source of truth and re-materialized whenever the
environment limit changes. Absolute overrides above the environment
limit are now **rejected with a 400** instead of being silently capped,
which is a behavior change on `POST
/api/v1/queues/:queue/concurrency/override`.

**The `health` report.** A server-computed verdict on whether work is
flowing, whether the runs that do start are healthy, and whether
telemetry is fresh, rendered as text with sparklines. Available as `GET
/api/v1/reports/:key`, `trigger report`, and the `get_report` MCP tool
(plus a `report` MCP prompt, which shows up as a slash command in hosts
that support prompts).

With the flags off, the Queues page renders the pre-metrics component
verbatim, nothing is emitted, and nothing is written to ClickHouse.

## Configuration

Two independent gates, on purpose. Emission is global so data accrues
for everyone before anyone can look at it; the view is per organization
so it can be turned on for one org at a time without a deploy.

**Runtime flags (no restart)**

| Flag | Store | Gates |
| --- | --- | --- |
| `queue_metrics:enabled` | run-queue Redis key (`"1"`/`"0"`, off by
default) | All emission, gauges and counters. Cached in-process for 10s
with stale-while-revalidate, warmed eagerly at boot so the first op
after a deploy is not dropped. |
| `queue_metrics:gauge_sample_rate` | run-queue Redis key, `0..1` |
Fraction of queue ops that emit a gauge. Counters are never sampled, so
throughput stays exact at any rate. |
| `queueMetricsUiEnabled` | feature-flag catalog: global `FeatureFlag`
row, per-org `Organization.featureFlags` override wins | Whether an org
sees the metrics view at all: the Queues list variant, the queue detail
route, the built-in Queues dashboard, the concurrency-keys endpoint, and
the metrics blocks on task pages and the run inspector. Off by default;
a gated org gets a 404 on the detail route rather than an empty page. |

Both Redis keys are readable and writable from `/admin/queue-metrics`
(super-admin UI, with a live per-shard stream-health table) and
`GET`/`POST /admin/api/v1/queue-metrics` (admin PAT). The admin surface
uses its own Redis client, so it works on any instance regardless of
whether that instance runs the emitter or the consumer.

**Environment variables (boot time)**

| Variable | Default | Notes |
| --- | --- | --- |
| `QUEUE_METRICS_EMIT_ENABLED` | `0` | Constructs the emitter and
injects it into the run engine. Without it the run queue has no emitter
at all. |
| `QUEUE_METRICS_CONSUMER_ENABLED` | `0` | Boots the stream consumer on
this instance. Independent of emission, so consumers can be sized
separately from the API. |
| `QUEUE_METRICS_STREAM_SHARD_COUNT` | `4` | Stream shards, hashed per
queue. |
| `QUEUE_METRICS_CONSUMER_BATCH_SIZE` | `1000` | Poll batch equals
insert batch, so an ack can never outrun a write. |
| `QUEUE_METRICS_REDIS_{HOST,PORT,USERNAME,PASSWORD,TLS_DISABLED}` |
falls back to the run-queue Redis | Set `HOST` to move the metrics
stream onto a dedicated instance so a metrics backlog cannot compete
with the run queue for memory. Self-hosters can leave it unset and get a
single-Redis deployment. |
| `QUEUE_METRICS_COUNTER_STREAM_MAXLEN` | `2000000` shared, `8000000`
dedicated | Bound on how much a stalled consumer can hold. The default
is deliberately lower when the stream shares the queue-critical Redis. |
| `QUEUE_METRICS_COUNTER_ODOMETER_TTL_SECONDS` | `604800` | TTL on the
per-queue cumulative counter key, refreshed on every write, so only
queues idle for the whole window are purged. |
| `QUEUE_METRICS_MAX_QUEUE_NAMES_PER_ENV` | `1000` | Distinct queue
names tracked per environment; overflow collapses into `__overflow__`. |
| `QUEUE_METRICS_MAX_CONCURRENCY_KEYS_PER_QUEUE` | `10000` | Same idea
one level down, per queue. |
| `QUEUE_METRICS_GAUGE_SAMPLE_RATE` | `1` | Default for the live
sample-rate key above. |
| `QUEUE_METRICS_QUERY_TABLES_VISIBLE` | `0` | Lists the queue-metrics
tables in the Query page, its schema docs, the schema API and the AI
query context. Off keeps them unlisted while the feature is dark; a
query naming them still runs either way. |
| `QUEUE_METRICS_CLICKHOUSE_URL` | falls back to the shared wiring |
Runs queue metrics on their own ClickHouse service: the consumer's
inserts and every queue-metrics read go through it, so a metrics-heavy
chart refresh never competes with runs-list or trace reads. Unset
reproduces the previous split exactly (inserts on `CLICKHOUSE_URL`,
reads on the query pool). |
| `QUEUE_METRICS_CLICKHOUSE_READER_URL` | the write URL | Reader split,
so the consumer's inserts can never land on a read endpoint. |
|
`QUEUE_METRICS_CLICKHOUSE_{KEEP_ALIVE_ENABLED,KEEP_ALIVE_IDLE_SOCKET_TTL_MS,MAX_OPEN_CONNECTIONS,LOG_LEVEL,COMPRESSION_REQUEST}`
| `1`, unset, `10`, `info`, `1` | Pool tuning, matching the other
per-workload ClickHouse clients. |

Migrations to apply: ClickHouse `036_create_queue_metrics_v1.sql`, and a
Postgres migration adding the nullable
`TaskQueue.concurrencyLimitOverridePercent`. Both are additive.

## How collection works

Queue operations produce two kinds of signal, and they have opposite
failure modes, so they are handled differently.

**Gauges** (queued, running, queue limit, env queued, env running, env
limit, throttled, plus keys-with-backlog and worst-key wait on keyed
queues) are read *inside* the same Redis script that performs the
enqueue or dequeue, so the reading is atomic with the operation it
describes rather than a racy follow-up read. The script returns them on
its reply and the app forwards them to the stream. Gauges are sampled
and drop-tolerant: they are aggregated with `max`, so a lost reading
costs resolution, never correctness.

**Counters** (enqueued, started, completed, plus nack and dead-lettered)
are cumulative odometers. Each event increments a per-queue key on the
metrics Redis and emits the absolute total, and ClickHouse takes the
difference across buckets at read time. This is the important property
of the design: a summed-delta counter undercounts permanently on any
lost event, while a cumulative one self-heals, because the next
surviving reading restates the whole total. Only bucket granularity can
be lost, never the total. A queue returning after its odometer TTL
expired restarts at 1 and reset detection handles it, which is safe
precisely because expiry only spans a window with no activity.

Both land on one sharded Redis stream. A consumer reads it with a
consumer group, reclaims stale pending entries on a 15s interval rather
than on every poll, maps one entry to one or two ClickHouse rows
(whole-queue and, for keyed queues, per-key), and acks only after the
insert lands. Each batch carries a dedup token derived from its
stream-entry ids, and the target tables set
`non_replicated_deduplication_window`, so a retried batch cannot
double-count either the raw rows or the aggregates that hang off them.
Consumer and emitter both emit OTel metrics
(`queue_metrics.emitter.emitted`,
`queue_metrics.consumer.{entries,rows_inserted,insert_errors,insert_duration,stream_depth,group_lag,pending,lag_unknown}`);
stream depth and group lag are the two worth alerting on, and
`lag_unknown` exists because Redis can report a null lag after a trim,
which must not be read as zero.

## Storage and read path

`queue_metrics_raw_v1` is a short landing table with a 6 hour TTL. Four
aggregate tiers are materialized straight from raw, never cascaded off
each other, each with a 30 day TTL:

- `queue_metrics_v1`, 10 second buckets per queue, the default read path
- `queue_metrics_5m_v1`, 5 minute buckets per queue, for wide ranges and
cross-queue ranking
- `env_metrics_v1`, 10 second buckets per environment, queue-independent
so it stays cheap at any range
- `queue_metrics_ck_v1`, 10 second buckets per concurrency key

Every tier is an MV from raw because the counter states do not survive a
cascade: their merge is order sensitive, so a `-MergeState` chain off
the 10s table inflates the result, and the same property means an
aggregate state may only be merged inside one queue. That constraint is
now enforced by the query engine rather than by reviewer discipline: a
column can declare a `mergeGroupKey`, and any query that references it
without grouping by, or pinning to a single value of, every named key
fails to compile with an actionable message.

On the read side, TRQL gains three tables (`queue_metrics`,
`env_metrics`, and a `queue_metrics_by_key` that is hidden from the
editor, schema docs and schema API but still queryable, so per-key rows
can never silently merge into a plain per-queue query), plus
`deltaSumTimestampMerge` and `quantilesTDigestMerge`. Two schema-level
optimizations ride along: a table can declare coarser rollups, so a
query whose bucket interval is 5 minutes or wider is routed to the 5m
table with no change to the query itself, and it can opt into the
ClickHouse query cache with time bounds floored to a fixed grid, so the
auto-refreshing dashboards actually share cache entries instead of
missing on every tick. Both are caller-side substitutions, so the
printer stays unaware of physical layout.

All of this can also live on its own ClickHouse service. A table
declares the pool its reads run on, the three queue-metrics tables name
the dedicated one, and the ingestion consumer writes through the same
client, so both directions move together with one env var and nothing
else routes differently.

The other engine change is opt-in gap filling: charts can request rows
for empty buckets, where counters zero-fill and gauges carry forward.
Grouped gauge series are densified per group and carried inside a
partition, so a quiet queue's line holds its last value without bleeding
another queue's value into it.

## Queue concurrency limits

`concurrencyLimitOverridePercent` on `TaskQueue` is the source of truth
when an override is set as a percentage; the absolute `concurrencyLimit`
is materialized from it (floored, clamped to at least 1 so a percentage
can never act as a pause, and never above the environment limit). Every
path that changes an environment limit now recalculates the
environment's percent-based overrides afterwards, outside the
transaction, and pushes changed limits to the engine. The push is
attempted even when the stored value did not change, so a previously
failed sync self-heals rather than leaving the database and the engine
diverged; paused queues are skipped so a recalculation cannot
effectively unpause one.

The API accepts exactly one of `concurrencyLimit` or `percent`, and the
reject-instead-of-clamp change above means a request asking for more
than the environment allows now fails loudly. The percent bound (greater
than 0, at most 100) is defined once and shared by the zod schema, the
dashboard mutation handler and the service, so the three cannot drift.

The concurrency-keys table on a queue is now paginated against the
ClickHouse per-key tier, ranked by peak backlog with the total on every
row from a single scan, and only the keys on the current page are
enriched with live counts from Redis. That replaces a hard top-50 cap
with something whose cost is a function of page size rather than key
cardinality.

## The health report

`GET /api/v1/reports/:key?period=&format=markdown|ansi|json`. The
verdict is computed on the server and is deterministic, not
model-generated. Three independent analyzers run over one input
snapshot: flow (is work moving, and if not, is the cause a limit,
throttling, one bad queue, or dead-lettering), execution (are the runs
that start succeeding, and at what latency), and liveness (how fresh is
the telemetry). When telemetry is genuinely stale, the first two are
forced to unknown and every actionable field is stripped, so no surface
ever advises action off stale data.

Authorization is per query table rather than a blanket query grant: a
JWT must be scoped to every table the report reads (`runs`,
`env_metrics`, `queue_metrics`), so a narrowly scoped token cannot pull
a report that reads more than it was granted. `period` is validated as a
shorthand with a 90 day ceiling at the edge. The report catalog is a
registry of `{ load, interpret }` entries, so the next report is a new
entry and no change to the route, the view model, the renderers, the CLI
or the MCP tool.

`trigger mcp` no longer launches the install wizard when stdout is a
TTY, which fixed a real failure: hosts spawn the server over a PTY, so
the wizard would open and the client would time out waiting for a server
that never started. The wizard now needs `trigger mcp --install`.

## The part that is live regardless of every flag

The enqueue and dequeue scripts now return a 2-tuple so a gauge reading
can ride back on the reply. Every return site in the eight affected
scripts is wrapped, and a `nil` original is converted to `false` on the
way out, because a raw `nil` in the first slot would make Lua truncate
the multi-bulk reply and silently drop the gauge on the throttled and
empty-queue paths. The reply shape and the destructuring on the app side
are exercised on every queue operation whether or not metrics are
enabled, so that is the part of `run-engine` worth the closest review.

One behavior fix in the same area: the scheduling-delay anchor is set
only on a run's first entry into the queue. Anchoring it to trigger time
on re-enqueues made waitpoint and checkpoint resumes report the entire
wait as scheduling delay. Queue ordering is untouched, so a re-enqueued
run keeps its position, and nacks deliberately keep the original anchor
because a rolled-back dequeue is the same continuous wait.

A pending-version promotion still anchors to trigger time, on purpose:
that promotion is the run's first real entry into the queue, since the
trigger deliberately held it back waiting for a worker version, and the
TTL is armed at the same point for the same reason. The consequence is
worth naming, because it is a judgement call: a run that waits on a
deployment reports that wait as scheduling delay on its queue, which is
time unrelated to queue capacity.

## Verification

Unit and integration suites across the new package, the run queue, the
mapping layer, the query engine and ClickHouse (including a test that
applies migration 036 through the same splitter CI uses, and a
regression test that inserts the same batch three times to prove the
aggregates do not inflate). Beyond that, the whole path was driven end
to end against a live stack with real runs: emitter to Redis stream to
consumer to ClickHouse to the dashboards, for both the local dev path
and the deployed path where a supervisor drives the dequeue, with
assertions on exact counter reconstruction per queue and per concurrency
key, throttling, environment saturation, scheduling delay, and a
deliberate mid-stream reading drop to confirm the cumulative counters
still reconstruct the correct total. The gated-off state was checked on
every touched surface.

The dedicated ClickHouse service was verified against a second,
separately-schema'd instance: with it configured, the driven counters
reconstruct exactly on the dedicated instance, the shared instance gains
no rows for that window, a read through the query API returns the value
that exists only on the dedicated instance, and a `runs` query still
succeeds (it would fail outright if it were mis-routed to a service
without that table). With the variable unset, the full suite passes
unchanged.

---------

Co-authored-by: Katia Bulatova <katia@trigger.dev>
Co-authored-by: Katia Bulatova <katherine.bulatova@gmail.com>
Co-authored-by: James Ritchie <james@trigger.dev>
 2026-07-29 16:45:24 +01:00
github-actions[bot] d189ce17d3 chore: release v4.5.8 (#4364)
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 1s
🚀 Publish Trigger.dev Docker / units (push) Failing after 0s
🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker-v4 (push) Has been skipped
🚀 Publish Trigger.dev Docker / scan-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / scan-supervisor (push) Has been skipped
🧭 Helm Chart Release / lint-and-test (push) Has been cancelled
🚀 Publish Trigger.dev Docker / 📣 Dispatch main image (push) Has been cancelled
🧭 Helm Chart Release / release (push) Has been cancelled
## Summary
2 new features, 9 improvements, 3 bug fixes.

## Highlights

- Allow additional environment API keys to create scoped public access
tokens through the Trigger.dev API. Use server-issued public access
tokens for batch operations so environment-scoped API keys can read
batch results.
([#4387](https://github.com/triggerdotdev/trigger.dev/pull/4387))

## Improvements
- Preserve the partial assistant message when a chat turn's model stream
fails mid-response. `chat.agent` now passes the recovered partial to
`onTurnComplete`, and `chat.createSession`'s `turn.complete()` keeps it
before rethrowing, instead of dropping the streamed-so-far output.
([#4348](https://github.com/triggerdotdev/trigger.dev/pull/4348))

## Server changes

These changes affect the self-hosted Docker image and Trigger.dev Cloud:

- Favorite any dashboard page to a new Favorites section in the side
menu, and customize the sidebar by renaming favorites, hiding items, and
reordering items and sections.
([#4375](https://github.com/triggerdotdev/trigger.dev/pull/4375))
- List API endpoints now clamp the page size to a maximum of 100.
Requests asking for a larger page size return up to 100 items and keep
paginating, rather than pulling an unbounded page.
([#4360](https://github.com/triggerdotdev/trigger.dev/pull/4360))
- Organizations without billing alerts now get default spend alert
thresholds, so you're notified before usage grows unexpectedly. The
billing limit page no longer pre-selects an option before you've set a
limit and prompts you to configure one. Alert previews now update
immediately after you change your billing limit.
([#4328](https://github.com/triggerdotdev/trigger.dev/pull/4328))
- When you create a Personal Access Token, the generated token now shows
its first and last few characters instead of being fully hidden, so you
can confirm you copied the right value.
([#4363](https://github.com/triggerdotdev/trigger.dev/pull/4363))
- Add metrics to the realtime backend that measure how often a single
changed run is served to multiple subscriptions in one batch.
([#4341](https://github.com/triggerdotdev/trigger.dev/pull/4341))
- Realtime run subscriptions can now be configured to read run data
straight from the primary database, so a run's latest state is never
served from a lagging replica. Off by default; replica reads are
unchanged unless you turn it on.
([#4378](https://github.com/triggerdotdev/trigger.dev/pull/4378))
- SSO and Directory Sync are no longer restricted to Enterprise plans —
get in touch and we can turn them on for your organization whatever plan
you're on.
([#4393](https://github.com/triggerdotdev/trigger.dev/pull/4393))
- Improved supervisor observability: it now reports metrics for its
outbound requests, making failed calls to upstream services easier to
monitor.
([#4350](https://github.com/triggerdotdev/trigger.dev/pull/4350))
- The runs list on a task's page now updates live — run statuses change
and newly triggered runs appear without a manual refresh, matching the
main Runs page.
([#4377](https://github.com/triggerdotdev/trigger.dev/pull/4377))
- Speed up the Batches list page for environments with a large number of
batches, which could previously time out while loading.
([#4361](https://github.com/triggerdotdev/trigger.dev/pull/4361))
- Container startup no longer prints database and ClickHouse connection
strings (with credentials) to the logs.
([#4346](https://github.com/triggerdotdev/trigger.dev/pull/4346))
- The tasks page no longer runs two queries whose results were never
displayed, cutting wasted work on every page load and removing a source
of hidden server errors
([#4380](https://github.com/triggerdotdev/trigger.dev/pull/4380))

<details>
<summary>Raw changeset output</summary>

# Releases
## @trigger.dev/build@4.5.8

### Patch Changes

- Updated dependencies:
  - `@trigger.dev/core@4.5.8`
## trigger.dev@4.5.8

### Patch Changes

- Updated dependencies:
  - `@trigger.dev/core@4.5.8`
  - `@trigger.dev/build@4.5.8`
  - `@trigger.dev/schema-to-json@4.5.8`
## @trigger.dev/core@4.5.8

### Patch Changes

- Allow additional environment API keys to create scoped public access
tokens through the Trigger.dev API. Use server-issued public access
tokens for batch operations so environment-scoped API keys can read
batch results.
([#4387](https://github.com/triggerdotdev/trigger.dev/pull/4387))
## @trigger.dev/python@4.5.8

### Patch Changes

- Updated dependencies:
  - `@trigger.dev/sdk@4.5.8`
  - `@trigger.dev/core@4.5.8`
  - `@trigger.dev/build@4.5.8`
## @trigger.dev/react-hooks@4.5.8

### Patch Changes

- Updated dependencies:
  - `@trigger.dev/core@4.5.8`
## @trigger.dev/redis-worker@4.5.8

### Patch Changes

- Updated dependencies:
  - `@trigger.dev/core@4.5.8`
## @trigger.dev/rsc@4.5.8

### Patch Changes

- Updated dependencies:
  - `@trigger.dev/core@4.5.8`
## @trigger.dev/schema-to-json@4.5.8

### Patch Changes

- Updated dependencies:
  - `@trigger.dev/core@4.5.8`
## @trigger.dev/sdk@4.5.8

### Patch Changes

- Preserve the partial assistant message when a chat turn's model stream
fails mid-response. `chat.agent` now passes the recovered partial to
`onTurnComplete`, and `chat.createSession`'s `turn.complete()` keeps it
before rethrowing, instead of dropping the streamed-so-far output.
([#4348](https://github.com/triggerdotdev/trigger.dev/pull/4348))
- Allow additional environment API keys to create scoped public access
tokens through the Trigger.dev API. Use server-issued public access
tokens for batch operations so environment-scoped API keys can read
batch results.
([#4387](https://github.com/triggerdotdev/trigger.dev/pull/4387))
- Updated dependencies:
  - `@trigger.dev/core@4.5.8`

</details>

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
 2026-07-27 16:40:31 +01:00
Chris Arderne efd0ee8d74 feat(core,sdk): support additional environment API keys (#4387) 
## Summary

Additional environment API keys can use SDK APIs that require public
access tokens. The SDK detects the additional-key format and asks the
Trigger.dev server to mint scoped tokens instead of attempting to sign
them locally.

Root environment keys retain their existing local-signing behavior.
Trigger and batch clients also prefer server-issued tokens returned in
response headers while preserving compatibility with older servers.

## Deployment notes

This package update is safe to publish before servers expose additional
key creation. Existing root keys continue to use the current path, while
an additional key used with an older server fails with an actionable
upgrade error.
 2026-07-27 15:19:27 +00:00
nicktrn 722e240e4d feat(supervisor): add prometheus metric for outbound http requests (#4350)
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 5s
Details
🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped
Details
🚀 Publish Trigger.dev Docker / publish-worker-v4 (push) Has been skipped
Details
🚀 Publish Trigger.dev Docker / scan-webapp (push) Has been skipped
Details
🚀 Publish Trigger.dev Docker / scan-supervisor (push) Has been skipped
Details
🚀 Publish Trigger.dev Docker / units (push) Failing after 6s
Details
🦋 Changesets PR / Create Release PR (push) Has been cancelled
Details
🚀 Publish Trigger.dev Docker / 📣 Dispatch main image (push) Has been cancelled
Details 
Adds Prometheus metrics so the supervisor's outbound HTTP calls are
observable - including client-side failures that previously only
surfaced as a log line.

- `supervisor_outbound_request_total{name, method, status, outcome}` -
counts every outbound request. `outcome` separates a transport failure
(`network_error`), an HTTP error response (`http_error`), a response
that failed schema validation (`invalid_response`), and success (`ok`).
- `supervisor_outbound_request_duration_seconds{name, outcome}` -
latency histogram. Leaner labels than the counter (no `status`) to avoid
bucket×label cardinality; buckets match the existing dequeue-latency
histogram since these calls share the same retrying HTTP client and
long-poll envelope.

Coverage:
- The warm-start request (a one-off `fetch`) - instrumented inline; the
response status code is now also included in the failure log (it was
previously dropped).
- All worker API client calls (`SupervisorHttpClient`: dequeue, run
attempt start/complete, heartbeats, snapshots, continue, suspend,
debug-log, connect) - routed through a single instrumented `request()`
helper that reports via an optional `onHttpRequestComplete` callback on
the client, which the supervisor wires into the counter + histogram.

Low cardinality by design: `name` is a **static per-endpoint label**
(e.g. `dequeue`, `start_run_attempt`), never the interpolated URL - so
no run/snapshot IDs land in labels, mirroring the templated `route`
labels on the inbound HTTP server.

Registered on the existing metrics registry, exposed on `/metrics` with
no new wiring. Internal-only change (no package release needed), so the
changelog note is a single `.server-changes` entry.
 2026-07-23 18:18:58 +01:00
github-actions[bot] aafc333523 chore: release v4.5.7 (#4319)
🚀 Publish Trigger.dev Docker / units (push) Failing after 5s
Details
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 15s
Details
🚀 Publish Trigger.dev Docker / scan-webapp (push) Has been skipped
Details
🚀 Publish Trigger.dev Docker / scan-supervisor (push) Has been skipped
Details
🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped
Details
🚀 Publish Trigger.dev Docker / publish-worker-v4 (push) Has been skipped
Details
🧭 Helm Chart Release / lint-and-test (push) Has been cancelled
Details
🧭 Helm Chart Release / release (push) Has been cancelled
Details
🚀 Publish Trigger.dev Docker / 📣 Dispatch main image (push) Has been cancelled
Details 
## Summary
5 improvements, 5 bug fixes.

## Improvements
- Add `node-24` and `node-26` as supported `runtime` options in
`trigger.config.ts`. The `experimental-node-24` and
`experimental-node-26` names are now deprecated aliases and emit a
deprecation warning; switch to `node-24` / `node-26` instead.
([#4337](https://github.com/triggerdotdev/trigger.dev/pull/4337))
  
  ```ts
  import { defineConfig } from "@trigger.dev/sdk";
  
  export default defineConfig({
  runtime: "node-24",
  project: "<your-project-ref>",
  });
  ```
- Avoid logging task run environment variable values at debug level
([#4336](https://github.com/triggerdotdev/trigger.dev/pull/4336))
- Custom chat agent loops get two ergonomic wins for owning the turn
loop. ([#4304](https://github.com/triggerdotdev/trigger.dev/pull/4304))
  
`chat.writeTurnComplete()` now returns the turn boundary's resume
cursors (`lastEventId` for the output stream and `sessionInEventId` for
the input stream), so you can persist them straight from the task
instead of round-tripping them back from the client.
  
  ```ts
const { lastEventId, sessionInEventId } = await
chat.writeTurnComplete();
  await db.chats.update(chatId, { lastEventId, sessionInEventId });
  ```
  
`chat.pipeAndCapture()` no longer throws when a stream is stopped or
fails. It now returns a `PipeAndCaptureResult` whose `message` holds any
partial output captured before the stop or failure, alongside a typed
`status` (`"complete" | "aborted" | "error"`) and, on failure, the
`error`. Read the message off the result:
  
  ```ts
  const { message, status, error } = await chat.pipeAndCapture(result, {
  signal,
  });
  if (message) conversation.addResponse(message);
  if (status === "error") logger.error("turn failed", { error });
  ```
  
Note: `pipeAndCapture` previously resolved to `UIMessage | undefined`.
Update call sites to read `.message` from the returned result.
- Suppress a build-time warning that could appear in Vite-based projects
when the optional `@ai-sdk/otel` package is not installed.
([#4188](https://github.com/triggerdotdev/trigger.dev/pull/4188))

## Bug fixes
- Fixes intermittent `trigger dev` run crashes where a run could fail at
boot with a cryptic `Cannot find module .../dev-run-worker.mjs` after a
rebuild had cleaned up the build directory the run was launched against.
Dev runs now retry cleanly instead of hard-crashing when their build
directory is missing, the dev watchdog no longer removes the build tree
of a still-running session, and a run assigned to a worker version that
was superseded by a rebuild now fails fast with a clear message instead
of silently hanging until it times out.
([#4276](https://github.com/triggerdotdev/trigger.dev/pull/4276))

## Server changes

These changes affect the self-hosted Docker image and Trigger.dev Cloud:

- Refreshed the side menu: separate organization and account menus, a
new project switcher, and the menu is now resizable by dragging its
edge. The account Profile page has also been redesigned.
([#4066](https://github.com/triggerdotdev/trigger.dev/pull/4066))
- Allow different organization members to use the same development
branch name without sharing or colliding with each other's branch
environments.
([#4323](https://github.com/triggerdotdev/trigger.dev/pull/4323))
- Limit account settings email input to 254 characters.
([#4330](https://github.com/triggerdotdev/trigger.dev/pull/4330))
- Prevent duplicate Staging and Preview environments when account setup
requests overlap
([#4261](https://github.com/triggerdotdev/trigger.dev/pull/4261))
- Fix the docs link on the empty Prompts page, which pointed to a page
that no longer exists.
([#4247](https://github.com/triggerdotdev/trigger.dev/pull/4247))

<details>
<summary>Raw changeset output</summary>

# Releases
## @trigger.dev/build@4.5.7

### Patch Changes

- Updated dependencies:
  - `@trigger.dev/core@4.5.7`
## trigger.dev@4.5.7

### Patch Changes

- Fixes intermittent `trigger dev` run crashes where a run could fail at
boot with a cryptic `Cannot find module .../dev-run-worker.mjs` after a
rebuild had cleaned up the build directory the run was launched against.
Dev runs now retry cleanly instead of hard-crashing when their build
directory is missing, the dev watchdog no longer removes the build tree
of a still-running session, and a run assigned to a worker version that
was superseded by a rebuild now fails fast with a clear message instead
of silently hanging until it times out.
([#4276](https://github.com/triggerdotdev/trigger.dev/pull/4276))
- Add `node-24` and `node-26` as supported `runtime` options in
`trigger.config.ts`. The `experimental-node-24` and
`experimental-node-26` names are now deprecated aliases and emit a
deprecation warning; switch to `node-24` / `node-26` instead.
([#4337](https://github.com/triggerdotdev/trigger.dev/pull/4337))

  ```ts
  import { defineConfig } from "@trigger.dev/sdk";

  export default defineConfig({
    runtime: "node-24",
    project: "<your-project-ref>",
  });
  ```

- Avoid logging task run environment variable values at debug level
([#4336](https://github.com/triggerdotdev/trigger.dev/pull/4336))
- Updated dependencies:
  - `@trigger.dev/core@4.5.7`
  - `@trigger.dev/build@4.5.7`
  - `@trigger.dev/schema-to-json@4.5.7`
## @trigger.dev/core@4.5.7

### Patch Changes

- Add `node-24` and `node-26` as supported `runtime` options in
`trigger.config.ts`. The `experimental-node-24` and
`experimental-node-26` names are now deprecated aliases and emit a
deprecation warning; switch to `node-24` / `node-26` instead.
([#4337](https://github.com/triggerdotdev/trigger.dev/pull/4337))

  ```ts
  import { defineConfig } from "@trigger.dev/sdk";

  export default defineConfig({
    runtime: "node-24",
    project: "<your-project-ref>",
  });
  ```
## @trigger.dev/python@4.5.7

### Patch Changes

- Updated dependencies:
  - `@trigger.dev/sdk@4.5.7`
  - `@trigger.dev/core@4.5.7`
  - `@trigger.dev/build@4.5.7`
## @trigger.dev/react-hooks@4.5.7

### Patch Changes

- Updated dependencies:
  - `@trigger.dev/core@4.5.7`
## @trigger.dev/redis-worker@4.5.7

### Patch Changes

- Updated dependencies:
  - `@trigger.dev/core@4.5.7`
## @trigger.dev/rsc@4.5.7

### Patch Changes

- Updated dependencies:
  - `@trigger.dev/core@4.5.7`
## @trigger.dev/schema-to-json@4.5.7

### Patch Changes

- Updated dependencies:
  - `@trigger.dev/core@4.5.7`
## @trigger.dev/sdk@4.5.7

### Patch Changes

- Custom chat agent loops get two ergonomic wins for owning the turn
loop. ([#4304](https://github.com/triggerdotdev/trigger.dev/pull/4304))

`chat.writeTurnComplete()` now returns the turn boundary's resume
cursors (`lastEventId` for the output stream and `sessionInEventId` for
the input stream), so you can persist them straight from the task
instead of round-tripping them back from the client.

  ```ts
const { lastEventId, sessionInEventId } = await
chat.writeTurnComplete();
  await db.chats.update(chatId, { lastEventId, sessionInEventId });
  ```

`chat.pipeAndCapture()` no longer throws when a stream is stopped or
fails. It now returns a `PipeAndCaptureResult` whose `message` holds any
partial output captured before the stop or failure, alongside a typed
`status` (`"complete" | "aborted" | "error"`) and, on failure, the
`error`. Read the message off the result:

  ```ts
  const { message, status, error } = await chat.pipeAndCapture(result, {
    signal,
  });
  if (message) conversation.addResponse(message);
  if (status === "error") logger.error("turn failed", { error });
  ```

Note: `pipeAndCapture` previously resolved to `UIMessage | undefined`.
Update call sites to read `.message` from the returned result.

- Suppress a build-time warning that could appear in Vite-based projects
when the optional `@ai-sdk/otel` package is not installed.
([#4188](https://github.com/triggerdotdev/trigger.dev/pull/4188))
- Updated dependencies:
  - `@trigger.dev/core@4.5.7`

</details>

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
 2026-07-22 15:51:29 +01:00
Chris Arderne 55a3bf2858 feat(core,cli): add node-24 and node-26 runtimes, deprecate experimental aliases (#4337) 
## Summary

Adds `node-24` and `node-26` as first-class `runtime` options in
`trigger.config.ts`. Previously these Node versions were only reachable
via the `experimental-node-24` / `experimental-node-26` names.

Those experimental names are now **deprecated aliases**: they still
resolve to `node-24` / `node-26` for backwards compatibility, but
loading a config that uses them prints a deprecation warning pointing at
the new name.

```ts
export default defineConfig({
  runtime: "node-24",
  project: "<your-project-ref>",
});
```

## Details

- `ConfigRuntime` (the public config schema) now accepts `node-24` and
`node-26` directly; the internal `BuildRuntime` already supported them,
so base images and the deploy path are unchanged.
- `resolveBuildRuntime` passes the new names straight through and keeps
mapping the experimental aliases to their replacements.
- Renamed the runtime helper from `isExperimentalConfigRuntime` to
`isDeprecatedConfigRuntime` and added `deprecatedRuntimeReplacement` so
the CLI can name the replacement in its warning.
- Docs snippet updated to list the new versions and flag the deprecated
names.
 2026-07-22 14:10:27 +01:00
Chris Arderne dc87b884e7 chore: upgrade to typescript 6 (#4310) 
## Summary

Upgrades the workspace to TypeScript 6.0.3 and applies the compiler,
type, and build configuration changes required to preserve package
layouts and existing runtime behavior, apart from correcting the HTTP
status field used for deployment connection errors.

## Compatibility

- Centralizes TypeScript 6.0.3 through the pnpm workspace catalog.
- Replaces compiler options and module resolution modes that TypeScript
6 no longer accepts.
- Restores explicit Node types where TypeScript 6 no longer includes
them transitively.
- Adds explicit declaration build roots that preserve each package's
existing output layout.
- Patches tsup to stop injecting the removed `baseUrl` option during
declaration builds.
- Uses type-only assertions for stricter typed-array and stream
definitions without changing runtime behavior.
- Reads the EventSource v3 HTTP status from `code`, so deployment
connection errors include it correctly.
- Keeps standalone CLI compatibility fixtures pinned to their existing
TypeScript version and lockfiles.

`turbo run typecheck` and the complete PR test suite are green.
 2026-07-21 13:57:52 +01:00
github-actions[bot] 325b906319 chore: release v4.5.6 (#4317)
🚀 Publish Trigger.dev Docker / units (push) Failing after 0s
Details
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 20s
Details
🚀 Publish Trigger.dev Docker / scan-webapp (push) Has been skipped
Details
🚀 Publish Trigger.dev Docker / scan-supervisor (push) Has been skipped
Details
🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped
Details
🚀 Publish Trigger.dev Docker / publish-worker-v4 (push) Has been skipped
Details
🧭 Helm Chart Release / release (push) Has been cancelled
Details
🧭 Helm Chart Release / lint-and-test (push) Has been cancelled
Details
🚀 Publish Trigger.dev Docker / 📣 Dispatch main image (push) Has been cancelled
Details 
## Summary
5 improvements, 9 bug fixes.

## Breaking changes
- Self-hosted deployments no longer ship shared default credentials;
fresh installs generate their own. If yours still uses a previously
published default, set a unique value before upgrading, or set
`ALLOW_INSECURE_DEFAULT_SECRETS=true` to keep booting while you migrate.
([#4316](https://github.com/triggerdotdev/trigger.dev/pull/4316))

## Improvements
- Require explicit browser approval for CLI and MCP login, with
resilient polling while approval is pending.
([#4316](https://github.com/triggerdotdev/trigger.dev/pull/4316))
- Deployed task telemetry now reports the deployment identifier (e.g.
`deployment_abc123`) in the `worker.id` attribute, instead of an opaque
internal value. Upgrade to get the readable identifier in your own
OpenTelemetry exporters.
([#4316](https://github.com/triggerdotdev/trigger.dev/pull/4316))
- Prevent prototype pollution when applying run metadata operations or
reconstructing nested telemetry attributes, while preserving legitimate
`constructor` and `prototype` fields.
([#4316](https://github.com/triggerdotdev/trigger.dev/pull/4316))
- Add helpers to mint and verify the deployment-scoped token used to
authenticate run controllers to the platform.
([#4316](https://github.com/triggerdotdev/trigger.dev/pull/4316))

## Server changes

These changes affect the self-hosted Docker image and Trigger.dev Cloud:

- Added optional request rate limiting for telemetry ingestion
endpoints.
([#4316](https://github.com/triggerdotdev/trigger.dev/pull/4316))
- Background-worker deployment lookups are now scoped to the
authenticated environment.
([#4316](https://github.com/triggerdotdev/trigger.dev/pull/4316))
- Updating a GitHub App installation from the callback flow is now
scoped to your own organization, so an installation ID belonging to
another organization can no longer be used to refresh that
organization's installation record. The GitHub App installation session
is also now single-use, so completing an installation callback
invalidates its state and it can no longer be replayed.
([#4316](https://github.com/triggerdotdev/trigger.dev/pull/4316))
- Scope schedule and environment-variable writes to the caller's project
and environment
([#4316](https://github.com/triggerdotdev/trigger.dev/pull/4316))
- Reject compute snapshot callbacks that do not match the snapshot
request that created them.
([#4316](https://github.com/triggerdotdev/trigger.dev/pull/4316))
- Require secret-key authentication to initialize the session out
(agent→client) stream, matching the append route.
([#4316](https://github.com/triggerdotdev/trigger.dev/pull/4316))
- Live run and trace subscriptions now validate their identifiers more
strictly and only return data from your own organization.
([#4316](https://github.com/triggerdotdev/trigger.dev/pull/4316))
- Window-function names in the query compiler are now validated against
the allowlist, matching how other function calls are handled.
([#4316](https://github.com/triggerdotdev/trigger.dev/pull/4316))
- Authenticate run controllers to the platform with a signed,
deployment-scoped token.
([#4316](https://github.com/triggerdotdev/trigger.dev/pull/4316))
- Verify that worker actions (starting, completing, and continuing a
run, and reading its snapshots) target a run belonging to the caller's
environment.
([#4316](https://github.com/triggerdotdev/trigger.dev/pull/4316))

<details>
<summary>Raw changeset output</summary>

# Releases
## @trigger.dev/build@4.5.6

### Patch Changes

- Updated dependencies:
  - `@trigger.dev/core@4.5.6`
## trigger.dev@4.5.6

### Patch Changes

- Require explicit browser approval for CLI and MCP login, with
resilient polling while approval is pending.
([#4316](https://github.com/triggerdotdev/trigger.dev/pull/4316))
- Deployed task telemetry now reports the deployment identifier (e.g.
`deployment_abc123`) in the `worker.id` attribute, instead of an opaque
internal value. Upgrade to get the readable identifier in your own
OpenTelemetry exporters.
([#4316](https://github.com/triggerdotdev/trigger.dev/pull/4316))
- Updated dependencies:
  - `@trigger.dev/core@4.5.6`
  - `@trigger.dev/build@4.5.6`
  - `@trigger.dev/schema-to-json@4.5.6`
## @trigger.dev/core@4.5.6

### Patch Changes

- Prevent prototype pollution when applying run metadata operations or
reconstructing nested telemetry attributes, while preserving legitimate
`constructor` and `prototype` fields.
([#4316](https://github.com/triggerdotdev/trigger.dev/pull/4316))
- Require explicit browser approval for CLI and MCP login, with
resilient polling while approval is pending.
([#4316](https://github.com/triggerdotdev/trigger.dev/pull/4316))
- Add helpers to mint and verify the deployment-scoped token used to
authenticate run controllers to the platform.
([#4316](https://github.com/triggerdotdev/trigger.dev/pull/4316))
## @trigger.dev/python@4.5.6

### Patch Changes

- Updated dependencies:
  - `@trigger.dev/core@4.5.6`
  - `@trigger.dev/build@4.5.6`
  - `@trigger.dev/sdk@4.5.6`
## @trigger.dev/react-hooks@4.5.6

### Patch Changes

- Updated dependencies:
  - `@trigger.dev/core@4.5.6`
## @trigger.dev/redis-worker@4.5.6

### Patch Changes

- Updated dependencies:
  - `@trigger.dev/core@4.5.6`
## @trigger.dev/rsc@4.5.6

### Patch Changes

- Updated dependencies:
  - `@trigger.dev/core@4.5.6`
## @trigger.dev/schema-to-json@4.5.6

### Patch Changes

- Updated dependencies:
  - `@trigger.dev/core@4.5.6`
## @trigger.dev/sdk@4.5.6

### Patch Changes

- Updated dependencies:
  - `@trigger.dev/core@4.5.6`

</details>

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
 2026-07-21 12:07:52 +01:00
Chris Arderne 6997aeb05e fix: security release 2026-07-08 (#4316)
⚒️ Publish Worker (v4) / build (supervisor) (push) Has been cancelled
Details
2026-07-21 12:00:58 +01:00
github-actions[bot] 1cbe25bd1d chore: release v4.5.5 (#4267)
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 1s
Details
🚀 Publish Trigger.dev Docker / scan-webapp (push) Has been skipped
Details
🚀 Publish Trigger.dev Docker / scan-supervisor (push) Has been skipped
Details
🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped
Details
🚀 Publish Trigger.dev Docker / publish-worker-v4 (push) Has been skipped
Details
🚀 Publish Trigger.dev Docker / units (push) Failing after 0s
Details
🧭 Helm Chart Release / lint-and-test (push) Has been cancelled
Details
🧭 Helm Chart Release / release (push) Has been cancelled
Details
🚀 Publish Trigger.dev Docker / 📣 Dispatch main image (push) Has been cancelled
Details 
## Summary
5 improvements, 5 bug fixes.

## Improvements
- Add experimental Node.js 24 and 26 task runtimes. Set `runtime` to
`experimental-node-24` or `experimental-node-26` in `trigger.config.ts`.
([#4085](https://github.com/triggerdotdev/trigger.dev/pull/4085))
- Add `defaultRegion` to the project GET and list API responses; null
when unset.
([#4146](https://github.com/triggerdotdev/trigger.dev/pull/4146))

## Server changes

These changes affect the self-hosted Docker image and Trigger.dev Cloud:

- Transient internal sync failures are now retried quietly instead of
surfacing as errors.
([#4270](https://github.com/triggerdotdev/trigger.dev/pull/4270))
- Optionally route ClickHouse read traffic to a read replica while
writes stay on the primary. Set `CLICKHOUSE_READER_URL` to move all
reads, or target the busiest paths with `RUNS_LIST_CLICKHOUSE_URL` (runs
list) and `EVENTS_READER_CLICKHOUSE_URL` (traces, spans, logs). All
optional; unset keeps current behavior.
([#4081](https://github.com/triggerdotdev/trigger.dev/pull/4081))
- Remove the deprecated realtime stream write endpoint used by retired
v3 task clients.
([#4250](https://github.com/triggerdotdev/trigger.dev/pull/4250))
- Fix batchTrigger requests that set a per-item idempotency key failing
with an error instead of creating and deduplicating the runs
([#4271](https://github.com/triggerdotdev/trigger.dev/pull/4271))
- Speed up idempotency checks on `batchTrigger` calls that use
idempotency keys. Large batches against a task with a big run history no
longer degrade to multi-second lookups.
([#4255](https://github.com/triggerdotdev/trigger.dev/pull/4255))
- The "Preview branches" usage on the Limits page now counts only
preview branches.
([#4283](https://github.com/triggerdotdev/trigger.dev/pull/4283))
- Avoid opening a redundant database connection pool when the legacy and
primary databases are the same server, preventing connection usage from
doubling.
([#4253](https://github.com/triggerdotdev/trigger.dev/pull/4253))
- Fix pages occasionally loading unstyled or failing to load during a
deploy. The dashboard now reloads automatically to recover.
([#4282](https://github.com/triggerdotdev/trigger.dev/pull/4282))

<details>
<summary>Raw changeset output</summary>

# Releases
## @trigger.dev/build@4.5.5

### Patch Changes

- Updated dependencies:
  - `@trigger.dev/core@4.5.5`
## trigger.dev@4.5.5

### Patch Changes

- Add experimental Node.js 24 and 26 task runtimes. Set `runtime` to
`experimental-node-24` or `experimental-node-26` in `trigger.config.ts`.
([#4085](https://github.com/triggerdotdev/trigger.dev/pull/4085))
- Updated dependencies:
  - `@trigger.dev/core@4.5.5`
  - `@trigger.dev/build@4.5.5`
  - `@trigger.dev/schema-to-json@4.5.5`
## @trigger.dev/core@4.5.5

### Patch Changes

- Add experimental Node.js 24 and 26 task runtimes. Set `runtime` to
`experimental-node-24` or `experimental-node-26` in `trigger.config.ts`.
([#4085](https://github.com/triggerdotdev/trigger.dev/pull/4085))
- Add `defaultRegion` to the project GET and list API responses; null
when unset.
([#4146](https://github.com/triggerdotdev/trigger.dev/pull/4146))
## @trigger.dev/python@4.5.5

### Patch Changes

- Updated dependencies:
  - `@trigger.dev/core@4.5.5`
  - `@trigger.dev/build@4.5.5`
  - `@trigger.dev/sdk@4.5.5`
## @trigger.dev/react-hooks@4.5.5

### Patch Changes

- Updated dependencies:
  - `@trigger.dev/core@4.5.5`
## @trigger.dev/redis-worker@4.5.5

### Patch Changes

- Updated dependencies:
  - `@trigger.dev/core@4.5.5`
## @trigger.dev/rsc@4.5.5

### Patch Changes

- Updated dependencies:
  - `@trigger.dev/core@4.5.5`
## @trigger.dev/schema-to-json@4.5.5

### Patch Changes

- Updated dependencies:
  - `@trigger.dev/core@4.5.5`
## @trigger.dev/sdk@4.5.5

### Patch Changes

- Updated dependencies:
  - `@trigger.dev/core@4.5.5`

</details>

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
 2026-07-20 14:01:37 +01:00
Chris Arderne d7ec75d5ad feat(runtime): add experimental Node.js 24 and 26 task runtimes (#4085)
⚒️ Publish Worker (v4) / build (supervisor) (push) Has been cancelled
Details 
## Summary

Adds experimental Node.js 24 and 26 task runtimes through the
`experimental-node-24` and `experimental-node-26` config values.

Existing runtime defaults and the `node`, `node-22`, and `bun` behavior
remain unchanged. The unprefixed `node-24` and `node-26` config values
remain unavailable until the runtimes are ready for general use.

## Design

Experimental config values normalize to canonical runtime identifiers
before build manifests are created, keeping deployment metadata and
execution behavior consistent. Kubernetes task pods also use the
runtime-default seccomp profile so modern Node.js versions fall back
from io_uring to checkpoint-compatible system calls.
 2026-07-16 12:19:03 +01:00
nicktrn 976171ea16 feat(webapp): management API for orgs, projects, members, and settings (#4146) 
## Summary

Adds a set of PAT-authenticated management API endpoints so orgs,
projects, members/invites, environment variables, and a few
project/environment settings can be managed programmatically (scripting,
automation) rather than only through the dashboard. Each route is a thin
wrapper over the **existing** service the dashboard already uses, with
the same authorization applied at the route layer - no new business
logic.

## Endpoints

**Organizations**
- `POST /api/v1/orgs` - create an org (`createOrganization`)
- `PATCH /api/v1/orgs/:orgParam` - rename (title)
- `DELETE /api/v1/orgs/:orgParam` - soft-delete
(`DeleteOrganizationService`; keeps the active-subscription guard)

**Members & invites**
- `GET /api/v1/orgs/:orgParam/members` - list members + pending invites
- `DELETE /api/v1/orgs/:orgParam/members/:memberId` - remove a member
(last-member guarded)
- `POST /api/v1/orgs/:orgParam/invites` - invite by email
(`inviteMembers`, sends the invite email)
- `DELETE /api/v1/orgs/:orgParam/invites/:inviteId` - revoke an invite

**Projects**
- `PATCH /api/v1/projects/:projectRef` - rename
(`ProjectSettingsService`)
- `DELETE /api/v1/projects/:projectRef` - soft-delete
(`DeleteProjectService`)
- `PUT /api/v1/projects/:projectRef/default-region` - set the default
region by worker-group name (`SetDefaultRegionService`)
- project GET/list now return `defaultRegion` (worker-group name, or
null when unset)

**Environments**
- `POST /api/v1/projects/:projectRef/:env/pause` and `/resume`
(`PauseEnvironmentService`)
- `POST /api/v1/projects/:projectRef/:env/regenerate-api-key` - rotate
the env secret key (`regenerateApiKey`, RBAC `write:apiKeys`)
- env var create now accepts an optional `isSecret` flag

## Auth & authorization

- All routes authenticate with a **Personal Access Token**
(`Authorization: Bearer tr_pat_...`).
- Org/project routes are built on the PAT route builders in
`apiBuilder.server.ts`: `createLoaderPATApiRoute` (already existed) and
**`createActionPATApiRoute`** (added here - the loader builder had no
mutation counterpart). The builder runs auth, resolves the org/project
role-floor via `context`, and enforces a declarative `authorization`
block using the same RBAC actions the dashboard applies
(`manage:organization` / `read:members` / `manage:members` /
`manage:project`). Handlers keep a membership-scoped query as the floor,
so a non-member gets a 404. This also gives these routes `tenantContext`
user attribution (Sentry) and `ServiceValidationError`-to-status mapping
for free.
- **Membership floor (important).** The OSS RBAC fallback grants a
permissive ability, so `ability.can(...)` can't reject a non-member on
self-hosted. Every handler therefore resolves the target scoped to the
caller's membership (`members: { some: { userId } }`) → 404 for
non-members. `authorization` is the *role* gate; this is the *tenant*
gate. `resolveOrganizationForApiUser`
(`organizationApiAccess.server.ts`) is the org-tier version of the
existing `findProjectByRef` - org-addressed PAT routes are new, so no
such helper existed before.
- Env-tier routes reuse the existing `authorizePatEnvironmentAccess`
(`write:apiKeys`).

### What `createActionPATApiRoute` gives you

A route is pure declaration - the builder handles auth, RBAC,
validation, tracing, and error mapping:

```ts
export const action = createActionPATApiRoute(
  {
    method: "PUT",                          // one verb, or ["PATCH", "DELETE"] for multi-verb routes
    params: ParamsSchema,
    body: SetDefaultRegionRequestBody,      // zod-validated
    context: async ({ projectRef }) => {    // resolve the org for the RBAC role-floor
      const project = await prisma.project.findFirst({
        where: { externalRef: projectRef, deletedAt: null },
        select: { organizationId: true },
      });
      return project ? { organizationId: project.organizationId } : {};
    },
    authorization: { action: "manage", resource: () => ({ type: "project" }) },
  },
  async ({ params, body, authentication, ability }) => {
    // auth + authz already enforced. Just do the work.
    // `throw new ServiceValidationError("Region not found", 400)` → mapped to that status.
    return json({ ok: true });
  }
);
```

Handled for you, so handlers stay thin:

- **Method allowlist** - `method` accepts a verb or an array; any other
verb → `405` with an `Allow` header, *before* auth runs:
  ```ts
const allowedMethods = method ? (Array.isArray(method) ? method :
[method]) : undefined;
if (allowedMethods && !(allowedMethods as
string[]).includes(request.method.toUpperCase())) {
return json({ error: "Method not allowed" }, { status: 405, headers: {
Allow: allowedMethods.join(", ") } });
  }
  ```
- **PAT / user-actor auth** in a single roundtrip → `401` on
missing/invalid/revoked token.
- **RBAC** - `context` computes the caller's role-floor for the target
org/project; `authorization` gates it → `403` with a structured error
body.
- **Sentry attribution** - `tenantContext.enrich({ userId })` so events
from the handler carry the acting user.
- **Typed errors** - a thrown `ServiceValidationError` is mapped to its
`.status` (default 400); anything else → `500`, and expected boundary
errors are logged as `warn` (kept out of Sentry).
- **Validation** - params / query / headers / body are all zod-checked →
`400` with details.

## Notes for reviewers

- Everything wraps an existing service; the intent is API parity for
things that are currently dashboard-only, not new behaviour.
- `createActionPATApiRoute` is new shared infra (the PAT + RBAC mutation
builder that didn't exist). It's self-contained - the loader builder and
existing routes are untouched.
- `@trigger.dev/core` gets one additive field (`defaultRegion` on the
project response, optional/nullable for client-server version skew) -
changeset included, patch.
- `removeTeamMember`'s last-member guard is now atomic (Serializable
transaction via the `$transaction` helper, with retry), so the dashboard
and API both get it server-side. Added a `## Transactions` rule to
`apps/webapp/CLAUDE.md` (always use the `$transaction` helper);
migrating the remaining direct usages is tracked in TRI-11698.

## Open questions

- ~~Is PAT the right auth (vs OAT for automation)?~~ **Resolved: PAT.**
Organization Access Tokens are currently internal-only (used by the
image builder) and not user-accessible, so they can't back this yet.
- Should any of these be gated behind a flag or scope?
- Naming/shape of the routes.
 2026-07-15 10:20:08 +01:00
Chris Arderne 313fe03481 test(core): fix flakey run-stream test depending on ordering (#4256) 2026-07-14 12:29:57 +01:00
github-actions[bot] 165955781d chore: release v4.5.4 (#4228)
📚 Publish docs / publish (push) Has been cancelled
Details
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 1s
Details
🚀 Publish Trigger.dev Docker / scan-webapp (push) Has been skipped
Details
🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped
Details
🚀 Publish Trigger.dev Docker / publish-worker-v4 (push) Has been skipped
Details
🚀 Publish Trigger.dev Docker / units (push) Failing after 0s
Details
🧭 Helm Chart Release / lint-and-test (push) Has been cancelled
Details
🧭 Helm Chart Release / release (push) Has been cancelled
Details
🚀 Publish Trigger.dev Docker / 📣 Dispatch main image (push) Has been cancelled
Details 
## Summary
2 new features, 11 improvements, 5 bug fixes.

## Breaking changes
- Trigger.dev v3 is no longer supported. For self-hosted deployments,
4.5.0 is the last version we officially support for running v3; stay on
4.5.0 or upgrade to v4. v3 triggers, batch triggers, reschedules, and
deploys now return a clear upgrade message instead of running.
([#4236](https://github.com/triggerdotdev/trigger.dev/pull/4236))

## Improvements
- You can now mark environment variables synced via the `syncEnvVars`
build extension as secrets. Return `{ name, value, isSecret: true }`
from your callback and those variables are stored redacted in the
dashboard, just like manually created secret env vars.
([#4203](https://github.com/triggerdotdev/trigger.dev/pull/4203))
- Remove the legacy `--mcp` and `--mcp-port` options from the `dev`
command. Run the dedicated `trigger mcp` command to start the
Trigger.dev MCP server.
([#4246](https://github.com/triggerdotdev/trigger.dev/pull/4246))
- Removed the unused `ResourceMonitor` export from
`@trigger.dev/core/v3/serverOnly`. It was a server-side logging helper
with no remaining consumers.
([#4244](https://github.com/triggerdotdev/trigger.dev/pull/4244))
- Removed the unused `@trigger.dev/core/v3/zodNamespace` export and the
legacy v3 socket message schemas. These were only used by the
now-retired v3 engine and have no v4 consumers.
([#4236](https://github.com/triggerdotdev/trigger.dev/pull/4236))

## Bug fixes
- Fix a `chat.agent` message-loss race where sending a message right
after an action (such as an undo) could drop the follow-up's response
from the UI until a refresh.
([#4234](https://github.com/triggerdotdev/trigger.dev/pull/4234))

## Server changes

These changes affect the self-hosted Docker image and Trigger.dev Cloud:

- Added `EVENT_REPOSITORY_POSTGRES_WRITES_DISABLED` to skip all
PostgreSQL task-event writes for deployments that store task events in
ClickHouse. Leave it off unless `EVENT_REPOSITORY_DEFAULT_STORE` is
`clickhouse_v2`, otherwise task events are lost.
([#4242](https://github.com/triggerdotdev/trigger.dev/pull/4242))
- Promo credits: a /promo signup landing page, redeeming a promo code
when a new org selects a plan, and showing remaining credits on the
usage page.
([#4138](https://github.com/triggerdotdev/trigger.dev/pull/4138))
- Speed up retrieving a background worker by version. The endpoint no
longer runs a slow lookup that scanned the full task table for large
deployments; it now reuses data it already loads, so the response is the
same but returns much faster.
([#4245](https://github.com/triggerdotdev/trigger.dev/pull/4245))
- Clearer login error when an email address is blocked by the
WHITELISTED_EMAILS setting: the message now explains the address isn't
allowed on this instance instead of the ambiguous "This email is
unauthorized".
([#4220](https://github.com/triggerdotdev/trigger.dev/pull/4220))
- Make the native build server the default in project build settings.
It's now opt-out, stored as a new `disableNativeBuildServer` key. Also
clarifies in the UI that build settings apply to GitHub-triggered and
native build server deployments.
([#3980](https://github.com/triggerdotdev/trigger.dev/pull/3980))
- Optionally process high-volume telemetry ingestion in parallel for
higher throughput under heavy load by setting
`OTEL_TRANSFORM_WORKER_POOL_ENABLED=1`. Off by default.
([#4232](https://github.com/triggerdotdev/trigger.dev/pull/4232))
- Add a `REALTIME_BACKEND_DEFAULT` env var to choose the default
realtime backend (`electric`, `native`, or `shadow`) for environments
whose org has no per-org override. Defaults to `electric`, so existing
behavior is unchanged.
([#4231](https://github.com/triggerdotdev/trigger.dev/pull/4231))
- Clarified on the Regions page that a region only affects where your
runs execute, not where your data is stored. This shows as a tooltip on
the Location column and in the confirmation dialog when you change your
default region.
([#4226](https://github.com/triggerdotdev/trigger.dev/pull/4226))
- Improved the reliability of how run data is read and written.
([#4237](https://github.com/triggerdotdev/trigger.dev/pull/4237))
- Fixed stale login errors: an error from a previous login attempt (for
example a rejected email address) no longer keeps reappearing on the
login page and no longer makes later, successful attempts look like they
failed.
([#4220](https://github.com/triggerdotdev/trigger.dev/pull/4220))
- The Errors page now shows better details for each error. Errors that
don't carry a message — such as errors thrown without a message, or
values thrown that aren't `Error` objects — get a meaningful title
instead of all reading "Unknown error", and are grouped by their name
(or value) rather than collapsed into a single group. The error type now
shows the actual error name, and stack traces now appear where
previously they were missing.
([#4225](https://github.com/triggerdotdev/trigger.dev/pull/4225))
- Return a clear client error when SSO form submissions use an
unsupported content type
([#4238](https://github.com/triggerdotdev/trigger.dev/pull/4238))
- Query page: extracting fields from a run's output with JSON functions
(such as JSONExtractString or JSONExtractInt) no longer fails with an
"illegal type: JSON" error.
([#4221](https://github.com/triggerdotdev/trigger.dev/pull/4221))

<details>
<summary>Raw changeset output</summary>

# Releases
## @trigger.dev/build@4.5.4

### Patch Changes

- You can now mark environment variables synced via the `syncEnvVars`
build extension as secrets. Return `{ name, value, isSecret: true }`
from your callback and those variables are stored redacted in the
dashboard, just like manually created secret env vars.
([#4203](https://github.com/triggerdotdev/trigger.dev/pull/4203))
- Updated dependencies:
  - `@trigger.dev/core@4.5.4`
## trigger.dev@4.5.4

### Patch Changes

- Remove the legacy `--mcp` and `--mcp-port` options from the `dev`
command. Run the dedicated `trigger mcp` command to start the
Trigger.dev MCP server.
([#4246](https://github.com/triggerdotdev/trigger.dev/pull/4246))
- Updated dependencies:
  - `@trigger.dev/core@4.5.4`
  - `@trigger.dev/build@4.5.4`
  - `@trigger.dev/schema-to-json@4.5.4`
## @trigger.dev/core@4.5.4

### Patch Changes

- Removed the unused `ResourceMonitor` export from
`@trigger.dev/core/v3/serverOnly`. It was a server-side logging helper
with no remaining consumers.
([#4244](https://github.com/triggerdotdev/trigger.dev/pull/4244))
- Removed the unused `@trigger.dev/core/v3/zodNamespace` export and the
legacy v3 socket message schemas. These were only used by the
now-retired v3 engine and have no v4 consumers.
([#4236](https://github.com/triggerdotdev/trigger.dev/pull/4236))
## @trigger.dev/python@4.5.4

### Patch Changes

- Updated dependencies:
  - `@trigger.dev/sdk@4.5.4`
  - `@trigger.dev/core@4.5.4`
  - `@trigger.dev/build@4.5.4`
## @trigger.dev/react-hooks@4.5.4

### Patch Changes

- Updated dependencies:
  - `@trigger.dev/core@4.5.4`
## @trigger.dev/redis-worker@4.5.4

### Patch Changes

- Updated dependencies:
  - `@trigger.dev/core@4.5.4`
## @trigger.dev/rsc@4.5.4

### Patch Changes

- Updated dependencies:
  - `@trigger.dev/core@4.5.4`
## @trigger.dev/schema-to-json@4.5.4

### Patch Changes

- Updated dependencies:
  - `@trigger.dev/core@4.5.4`
## @trigger.dev/sdk@4.5.4

### Patch Changes

- Fix a `chat.agent` message-loss race where sending a message right
after an action (such as an undo) could drop the follow-up's response
from the UI until a refresh.
([#4234](https://github.com/triggerdotdev/trigger.dev/pull/4234))
- Updated dependencies:
  - `@trigger.dev/core@4.5.4`

</details>

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
 2026-07-14 10:07:58 +01:00
Chris Arderne 64e5d732ad chore(webapp,core): remove the unused ResourceMonitor server logging helper (#4244) 
The `ResourceMonitor` server-side logging helper is no longer used. It
periodically logged the webapp process own memory, disk, and CPU usage
behind the `RESOURCE_MONITOR_ENABLED` flag (off by default), and was
also exported from `@trigger.dev/core/v3/serverOnly` with no other
consumers.

This removes the helper, its `@trigger.dev/core` export, the webapp
wiring, and the `RESOURCE_MONITOR_ENABLED` env var. The supervisor has
its own unrelated `ResourceMonitor` class, which is left untouched.
 2026-07-13 20:32:29 +01:00
Eric Allam 5ba8557a51 chore(webapp,core): remove the end-of-life v3 (engine V1) execution stack (#4236) 
## Summary

v3 (the engine that ran the SDK v3 era, internally
`RunEngineVersion.V1`) is end-of-life. Following the removal of the v3
execution apps
([#4194](https://github.com/triggerdotdev/trigger.dev/pull/4194)) and
the legacy dev websocket
([#4198](https://github.com/triggerdotdev/trigger.dev/pull/4198)), this
removes the remaining v3 execution stack from the server.

Clients still on v3 (an old SDK or CLI that has not upgraded) keep
getting a clear "upgrade to v4" response. Triggers, batch triggers,
reschedules, and deploys that resolve to v3 are rejected with a graceful
4xx pointing at the migration guide, never a 5xx, so a stale client
cannot affect server health. Self-hosted instances still running v3
should stay on the 4.5.x release line until they migrate.

## What is removed

- The MarQS queue and its shared/dev queue consumers.
- The v3 socket.io namespaces (coordinator, provider, shared-queue) and
the v3 run lifecycle services (attempt, checkpoint, and batch-resume).
- The graphile-worker background job system; all live jobs already run
on `@trigger.dev/redis-worker`.
- The `DEPRECATE_V3_ENABLED` flag: v3 is now rejected unconditionally,
so the flag is gone.
- Unused v3 exports from `@trigger.dev/core` (the `v3/zodNamespace`
subpath and the legacy socket message catalogs) and the now-dead MarQS
environment variables.

## What stays

The v4 engine is untouched. The graceful v3 rejection boundary stays,
`determineEngineVersion` still detects a v3 project so it can reject it,
and the batch service plus batch-completion worker stay for current
clients. Live queue concurrency limits and metrics now read from the v4
run engine instead of MarQS, and a brand-new dev environment now
defaults to v4.



## Dependency cleanup

Removes webapp dependencies left unused by this change: `seedrandom` and
`semver` (only the removed v3 code used them) plus a set that was
already dead, their orphaned `@types` packages, and two dead files. Adds
a `knip:deps` script and a `knip.json` config so unused dependencies can
be found the same way going forward.
 2026-07-13 11:32:06 +01:00
Iss 983bd03131 feat: support isSecret in syncEnvVars (#4203) 
## What

Adds per-variable secret support to the `syncEnvVars` build extension.
Return `{ name, value, isSecret: true }` and the variable is stored as a
secret (redacted in the dashboard, value non-revealable), just like a
manually created secret env var. Secret and non-secret variables can be
mixed in one callback.

```ts
syncEnvVars(async () => [
  { name: "PUBLIC_API_URL", value: "https://api.example.com" },
  { name: "DATABASE_URL", value: "postgres://...", isSecret: true },
]);
```

## How

Env vars flow through the build pipeline as a flat name→value map, and
the import API's `isSecret` is per-call. So secret vars are carried
through the layer + manifest in parallel `secretEnv` / `secretParentEnv`
maps, and at deploy time they go up in a second `importEnvVars` call
with `isSecret: true` (the plain vars in the first call). The record
form (`{ KEY: "value" }`) is unchanged and stays non-secret.

## Commits

- `feat(core)`: carry secret env vars through the build layer + manifest
schema
- `feat(build)`: partition `isSecret` vars in `syncEnvVars`
- `feat(cli)`: merge secret layers and import them with `isSecret: true`
at deploy
- `test(build)`: cover the partitioning + document `isSecret`

## Testing

- vitest covers the partitioning (secret/non-secret × child/parent) and
that the record form stays non-secret.
- Verified against a local webapp that the deploy's import contract
stores the secret var redacted (`isSecret: true`) and the plain var
visible.

Closes TRI-11099
 2026-07-10 11:03:28 -04:00
github-actions[bot] 9f76c92021 chore: release v4.5.3 (#4219)
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 3s
Details
🚀 Publish Trigger.dev Docker / scan-webapp (push) Has been skipped
Details
🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped
Details
🚀 Publish Trigger.dev Docker / publish-worker-v4 (push) Has been skipped
Details
🚀 Publish Trigger.dev Docker / units (push) Failing after 4s
Details
🧭 Helm Chart Release / lint-and-test (push) Has been cancelled
Details
🧭 Helm Chart Release / release (push) Has been cancelled
Details
🚀 Publish Trigger.dev Docker / 📣 Dispatch main image (push) Has been cancelled
Details 
## Summary
1 improvement, 2 bug fixes.

## Breaking changes
- Removed support for the end-of-life v3 `trigger dev` CLI. Starting a
dev session with an old v3 CLI now returns an upgrade message instead of
connecting - upgrade to the v4 CLI to continue using `trigger dev`.
([#4198](https://github.com/triggerdotdev/trigger.dev/pull/4198))

## Bug fixes
- Fix TS2742 ("inferred type cannot be named") when exporting a
`chat.agent` from a project with declaration emit: `ChatTaskWirePayload`
and `ChatInputChunk` are now declared in the public
`@trigger.dev/sdk/chat` subpath, so inferred agent types emit portable
declarations and the wire types are directly importable.
([#4218](https://github.com/triggerdotdev/trigger.dev/pull/4218))

## Server changes

These changes affect the self-hosted Docker image and Trigger.dev Cloud:

- Reduce primary database load on the runs page by serving its
empty-state check from ClickHouse instead of Postgres.
([#4202](https://github.com/triggerdotdev/trigger.dev/pull/4202))
- Fixed submitting your email on the login page reloading back to an
empty form instead of showing the magic link confirmation screen.
([#4215](https://github.com/triggerdotdev/trigger.dev/pull/4215))

<details>
<summary>Raw changeset output</summary>

# Releases
## @trigger.dev/build@4.5.3

### Patch Changes

- Updated dependencies:
  - `@trigger.dev/core@4.5.3`
## trigger.dev@4.5.3

### Patch Changes

- Updated dependencies:
  - `@trigger.dev/build@4.5.3`
  - `@trigger.dev/core@4.5.3`
  - `@trigger.dev/schema-to-json@4.5.3`
## @trigger.dev/python@4.5.3

### Patch Changes

- Updated dependencies:
  - `@trigger.dev/sdk@4.5.3`
  - `@trigger.dev/build@4.5.3`
  - `@trigger.dev/core@4.5.3`
## @trigger.dev/react-hooks@4.5.3

### Patch Changes

- Updated dependencies:
  - `@trigger.dev/core@4.5.3`
## @trigger.dev/redis-worker@4.5.3

### Patch Changes

- Updated dependencies:
  - `@trigger.dev/core@4.5.3`
## @trigger.dev/rsc@4.5.3

### Patch Changes

- Updated dependencies:
  - `@trigger.dev/core@4.5.3`
## @trigger.dev/schema-to-json@4.5.3

### Patch Changes

- Updated dependencies:
  - `@trigger.dev/core@4.5.3`
## @trigger.dev/sdk@4.5.3

### Patch Changes

- Fix TS2742 ("inferred type cannot be named") when exporting a
`chat.agent` from a project with declaration emit: `ChatTaskWirePayload`
and `ChatInputChunk` are now declared in the public
`@trigger.dev/sdk/chat` subpath, so inferred agent types emit portable
declarations and the wire types are directly importable.
([#4218](https://github.com/triggerdotdev/trigger.dev/pull/4218))
- Updated dependencies:
  - `@trigger.dev/core@4.5.3`
## @trigger.dev/core@4.5.3

</details>

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
 2026-07-10 08:42:34 +01:00
github-actions[bot] 188f008715 chore: release v4.5.2 (#4180) 
## Summary
4 improvements, 5 bug fixes.

## Improvements
- Add SDK and API client helpers for run bulk actions.
([#4105](https://github.com/triggerdotdev/trigger.dev/pull/4105))
- Large batch payloads now offload to object storage instead of riding
inline in the trigger request. `batchTrigger` and `batchTriggerAndWait`
(and the by-id and by-task variants) offload any per-item payload over
128KB before sending, the same way single `trigger` and `triggerAndWait`
already do, so a big batch no longer blows past the API body limit.
([#4165](https://github.com/triggerdotdev/trigger.dev/pull/4165))
- Removed internal helpers that were only used by the end-of-life v3
self-hosted compute providers.
([#4194](https://github.com/triggerdotdev/trigger.dev/pull/4194))
- Add an `onEvent` callback to `TriggerChatTransport` /
`useTriggerChatTransport` that emits typed lifecycle events for sends,
stream connects, first chunk, and turn completion. Send-success metrics,
time-to-first-token, and "sent but never answered" watchdogs become a
few lines of client code.
([#4187](https://github.com/triggerdotdev/trigger.dev/pull/4187))
  
  ```ts
  onEvent: (event) => {
if (event.type === "message-sent") metrics.timing("chat.send_ms",
event.durationMs);
if (event.type === "first-chunk") metrics.timing("chat.ttft_ms",
event.sinceSendMs ?? 0);
  },
  ```

## Bug fixes
- fix(cli): honor the MCP server's `--dev-only` flag
([#4199](https://github.com/triggerdotdev/trigger.dev/pull/4199))
- Fix chat turns that throw (for example from an `onTurnStart` hook)
leaking their message listener, which lost or duplicated messages sent
during later turns.
([#4176](https://github.com/triggerdotdev/trigger.dev/pull/4176))
- Fix `chat.agent` and `chat.createSession` permanently dropping user
messages when several arrived during a single turn: every buffered
message is now dispatched as its own turn instead of only the first.
([#4176](https://github.com/triggerdotdev/trigger.dev/pull/4176))
- Fix chat continuation runs replaying already-answered messages: turns
delivered while the run was suspended now advance the session.in resume
cursor, so a new run picks up exactly where the previous one left off.
([#4176](https://github.com/triggerdotdev/trigger.dev/pull/4176))
- Fix `chat.createSession` swallowing a message sent shortly after
stopping a turn: the turn's message listener now detaches when the
stream settles, so those messages run as the next turn.
([#4176](https://github.com/triggerdotdev/trigger.dev/pull/4176))

<details>
<summary>Raw changeset output</summary>

# Releases
## @trigger.dev/build@4.5.2

### Patch Changes

- Updated dependencies:
  - `@trigger.dev/core@4.5.2`
## trigger.dev@4.5.2

### Patch Changes

- fix(cli): honor the MCP server's `--dev-only` flag
([#4199](https://github.com/triggerdotdev/trigger.dev/pull/4199))
- Updated dependencies:
  - `@trigger.dev/core@4.5.2`
  - `@trigger.dev/build@4.5.2`
  - `@trigger.dev/schema-to-json@4.5.2`
## @trigger.dev/core@4.5.2

### Patch Changes

- Add SDK and API client helpers for run bulk actions.
([#4105](https://github.com/triggerdotdev/trigger.dev/pull/4105))
- Large batch payloads now offload to object storage instead of riding
inline in the trigger request. `batchTrigger` and `batchTriggerAndWait`
(and the by-id and by-task variants) offload any per-item payload over
128KB before sending, the same way single `trigger` and `triggerAndWait`
already do, so a big batch no longer blows past the API body limit.
([#4165](https://github.com/triggerdotdev/trigger.dev/pull/4165))
- Removed internal helpers that were only used by the end-of-life v3
self-hosted compute providers.
([#4194](https://github.com/triggerdotdev/trigger.dev/pull/4194))
## @trigger.dev/python@4.5.2

### Patch Changes

- Updated dependencies:
  - `@trigger.dev/core@4.5.2`
  - `@trigger.dev/sdk@4.5.2`
  - `@trigger.dev/build@4.5.2`
## @trigger.dev/react-hooks@4.5.2

### Patch Changes

- Updated dependencies:
  - `@trigger.dev/core@4.5.2`
## @trigger.dev/redis-worker@4.5.2

### Patch Changes

- Updated dependencies:
  - `@trigger.dev/core@4.5.2`
## @trigger.dev/rsc@4.5.2

### Patch Changes

- Updated dependencies:
  - `@trigger.dev/core@4.5.2`
## @trigger.dev/schema-to-json@4.5.2

### Patch Changes

- Updated dependencies:
  - `@trigger.dev/core@4.5.2`
## @trigger.dev/sdk@4.5.2

### Patch Changes

- Add SDK and API client helpers for run bulk actions.
([#4105](https://github.com/triggerdotdev/trigger.dev/pull/4105))
- Fix chat turns that throw (for example from an `onTurnStart` hook)
leaking their message listener, which lost or duplicated messages sent
during later turns.
([#4176](https://github.com/triggerdotdev/trigger.dev/pull/4176))
- Fix `chat.agent` and `chat.createSession` permanently dropping user
messages when several arrived during a single turn: every buffered
message is now dispatched as its own turn instead of only the first.
([#4176](https://github.com/triggerdotdev/trigger.dev/pull/4176))
- Fix chat continuation runs replaying already-answered messages: turns
delivered while the run was suspended now advance the session.in resume
cursor, so a new run picks up exactly where the previous one left off.
([#4176](https://github.com/triggerdotdev/trigger.dev/pull/4176))
- Fix `chat.createSession` swallowing a message sent shortly after
stopping a turn: the turn's message listener now detaches when the
stream settles, so those messages run as the next turn.
([#4176](https://github.com/triggerdotdev/trigger.dev/pull/4176))
- Add an `onEvent` callback to `TriggerChatTransport` /
`useTriggerChatTransport` that emits typed lifecycle events for sends,
stream connects, first chunk, and turn completion. Send-success metrics,
time-to-first-token, and "sent but never answered" watchdogs become a
few lines of client code.
([#4187](https://github.com/triggerdotdev/trigger.dev/pull/4187))

  ```ts
  onEvent: (event) => {
if (event.type === "message-sent") metrics.timing("chat.send_ms",
event.durationMs);
if (event.type === "first-chunk") metrics.timing("chat.ttft_ms",
event.sinceSendMs ?? 0);
  },
  ```

- Large batch payloads now offload to object storage instead of riding
inline in the trigger request. `batchTrigger` and `batchTriggerAndWait`
(and the by-id and by-task variants) offload any per-item payload over
128KB before sending, the same way single `trigger` and `triggerAndWait`
already do, so a big batch no longer blows past the API body limit.
([#4165](https://github.com/triggerdotdev/trigger.dev/pull/4165))
- Updated dependencies:
  - `@trigger.dev/core@4.5.2`
## @trigger.dev/plugins@4.5.2

### Patch Changes

- Updated dependencies:
  - `@trigger.dev/core@4.5.2`

</details>

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
 2026-07-09 12:02:34 +00:00
nicktrn a6bd370e42 chore: remove end-of-life v3 execution components (#4194) 
v3 (engine V1) is end-of-lifed and the v3 clusters are gone, so this
removes the dead v3 execution code from the monorepo. It's the first
pass of TRI-11824 - the webapp v3 code paths are deliberately left
untouched and gated for a follow-up.

## Apps

Deletes the three v3-only execution apps and their build wiring:

- `apps/coordinator`, `apps/kubernetes-provider`, `apps/docker-provider`
- `.github/workflows/publish-worker.yml` - it built only those three;
the v4 worker publish is a separate workflow
- Their references in `.changeset/config.json`, `.cursorignore`,
`CHANGESETS.md`, `CONTRIBUTING.md`, `.server-changes/README.md`
- `pnpm-lock.yaml` regenerated to prune the apps and their app-only
dependencies (`socket.io`, `@kubernetes/client-node`, `p-queue`,
`execa`, `prom-client`, `tinyexec`)

## Core

Removes the helpers in `@trigger.dev/core` that only those apps used -
`ProviderShell`, `SimpleLogger`, the `Exec`/process helpers,
`isExecaChildProcess`, `getTextBody`, and `testDockerCheckpoint`. Each
was verified to have no remaining consumers anywhere in the repo.

Kept the helpers still used elsewhere: `ExponentialBackoff` (warm-start
client), `HttpReply`/`getJsonBody` (serverOnly http server),
`SimpleStructuredLogger` (widely used), and
`ZodNamespace`/`ZodSocketConnection` (still referenced by legacy v3
webapp code, hence the follow-up pass).

The `./v3/apps` and `./v3/serverOnly` export subpaths remain - only dead
members were trimmed from their barrels, so no `package.json` exports
changed.

## Verification

`@trigger.dev/core` builds, and `typecheck` passes for core, supervisor,
cli-v3, run-engine, redis-worker, and webapp.

refs TRI-11824
 2026-07-08 19:47:20 +02:00
Wes Mason 76c37ecd24 feat(sdk,core,webapp): offload large batch payloads to object storage (#4165) 
## Summary

`batchTrigger` and `batchTriggerAndWait` (and the by-id and by-task
variants) now offload any per-item payload over 128KB to object storage
before sending, the same way single `trigger`/`triggerAndWait` already
do since
[#3785](https://github.com/triggerdotdev/trigger.dev/pull/3785). A batch
of large items no longer inflates the request body past the API limit.

## Demo

A live local run: `batchTriggerAndWait` of 5 items × 300KB (1.5MB
total). Each item offloads to object storage, so the receiver run rows
hold a 65-byte `application/store` pointer instead of the 300KB body,
and every item round-trips (received == sent).

<img width="1000" height="494" alt="batch large-payload offload demo"
src="https://github.com/user-attachments/assets/77ae3958-97d6-4b5c-ab25-39b217caefbc"
/>


## Design

Both the array and streaming batch paths funnel through
`executeBatchTwoPhase`, so offloading happens once there: each item is
measured, then offloaded through the existing
`conditionallyExportPacket` when it crosses 128KB, with bounded
concurrency so a big batch doesn't fire an unbounded number of presigned
PUTs.

Because items are offloaded before the request, SDK batches arrive as
small `application/store` references, so the server-side inline offload
during item ingest (parallelised in
[#3777](https://github.com/triggerdotdev/trigger.dev/pull/3777)) mostly
no longer fires for them.

Every trigger and item also carries its pre-offload serialised size as
`options.payloadSize`. The trigger span records that value, so an
offloaded payload shows its real size instead of the size of the small
object-store reference (previously the span measured the reference).
 2026-07-07 16:21:48 +00:00
Chris Arderne aa74e68c71 feat(sdk): add bulk replay to api and sdk (#4105) 
## Summary

Adds SDK and API support for run bulk actions. You can now create bulk
cancel or replay actions from `@trigger.dev/sdk` using run IDs or the
same filters as `runs.list()`, then retrieve, list, poll, or abort the
action by its `bulk_` handle.

Tests, docs, changesets added.

## Design

The dashboard bulk action service now accepts structured filters instead
of reading directly from a dashboard request, so the dashboard and API
share the same creation path. Replay actions created through the API are
attributed with the existing `api` trigger source, while
dashboard-created actions keep `dashboard`.

The SDK exposes the new surface under `runs.bulk.*`, including
`targetRegion` for replay region overrides and cursor pagination for
listing bulk actions.

## Filters and runIds

Nuance on filters. If `filter` is provided, it MUST have at least one
key. This is to remove the footgun of passing no filter and selecting
all runs.

```typescript
   { action: "cancel", runIds: ["run_1"] } // valid
   { action: "cancel", runIds: [] } // invalid, min(1)
   { action: "cancel", filter: { status: "FAILED" } } // valid
   { action: "cancel", filter: {} } // invalid
   { action: "cancel", filter: {}, runIds: ["run_1"] } // invalid
```
 2026-07-07 15:43:30 +01:00
Eric Allam add0a7da0a fix(sdk,core): stop chat sessions dropping messages that arrive during a turn (#4176) 
## Summary

Sending a message to a chat whose run had ended could make the message
vanish: the continuation run replayed already-answered messages, never
processed the new one, and a page refresh lost it entirely. Chasing that
report surfaced four composing message-loss bugs in the chat session
runtime; this PR fixes all of them, each with a regression test.

## The fixes

1. **Stale resume cursor.** Records delivered while a run was suspended
(the waitpoint path) advanced the SSE resume counter but not the
committed-consume cursor, so the `session-in-event-id` header stamped on
turn-completes went stale by one record per suspended turn. Continuation
boots seed from that header, which is what made them replay
already-processed messages. `session.in.wait()` now advances both
cursors.

2. **Only the first buffered message dispatched.** Messages arriving
during a turn are consumed into a buffer whose end-of-turn pickup
dispatched only the first entry; the buffer was recreated each turn, so
the rest were discarded, and since consuming a record commits the cursor
the loss was permanent. A continuation boot's replay delivers several
records back-to-back, which put the user's new message at index 1 or
later. The buffer now outlives the turn and drains one message per turn
in both `chat.agent` and `chat.createSession` (whose equivalent buffer
was never read at all).

3. **Post-stop window in `chat.createSession`.** The turn's message
listener stayed attached through the stopped turn's post-stream work, so
a message sent shortly after stopping a turn was consumed into the dead
steering queue and lost. The listener now detaches when the stream
settles, matching the `chat.agent` loop.

4. **Handler leak on errored turns.** A turn that threw outside the
streaming section (for example from an `onTurnStart` hook) leaked its
message listener. Previously that silently lost mid-turn messages; with
the loop-level buffer it would have duplicated them instead. The
subscription handle is now detached by the turn's catch/finally, and
`chat.createSession` defensively detaches its prior turn's listener when
user code exits a turn without `complete()`/`done()`.

## Verification

Reproduced end-to-end with the ai-chat reference project before the fix
(message consumed but never answered, two replayed turns, gone on
refresh) and verified after (single clean turn, survives refresh,
turn-complete cursors strictly advancing). Regression tests in
`packages/trigger-sdk/test/pending-message-drain.test.ts` cover all
four, each verified red against the unfixed behavior. A smoke sweep of
the standard chat scenarios (basic send, multi-turn, suspend/resume,
mid-stream refresh, stop, steering, cancel + continue, and the
`createSession` variant) passes on the final branch state.
 2026-07-07 14:10:46 +01:00
Matt Aitken 1a033b665b fix(webapp,core): retry run resume through transient database outages (#4161) 
## Summary

When the platform database is briefly unreachable while a run is
resuming from a wait, the run no longer fails with
`TASK_EXECUTION_ABORTED`. The worker now retries the resume through the
outage instead of aborting on the first blip.

## Root cause

Resuming a run calls the engine's `continue` worker-action endpoint.
That route caught every error and returned a `422`, which the worker's
HTTP client treats as non-retryable. So a transient Prisma
infrastructure error (for example `P1001` "Can't reach database server")
was flattened into a permanent failure: the worker gave up, force-killed
the run process, and completed it with `TASK_EXECUTION_ABORTED`.

## Fix

- The `continue` route now lets infrastructure errors propagate to the
generic 500 handler (message scrubbed, and retryable by the worker's
HTTP client), the same treatment the trigger path already gives them via
`isInfrastructureError`. Genuine validation errors (snapshot mismatch,
invalid state) still return `422`, so a stale retry stays non-retryable.
Resuming is idempotent server-side (guarded by the snapshot id), so
retrying is safe.
- The worker's `continueRunExecution` calls (both the
runner-to-supervisor and supervisor-to-engine hops) retry with a longer,
jittered backoff so they can ride out an outage lasting tens of seconds,
and the jitter keeps a fleet of resuming runs from stampeding the
database the moment it recovers.

Builds on #3960, which scrubbed the leaked message on these routes but
left the status non-retryable.

No changeset: this is a server-side behaviour fix recorded via
`.server-changes`. The `@trigger.dev/core` edits are internal run-engine
worker plumbing, not a public API change.
 2026-07-07 11:51:59 +01:00
github-actions[bot] c584236937 chore: release v4.5.1 (#4126)
🚀 Publish Trigger.dev Docker / units (push) Failing after 22s
Details
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 22s
Details
🚀 Publish Trigger.dev Docker / publish-worker-v4 (push) Has been skipped
Details
🚀 Publish Trigger.dev Docker / scan-webapp (push) Has been skipped
Details
🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped
Details
🚀 Publish Trigger.dev Docker / publish-worker (push) Has been skipped
Details
🧭 Helm Chart Release / lint-and-test (push) Has been cancelled
Details
🧭 Helm Chart Release / release (push) Has been cancelled
Details
🚀 Publish Trigger.dev Docker / 📣 Dispatch main image (push) Has been cancelled
Details 
## Summary
1 improvement.

## Improvements
- Extend the SSO plugin contract with WorkOS Directory Sync (SCIM)
support.
([#4148](https://github.com/triggerdotdev/trigger.dev/pull/4148))

<details>
<summary>Raw changeset output</summary>

# Releases
## @trigger.dev/build@4.5.1

### Patch Changes

- Updated dependencies:
  - `@trigger.dev/core@4.5.1`
## trigger.dev@4.5.1

### Patch Changes

- Updated dependencies:
  - `@trigger.dev/build@4.5.1`
  - `@trigger.dev/core@4.5.1`
  - `@trigger.dev/schema-to-json@4.5.1`
## @trigger.dev/python@4.5.1

### Patch Changes

- Updated dependencies:
  - `@trigger.dev/build@4.5.1`
  - `@trigger.dev/core@4.5.1`
  - `@trigger.dev/sdk@4.5.1`
## @trigger.dev/react-hooks@4.5.1

### Patch Changes

- Updated dependencies:
  - `@trigger.dev/core@4.5.1`
## @trigger.dev/redis-worker@4.5.1

### Patch Changes

- Updated dependencies:
  - `@trigger.dev/core@4.5.1`
## @trigger.dev/rsc@4.5.1

### Patch Changes

- Updated dependencies:
  - `@trigger.dev/core@4.5.1`
## @trigger.dev/schema-to-json@4.5.1

### Patch Changes

- Updated dependencies:
  - `@trigger.dev/core@4.5.1`
## @trigger.dev/sdk@4.5.1

### Patch Changes

- Updated dependencies:
  - `@trigger.dev/core@4.5.1`
## @trigger.dev/core@4.5.1


## @trigger.dev/plugins@4.5.1

### Patch Changes

- Extend the SSO plugin contract with WorkOS Directory Sync (SCIM)
support.
([#4148](https://github.com/triggerdotdev/trigger.dev/pull/4148))
- Updated dependencies:
  - `@trigger.dev/core@4.5.1`

</details>

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
 2026-07-06 19:17:36 +01:00
Daniel Sutton 092b9ef07a fix(run-ops): DNS-safe, sortable base32hex run id (replace base62 KSUID) (#4154) 
## Problem

The run-ops split mints NEW-store run ids as **27-char base62 KSUIDs**.
The supervisor writes the run id into the Kubernetes pod name
(`runner-<id>`), and pod names must be DNS-1123 labels (lowercase
`[a-z0-9-]`) — so uppercase base62 ids make k8s reject the pod (422) and
**those runs never launch** (they loop in `PENDING_EXECUTING` until the
heartbeat-stall handler nacks them, forever). `.toLowerCase()` can't fix
it: base62 has both `A`(10) and `a`(36) as distinct symbols, so folding
collides distinct ids and destroys sort order.

## Fix: change the encoding, not the structure

Mint a **26-char lowercase base32hex** run id:

```
run_<24-char base32hex core><region char><version char>
      [ 6-byte ms timestamp ][ 9 CSPRNG bytes ]
```

- **base32hex** (RFC 4648 §7, alphabet `0-9a-v`): lowercase,
order-preserving, DNS-safe; 15 bytes → exactly 24 chars, no padding.
Hand-rolled encode/decode (no new dependency).
- **48-bit ms timestamp** in the leading bytes → plain string sort ==
creation order at millisecond resolution.
- **72 bits CSPRNG** entropy; PK unique constraint is the backstop (no
retry loop).
- **region / version** are raw positional chars (read via one `charAt`
before decoding/routing), version = `"1"`.

DNS-safe from birth and hyphen-free, so **firekeeper is unchanged** —
`runner-<id>-attempt-N` → strip `runner-`, cut at first hyphen still
recovers the exact id incl. region+version.

## Residency discriminator: length → version char

`classifyKind`/`classifyResidency` (`runOpsResidency.ts`) previously
distinguished NEW vs LEGACY by **id length**. That gets ambiguous with a
third format. It now discriminates on the **version char at a fixed
position** (`isRunOpsIdBody`: 26 chars, `[25] === "1"`, base32hex
alphabet) → NEW; everything else → LEGACY. Total, never throws. The
`Residency` (NEW/LEGACY) contract the routing store consumes is
unchanged; the `"ksuid"` `ResidencyKind` label is retained only because
it's the persisted `runOpsMintKsuid` feature-flag value.

## Scope / verification

- Generator + discriminator in `@trigger.dev/core` isomorphic; mint path
+ all id-shape call sites swept (~40 webapp files); changeset added
(`@trigger.dev/core` patch).
- Core unit tests (encode/decode round-trip + property, generator shape,
ms sort-order incl. intra-second, parse partitioned-vs-legacy,
firekeeper round-trip): **24 pass**. `@trigger.dev/core` builds; webapp
typechecks; format/lint clean.

## Open decisions (flagged, not silently chosen)

1. **Backward-compat**: existing 27-char base62 KSUID runs now classify
LEGACY. On test cloud these are the broken/looping runs that never
completed, so this is acceptable — but worth a conscious call before
prod. No transitional length-recognition added (keeps the discriminator
clean).
2. **Storage collation**: the sort guarantee is byte-order — if the
run-ops id column is `TEXT` with default locale collation it's silently
not honored. Confirm whether `COLLATE "C"` / `BYTEA` is needed on the
run-ops schema.
3. **Region sourcing** wiring — see `regionCharForRegion` /
`REGION_CODES`.


---

## ⚠️ Required migration — deploy in lockstep

This PR renames a persisted feature-flag key/value and an env var. These
are **not** changed by the code alone and must be migrated when this
deploys, or affected orgs silently fall back to `cuid` minting (no crash
— `defaultValue: "cuid"`):

1. **Env var** (terraform): `RUN_OPS_MINT_KSUID_ENABLED` →
`RUN_OPS_MINT_ENABLED` (carry the value over).
2. **DB** `organization.featureFlags`: migrate both the key and value
together:
   - key `runOpsMintKsuid` → `runOpsMintKind`
   - value `"ksuid"` → `"runOpsId"`

Until an org's flag row is migrated, its `runOpsMintKind` lookup misses
and it mints `cuid` (legacy) — so no NEW-store ids for that org until
the data lands.

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-07-05 10:05:54 +01:00
Daniel Sutton 4cfa596271 feat(core): id-shape residency classifier + ksuid mint primitives (run-ops split base) (#4112) 
## What

Foundation for the run-ops database split: an isomorphic **id-shape
residency classifier** and the **ksuid mint primitives**, added to
`@trigger.dev/core` under `v3/isomorphic`.

- **`runOpsResidency.ts`** — classifies a run id by its shape: 25-char
cuid → `LEGACY`, 27-char ksuid → `NEW`. Pure and environment-free (safe
on both client and server).
- **`friendlyId.ts`** — ksuid mint primitives and id helpers.
- Both exported via `v3/isomorphic/index.ts`.

## Why

This is the **base of a stacked series** implementing the run-ops DB
split (routing run-execution data to a dedicated database by id-shape).
Later PRs in the series consume this classifier and these primitives to
route reads and writes across the two databases.

On its own this PR is **purely additive** — new isomorphic helpers with
unit tests, no runtime wiring, and no behaviour change to existing code
paths.

## Tests

Unit tests for the classifier (`runOpsResidency.test.ts`) and the id /
mint primitives (`friendlyId.test.ts`).

## Notes

- Draft, stacked on `main`; subsequent PRs in the series build on top of
this one.
- A changeset for `@trigger.dev/core` will be added before this is
marked ready for review.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-07-03 11:05:39 +01:00
Oskar Otwinowski c7f6ed501c feat(cli,core): add opt-in dev-only telnet log streaming (#4110) 
Stream dev logs over a local telnet/TCP socket. `trigger dev` mirrors
its terminal output on port 6767 by default (override with
--telnet-logs-port or TRIGGER_DEV_TELNET_LOGS_PORT, 0 disables). webapp,
supervisor, and coordinator each expose an opt-in stream gated on a
per-service *_TELNET_LOGS_PORT env var. New
@trigger.dev/core/v3/telnetLogServer module (localhost-only,
backpressure-safe, plain-text) plus optional static Logger.onLog /
SimpleStructuredLogger.onLog sinks.

Then you (or your agent) can use `nc` to connect and filter out the
stream.
<img width="1103" height="239" alt="image"
src="https://github.com/user-attachments/assets/b4d47efc-8a57-4185-a159-10f2806627ae"
/>
 2026-07-02 19:16:14 +00:00
Chris Arderne c7861be520 chore: activate no-unused-vars and import linters (#4096) 
Once this is merged, oxlint is at a pretty sensible baseline.

**Enable `no-unused-vars`, `typescript/consistent-type-imports`, and
`import/no-duplicates` lint rules**

Turns on three previously-disabled oxlint rules across the monorepo and
fixes all violations:

- **`no-unused-vars`** – enabled as an error with standard ignore
patterns: unused function arguments are ignored by default (`args:
"none"`), variables/caught errors/destructured array elements prefixed
with `_` are allowed, and rest siblings are permitted.
- **`typescript/consistent-type-imports`** – enforced as an error; all
type-only imports now use the `import type` syntax.
- **`import/no-duplicates`** – enforced as an error; duplicate import
statements from the same module have been merged.

The remaining commits clean up the violations found across the codebase:
removing unused variables/imports/type aliases, adding `_` prefixes to
intentionally unused bindings, fixing duplicate imports, and converting
value imports to `import type` where appropriate.
 2026-07-02 11:37:05 +01:00
github-actions[bot] 86ef3c4979 chore: release v4.5.0 (#3998)
📚 Publish docs / publish (push) Has been cancelled
Details
🚀 Publish Trigger.dev Docker / units (push) Failing after 20s
Details
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 20s
Details
🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped
Details
🚀 Publish Trigger.dev Docker / publish-worker (push) Has been skipped
Details
🚀 Publish Trigger.dev Docker / publish-worker-v4 (push) Has been skipped
Details
🚀 Publish Trigger.dev Docker / scan-webapp (push) Has been skipped
Details
🧭 Helm Chart Release / lint-and-test (push) Has been cancelled
Details
🧭 Helm Chart Release / release (push) Has been cancelled
Details
🚀 Publish Trigger.dev Docker / 📣 Dispatch main image (push) Has been cancelled
Details 
# Trigger.dev v4.5.0

4.5.0 is the GA of the AI Agents platform. Everything built during the
prerelease line (durable agents, Sessions, AI Prompts) is now stable on
the `latest` tag, alongside a set of SDK and runtime improvements.

## AI Agents (`chat.agent`)

Run Vercel AI SDK chat completions as durable Trigger.dev tasks instead
of fragile API routes. A conversation runs as one long-lived task keyed
on `chatId`, so it survives page refreshes, network blips, redeploys,
and crashes, and every turn is a span in the dashboard.

```ts
import { chat } from "@trigger.dev/sdk/ai";
import { streamText, stepCountIs } from "ai";
import { anthropic } from "@ai-sdk/anthropic";

export const myChat = chat.agent({
  id: "my-chat",
  run: async ({ messages, signal }) => {
    return streamText({
      ...chat.toStreamTextOptions(), // system prompt, compaction, steering, telemetry
      model: anthropic("claude-sonnet-4-5"),
      messages,
      abortSignal: signal,
      stopWhen: stepCountIs(15),
    });
  },
});
```

## Sessions

The durable primitive underneath `chat.agent`, usable on its own: a
run-aware, bidirectional stream channel keyed on a stable `externalId`
whose `.in` / `.out` streams survive run boundaries (suspend, crash,
idle-timeout, redeploy). One Session spans many runs, which makes it a
good fit for agent inboxes and approval flows.

```ts
import { sessions } from "@trigger.dev/sdk";

// Create the session and trigger its first run (idempotent on externalId)
await sessions.start({
  type: "inbox",
  externalId: userId,
  taskIdentifier: "inbox-agent",
});

const session = sessions.open(userId);
await session.in.send({ text: "hello" });

const stream = await session.out.read({ signal: AbortSignal.timeout(30_000) });
for await (const chunk of stream) console.log(chunk); // durable across run swaps
```

## AI Prompts

Define prompt templates as code, versioned on every deploy, and override
the text or model from the dashboard without redeploying
(environment-scoped). Each generation links back to its prompt version
for usage, cost, and latency.

```ts
import { prompts } from "@trigger.dev/sdk";
import { z } from "zod";

export const supportPrompt = prompts.define({
  id: "customer-support",
  model: "gpt-4o",
  variables: z.object({ customerName: z.string(), issue: z.string() }),
  content: `You are a support agent for Acme.
Customer: {{customerName}}
Issue: {{issue}}`,
});

// Honors any active dashboard override, else the current deployed version
const resolved = await supportPrompt.resolve({ customerName: "Alice", issue: "Can't log in" });
// resolved.text, resolved.model, resolved.version
```

## `useChat` integration

`useTriggerChatTransport` is a Vercel AI SDK `ChatTransport` that runs
`useChat` over Trigger.dev realtime with no API routes. Text, tool
calls, reasoning, and `data-*` parts stream natively, and it works with
AI SDK v5, v6, and now v7.

## First-turn fast path (`chat.headStart`)

Runs the first turn in your warm server process while the agent boots in
parallel, cutting cold-start time-to-first-chunk roughly in half
(measured ~2.8s to ~1.2s). Available via the new
`@trigger.dev/sdk/chat-server` subpath.

## Human-in-the-loop, stop, and steering

The agent control surface: tool approvals (`needsApproval` +
`addToolApprovalResponse`), client-driven stop-generation, mid-execution
steering (`pendingMessages`), and between-turn context injection
(`chat.inject` / `chat.defer`), all durable across the conversation.

## Agent Skills

`skills.define({ id, path })` bundles a `SKILL.md` folder into your
deploy image. The agent gets a one-line summary up front and loads the
full instructions plus scoped `bash` / `readFile` tools on demand
(progressive disclosure), so a capability is something the model reaches
for rather than a pre-declared typed tool.

## `trigger skills` for coding assistants

`trigger skills` installs version-pinned Trigger.dev skills plus a
bundled docs snapshot into Claude Code, Cursor, GitHub Copilot, and
Codex, so your assistant's Trigger.dev knowledge stays current with your
installed SDK version. `trigger init` now offers to set up the MCP
server and skills too.

## Model library

A new Models page in the dashboard: a catalog of models grouped by
provider with context window, capabilities, and input / output pricing
per 1M tokens, plus a "Your models" tab showing per-model usage, cost,
and cache-hit sparklines from your actual traffic.

## Dev branches

Run multiple local `trigger dev` sessions in parallel (separate git
worktrees or coding agents) without runs colliding, each isolated with
its own dashboard, via `trigger dev --branch <name>`.

## `TriggerClient`

An instantiable client so one process can trigger and read across
projects, environments, and preview branches, each with its own auth and
baseURL, with no shared global state.

```ts
import { TriggerClient } from "@trigger.dev/sdk";

const prod = new TriggerClient({ accessToken: process.env.TRIGGER_PROD_KEY });
const preview = new TriggerClient({
  accessToken: process.env.TRIGGER_PREVIEW_KEY,
  previewBranch: "signup-flow",
});

await prod.tasks.trigger("send-email", { to: "user@example.com" });
await preview.runs.list({ status: ["COMPLETED"] });
```

## SDK and runtime

- AI SDK 7 support (v5 and v6 still supported), with OpenTelemetry
telemetry auto-wired
- Large trigger-payload offload: trigger payloads at or above 128KB
upload to object storage automatically, using the same auth and baseURL
as the trigger call
- Region support on the runs API: filter runs by region and read each
run's executing region (also on MCP `list_runs`)
- Duplicate task-id detection: `dev` and `deploy` fail with a clear
error instead of silently overwriting
- `envvars.upload` gains an `isSecret` flag to import redacted secret
variables
- Retry hardening: `TASK_MIDDLEWARE_ERROR` now retries under the task's
retry policy

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
 2026-07-02 11:26:52 +01:00
Matt Aitken 5df9fb6463 fix(core,cli): retain idempotency key metadata beyond 1000 keys per run (#4094) 
Fixes #4046

## Problem

When a single run creates more than 1000 idempotency keys (e.g. a large
batch trigger where each item calls `idempotencyKeys.create()`), the
original key and scope metadata is silently dropped for all but the most
recent 1000 keys.

`idempotencyKeys.create()` returns a plain 64-char hash and stores the
`{ key, scope }` mapping in an in-process catalog keyed by that hash.
That catalog was a fixed-size **LRU capped at 1000 entries**. Once a run
creates more than 1000 keys, the earliest mappings are evicted, so when
the SDK later looks them up to attach `idempotencyKeyOptions` to the
trigger call, it finds nothing and sends `undefined`. The affected runs
then:

- report `ctx.run.idempotencyKey` as the raw hash instead of the
user-provided key
- have no `idempotencyKeyScope`
- show empty `idempotency_key` / `idempotency_key_scope` in the
dashboard and analytics

Deduplication still works (the hash is intact); only the human-readable
metadata is lost, which makes the failure silent and hard to notice.

## Fix

- Replace the LRU catalog with an unbounded in-memory catalog, so every
key created within a run keeps its metadata regardless of how many are
created.
- Clear the catalog at each run boundary via
`resetExecutionEnvironment()` (both dev and managed workers), matching
how every other per-run manager is reset. Deployed workers reuse one
process across many runs (warm starts), so this bounds memory to a
single run's keys instead of accumulating across runs — which is the
reason the size cap existed in the first place.

## Tests

- New public-API test creates 3000 keys and asserts all of them
(including the first) retain their key/scope — this fails on `main` and
passes with the fix.
- New test for the in-memory catalog covers store/retrieve/overwrite,
large-N retention (no eviction), and `clear()`.
- New test asserts the catalog is emptied after a run-boundary reset.
- Replaces the previous LRU catalog + its eviction tests.

Verified: `@trigger.dev/core` and `trigger.dev` both build; all
idempotency tests pass. Changeset added (patch).
 2026-07-01 12:15:07 +01:00