feat/multi-keys-auth
533 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Chris Arderne
|
e15a7893fc | feat(database,rbac): add multiple environment API key foundations | ||
|
Chris Arderne
|
efd0ee8d74 |
feat(core,sdk): support additional environment API keys (#4387)
## Summary Additional environment API keys can use SDK APIs that require public access tokens. The SDK detects the additional-key format and asks the Trigger.dev server to mint scoped tokens instead of attempting to sign them locally. Root environment keys retain their existing local-signing behavior. Trigger and batch clients also prefer server-issued tokens returned in response headers while preserving compatibility with older servers. ## Deployment notes This package update is safe to publish before servers expose additional key creation. Existing root keys continue to use the current path, while an additional key used with an older server fails with an actionable upgrade error. |
||
|
nicktrn
|
722e240e4d |
feat(supervisor): add prometheus metric for outbound http requests (#4350)
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 5s
🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker-v4 (push) Has been skipped
🚀 Publish Trigger.dev Docker / scan-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / scan-supervisor (push) Has been skipped
🚀 Publish Trigger.dev Docker / units (push) Failing after 6s
🦋 Changesets PR / Create Release PR (push) Has been cancelled
🚀 Publish Trigger.dev Docker / 📣 Dispatch main image (push) Has been cancelled
Adds Prometheus metrics so the supervisor's outbound HTTP calls are
observable - including client-side failures that previously only
surfaced as a log line.
- `supervisor_outbound_request_total{name, method, status, outcome}` -
counts every outbound request. `outcome` separates a transport failure
(`network_error`), an HTTP error response (`http_error`), a response
that failed schema validation (`invalid_response`), and success (`ok`).
- `supervisor_outbound_request_duration_seconds{name, outcome}` -
latency histogram. Leaner labels than the counter (no `status`) to avoid
bucket×label cardinality; buckets match the existing dequeue-latency
histogram since these calls share the same retrying HTTP client and
long-poll envelope.
Coverage:
- The warm-start request (a one-off `fetch`) - instrumented inline; the
response status code is now also included in the failure log (it was
previously dropped).
- All worker API client calls (`SupervisorHttpClient`: dequeue, run
attempt start/complete, heartbeats, snapshots, continue, suspend,
debug-log, connect) - routed through a single instrumented `request()`
helper that reports via an optional `onHttpRequestComplete` callback on
the client, which the supervisor wires into the counter + histogram.
Low cardinality by design: `name` is a **static per-endpoint label**
(e.g. `dequeue`, `start_run_attempt`), never the interpolated URL - so
no run/snapshot IDs land in labels, mirroring the templated `route`
labels on the inbound HTTP server.
Registered on the existing metrics registry, exposed on `/metrics` with
no new wiring. Internal-only change (no package release needed), so the
changelog note is a single `.server-changes` entry.
|
||
|
Chris Arderne
|
55a3bf2858 |
feat(core,cli): add node-24 and node-26 runtimes, deprecate experimental aliases (#4337)
## Summary
Adds `node-24` and `node-26` as first-class `runtime` options in
`trigger.config.ts`. Previously these Node versions were only reachable
via the `experimental-node-24` / `experimental-node-26` names.
Those experimental names are now **deprecated aliases**: they still
resolve to `node-24` / `node-26` for backwards compatibility, but
loading a config that uses them prints a deprecation warning pointing at
the new name.
```ts
export default defineConfig({
runtime: "node-24",
project: "<your-project-ref>",
});
```
## Details
- `ConfigRuntime` (the public config schema) now accepts `node-24` and
`node-26` directly; the internal `BuildRuntime` already supported them,
so base images and the deploy path are unchanged.
- `resolveBuildRuntime` passes the new names straight through and keeps
mapping the experimental aliases to their replacements.
- Renamed the runtime helper from `isExperimentalConfigRuntime` to
`isDeprecatedConfigRuntime` and added `deprecatedRuntimeReplacement` so
the CLI can name the replacement in its warning.
- Docs snippet updated to list the new versions and flag the deprecated
names.
|
||
|
Chris Arderne
|
dc87b884e7 |
chore: upgrade to typescript 6 (#4310)
## Summary Upgrades the workspace to TypeScript 6.0.3 and applies the compiler, type, and build configuration changes required to preserve package layouts and existing runtime behavior, apart from correcting the HTTP status field used for deployment connection errors. ## Compatibility - Centralizes TypeScript 6.0.3 through the pnpm workspace catalog. - Replaces compiler options and module resolution modes that TypeScript 6 no longer accepts. - Restores explicit Node types where TypeScript 6 no longer includes them transitively. - Adds explicit declaration build roots that preserve each package's existing output layout. - Patches tsup to stop injecting the removed `baseUrl` option during declaration builds. - Uses type-only assertions for stricter typed-array and stream definitions without changing runtime behavior. - Reads the EventSource v3 HTTP status from `code`, so deployment connection errors include it correctly. - Keeps standalone CLI compatibility fixtures pinned to their existing TypeScript version and lockfiles. `turbo run typecheck` and the complete PR test suite are green. |
||
|
Chris Arderne
|
6997aeb05e |
fix: security release 2026-07-08 (#4316)
⚒️ Publish Worker (v4) / build (supervisor) (push) Has been cancelled
|
||
|
Chris Arderne
|
d7ec75d5ad |
feat(runtime): add experimental Node.js 24 and 26 task runtimes (#4085)
⚒️ Publish Worker (v4) / build (supervisor) (push) Has been cancelled
## Summary Adds experimental Node.js 24 and 26 task runtimes through the `experimental-node-24` and `experimental-node-26` config values. Existing runtime defaults and the `node`, `node-22`, and `bun` behavior remain unchanged. The unprefixed `node-24` and `node-26` config values remain unavailable until the runtimes are ready for general use. ## Design Experimental config values normalize to canonical runtime identifiers before build manifests are created, keeping deployment metadata and execution behavior consistent. Kubernetes task pods also use the runtime-default seccomp profile so modern Node.js versions fall back from io_uring to checkpoint-compatible system calls. |
||
|
nicktrn
|
976171ea16 |
feat(webapp): management API for orgs, projects, members, and settings (#4146)
## Summary
Adds a set of PAT-authenticated management API endpoints so orgs,
projects, members/invites, environment variables, and a few
project/environment settings can be managed programmatically (scripting,
automation) rather than only through the dashboard. Each route is a thin
wrapper over the **existing** service the dashboard already uses, with
the same authorization applied at the route layer - no new business
logic.
## Endpoints
**Organizations**
- `POST /api/v1/orgs` - create an org (`createOrganization`)
- `PATCH /api/v1/orgs/:orgParam` - rename (title)
- `DELETE /api/v1/orgs/:orgParam` - soft-delete
(`DeleteOrganizationService`; keeps the active-subscription guard)
**Members & invites**
- `GET /api/v1/orgs/:orgParam/members` - list members + pending invites
- `DELETE /api/v1/orgs/:orgParam/members/:memberId` - remove a member
(last-member guarded)
- `POST /api/v1/orgs/:orgParam/invites` - invite by email
(`inviteMembers`, sends the invite email)
- `DELETE /api/v1/orgs/:orgParam/invites/:inviteId` - revoke an invite
**Projects**
- `PATCH /api/v1/projects/:projectRef` - rename
(`ProjectSettingsService`)
- `DELETE /api/v1/projects/:projectRef` - soft-delete
(`DeleteProjectService`)
- `PUT /api/v1/projects/:projectRef/default-region` - set the default
region by worker-group name (`SetDefaultRegionService`)
- project GET/list now return `defaultRegion` (worker-group name, or
null when unset)
**Environments**
- `POST /api/v1/projects/:projectRef/:env/pause` and `/resume`
(`PauseEnvironmentService`)
- `POST /api/v1/projects/:projectRef/:env/regenerate-api-key` - rotate
the env secret key (`regenerateApiKey`, RBAC `write:apiKeys`)
- env var create now accepts an optional `isSecret` flag
## Auth & authorization
- All routes authenticate with a **Personal Access Token**
(`Authorization: Bearer tr_pat_...`).
- Org/project routes are built on the PAT route builders in
`apiBuilder.server.ts`: `createLoaderPATApiRoute` (already existed) and
**`createActionPATApiRoute`** (added here - the loader builder had no
mutation counterpart). The builder runs auth, resolves the org/project
role-floor via `context`, and enforces a declarative `authorization`
block using the same RBAC actions the dashboard applies
(`manage:organization` / `read:members` / `manage:members` /
`manage:project`). Handlers keep a membership-scoped query as the floor,
so a non-member gets a 404. This also gives these routes `tenantContext`
user attribution (Sentry) and `ServiceValidationError`-to-status mapping
for free.
- **Membership floor (important).** The OSS RBAC fallback grants a
permissive ability, so `ability.can(...)` can't reject a non-member on
self-hosted. Every handler therefore resolves the target scoped to the
caller's membership (`members: { some: { userId } }`) → 404 for
non-members. `authorization` is the *role* gate; this is the *tenant*
gate. `resolveOrganizationForApiUser`
(`organizationApiAccess.server.ts`) is the org-tier version of the
existing `findProjectByRef` - org-addressed PAT routes are new, so no
such helper existed before.
- Env-tier routes reuse the existing `authorizePatEnvironmentAccess`
(`write:apiKeys`).
### What `createActionPATApiRoute` gives you
A route is pure declaration - the builder handles auth, RBAC,
validation, tracing, and error mapping:
```ts
export const action = createActionPATApiRoute(
{
method: "PUT", // one verb, or ["PATCH", "DELETE"] for multi-verb routes
params: ParamsSchema,
body: SetDefaultRegionRequestBody, // zod-validated
context: async ({ projectRef }) => { // resolve the org for the RBAC role-floor
const project = await prisma.project.findFirst({
where: { externalRef: projectRef, deletedAt: null },
select: { organizationId: true },
});
return project ? { organizationId: project.organizationId } : {};
},
authorization: { action: "manage", resource: () => ({ type: "project" }) },
},
async ({ params, body, authentication, ability }) => {
// auth + authz already enforced. Just do the work.
// `throw new ServiceValidationError("Region not found", 400)` → mapped to that status.
return json({ ok: true });
}
);
```
Handled for you, so handlers stay thin:
- **Method allowlist** - `method` accepts a verb or an array; any other
verb → `405` with an `Allow` header, *before* auth runs:
```ts
const allowedMethods = method ? (Array.isArray(method) ? method :
[method]) : undefined;
if (allowedMethods && !(allowedMethods as
string[]).includes(request.method.toUpperCase())) {
return json({ error: "Method not allowed" }, { status: 405, headers: {
Allow: allowedMethods.join(", ") } });
}
```
- **PAT / user-actor auth** in a single roundtrip → `401` on
missing/invalid/revoked token.
- **RBAC** - `context` computes the caller's role-floor for the target
org/project; `authorization` gates it → `403` with a structured error
body.
- **Sentry attribution** - `tenantContext.enrich({ userId })` so events
from the handler carry the acting user.
- **Typed errors** - a thrown `ServiceValidationError` is mapped to its
`.status` (default 400); anything else → `500`, and expected boundary
errors are logged as `warn` (kept out of Sentry).
- **Validation** - params / query / headers / body are all zod-checked →
`400` with details.
## Notes for reviewers
- Everything wraps an existing service; the intent is API parity for
things that are currently dashboard-only, not new behaviour.
- `createActionPATApiRoute` is new shared infra (the PAT + RBAC mutation
builder that didn't exist). It's self-contained - the loader builder and
existing routes are untouched.
- `@trigger.dev/core` gets one additive field (`defaultRegion` on the
project response, optional/nullable for client-server version skew) -
changeset included, patch.
- `removeTeamMember`'s last-member guard is now atomic (Serializable
transaction via the `$transaction` helper, with retry), so the dashboard
and API both get it server-side. Added a `## Transactions` rule to
`apps/webapp/CLAUDE.md` (always use the `$transaction` helper);
migrating the remaining direct usages is tracked in TRI-11698.
## Open questions
- ~~Is PAT the right auth (vs OAT for automation)?~~ **Resolved: PAT.**
Organization Access Tokens are currently internal-only (used by the
image builder) and not user-accessible, so they can't back this yet.
- Should any of these be gated behind a flag or scope?
- Naming/shape of the routes.
|
||
|
Chris Arderne
|
64e5d732ad |
chore(webapp,core): remove the unused ResourceMonitor server logging helper (#4244)
The `ResourceMonitor` server-side logging helper is no longer used. It periodically logged the webapp process own memory, disk, and CPU usage behind the `RESOURCE_MONITOR_ENABLED` flag (off by default), and was also exported from `@trigger.dev/core/v3/serverOnly` with no other consumers. This removes the helper, its `@trigger.dev/core` export, the webapp wiring, and the `RESOURCE_MONITOR_ENABLED` env var. The supervisor has its own unrelated `ResourceMonitor` class, which is left untouched. |
||
|
Eric Allam
|
5ba8557a51 |
chore(webapp,core): remove the end-of-life v3 (engine V1) execution stack (#4236)
## Summary v3 (the engine that ran the SDK v3 era, internally `RunEngineVersion.V1`) is end-of-life. Following the removal of the v3 execution apps ([#4194](https://github.com/triggerdotdev/trigger.dev/pull/4194)) and the legacy dev websocket ([#4198](https://github.com/triggerdotdev/trigger.dev/pull/4198)), this removes the remaining v3 execution stack from the server. Clients still on v3 (an old SDK or CLI that has not upgraded) keep getting a clear "upgrade to v4" response. Triggers, batch triggers, reschedules, and deploys that resolve to v3 are rejected with a graceful 4xx pointing at the migration guide, never a 5xx, so a stale client cannot affect server health. Self-hosted instances still running v3 should stay on the 4.5.x release line until they migrate. ## What is removed - The MarQS queue and its shared/dev queue consumers. - The v3 socket.io namespaces (coordinator, provider, shared-queue) and the v3 run lifecycle services (attempt, checkpoint, and batch-resume). - The graphile-worker background job system; all live jobs already run on `@trigger.dev/redis-worker`. - The `DEPRECATE_V3_ENABLED` flag: v3 is now rejected unconditionally, so the flag is gone. - Unused v3 exports from `@trigger.dev/core` (the `v3/zodNamespace` subpath and the legacy socket message catalogs) and the now-dead MarQS environment variables. ## What stays The v4 engine is untouched. The graceful v3 rejection boundary stays, `determineEngineVersion` still detects a v3 project so it can reject it, and the batch service plus batch-completion worker stay for current clients. Live queue concurrency limits and metrics now read from the v4 run engine instead of MarQS, and a brand-new dev environment now defaults to v4. ## Dependency cleanup Removes webapp dependencies left unused by this change: `seedrandom` and `semver` (only the removed v3 code used them) plus a set that was already dead, their orphaned `@types` packages, and two dead files. Adds a `knip:deps` script and a `knip.json` config so unused dependencies can be found the same way going forward. |
||
|
Iss
|
983bd03131 |
feat: support isSecret in syncEnvVars (#4203)
## What
Adds per-variable secret support to the `syncEnvVars` build extension.
Return `{ name, value, isSecret: true }` and the variable is stored as a
secret (redacted in the dashboard, value non-revealable), just like a
manually created secret env var. Secret and non-secret variables can be
mixed in one callback.
```ts
syncEnvVars(async () => [
{ name: "PUBLIC_API_URL", value: "https://api.example.com" },
{ name: "DATABASE_URL", value: "postgres://...", isSecret: true },
]);
```
## How
Env vars flow through the build pipeline as a flat name→value map, and
the import API's `isSecret` is per-call. So secret vars are carried
through the layer + manifest in parallel `secretEnv` / `secretParentEnv`
maps, and at deploy time they go up in a second `importEnvVars` call
with `isSecret: true` (the plain vars in the first call). The record
form (`{ KEY: "value" }`) is unchanged and stays non-secret.
## Commits
- `feat(core)`: carry secret env vars through the build layer + manifest
schema
- `feat(build)`: partition `isSecret` vars in `syncEnvVars`
- `feat(cli)`: merge secret layers and import them with `isSecret: true`
at deploy
- `test(build)`: cover the partitioning + document `isSecret`
## Testing
- vitest covers the partitioning (secret/non-secret × child/parent) and
that the record form stays non-secret.
- Verified against a local webapp that the deploy's import contract
stores the secret var redacted (`isSecret: true`) and the plain var
visible.
Closes TRI-11099
|
||
|
nicktrn
|
a6bd370e42 |
chore: remove end-of-life v3 execution components (#4194)
v3 (engine V1) is end-of-lifed and the v3 clusters are gone, so this removes the dead v3 execution code from the monorepo. It's the first pass of TRI-11824 - the webapp v3 code paths are deliberately left untouched and gated for a follow-up. ## Apps Deletes the three v3-only execution apps and their build wiring: - `apps/coordinator`, `apps/kubernetes-provider`, `apps/docker-provider` - `.github/workflows/publish-worker.yml` - it built only those three; the v4 worker publish is a separate workflow - Their references in `.changeset/config.json`, `.cursorignore`, `CHANGESETS.md`, `CONTRIBUTING.md`, `.server-changes/README.md` - `pnpm-lock.yaml` regenerated to prune the apps and their app-only dependencies (`socket.io`, `@kubernetes/client-node`, `p-queue`, `execa`, `prom-client`, `tinyexec`) ## Core Removes the helpers in `@trigger.dev/core` that only those apps used - `ProviderShell`, `SimpleLogger`, the `Exec`/process helpers, `isExecaChildProcess`, `getTextBody`, and `testDockerCheckpoint`. Each was verified to have no remaining consumers anywhere in the repo. Kept the helpers still used elsewhere: `ExponentialBackoff` (warm-start client), `HttpReply`/`getJsonBody` (serverOnly http server), `SimpleStructuredLogger` (widely used), and `ZodNamespace`/`ZodSocketConnection` (still referenced by legacy v3 webapp code, hence the follow-up pass). The `./v3/apps` and `./v3/serverOnly` export subpaths remain - only dead members were trimmed from their barrels, so no `package.json` exports changed. ## Verification `@trigger.dev/core` builds, and `typecheck` passes for core, supervisor, cli-v3, run-engine, redis-worker, and webapp. refs TRI-11824 |
||
|
Wes Mason
|
76c37ecd24 |
feat(sdk,core,webapp): offload large batch payloads to object storage (#4165)
## Summary `batchTrigger` and `batchTriggerAndWait` (and the by-id and by-task variants) now offload any per-item payload over 128KB to object storage before sending, the same way single `trigger`/`triggerAndWait` already do since [#3785](https://github.com/triggerdotdev/trigger.dev/pull/3785). A batch of large items no longer inflates the request body past the API limit. ## Demo A live local run: `batchTriggerAndWait` of 5 items × 300KB (1.5MB total). Each item offloads to object storage, so the receiver run rows hold a 65-byte `application/store` pointer instead of the 300KB body, and every item round-trips (received == sent). <img width="1000" height="494" alt="batch large-payload offload demo" src="https://github.com/user-attachments/assets/77ae3958-97d6-4b5c-ab25-39b217caefbc" /> ## Design Both the array and streaming batch paths funnel through `executeBatchTwoPhase`, so offloading happens once there: each item is measured, then offloaded through the existing `conditionallyExportPacket` when it crosses 128KB, with bounded concurrency so a big batch doesn't fire an unbounded number of presigned PUTs. Because items are offloaded before the request, SDK batches arrive as small `application/store` references, so the server-side inline offload during item ingest (parallelised in [#3777](https://github.com/triggerdotdev/trigger.dev/pull/3777)) mostly no longer fires for them. Every trigger and item also carries its pre-offload serialised size as `options.payloadSize`. The trigger span records that value, so an offloaded payload shows its real size instead of the size of the small object-store reference (previously the span measured the reference). |
||
|
Chris Arderne
|
aa74e68c71 |
feat(sdk): add bulk replay to api and sdk (#4105)
## Summary
Adds SDK and API support for run bulk actions. You can now create bulk
cancel or replay actions from `@trigger.dev/sdk` using run IDs or the
same filters as `runs.list()`, then retrieve, list, poll, or abort the
action by its `bulk_` handle.
Tests, docs, changesets added.
## Design
The dashboard bulk action service now accepts structured filters instead
of reading directly from a dashboard request, so the dashboard and API
share the same creation path. Replay actions created through the API are
attributed with the existing `api` trigger source, while
dashboard-created actions keep `dashboard`.
The SDK exposes the new surface under `runs.bulk.*`, including
`targetRegion` for replay region overrides and cursor pagination for
listing bulk actions.
## Filters and runIds
Nuance on filters. If `filter` is provided, it MUST have at least one
key. This is to remove the footgun of passing no filter and selecting
all runs.
```typescript
{ action: "cancel", runIds: ["run_1"] } // valid
{ action: "cancel", runIds: [] } // invalid, min(1)
{ action: "cancel", filter: { status: "FAILED" } } // valid
{ action: "cancel", filter: {} } // invalid
{ action: "cancel", filter: {}, runIds: ["run_1"] } // invalid
```
|
||
|
Eric Allam
|
add0a7da0a |
fix(sdk,core): stop chat sessions dropping messages that arrive during a turn (#4176)
## Summary Sending a message to a chat whose run had ended could make the message vanish: the continuation run replayed already-answered messages, never processed the new one, and a page refresh lost it entirely. Chasing that report surfaced four composing message-loss bugs in the chat session runtime; this PR fixes all of them, each with a regression test. ## The fixes 1. **Stale resume cursor.** Records delivered while a run was suspended (the waitpoint path) advanced the SSE resume counter but not the committed-consume cursor, so the `session-in-event-id` header stamped on turn-completes went stale by one record per suspended turn. Continuation boots seed from that header, which is what made them replay already-processed messages. `session.in.wait()` now advances both cursors. 2. **Only the first buffered message dispatched.** Messages arriving during a turn are consumed into a buffer whose end-of-turn pickup dispatched only the first entry; the buffer was recreated each turn, so the rest were discarded, and since consuming a record commits the cursor the loss was permanent. A continuation boot's replay delivers several records back-to-back, which put the user's new message at index 1 or later. The buffer now outlives the turn and drains one message per turn in both `chat.agent` and `chat.createSession` (whose equivalent buffer was never read at all). 3. **Post-stop window in `chat.createSession`.** The turn's message listener stayed attached through the stopped turn's post-stream work, so a message sent shortly after stopping a turn was consumed into the dead steering queue and lost. The listener now detaches when the stream settles, matching the `chat.agent` loop. 4. **Handler leak on errored turns.** A turn that threw outside the streaming section (for example from an `onTurnStart` hook) leaked its message listener. Previously that silently lost mid-turn messages; with the loop-level buffer it would have duplicated them instead. The subscription handle is now detached by the turn's catch/finally, and `chat.createSession` defensively detaches its prior turn's listener when user code exits a turn without `complete()`/`done()`. ## Verification Reproduced end-to-end with the ai-chat reference project before the fix (message consumed but never answered, two replayed turns, gone on refresh) and verified after (single clean turn, survives refresh, turn-complete cursors strictly advancing). Regression tests in `packages/trigger-sdk/test/pending-message-drain.test.ts` cover all four, each verified red against the unfixed behavior. A smoke sweep of the standard chat scenarios (basic send, multi-turn, suspend/resume, mid-stream refresh, stop, steering, cancel + continue, and the `createSession` variant) passes on the final branch state. |
||
|
Matt Aitken
|
1a033b665b |
fix(webapp,core): retry run resume through transient database outages (#4161)
## Summary When the platform database is briefly unreachable while a run is resuming from a wait, the run no longer fails with `TASK_EXECUTION_ABORTED`. The worker now retries the resume through the outage instead of aborting on the first blip. ## Root cause Resuming a run calls the engine's `continue` worker-action endpoint. That route caught every error and returned a `422`, which the worker's HTTP client treats as non-retryable. So a transient Prisma infrastructure error (for example `P1001` "Can't reach database server") was flattened into a permanent failure: the worker gave up, force-killed the run process, and completed it with `TASK_EXECUTION_ABORTED`. ## Fix - The `continue` route now lets infrastructure errors propagate to the generic 500 handler (message scrubbed, and retryable by the worker's HTTP client), the same treatment the trigger path already gives them via `isInfrastructureError`. Genuine validation errors (snapshot mismatch, invalid state) still return `422`, so a stale retry stays non-retryable. Resuming is idempotent server-side (guarded by the snapshot id), so retrying is safe. - The worker's `continueRunExecution` calls (both the runner-to-supervisor and supervisor-to-engine hops) retry with a longer, jittered backoff so they can ride out an outage lasting tens of seconds, and the jitter keeps a fleet of resuming runs from stampeding the database the moment it recovers. Builds on #3960, which scrubbed the leaked message on these routes but left the status non-retryable. No changeset: this is a server-side behaviour fix recorded via `.server-changes`. The `@trigger.dev/core` edits are internal run-engine worker plumbing, not a public API change. |
||
|
Daniel Sutton
|
092b9ef07a |
fix(run-ops): DNS-safe, sortable base32hex run id (replace base62 KSUID) (#4154)
## Problem
The run-ops split mints NEW-store run ids as **27-char base62 KSUIDs**.
The supervisor writes the run id into the Kubernetes pod name
(`runner-<id>`), and pod names must be DNS-1123 labels (lowercase
`[a-z0-9-]`) — so uppercase base62 ids make k8s reject the pod (422) and
**those runs never launch** (they loop in `PENDING_EXECUTING` until the
heartbeat-stall handler nacks them, forever). `.toLowerCase()` can't fix
it: base62 has both `A`(10) and `a`(36) as distinct symbols, so folding
collides distinct ids and destroys sort order.
## Fix: change the encoding, not the structure
Mint a **26-char lowercase base32hex** run id:
```
run_<24-char base32hex core><region char><version char>
[ 6-byte ms timestamp ][ 9 CSPRNG bytes ]
```
- **base32hex** (RFC 4648 §7, alphabet `0-9a-v`): lowercase,
order-preserving, DNS-safe; 15 bytes → exactly 24 chars, no padding.
Hand-rolled encode/decode (no new dependency).
- **48-bit ms timestamp** in the leading bytes → plain string sort ==
creation order at millisecond resolution.
- **72 bits CSPRNG** entropy; PK unique constraint is the backstop (no
retry loop).
- **region / version** are raw positional chars (read via one `charAt`
before decoding/routing), version = `"1"`.
DNS-safe from birth and hyphen-free, so **firekeeper is unchanged** —
`runner-<id>-attempt-N` → strip `runner-`, cut at first hyphen still
recovers the exact id incl. region+version.
## Residency discriminator: length → version char
`classifyKind`/`classifyResidency` (`runOpsResidency.ts`) previously
distinguished NEW vs LEGACY by **id length**. That gets ambiguous with a
third format. It now discriminates on the **version char at a fixed
position** (`isRunOpsIdBody`: 26 chars, `[25] === "1"`, base32hex
alphabet) → NEW; everything else → LEGACY. Total, never throws. The
`Residency` (NEW/LEGACY) contract the routing store consumes is
unchanged; the `"ksuid"` `ResidencyKind` label is retained only because
it's the persisted `runOpsMintKsuid` feature-flag value.
## Scope / verification
- Generator + discriminator in `@trigger.dev/core` isomorphic; mint path
+ all id-shape call sites swept (~40 webapp files); changeset added
(`@trigger.dev/core` patch).
- Core unit tests (encode/decode round-trip + property, generator shape,
ms sort-order incl. intra-second, parse partitioned-vs-legacy,
firekeeper round-trip): **24 pass**. `@trigger.dev/core` builds; webapp
typechecks; format/lint clean.
## Open decisions (flagged, not silently chosen)
1. **Backward-compat**: existing 27-char base62 KSUID runs now classify
LEGACY. On test cloud these are the broken/looping runs that never
completed, so this is acceptable — but worth a conscious call before
prod. No transitional length-recognition added (keeps the discriminator
clean).
2. **Storage collation**: the sort guarantee is byte-order — if the
run-ops id column is `TEXT` with default locale collation it's silently
not honored. Confirm whether `COLLATE "C"` / `BYTEA` is needed on the
run-ops schema.
3. **Region sourcing** wiring — see `regionCharForRegion` /
`REGION_CODES`.
---
## ⚠️ Required migration — deploy in lockstep
This PR renames a persisted feature-flag key/value and an env var. These
are **not** changed by the code alone and must be migrated when this
deploys, or affected orgs silently fall back to `cuid` minting (no crash
— `defaultValue: "cuid"`):
1. **Env var** (terraform): `RUN_OPS_MINT_KSUID_ENABLED` →
`RUN_OPS_MINT_ENABLED` (carry the value over).
2. **DB** `organization.featureFlags`: migrate both the key and value
together:
- key `runOpsMintKsuid` → `runOpsMintKind`
- value `"ksuid"` → `"runOpsId"`
Until an org's flag row is migrated, its `runOpsMintKind` lookup misses
and it mints `cuid` (legacy) — so no NEW-store ids for that org until
the data lands.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
Daniel Sutton
|
4cfa596271 |
feat(core): id-shape residency classifier + ksuid mint primitives (run-ops split base) (#4112)
## What Foundation for the run-ops database split: an isomorphic **id-shape residency classifier** and the **ksuid mint primitives**, added to `@trigger.dev/core` under `v3/isomorphic`. - **`runOpsResidency.ts`** — classifies a run id by its shape: 25-char cuid → `LEGACY`, 27-char ksuid → `NEW`. Pure and environment-free (safe on both client and server). - **`friendlyId.ts`** — ksuid mint primitives and id helpers. - Both exported via `v3/isomorphic/index.ts`. ## Why This is the **base of a stacked series** implementing the run-ops DB split (routing run-execution data to a dedicated database by id-shape). Later PRs in the series consume this classifier and these primitives to route reads and writes across the two databases. On its own this PR is **purely additive** — new isomorphic helpers with unit tests, no runtime wiring, and no behaviour change to existing code paths. ## Tests Unit tests for the classifier (`runOpsResidency.test.ts`) and the id / mint primitives (`friendlyId.test.ts`). ## Notes - Draft, stacked on `main`; subsequent PRs in the series build on top of this one. - A changeset for `@trigger.dev/core` will be added before this is marked ready for review. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Oskar Otwinowski
|
c7f6ed501c |
feat(cli,core): add opt-in dev-only telnet log streaming (#4110)
Stream dev logs over a local telnet/TCP socket. `trigger dev` mirrors its terminal output on port 6767 by default (override with --telnet-logs-port or TRIGGER_DEV_TELNET_LOGS_PORT, 0 disables). webapp, supervisor, and coordinator each expose an opt-in stream gated on a per-service *_TELNET_LOGS_PORT env var. New @trigger.dev/core/v3/telnetLogServer module (localhost-only, backpressure-safe, plain-text) plus optional static Logger.onLog / SimpleStructuredLogger.onLog sinks. Then you (or your agent) can use `nc` to connect and filter out the stream. <img width="1103" height="239" alt="image" src="https://github.com/user-attachments/assets/b4d47efc-8a57-4185-a159-10f2806627ae" /> |
||
|
Chris Arderne
|
c7861be520 |
chore: activate no-unused-vars and import linters (#4096)
Once this is merged, oxlint is at a pretty sensible baseline. **Enable `no-unused-vars`, `typescript/consistent-type-imports`, and `import/no-duplicates` lint rules** Turns on three previously-disabled oxlint rules across the monorepo and fixes all violations: - **`no-unused-vars`** – enabled as an error with standard ignore patterns: unused function arguments are ignored by default (`args: "none"`), variables/caught errors/destructured array elements prefixed with `_` are allowed, and rest siblings are permitted. - **`typescript/consistent-type-imports`** – enforced as an error; all type-only imports now use the `import type` syntax. - **`import/no-duplicates`** – enforced as an error; duplicate import statements from the same module have been merged. The remaining commits clean up the violations found across the codebase: removing unused variables/imports/type aliases, adding `_` prefixes to intentionally unused bindings, fixing duplicate imports, and converting value imports to `import type` where appropriate. |
||
|
Matt Aitken
|
5df9fb6463 |
fix(core,cli): retain idempotency key metadata beyond 1000 keys per run (#4094)
Fixes #4046 ## Problem When a single run creates more than 1000 idempotency keys (e.g. a large batch trigger where each item calls `idempotencyKeys.create()`), the original key and scope metadata is silently dropped for all but the most recent 1000 keys. `idempotencyKeys.create()` returns a plain 64-char hash and stores the `{ key, scope }` mapping in an in-process catalog keyed by that hash. That catalog was a fixed-size **LRU capped at 1000 entries**. Once a run creates more than 1000 keys, the earliest mappings are evicted, so when the SDK later looks them up to attach `idempotencyKeyOptions` to the trigger call, it finds nothing and sends `undefined`. The affected runs then: - report `ctx.run.idempotencyKey` as the raw hash instead of the user-provided key - have no `idempotencyKeyScope` - show empty `idempotency_key` / `idempotency_key_scope` in the dashboard and analytics Deduplication still works (the hash is intact); only the human-readable metadata is lost, which makes the failure silent and hard to notice. ## Fix - Replace the LRU catalog with an unbounded in-memory catalog, so every key created within a run keeps its metadata regardless of how many are created. - Clear the catalog at each run boundary via `resetExecutionEnvironment()` (both dev and managed workers), matching how every other per-run manager is reset. Deployed workers reuse one process across many runs (warm starts), so this bounds memory to a single run's keys instead of accumulating across runs — which is the reason the size cap existed in the first place. ## Tests - New public-API test creates 3000 keys and asserts all of them (including the first) retain their key/scope — this fails on `main` and passes with the fix. - New test for the in-memory catalog covers store/retrieve/overwrite, large-N retention (no eviction), and `clear()`. - New test asserts the catalog is emptied after a run-boundary reset. - Replaces the previous LRU catalog + its eviction tests. Verified: `@trigger.dev/core` and `trigger.dev` both build; all idempotency tests pass. Changeset added (patch). |
||
|
Chris Arderne
|
bfa902bd18 |
chore: enable more linters (#4080)
Re-enables ~15 oxlint rules that were blanket-disabled before. |
||
|
nicktrn
|
baaecfcff3 |
chore(core): redact sensitive flag values from exec command logs (#4087)
The `Exec` helper in `@trigger.dev/core` logs command args at debug level (and in its output/error metadata). For commands that take a credential directly on the command line - `--password`, `--token`, `--secret`, etc. - that value is logged verbatim, so turning on debug logging can surface secrets in log sinks. This masks the value of known credential-bearing flags (both `--flag value` and `--flag=value` forms) before the args are logged. The executed command is untouched - only the logged copy is redacted. Added a small unit test for the redaction helper. |
||
|
Chris Arderne
|
b54201f986 | chore: switch to oxfmt, oxlint - add ci checks (#3977) | ||
|
Chris Arderne
|
df78ef96d9 |
feat: multi dev branches (#4023)
Closes this feature request: [https://triggerdev.featurebase.app/p/isolated-dev-sessions-for-multiple-local-trigger-dev-instances](https://triggerdev.featurebase.app/p/isolated-dev-sessions-for-multiple-local-trigger-dev-instances) ### Feature notes: - CLI `trigger dev` works as before - `trigger dev --branch my-branch` to create a new branch and run against it. - `trigger dev archive --branch my-branch` to archive (or in webapp). - New webapp page to manage and archive dev branches, currently feature flagged. ### Implementation details: - No changes to data model, no backfill. `isBranchableEnvironment` column is ignored for dev branches, we use `parentEnvironmentId IS NULL` instead. - `x-trigger-branch` overloaded for preview and dev branches - New `TRIGGER_DEV_BRANCH` env var available locally. `TRIGGER_PREVIEW_BRANCH` overloaded for child runs. - Lots of new glue code to sanitise the branch checks. ### Rollout - Deploy webapp/API changes (all backwards compatible) - Manual tests on some orgs - Deploy docs, release CLI, flip feature flag for webapp feature ### NB - `api.v1.projects.$projectRef.environments.ts` will return `isBranchableEnvironment: true` for all dev environments. ### Prerequisites - [x] Typecheck will not pass until we make a new release of `@trigger.dev/platform` and bump it here |
||
|
nicktrn
|
7621601ecd |
fix(supervisor): drop debug-log requests cheaply when disabled (#4009)
Follow-up to #3992, which gated the send runner-side - but only for new runner images. Existing runners still POST a debug log per line. When `SEND_RUN_DEBUG_LOGS` is off (default), the route now drops the request immediately: `skipBodyParsing` skips the body read/parse, a bare handler returns 204, no wide event. The route stays registered so it avoids the `No route match` error log; the only per-request log left is the framework's `logger.debug` trace, suppressed at the default `info` level. Still counted by request metrics, and 204 is non-retryable so no retry storm. Adds a `skipBodyParsing` flag to the internal HTTP server. |
||
|
Eric Allam
|
5052d895b3 |
feat(webapp,core): add a public HTTP API for errors (#4005)
## Summary
Adds an environment-scoped HTTP API over the Errors feature, mirroring
the runs API. Task-run failures are grouped by a fingerprint into "error
groups," and this exposes everything you can do with them in the
dashboard:
- `GET /api/v1/errors` lists error groups, with
`filter[taskIdentifier]`, `filter[version]`, `filter[status]`
(`unresolved`/`resolved`/`ignored`), `filter[search]`, a time range, and
cursor pagination.
- `GET /api/v1/errors/{errorId}` retrieves a single group (summary,
lifecycle state, affected versions).
- `POST /api/v1/errors/{errorId}/{resolve,ignore,unresolve}` changes its
state.
- `GET /api/v1/runs?filter[error]={errorId}` lists the runs behind a
group.
Request and response schemas are exported from `@trigger.dev/core/v3` so
the SDK can reuse them, and all endpoints are documented in the API
reference (OpenAPI). `errorId` is the `error_<fingerprint>` friendly id.
## Attribution
State changes record who made them. A plain environment API key has no
user, so `resolvedBy`/`ignoredByUserId` stay null. When the caller uses
an environment JWT obtained by exchanging a personal access token or a
delegated user token at `POST /api/v1/projects/:ref/:env/jwt`, that
exchange now stamps an `act` delegation claim, and the write endpoints
read `act.sub` to attribute the change to the acting user. This is the
first endpoint to consume the `act` claim, so two small pieces of
plumbing ride along: the exchange stamps `act` for personal-access-token
subjects too (it was delegated-token-only), and the public-JWT
bearer-auth path surfaces `act.sub` to the handler.
Built on the delegated-token work in #3997.
|
||
|
Eric Allam
|
07a0e4ade9 |
feat(webapp): split Models into Your models and Model library tabs (#3958)
## Summary The Models page is now split into two tabs. **Your models** shows the models your project has actually used in the selected time range, with usage charts (cost over time, tokens over time, calls by model), a per-model table of calls / cost / avg TTFC / avg tokens-per-sec, and calls/tokens trend sparklines. **Model library** is the full catalog, reordered from alphabetical to a relevance-based provider order (Anthropic, OpenAI, Google, then the rest), newest models first within each provider, with a "New" badge on models released in the last 7 days. One time-range selector drives the whole Your models tab, so the charts, the table, and the sparklines all share the same window. Opening a model shows its own metrics with an independent range picker and a "View in AI metrics" link that opens the AI metrics dashboard filtered to that model. The active tab is kept in the URL so it survives a refresh and is shareable. ## Prompt caching & cost accuracy Both the Your models tab and the AI metrics dashboard now surface prompt-cache usage: a cache-savings column plus per-model cached-tokens and cache-hit-rate views, and a caching section on the dashboard (hit rate, cached tokens, estimated savings, and hit rate by model). Building this surfaced a cost bug. `input_tokens` is the total prompt count and already includes cache-read and cache-creation tokens, but the cost pipeline charged the full input at the input price and then added a separate cache line, so cached tokens were billed twice (and on Anthropic, cache reads were never discounted because their price is keyed differently). The input price now applies only to the non-cached remainder, with cache prices resolved across the provider-specific keys, so LLM cost and the cache hit-rate metric are accurate. Hit rate is computed as cached reads over total input. ## Notes Also fixes React "invalid DOM property" console warnings from the provider icons (the Llama and DeepSeek SVGs used raw `fill-rule` / `clip-rule` / `clip-path` attributes), which this page surfaces by rendering more provider icons. ## Screenshots **Your models tab:** usage charts and a per-model table with calls/tokens trend sparklines. <img width="2560" height="1267" alt="1-your-models-tab" src="https://github.com/user-attachments/assets/859bd24f-9047-4828-8bbb-83e5882846d6" /> **Model library:** provider-relevance ordering with a "New" badge on models released in the last 7 days. <img width="2560" height="1267" alt="2-model-library-tab" src="https://github.com/user-attachments/assets/46dd54b9-80f9-4922-ade9-5935b08dfebc" /> **Model detail, Metrics tab:** per-model range picker and a "View in AI metrics" link. <img width="2560" height="1267" alt="3-model-detail-metrics" src="https://github.com/user-attachments/assets/0f65d9d0-6142-4918-93f0-110bb277101a" /> **View in AI metrics:** the dashboard deep-linked and filtered to the selected model. <img width="2560" height="1267" alt="4-ai-metrics-filtered" src="https://github.com/user-attachments/assets/821f256c-e305-493c-98c7-eafaf2f57f83" /> |
||
|
Eric Allam
|
2b6d2492fe |
fix(sdk,core): head-start handover correctness and continuation boot latency (#3907)
## Summary Three related fixes for `chat.headStart` and continuation boots, found while investigating customer reports. **1. `chat.headStart` now works with `hydrateMessages`.** The turn-0 handover splice only ran on the default accumulation path, so agents registering `hydrateMessages` silently lost the warm route's step-1 response: pure-text turns fired `onTurnComplete` with no assistant message (and an empty durable write), tool-call turns re-ran step 1 from scratch under a fresh `messageId`, and the head-start user message never reached the hydrate hook at all. The first-turn history now reaches `hydrateMessages` as `incomingMessages`, and the splice runs after both accumulation branches, deduplicated by the handover `messageId`. **2. Reasoning parts survive the handover.** The synthesized partial only mapped text and tool-call parts, so an extended-thinking model's step-1 reasoning streamed to the browser but never reached durable history. Reasoning parts now map through with provider metadata, so Anthropic thinking signatures survive a UIMessage round trip on hydrate replays. **3. Continuation boots no longer stall for ~10 seconds.** The `.in` resume cursor was found by draining an SSE subscription that only closes after its 5 second inactivity window, and the scan ran twice per boot. It is now a non-blocking records read of the latest turn-complete header, runs at most once per boot, the boot reads run concurrently, and chat snapshots carry the cursor so subsequent boots skip the scan entirely. Measured locally on a cancel-then-continue repro: pre-turn continuation latency dropped from ~11s to ~0.5s. Every fix was verified red-green: new unit tests reproduced each failure before the fix, and end-to-end smoke tests against a live local stack covered both handover legs, reasoning persistence with extended thinking (including a follow-up turn that round-trips the persisted signed reasoning back to the provider), and the boot timing comparison. ## Rollout SDK-only; no server change required. A new SDK against a server that does not serialize record headers degrades to the existing no-cursor fallback. Old SDKs ignore the new snapshot field, and new SDKs fall back to the records scan on snapshots written before it existed. |
||
|
Eric Allam
|
f5f29ceb26 |
fix(sdk,core): chat.agent delivery, idempotency, and recovery fixes (#3891)
## Summary A batch of reliability fixes for `chat.agent`: - A user message sent while the agent is streaming is no longer delivered twice (which could run a duplicate turn). - Input appends carry an idempotency key (`X-Part-Id`) so a retried send can't duplicate a message. - `onTurnComplete` now fires on errored turns with the thrown error attached, and the failed turn's user message is persisted so it isn't lost on the next run. - Stopping a generation clears the streaming state, so a page reload doesn't replay the stopped turn. - Custom agents and manual `chat.writeTurnComplete` callers trim the output stream, sending a custom action no longer leaves a second stream reader running, a long-lived `watch` subscription no longer grows its dedupe set without bound, promoting a queued message to steering no longer risks a double-send, and runs keep the full set of dashboard tags. The `X-Part-Id` header is accepted by current servers (they just don't dedupe on it yet), so this is safe to ship ahead of the matching server change. |
||
|
Saadi Myftija
|
081b6bac17 |
feat(supervisor): publish client-side dequeue API latency as a Prometheus histogram (#3887)
⚒️ Publish Worker (v4) / build (supervisor) (push) Has been cancelled
The supervisor's dequeue round-trip time (`POST /engine/v1/worker-actions/dequeue`) was measured but only flowed into wide events and OTel span attributes — there was no Prometheus series, so latency percentiles and error rates weren't queryable. This adds `queue_consumer_pool_dequeue_duration_seconds` (histogram, label `outcome=success|empty|error`) to the existing consumer-pool metrics, scraped automatically by the existing ServiceMonitors on queue-raider/schedule-raider/supervisor. - Records every dequeue call, including failed ones, which previously emitted no timing at all - The pool's shared `ConsumerPoolMetrics` instance is injected into each consumer (mirrors the `BackpressureMetrics` → `BackpressureMonitor` wiring) - Buckets extend to 30s because `wrapZodFetch` retries internally (5 attempts, ≥7.5s backoff before a retryable error surfaces) - Existing `dequeueResponseMs` wide-event/span behavior unchanged |
||
|
Eric Allam
|
87448ccaf2 |
feat(webapp,core): add an endpoint to list a project's environments (#3880)
## Summary
Adds `GET /api/v1/projects/{projectRef}/environments` (personal access
token auth), which lists the base environments a user can access for a
project — their own dev environment plus the project's staging, preview,
and production environments.
## Details
- Built on the PAT route builder, so it inherits org-membership auth and
the per-resource ability check.
- `dev` is scoped to the token owner; archived environments are
excluded.
- Returns the branchable **parent** preview environment — preview branch
children are not included. A consumer targets the parent; branch-level
overrides are handled separately.
- Sorted to match the dashboard's environment switcher (dev → staging →
preview → prod), and never returns API keys.
Example response:
```json
[
{ "id": "...", "slug": "dev", "type": "DEVELOPMENT", "isBranchableEnvironment": false, "branchName": null, "paused": false },
{ "id": "...", "slug": "stg", "type": "STAGING", "isBranchableEnvironment": false, "branchName": null, "paused": false },
{ "id": "...", "slug": "preview", "type": "PREVIEW", "isBranchableEnvironment": true, "branchName": null, "paused": false },
{ "id": "...", "slug": "prod", "type": "PRODUCTION", "isBranchableEnvironment": false, "branchName": null, "paused": false }
]
```
|
||
|
Matt Aitken
|
f4a96bdf84 |
Fail dev and deploy on duplicate task ids (#3865)
## What `dev` and `deploy` now fail with a clear error when two tasks are defined with the same id — including across task types (e.g. a scheduled task and a regular task sharing an id). ## Why Tasks are registered into the resource catalog keyed by id, so a second definition with the same id silently overwrote the first. One of the tasks would just vanish from the worker with no warning — easy to miss, hard to debug. (Any earlier duplicate-id check ran against the post-registration task list, which is already de-duplicated, so it never actually fired.) ## How - **Detect at registration** (`@trigger.dev/core`): `StandardResourceCatalog` records a collision when a task id is registered more than once, capturing the files involved — the only point where both definitions are visible before the id-keyed map collapses them. Exposed via `listTaskIdCollisions()`. - **Fail indexing** (`trigger.dev` CLI): both index workers report collisions via a new `TASKS_FAILED_TO_INDEX` message; `indexWorkerManifest` rejects with a new `DuplicateTaskIdsError`. `dev` renders a dedicated error (offending ids + files + docs link); `deploy` fails with the same message. Runtime worker boot is unaffected — it never reads the collisions. - **Server-side backstop** (webapp): background-worker registration also rejects duplicate ids with a clear `ServiceValidationError`, so duplicates are caught even from an older CLI. ## Testing - Unit tests for collision collection in the catalog and for the error-message formatting (standard, same-file, and 3+-definition cases). - Verified end to end against a local webapp: a project with a regular task and a scheduled task sharing an id now fails `dev` with the dedicated error; a project with distinct ids still starts normally. ## Changeset Patch for `@trigger.dev/core` and `trigger.dev`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
nicktrn
|
35c56f1d09 |
feat(supervisor): add opt-in dequeue backpressure (#3836)
The supervisor can now pause dequeuing - and freeze consumer-pool scale-up - when a backpressure signal says the cluster can't place more work, then ramp dequeuing back up gradually once it clears. The signal is a verdict published to a Redis key by a cluster-side component; the supervisor reads it on a short refresh and gates `preDequeue` on it. Off by default (`TRIGGER_DEQUEUE_BACKPRESSURE_ENABLED`). Everything fails open: a missing, stale, or unreadable verdict never pins the brake, and the hot-path read is a synchronous cached lookup with no I/O. The scale-up freeze leaves scale-down untouched, and on release the resume is ramped so a deep queue isn't hammered all at once. Dry-run is on by default (`TRIGGER_DEQUEUE_BACKPRESSURE_DRY_RUN`): even once enabled it only logs what it would have done, and surfaces the computed state through metrics, until explicitly set to act. Prometheus: `supervisor_backpressure_engaged`, `_dry_run`, `_skipped_dequeues_total`. Refs TRI-5354 |
||
|
Eric Allam
|
85886b96da |
feat(webapp,supervisor): isolate scheduled runs on a dedicated worker queue (#3839)
## Summary Scheduled runs and their descendants can now be routed to a dedicated per-region worker queue, processed by a separate worker fleet, so a burst of scheduled crons no longer competes with standard and agent runs for the same queue and inflates their startup latency. It is off by default and enabled per organization via a feature flag (with a global default), so nothing changes until it is turned on. ## Design At trigger time, any run whose lineage originates from a schedule (`rootTriggerSource === "schedule"`, which already propagates from a scheduled run down to all of its children) gets its worker queue suffixed with `:scheduled`. The worker queue name is an opaque string persisted on the run and used verbatim by enqueue and dequeue, so this needs no Lua, message-envelope, or concurrency changes. Concurrency stays keyed by environment and queue, not by worker queue. On the consumer side, the dequeue endpoint gains an optional `queueClass` selector. A supervisor sends `queueClass: "scheduled"` and the server derives the actual queue from the worker's own group, so a token can only ever reach its own region's queues. A fleet picks its class with the `TRIGGER_WORKER_QUEUE_CLASS` env var (`default` or `scheduled`), so a dedicated scheduled fleet can run alongside the standard one. Verified end to end against a local managed-worker setup: scheduled runs route to the dedicated queue, are drained only by the scheduled fleet, and standard runs are left untouched. |
||
|
Katia Bulatova
|
cae3dcb7dd |
Env vars page performance fix (#3829)
## Summary This PR improves performance across the Environment Variables page. ## Changes ### Targeted value loading - load only the non-secret (environmentId, key) pairs required by the page. Secret values continue to be redacted in the UI. ### SSR windowing + virtualization - SSR-render only the first 50 rows - hydrate those rows - virtualize the remaining dataset client-side - search is now URL-driven during SSR, ensuring deep links such as `?search=DATABASE_URL` ### Lightweight 'Create' flow - 'Create' page no longer loads the full Environment Variables dataset. ## Results Large projects no longer render thousands of rows during SSR. Example (~11k rendered rows): Metric | Before | After -- | -- | -- Document size | ~150 MB | ~5 MB SSR rows | ~11k | 50 Browser DOM rows | Thousands | ~26–38 |
||
|
Katia Bulatova
|
bb7d7dc7d1 |
feat(sdk,core): offload large trigger payloads via object storage (#3785)
Adds backward-compatible support for large trigger payloads by reusing the existing object-storage packet flow. Large payloads are uploaded to object storage before the trigger request is sent. The trigger API receives a small application/store pointer payload instead of embedding large JSON bodies in the request. Small payload behavior is unchanged. |
||
|
nicktrn
|
a4d8c9f65f |
chore(deps): update OpenTelemetry suite to 0.218.0 / 2.7.1 (#3810)
Brings the OpenTelemetry packages up to the latest coherent release
across the webapp and the published packages (`@trigger.dev/core`, the
CLI, `@trigger.dev/sdk`) plus
`internal-packages/{tracing,testcontainers}`:
- `@opentelemetry/sdk-node` 0.218.0
- `@opentelemetry/core` 2.7.1
- `@opentelemetry/host-metrics` 0.38.3
We were already on the otel 2.x line, so this is a same-major minor move
- the versions are pinned to `@opentelemetry/sdk-node@0.218.0`'s own
declared dependency set so the experimental (0.2xx) and stable (2.x)
packages stay coherent (mixing them is the usual otel breakage).
**One code change:** otel 0.215 made `forceFlush()` a required method on
`LogRecordExporter`, so `ExternalLogRecordExporterWrapper` (core's
tracing SDK) gains a `forceFlush()` that delegates to the underlying
exporter.
**Notable upgrades along the way:** OTLP exporters can take a custom
HTTP agent (connection pooling/keepAlive on the export path), HTTP
request headers are captured at span creation, and core hot-path perf
improvements in 2.6.1/2.7. `host-metrics` 0.37→0.38 is a clean upgrade.
Patch changeset added for the three published packages. References
projects are intentionally untouched.
Verified: `@trigger.dev/core` / CLI / `@trigger.dev/sdk` build, webapp +
`@internal/tracing` typecheck - all green.
|
||
|
Eric Allam
|
4b78d7e84e |
feat(core,webapp): support isSecret on environment variable imports (#3809)
## Summary
The environment variables import API now accepts an optional `isSecret`
flag, so imported variables can be created as secret (redacted)
environment variables instead of plaintext. When the flag is omitted,
variables default to non-secret, preserving existing behavior for CLI
deploys and dashboard imports.
This is useful for tools that push secrets into Trigger.dev (for
example, syncing from a secrets manager) and want them stored as secrets
rather than plain environment variables.
It's available through `envvars.import` in the SDK and the `POST
/api/v1/projects/{projectRef}/envvars/{slug}/import` endpoint, and is
honored for both regular and preview-branch environments.
```ts
await envvars.import("proj_1234", "prod", {
variables: { STRIPE_SECRET_KEY: "sk_live_..." },
isSecret: true,
});
```
|
||
|
Daniel Sutton
|
20a676d2ed |
fix(core,webapp): coerce numeric concurrencyKey to string (#3789)
## Summary `concurrencyKey` validation accepted only `z.string().optional()` on the single-trigger and V2/V3 batch endpoints, and the Phase-2 streaming NDJSON endpoint accepted `z.record(z.unknown()).optional()` for the entire `options` field. Callers passing `concurrencyKey: someNumericId` (e.g. `payload.userId`) either failed schema validation on the first two paths or sailed through on Phase-2 and then failed downstream at `prisma.taskRun.create` with `Argument concurrencyKey: Expected String or Null, provided Int`. The schema now accepts `string | number` for `concurrencyKey` and stringifies on the way in, across all three paths. The Phase-2 NDJSON `options` is tightened to reuse the strict `BatchTriggerTaskItem.options` shape so it validates identically to the V2/V3 batch endpoints. A defensive `typeof === "number"` coercion at the `engine.trigger` call site in `RunEngineTriggerTaskService` covers in-flight Redis-stored batch items enqueued before the schema fix — those items are rebuilt from a `Record<string, unknown>` shape that bypasses the new schema and would otherwise continue failing for up to their TTL. ## Test plan - [x] `packages/core/src/v3/schemas/batchItemNDJSON.test.ts` — unit tests covering numeric→string coercion, string passthrough, no-options, and rejection of non-string/non-number shapes across `TriggerTaskRequestBody`, `BatchTriggerTaskItem`, and `BatchItemNDJSON`. - [x] `apps/webapp/test/engine/triggerTask.test.ts` — `containerTest` simulating the in-flight Redis batch-item shape (numeric `concurrencyKey` via `Record<string, unknown>`), verifies the run is created with `concurrencyKey: "51262"`. Without the worker coercion, the test reproduces the production stack at `prisma.taskRun.create`. - [x] `pnpm run typecheck --filter webapp` clean. - [x] `pnpm run build --filter @trigger.dev/core --filter @trigger.dev/sdk --filter trigger.dev` clean. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
nicktrn
|
c043c4a6ad |
fix(core): external trace context leaks across warm-started runs (#3768)
On warm-started workers with `processKeepAlive` enabled, every run's attempt span was exported with the first run's `traceId` and `parentSpanId`. `ExternalSpanExporterWrapper` and `ExternalLogRecordExporterWrapper` captured `externalTraceContext` at `TracingSDK` construction, and the SDK is memoized for the worker's lifetime - so the per-run reset of `StandardTraceContextManager.traceContext` never reached the wrappers. Reported by a customer running v4.4.x: 33 distinct runs on the same host/pid showed up in their APM as siblings of one parent span. Fix: the wrappers now read external context live from the trace context manager per export. Runs without an external trace context fall through to the unchanged `externalTraceId` fallback - no behaviour change for them. Regression test in `packages/core/test/externalSpanExporterWrapper.test.ts` asserts that reassigning the manager between exports produces correctly-parented spans. |
||
|
Iss
|
8399aa2052 |
fix(core): retry TASK_MIDDLEWARE_ERROR under the task's retry policy (#3676)
Retries `TASK_MIDDLEWARE_ERROR` under the task's retry policy. `shouldRetryError` already classed it as retryable, but `shouldLookupRetrySettings` did not, so the run fell through to `fail_run` on attempt 1 instead of using the task's `retry` config. Fixes #3231. |
||
|
Oskar Otwinowski
|
5ba1b32d53 | fix(core): handle conflicting dotted paths in unflattenAttributes (#3762) | ||
|
Eric Allam
|
832cf7220b |
feat(sdk,core): add TriggerClient for per-instance SDK configuration (#3683)
## Summary
`new TriggerClient({...})` exposes the management API (tasks, runs,
schedules, envvars, batch, queues, deployments, prompts, auth) as an
explicit instance with its own auth, preview branch, and baseURL.
Multiple clients can coexist in one process without mutating shared
global state — useful when a single service triggers across multiple
projects, environments, or preview branches.
```ts
import { TriggerClient } from "@trigger.dev/sdk";
const prod = new TriggerClient({ accessToken: process.env.TRIGGER_PROD_KEY });
const preview = new TriggerClient({
accessToken: process.env.TRIGGER_PREVIEW_KEY,
previewBranch: "signup-flow",
});
await prod.tasks.trigger("send-email", payload);
await preview.runs.list({ status: ["COMPLETED"] });
```
The existing global `configure()` API keeps working unchanged.
## Design
Instance methods enter an `AsyncLocalStorage`-backed scope (`sdkScope`)
before delegating to the existing module-level functions. The four
"pollution" points that previously read globals now consult the scope
first:
- `apiClientManager.{baseURL, accessToken, branchName}` and
`clientOrThrow` — identity fields are scope-only when scoped; `baseURL`
still falls back to `TRIGGER_API_URL` because plumbing (where the API
lives) is not identity.
- `taskContext.{ctx, worker, isWarmStart, isInsideTask}` — masked inside
an isolated scope so a `client.tasks.trigger(...)` from inside a task
doesn't leak the parent's `parentRunId` / `lockToVersion` / `isTest`
into a trigger that hits a different project.
- Inline `getEnvVar("TRIGGER_VERSION")` reads in `shared.ts` go through
a `scopedEnvVar` helper that returns `undefined` inside an isolated
scope.
The `TriggerClient` class itself is a thin wrapper that captures the
scope in its constructor and proxies each namespace method to enter that
scope before calling the existing impl. Generic inference (e.g.
`client.tasks.trigger<typeof t>(...)`) is preserved via `Pick<typeof ns,
keyof curatedSubset>` typings.
Two correctness fixes uncovered along the way are folded in:
- `apiClientManager.setGlobalAPIClientConfiguration` no longer silently
no-ops on the second call. `configure()` now actually overrides as users
expect (this is the root cause behind some "I changed the config but
nothing happened" reports).
- `apiClientManager.runWithConfig` (and therefore `auth.withAuth`) is
now backed by `sdkScope.withScope` instead of "mutate the global and
restore in finally". Two parallel `withAuth` calls with different
configs no longer stomp each other.
Surface curation: instance namespaces drop methods that don't make sense
per-instance — `batch.*AndWait` (runtime-dependent), `schedules.task` /
`schedules.timezones` (definition-time / stateless), `prompts.define`
(definition-time), `auth.configure` / `auth.withAuth` (global-only).
## Test plan
- [x] 9 runtime unit tests in `triggerClient.test.ts` cover: required
accessToken, instance auth + branch headers, no env fallback for
identity fields, no leakage between global and instance, four parallel
calls across two clients stay isolated, taskContext masking +
`inheritContext: true` override, `configure()` second-call override,
parallel `auth.withAuth` isolation.
- [x] 10 type-level assertions in `triggerClient.types.test.ts` using
`expectTypeOf` + `@ts-expect-error` lock in generic inference, return
type passthrough, overload preservation, and curated-surface drift.
- [x] Full SDK suite (219 tests) and core suite (530 tests) pass.
- [x] Webapp typecheck clean.
- [x] End-to-end smoke test against local webapp and a
freshly-provisioned cloud project — six concurrent multi-client triggers
all returned 200 with run IDs, headers per-client as expected.
- [ ] Reviewer: run `references/multi-client` per its `README.md` to
reproduce the smoke test locally.
## Try it
`references/multi-client` is a new reference workspace that exercises
this end-to-end:
- `src/trigger/echo.ts` — trivial target task
- `src/trigger/fanOut.ts` — opens two `TriggerClient`s from inside a
task, fires `echo` through each in parallel
- `src/external/main.ts` — external Node script with two clients
triggering `echo` sequentially and concurrently; logs every outgoing
request's `authorization` + `x-trigger-branch`
- `src/external/isolation.ts` — interleaves global `configure()` and an
instance call, asserts the captured fetch sequence shows no leakage
either way
|
||
|
nicktrn
|
dbf9b4e7e5 |
fix(core,cli): register tasks loaded via dynamic import during run (#3688)
On a warm worker process, a task whose `task()` definition is loaded via `await import(...)` from inside another task's `run()` could end up permanently missing from the catalog: the `task()` call fired with no `_currentFileContext` set, `registerTaskMetadata` silently returned, and Node's ESM module cache then blocked the worker's setContext + re-import recovery from ever firing the call again. Subsequent runs of that task on the same warm process failed with `COULD_NOT_FIND_EXECUTOR` until the process hit `maxExecutionsPerProcess` and exited. All five of these had to coincide on the same worker for the bug to surface: 1. `processKeepAlive` enabled (so catalog state survives across runs). 2. A `run()` function (or lifecycle hook) does `await import(...)`. 3. The import's transitive static graph reaches a `task()` / `schemaTask()` call. 4. The task containing the dynamic import is the **first** task to run on a given warm worker process — so the dropped `task()` calls fire on this process for the first time, are silently dropped, and Node's module cache locks the wrong outcome in. 5. A subsequent run for one of the dropped task ids lands on the same warm worker before it recycles. The runtime workers now set a sentinel file context (`<no-context>`) around the `executor.execute(...)` call, so `task()` invocations firing during a run register normally. The catalog detects the sentinel and emits a one-time `console.warn` per task id so the pattern stays visible without spamming. The indexer never sets this context, so deploy-time behavior is unchanged. Repro is `references/hello-world/src/trigger/dynamicImportRepro*.ts`. Verified end-to-end against a deployed image with firestarter warm-starts on: pre-fix saw `COULD_NOT_FIND_EXECUTOR` on children that landed on the parent-poisoned worker; post-fix all 23/23 runs succeeded and the warning surfaces in the parent's run trace. |
||
|
Eric Allam
|
d343727021 |
fix(webapp,sdk): keep chat.agent snapshots on one object store (#3679)
(`OBJECT_STORE_BASE_URL`) and a named protocol provider
(`OBJECT_STORE_DEFAULT_PROTOCOL=s3`), chat.agent session snapshot writes
landed in the named provider but reads fell through to the default — so
the recovery boot couldn't find the snapshot it had just written.
After a mid-stream cancel, the missing snapshot triggered a fallback
replay path that dropped the user's follow-up message, leaving the chat
stuck in `submitted` indefinitely.
Fix:
- New `/api/v1/sessions/:id/snapshot-url` route handles PUT + GET
symmetrically — both prefix unprefixed keys with
`OBJECT_STORE_DEFAULT_PROTOCOL` so they always round-trip through the
same store.
- `Session.chatSnapshotStoragePath` persists the resolved URI on first
write so future protocol changes don't strand existing snapshots.
Reads prefer the stored URI and fall back to the computed default for
pre-column sessions.
- SDK calls `createChatSnapshotUploadUrl` / `getChatSnapshotUrl`; the
generic v1/v2 packets endpoints are unchanged.
## Test plan
- [x] Configure local with two providers (R2 default + MinIO `s3` named)
and `OBJECT_STORE_DEFAULT_PROTOCOL=s3`.
- [x] Reproduce hang: send a message, cancel mid-stream, send another —
without the fix it hangs in `submitted`; with the fix it streams.
- [x] Snapshot lands in the `s3`-protocol bucket and
`Session.chatSnapshotStoragePath` is set after first write.
- [x] SDK unit tests pass; webapp typecheck passes.
|
||
|
Eric Allam
|
f91b96efb7 | feat(sdk,core): preserve chat.agent context after cancel / OOM / crash (#3671) | ||
|
Eric Allam
|
12d2125148 |
fix(sdk,core,build): SDK hardening pass (#3670)
## Summary Five hardening fixes across `@trigger.dev/sdk`, `@trigger.dev/core`, and `@trigger.dev/build`. - `tasks.triggerAndSubscribe` now forwards caller `requestOptions` (custom API keys, per-request overrides) to the underlying `apiClient.triggerTask` call instead of silently dropping them. - `SSEStreamSubscription` no longer retries permanent client errors forever. The default `nonRetryableStatuses` widens from `[404, 410]` to `[400, 404, 409, 410, 422]`, so a malformed session-stream request fails fast instead of busy-looping under bounded backoff. - Session writer falls back to manually wiring the caller's `AbortSignal` on Node 18, where `AbortSignal.any` is unavailable. Caller-driven cancellation now propagates on every supported runtime. - `TriggerChatTransport` throws immediately when a `chat.handover` response is missing `X-Trigger-Chat-Access-Token`, instead of silently downgrading every subsequent turn back to the handover path. `dispose()` aborts every active `session.out` subscription before tearing the coordinator down, so unmount/navigation no longer leaves SSE readers in flight. - Removed the experimental `@trigger.dev/build/extensions/secureExec` build extension. It will return alongside the sandbox feature it was built to support. ## Test plan - [ ] `pnpm run build --filter @trigger.dev/sdk --filter @trigger.dev/core --filter @trigger.dev/build` - [ ] `pnpm --filter @trigger.dev/sdk test --run` (183 tests, including chat / chat-server / sessions / handover) - [ ] `pnpm --filter @trigger.dev/core test --run` - [ ] Manually trigger a `chat.handover` whose response strips `X-Trigger-Chat-Access-Token`, and confirm the transport throws synchronously rather than degrading. - [ ] Unmount a chat UI mid-stream and confirm the active `session.out` SSE connection closes immediately. |
||
|
Eric Allam
|
427d9e078a |
feat(sdk): functional baseURL and fetch override on chat transports (#3655)
## Summary
`TriggerChatTransport`, `AgentChat`, and `chat.createStartSessionAction`
now accept a string-or-function `baseURL` so callers can route per
endpoint — e.g. `.in/append` through a trusted edge proxy while keeping
`.out` SSE direct. The same surfaces add a `fetch` override for header
injection, custom retries, or proxy rewrites that go beyond URL routing.
SSE GETs are covered too via a new `fetchClient` option on
`SSEStreamSubscription`.
```ts
// TriggerChatTransport / AgentChat — endpoints: "in" | "out"
baseURL: ({ endpoint }) =>
endpoint === "out" ? DIRECT : PROXY,
fetch: (url, init, ctx) => {
init.headers = new Headers(init.headers);
init.headers.set("traceparent", currentTraceparent());
return globalThis.fetch(url, init);
},
// chat.createStartSessionAction — endpoints: "sessions" | "auth"
chat.createStartSessionAction("my-agent", {
baseURL: ({ endpoint }) => (endpoint === "sessions" ? PROXY : DIRECT),
});
```
`streamBaseURL` on `TriggerChatTransport` is kept as a backwards-compat
alias and continues to win for the `"out"` endpoint when set.
Plain-string `baseURL` still applies to every endpoint, matching prior
behavior.
|
||
|
Eric Allam
|
8b98e21b4b |
fix(sdk,core): cache realtime-stream credentials per slot with refresh on writer failure (#3658)
## Summary Hot-loop writers — `streams.writer` / `streams.pipe` on the run-scoped side, `chat.response.write` / `chat.stream.*` on the session side — were issuing a fresh `PUT` to mint S2 credentials for every chunk. On run streams, each PUT also pushed the streamId onto `TaskRun.realtimeStreams`, so a chat-agent turn writing N chunks produced N PUTs and N duplicate array pushes against the same row. The SDK now caches the initialize response per cache slot: `(runId, key)` for run streams, the session id for session streams. First call PUTs as before; subsequent calls reuse the cached promise. Hot-loop writers do one PUT per slot for the lifetime of the cache. S2 access tokens have a 1-day TTL. If a writer's `wait()` rejects (auth error, expired token, network blip), the cache evicts the matching slot so the next call re-PUTs and mints fresh credentials, identity-checked so a concurrent caller's fresh promise isn't accidentally cleared. ## chat.agent guardrail `streams.pipe / writer / append / read` called inside a `chat.agent` run now logs a one-time warning pointing at `chat.response.write` / `chat.stream.*` — `streams.*` is run-scoped and isn't visible on the chat session. The ai-chat docs are updated to drop the old guidance toward run-scoped streams. |