Updates the ClickHouse chat agent example page to match the upgraded
example (triggerdotdev/examples#124), which is now a fullstack
generative-UI chat app rather than an agent-only project.
## What changed
- **Overview / tech stack / features** rewritten: Next.js chat app
(`useChat` + `useTriggerChatTransport`, no API route), a
`renderVisualization` tool taking json-render specs rendered with
`@json-render/shadcn` + shadcn charts (Recharts) + mapcn point maps, and
a shared catalog that generates both the system-prompt component
reference and tool-call validation.
- **The agent section** now shows the versioned [AI
Prompt](https://trigger.dev/docs/ai/prompts) pattern (`prompts.define()`
+ `chat.prompt.set()` + `chat.toStreamTextOptions({ registry })`), with
a warning that `experimental_telemetry` comes from the stored prompt —
the docs previously showed a static `system:` string, which silently
ships no LLM observability.
- **New sections** for the shared catalog, the `renderVisualization`
tool, the Next.js chat UI and registry.
- **Relevant code links** updated to the new `src/` layout.
- **Learn more** cards now include Frontend and AI Prompts.
Note: merge after triggerdotdev/examples#124 lands, so the GitHub file
links resolve.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
The dashboard was redesigned and two pages moved, but the docs still
described the old sidebar:
- **Schedules** no longer has its own sidebar page — schedules are
managed from the **Tasks** page (open a scheduled task to create / view
/ edit / enable-disable / delete them).
- The standalone list-based **Test** page is deprecated — you test a
task from its own **Test** button now.
## Changes
- `tasks/scheduled.mdx`: rewrote the "attaching schedules" and "testing
schedules" sections for the Tasks-based flow, added a "managing
schedules in the dashboard" section, and added explicit callouts noting
both pages moved (so readers — and search — aren't pointed at a page
that no longer exists). Re-shot the four schedule screenshots and fixed
a mislabeled alt text.
- `run-tests.mdx`, `snippets/step-run-test.mdx`,
`guides/examples/sentry-error-tracking.mdx`: replaced "select the Test
page in the sidebar" with the task-first flow plus a callout, and
refreshed `test-dashboard.png`.
TRI-11939
## Summary
Adds `EVENT_REPOSITORY_POSTGRES_WRITES_DISABLED` (default off), which
makes the task-event store skip all PostgreSQL `TaskEvent` writes. It's
for deployments that store task events in ClickHouse
(`EVENT_REPOSITORY_DEFAULT_STORE=clickhouse_v2`) and no longer want the
PostgreSQL copy.
## How it works
The guard sits at the single postgres write boundary,
`TaskEventStore.create` / `createMany`, so it covers every write path
(OTLP ingestion and run-lifecycle events) with one check. Reads are
untouched (`findMany` / trace queries / streaming), so existing
PostgreSQL events remain readable.
Leave it off unless the default store is `clickhouse_v2`, otherwise task
events for any run still routed to PostgreSQL would be dropped.
## Summary
Refreshes the docs for the v3 sunset: v3 (SDK v3) is end of life, and
4.5.0 is the last version we officially support for running v3.
- The self-hosting overview, plus the docker and kubernetes
version-locking sections, now tell self-hosters on v3 to stay on 4.5.0
or migrate to v4. 4.5.1 and later reject v3 triggers and deploys with an
upgrade message.
- The migration guide's deprecation notice was still written in the
future tense (with dates that have since passed); it now describes v3 as
retired and adds the self-hosted 4.5.0 cutoff. This is the page the
server's upgrade message links to.
- Fixes a stale "v3 project" reference in the CLI overview.
The Mintlify preview will render the callouts for a visual check.
## Summary
v3 (the engine that ran the SDK v3 era, internally
`RunEngineVersion.V1`) is end-of-life. Following the removal of the v3
execution apps
([#4194](https://github.com/triggerdotdev/trigger.dev/pull/4194)) and
the legacy dev websocket
([#4198](https://github.com/triggerdotdev/trigger.dev/pull/4198)), this
removes the remaining v3 execution stack from the server.
Clients still on v3 (an old SDK or CLI that has not upgraded) keep
getting a clear "upgrade to v4" response. Triggers, batch triggers,
reschedules, and deploys that resolve to v3 are rejected with a graceful
4xx pointing at the migration guide, never a 5xx, so a stale client
cannot affect server health. Self-hosted instances still running v3
should stay on the 4.5.x release line until they migrate.
## What is removed
- The MarQS queue and its shared/dev queue consumers.
- The v3 socket.io namespaces (coordinator, provider, shared-queue) and
the v3 run lifecycle services (attempt, checkpoint, and batch-resume).
- The graphile-worker background job system; all live jobs already run
on `@trigger.dev/redis-worker`.
- The `DEPRECATE_V3_ENABLED` flag: v3 is now rejected unconditionally,
so the flag is gone.
- Unused v3 exports from `@trigger.dev/core` (the `v3/zodNamespace`
subpath and the legacy socket message catalogs) and the now-dead MarQS
environment variables.
## What stays
The v4 engine is untouched. The graceful v3 rejection boundary stays,
`determineEngineVersion` still detects a v3 project so it can reject it,
and the batch service plus batch-completion worker stay for current
clients. Live queue concurrency limits and metrics now read from the v4
run engine instead of MarQS, and a brand-new dev environment now
defaults to v4.
## Dependency cleanup
Removes webapp dependencies left unused by this change: `seedrandom` and
`semver` (only the removed v3 code used them) plus a set that was
already dead, their orphaned `@types` packages, and two dead files. Adds
a `knip:deps` script and a `knip.json` config so unused dependencies can
be found the same way going forward.
## Summary
Sending a chat message immediately after an action (for example an undo)
could make the message's response vanish from the UI. The transport
opened a response stream that closed on the *earlier* turn's completion
instead of waiting for the send's own turn. The agent still produced and
persisted the answer, so it reappeared on refresh. Same "disappearing
message" class as
[#4176](https://github.com/triggerdotdev/trigger.dev/pull/4176),
different cause.
## Fix
A send's response stream had no way to tell whether a `turn-complete`
belonged to its turn. `POST /realtime/v1/sessions/:id/in/append` now
returns the appended record's sequence number, and the transport skips
any turn-complete whose `session-in-event-id` (the agent's committed
`.in` cursor) is below that seq, closing only on its own turn. Older
webapps omit the seq, in which case the transport falls back to the
previous behavior, so the SDK and server can ship independently.
Because the fix spans the SDK and the server, both a webapp deploy and
an SDK release are needed for the full effect.
Verified end to end with the ai-chat reference app:
undo-then-immediate-send loses the follow-up's answer before the fix and
streams it inline after, with a revert-the-guard run reproducing the
loss on the same script. Unit tests cover the skip and the no-seq
fallback.
## What
Adds per-variable secret support to the `syncEnvVars` build extension.
Return `{ name, value, isSecret: true }` and the variable is stored as a
secret (redacted in the dashboard, value non-revealable), just like a
manually created secret env var. Secret and non-secret variables can be
mixed in one callback.
```ts
syncEnvVars(async () => [
{ name: "PUBLIC_API_URL", value: "https://api.example.com" },
{ name: "DATABASE_URL", value: "postgres://...", isSecret: true },
]);
```
## How
Env vars flow through the build pipeline as a flat name→value map, and
the import API's `isSecret` is per-call. So secret vars are carried
through the layer + manifest in parallel `secretEnv` / `secretParentEnv`
maps, and at deploy time they go up in a second `importEnvVars` call
with `isSecret: true` (the plain vars in the first call). The record
form (`{ KEY: "value" }`) is unchanged and stays non-secret.
## Commits
- `feat(core)`: carry secret env vars through the build layer + manifest
schema
- `feat(build)`: partition `isSecret` vars in `syncEnvVars`
- `feat(cli)`: merge secret layers and import them with `isSecret: true`
at deploy
- `test(build)`: cover the partitioning + document `isSecret`
## Testing
- vitest covers the partitioning (secret/non-secret × child/parent) and
that the record form stays non-secret.
- Verified against a local webapp that the deploy's import contract
stores the secret var redacted (`isSecret: true`) and the plain var
visible.
Closes TRI-11099
- Combine SDK and dashboard bulk actions docs
- Fix API reference pages for bulk actions
- Fix weird rendering on bulk actions page
## Todo
- [ ] not sure about having the SDK+dashboard combined and under "Using
the dashboard"... need to find the right place
## Summary
The limits page didn't spell out that large payloads offload to object
storage automatically, and its single "512KB" note conflated two
different thresholds. This clarifies the behaviour.
On the way in, the SDK uploads any trigger or batch-item payload over
128KB to object storage before sending, so large triggers and batches
don't hit the request body limit (`trigger` / `triggerAndWait` since
4.5.0, `batchTrigger` / `batchTriggerAndWait` since 4.5.2). On
retrieval, payloads and outputs over 512KB are stored in object storage
and returned as a presigned URL from `runs.retrieve`.
Follow-up to #4194 (v3 execution app + core-helper removal). The v3
(engine V1) is end-of-lifed and enforced off in prod, so this removes a
self-contained slice of the remaining dead v3 code while **keeping every
user-facing deprecation message** - a user still on v3 must still be
told to upgrade.
## Legacy dev websocket
`app/v3/handleWebsockets.server.ts` backs the `/ws` transport used
**only** by the legacy v3 `trigger dev` CLI (v4 dev uses a different
transport). It's now authenticate-then-close with
`V3_DEV_DEPRECATION_MESSAGE`, so an old CLI is still told what to do -
only the legacy `AuthenticatedSocketConnection` / `DevQueueConsumer`
execution behind it (which can no longer run) is removed.
- Deleted `app/v3/authenticatedSocketConnection.server.ts` (its only
consumer).
- `engineDeprecation.server.ts` and the deprecation message constants
are untouched.
## Docs
Deleted the intentionally-legacy "Docker (legacy)" self-hosting page
(`open-source-self-hosting.mdx`) and redirected
`/open-source-self-hosting` (+ the existing
`/v3/open-source-self-hosting` alias) to `/self-hosting/overview`;
repointed the two inbound links. The current `self-hosting/*` docs
already describe the v4 (single supervisor) setup.
## Deliberately out of scope
Despite the branch name, this PR does **not** touch MarQS or the
socket.io coordinator/provider namespaces. Investigation found MarQS is
entangled with **live v2** queue/metrics/concurrency/project-cleanup
code (`runQueue`, `queueSizeLimits`, `taskRunConcurrencyTracker`,
`EnvironmentQueuePresenter`, `registerProjectMetrics`, `deleteProject`),
so it needs a per-file reviewed pass, not a bulk delete. That remainder
stays on TRI-11883.
refs TRI-11883
## What
Adds a new Example projects page: **ClickHouse chat agent** — a
`chat.agent()` that answers questions about your data by writing and
running SQL against ClickHouse Cloud via the official Node.js ClickHouse
client.
The page follows the existing example-project format (overview,
features, GitHub repo card, how-it-works with code excerpts, relevant
code links) and is registered in `docs.json` in alphabetical order.
## Note
The GitHub repo card links to
`triggerdotdev/examples/tree/main/clickhouse-chat-agent`, which lands in
a companion examples PR — merge that one first.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
## Summary
Realtime streams (AI-agent token streaming and run streams) now default
to v2 for self-hosters, backed by a bundled [s2-lite](https://s2.dev)
service. Self-hosting previously shipped no S2 configuration, so streams
ran on the Redis-backed v1 path and there were no docs for wiring up v2.
Both the Docker Compose stack and the Helm chart now provision s2-lite
with persistent storage and set the stream env vars out of the box.
## What's included
- **Docker Compose**: a persistent `s2` service (s2-lite), a basin init
spec, and the `REALTIME_STREAMS_S2_*` plus
`REALTIME_STREAMS_DEFAULT_VERSION=v2` env on the webapp. `.env.example`
documents the v1 fallback and hosted-S2 options.
- **Helm**: an `s2` StatefulSet, PVC, Service and ConfigMap (runs as the
non-root image user via `fsGroup`), an `s2` values block, and webapp env
wiring with an existing-secret path for hosted S2.
- **Docs**: the `REALTIME_STREAMS_S2_*` and
`REALTIME_STREAMS_DEFAULT_VERSION` vars in the webapp env reference,
plus a "Realtime streams" section in the Docker and Kubernetes
self-hosting guides.
## Notes
- The OSS code default stays `v1`; v2 becomes the default purely through
the self-hosting artifacts, so non-self-host deployments are unaffected.
Disabling s2, or setting the version back to `v1`, cleanly reverts to
Redis-backed v1.
- With v2 enabled, the bundled s2 service is a required dependency for
streaming: if it is down, streams error while the task itself still
runs. That is the intended trade for the better v2 path.
- You can point at a hosted S2 at s2.dev instead of the bundled server.
New `/billing-limits` page covering the full [billing limits
feature](https://trigger.dev/changelog/billing-limits) : the three limit
options (plan / custom / no limit), billing alerts (% of limit or dollar
thresholds), what happens when the limit is reached, the recovery flow,
the soft-limits caveat, and the billing limit marker on the Usage page.
## Summary
`sendMessage` from `useChat` gives no feedback about whether a message
actually reached the backend, and the `fetch` override is wire-level: it
requires knowing endpoint semantics, cannot attribute requests to
messages, and misses the headStart first-turn POST entirely. This adds a
typed `onEvent` observability callback to `TriggerChatTransport` /
`useTriggerChatTransport` so send-success metrics, time-to-first-token,
and "sent but never answered" watchdogs become a few lines of client
code.
## Example
```ts
const transport = useTriggerChatTransport({
task: "my-chat",
accessToken: ({ chatId }) => mintChatAccessToken(chatId),
onEvent: (event) => {
switch (event.type) {
case "message-sent":
// Durably acknowledged by the session's input stream, not just "request accepted".
metrics.increment("chat.message_sent", { source: event.source });
metrics.timing("chat.send_duration_ms", event.durationMs);
break;
case "message-send-failed":
metrics.increment("chat.message_send_failed", { status: event.status });
break;
case "first-chunk":
metrics.timing("chat.ttft_ms", event.sinceSendMs ?? 0);
break;
case "turn-completed":
metrics.timing("chat.turn_duration_ms", event.sinceSendMs ?? 0);
break;
}
},
});
```
## Design
One callback, one discriminated union (`ChatTransportEvent`):
- `message-sent` / `message-send-failed`: terminal send outcomes with
`messageId`, a `source` discriminator (submit, regenerate, steer,
action, stop, head-start), `durationMs`, `bodyBytes`, the append's
idempotency key (`partId`, also stored on the server-side record), and
error + HTTP status on failure. `message-sent` means the append was
durably acknowledged, after any internal token-refresh retries.
- `stream-connected` (with a `resumed` flag and the cursor it connected
from), `first-chunk` (chunk type plus `sinceSendMs` for
time-to-first-token), `turn-completed` (`sinceSendMs` full-turn latency
and the agent's committed input cursor), and `stream-error` follow the
response side, so a send can be paired with the answer that should
follow it. `messageId` on response events is client-side attribution
from the last turn-producing send on that chat.
Emissions sit at the transport's existing choke points, covering every
send path uniformly (including steering and headStart, which the fetch
override cannot observe). Exceptions thrown by the callback are
swallowed: observability can never break the chat. The React hook keeps
the callback live across renders instead of freezing the first-render
closure.
## Verification
Unit tests drive the transport directly with the `fetch` override as the
network stub (send success/failure per source, stream lifecycle, resumed
flag, field enrichment, callback exceptions swallowed). Verified
end-to-end against a realistic metrics setup in the ai-chat reference
app (counters, send-duration and TTFT histograms, and both watchdogs
built purely on these events): a healthy two-turn chat produces exactly
the expected event sequence and TTFT values; an oversized append records
`message_send_failed` with status 413; and killing the worker after a
durable send fires both `sent_but_no_stream` and `sent_but_unanswered`,
reproducing and detecting the "message disappeared" failure mode that
motivated this feature.
## Summary
Adds SDK and API support for run bulk actions. You can now create bulk
cancel or replay actions from `@trigger.dev/sdk` using run IDs or the
same filters as `runs.list()`, then retrieve, list, poll, or abort the
action by its `bulk_` handle.
Tests, docs, changesets added.
## Design
The dashboard bulk action service now accepts structured filters instead
of reading directly from a dashboard request, so the dashboard and API
share the same creation path. Replay actions created through the API are
attributed with the existing `api` trigger source, while
dashboard-created actions keep `dashboard`.
The SDK exposes the new surface under `runs.bulk.*`, including
`targetRegion` for replay region overrides and cursor pagination for
listing bulk actions.
## Filters and runIds
Nuance on filters. If `filter` is provided, it MUST have at least one
key. This is to remove the footgun of passing no filter and selecting
all runs.
```typescript
{ action: "cancel", runIds: ["run_1"] } // valid
{ action: "cancel", runIds: [] } // invalid, min(1)
{ action: "cancel", filter: { status: "FAILED" } } // valid
{ action: "cancel", filter: {} } // invalid
{ action: "cancel", filter: {}, runIds: ["run_1"] } // invalid
```
## Summary
Sending a message to a chat whose run had ended could make the message
vanish: the continuation run replayed already-answered messages, never
processed the new one, and a page refresh lost it entirely. Chasing that
report surfaced four composing message-loss bugs in the chat session
runtime; this PR fixes all of them, each with a regression test.
## The fixes
1. **Stale resume cursor.** Records delivered while a run was suspended
(the waitpoint path) advanced the SSE resume counter but not the
committed-consume cursor, so the `session-in-event-id` header stamped on
turn-completes went stale by one record per suspended turn. Continuation
boots seed from that header, which is what made them replay
already-processed messages. `session.in.wait()` now advances both
cursors.
2. **Only the first buffered message dispatched.** Messages arriving
during a turn are consumed into a buffer whose end-of-turn pickup
dispatched only the first entry; the buffer was recreated each turn, so
the rest were discarded, and since consuming a record commits the cursor
the loss was permanent. A continuation boot's replay delivers several
records back-to-back, which put the user's new message at index 1 or
later. The buffer now outlives the turn and drains one message per turn
in both `chat.agent` and `chat.createSession` (whose equivalent buffer
was never read at all).
3. **Post-stop window in `chat.createSession`.** The turn's message
listener stayed attached through the stopped turn's post-stream work, so
a message sent shortly after stopping a turn was consumed into the dead
steering queue and lost. The listener now detaches when the stream
settles, matching the `chat.agent` loop.
4. **Handler leak on errored turns.** A turn that threw outside the
streaming section (for example from an `onTurnStart` hook) leaked its
message listener. Previously that silently lost mid-turn messages; with
the loop-level buffer it would have duplicated them instead. The
subscription handle is now detached by the turn's catch/finally, and
`chat.createSession` defensively detaches its prior turn's listener when
user code exits a turn without `complete()`/`done()`.
## Verification
Reproduced end-to-end with the ai-chat reference project before the fix
(message consumed but never answered, two replayed turns, gone on
refresh) and verified after (single clean turn, survives refresh,
turn-complete cursors strictly advancing). Regression tests in
`packages/trigger-sdk/test/pending-message-drain.test.ts` cover all
four, each verified red against the unfixed behavior. A smoke sweep of
the standard chat scenarios (basic send, multi-turn, suspend/resume,
mid-stream refresh, stop, steering, cancel + continue, and the
`createSession` variant) passes on the final branch state.
## Summary
Depends on #4136.
Docker Compose self-hosting now uses the maintained `latest` image tag
by default instead of the frozen prerelease tag. The version-locking
docs keep pointing production users at explicit versioned tags when they
want pinned upgrades.
## Summary
Stable v4 Docker image builds now also publish `v4` and `latest` tags,
giving Docker-based self-hosters a maintained floating tag to use after
a stable release. The Kubernetes guide now uses the current Helm chart
line and current pinned examples, so new installs and upgrades resolve
to the 4.5 chart line instead of the 4.0 line.
## Design
The publish workflows add the floating tags only for stable `v4.x.x`
image tags. Prerelease and `main` builds keep their existing tags.
Removes the release-candidate framing from the docs for the 4.5 GA: the
`@rc` install caveats on the `skills` CLI command, and the AI Agents and
Prompts release-candidate banner (a shared snippet used across the
ai-chat and prompts pages) plus the `>=4.5.0-rc.0` compatibility pin in
the AI reference.
## Summary
Adds the `4.5.0` GA entry to the AI chat agents changelog covering the
`chat.agent` changes in that release: a new `apiClient` option on
`chat.headStart` and `chat.createStartSessionAction` for pointing chat
sessions at a different project or environment, the fix for messaging
chat agents deployed to a preview branch, and the fix for Head Start
handovers when the agent also defines a `prepareMessages` hook.
All four changes are from
[#4018](https://github.com/triggerdotdev/trigger.dev/pull/4018).
Fixed trace rendering for child and nested runs in large traces.
Dashboard and trace API responses now load the requested run's trace
subtree instead of depending on the run span appearing in the initial
trace slice.
## Summary
The `skills` installer command (`npx trigger.dev@latest skills`) ships
in the release candidate but is not yet on the stable release, so
running it with `@latest` fails today. Adds a warning on each docs page
that shows the command, telling users to run it with the `@rc` tag until
it lands in `@latest`.
The commands themselves stay on `@latest`, so nothing needs reverting
once the next stable release ships, just the warnings.
Pages updated:
- `/skills`
- `/mcp-agent-rules`
- `/building-with-ai`
## Summary
Adds Billing Limits to the webapp.
Customers can set a monthly spend cap. When usage crosses the limit,
billable environments enter a grace period. If the limit is not resolved
before grace expires, new triggers are rejected until the organization
increases or removes the limit.
Closes this feature request:
[https://triggerdev.featurebase.app/p/isolated-dev-sessions-for-multiple-local-trigger-dev-instances](https://triggerdev.featurebase.app/p/isolated-dev-sessions-for-multiple-local-trigger-dev-instances)
### Feature notes:
- CLI `trigger dev` works as before
- `trigger dev --branch my-branch` to create a new branch and run
against it.
- `trigger dev archive --branch my-branch` to archive (or in webapp).
- New webapp page to manage and archive dev branches, currently feature
flagged.
### Implementation details:
- No changes to data model, no backfill. `isBranchableEnvironment`
column is ignored for dev branches, we use `parentEnvironmentId IS NULL`
instead.
- `x-trigger-branch` overloaded for preview and dev branches
- New `TRIGGER_DEV_BRANCH` env var available locally.
`TRIGGER_PREVIEW_BRANCH` overloaded for child runs.
- Lots of new glue code to sanitise the branch checks.
### Rollout
- Deploy webapp/API changes (all backwards compatible)
- Manual tests on some orgs
- Deploy docs, release CLI, flip feature flag for webapp feature
### NB
- `api.v1.projects.$projectRef.environments.ts` will return
`isBranchableEnvironment: true` for all dev environments.
### Prerequisites
- [x] Typecheck will not pass until we make a new release of
`@trigger.dev/platform` and bump it here
The wait.for() idempotency example passed the idempotency options as a
second argument, but wait.for() takes a single options object — so the
second object was silently ignored and the key never applied. Merged the
fields into the single options object, matching the wait.until()
example.
Supersedes #4021.
## Summary
Adds an in-dashboard AI agent: a chat panel, reachable from any
environment
page, that answers questions about your runs, errors, tasks, and
analytics,
diagnoses why a run failed, charts your data, reads your connected
repo's
source, and answers product and how-to questions. It is gated behind the
`hasDashboardAgentAccess` feature flag (global or per-org, default off),
so
this PR ships disabled: the launcher is hidden unless the flag is
enabled.
## Design
The agent runs as a standalone `chat.agent` Trigger task in its own
internal
package, with no access to the webapp database, Prisma, or ClickHouse.
It reads
the user's data over the public API, acting as the user via a
short-lived
delegated user-actor token minted server-side each turn (never in the
browser),
building on
[#3997](https://github.com/triggerdotdev/trigger.dev/pull/3997). The
error and analytics tools use
[#4005](https://github.com/triggerdotdev/trigger.dev/pull/4005)
and the TRQL query API.
The first turn of a new chat streams from a warm webapp route (Head
Start) while
the durable agent boots in parallel. Structured answers (a run-failure
diagnosis
card, a live chart) render through a small typed view catalog rather
than
arbitrary markup. A knowledge lane forwards product and how-to questions
to the
support assistant.
Conversation history lives in a separate Drizzle-backed store on its own
Postgres schema, kept as a display read-model so it can never corrupt
the
agent's model context.
The SDK changes add an `apiClient` option to
`chat.createStartSessionAction` and
`chat.headStart`, and keep the Head Start tool-approval tail intact
across a
custom `prepareMessages` hook so prompt caching and Head Start compose.
## Summary
Adds an environment-scoped HTTP API over the Errors feature, mirroring
the runs API. Task-run failures are grouped by a fingerprint into "error
groups," and this exposes everything you can do with them in the
dashboard:
- `GET /api/v1/errors` lists error groups, with
`filter[taskIdentifier]`, `filter[version]`, `filter[status]`
(`unresolved`/`resolved`/`ignored`), `filter[search]`, a time range, and
cursor pagination.
- `GET /api/v1/errors/{errorId}` retrieves a single group (summary,
lifecycle state, affected versions).
- `POST /api/v1/errors/{errorId}/{resolve,ignore,unresolve}` changes its
state.
- `GET /api/v1/runs?filter[error]={errorId}` lists the runs behind a
group.
Request and response schemas are exported from `@trigger.dev/core/v3` so
the SDK can reuse them, and all endpoints are documented in the API
reference (OpenAPI). `errorId` is the `error_<fingerprint>` friendly id.
## Attribution
State changes record who made them. A plain environment API key has no
user, so `resolvedBy`/`ignoredByUserId` stay null. When the caller uses
an environment JWT obtained by exchanging a personal access token or a
delegated user token at `POST /api/v1/projects/:ref/:env/jwt`, that
exchange now stamps an `act` delegation claim, and the write endpoints
read `act.sub` to attribute the change to the acting user. This is the
first endpoint to consume the `act` claim, so two small pieces of
plumbing ride along: the exchange stamps `act` for personal-access-token
subjects too (it was delegated-token-only), and the public-JWT
bearer-auth path surfaces `act.sub` to the handler.
Built on the delegated-token work in #3997.
## Summary
Adds the `4.5.0-rc.7` entry to the AI chat changelog, covering the
agent-facing changes in
[v4.5.0-rc.7](https://github.com/triggerdotdev/trigger.dev/releases/tag/v4.5.0-rc.7):
- `chat.headStart` now works with the `chat.customAgent` and
`chat.createSession` backends, not just `chat.agent`
- Opt-in Anthropic system-prompt caching via
`chat.toStreamTextOptions()`
- Three custom-agent-loop fixes: continuation replay, mid-stream
steering, and task-backed tools
- `trigger skills` follow-ups: `trigger-` namespacing, SDK-bundled docs,
and a new cost-savings skill
Generic, non-agent rc.7 items (the CLI uninitialized-project error
message, run-span cost fields) are intentionally left out to keep this
changelog scoped to AI chat agents.
A batch of technical-SEO fixes across the docs, all reader-facing
(titles, links, redirects):
- Canonicalize the duplicate CLI command pages: the bare `/cli-dev` and
`/cli-deploy` paths now permanently redirect to their `-commands`
equivalents, and a duplicate navigation entry is removed.
- Give the three pages that all rendered as "Overview" distinct titles
(Building with AI, self-hosting overview, Management API overview), with
sidebar labels unchanged.
- Replace the generic "Learn more" links in the introduction's
build-extension list with descriptive anchor text.
- Switch two http links to https in the Supabase guides, point a
troubleshooting page's help link to Discord, and add missing meta
descriptions to three help and troubleshooting pages.
## Summary
Adds a "Stopping generation" section to the Custom agents page. It
documents how stop works when you drop down from `chat.agent` to
`chat.createSession`: pass `turn.signal` (a combined stop-and-cancel
`AbortSignal`) to `streamText`, and `turn.complete()` cleans up the
aborted partial, accumulates it as its own assistant message, and keeps
the run alive for the next turn. `turn.stopped` distinguishes a user
stop from a full run cancel.
Until now the createSession stop story only existed as scattered fields
in the reference table; the client side (`transport.stopGeneration`) and
the `chat.agent` run-callback signals were documented, but not the
custom-agent turn loop. Steering for these backends is already covered
on the pending messages page, which this page links to.
## Summary
Adds a "Duration and cost while paused" section to the human-in-the-loop
page. It explains that a HITL pause (a no-execute tool waiting on
`addToolOutput`) suspends the run and frees compute, so the human's
thinking time does not count against `maxDuration` (which measures
active CPU time and excludes suspended waitpoint time, the same as
`wait.for`). Customers don't need to raise `maxDuration` or end the run
to support long human waits.
This was a recurring point of confusion: readers assumed the pause holds
the run open and burns the budget. Also updates the how-it-works
pseudocode ("Agent suspends (compute freed)") and links `wait.for` and
`maxDuration` on first mention.
## Summary
Three improvements to the SDK-bundled agent skills (follow-up to the
skills installer):
- **`trigger-` namespace.** The installed skills (`authoring-tasks`,
`getting-started`, …) had generic names that collide with unrelated
skills in a shared agent skills directory. They're now prefixed —
`trigger-authoring-tasks`, `trigger-getting-started`, etc. — matching
the convention the public skills repo already uses.
- **New `trigger-cost-savings` skill.** An MCP-driven cost audit:
right-sizes machines, flags missing `maxDuration`, spots sequential
triggers that could batch, and reviews schedule frequency, using
`list_runs` / `get_run_details` for live analysis.
- **Bundle the full docs.** `@trigger.dev/sdk` now bundles the entire
"Documentation" section of the docs (157 pages) instead of a curated
55-page subset, so an agent has the complete, version-pinned reference
in `node_modules`.
## How the bundling works
`scripts/bundleSdkDocs.ts` now reads `docs/docs.json`, walks the
"Documentation" dropdown, and copies every page under it into the SDK.
The set tracks the docs navigation automatically — add a page to the nav
and it ships, no skill edits needed. The API reference and Guides &
examples dropdowns are intentionally excluded. A skill's `sources:`
frontmatter is now informational only.
The dropped idea of a dedicated `trigger-config` skill is replaced by
references to the bundled build-extension docs (`config/extensions/*`)
from the `trigger-authoring-tasks` config section and the chat-agent
skills.
## Summary
The agent skills' deep guidance now ships inside `@trigger.dev/sdk` and
is read from `node_modules`, so it tracks the `@trigger.dev/sdk` version
installed in your project automatically. This updates the Skills page,
the Building with AI step, and the rules-redirect page to drop the old
"pinned to the CLI version, re-run to refresh" framing and describe the
version-pinned reference instead.
Pairs with the SDK/CLI change in #3937. Keep this draft until that
ships, since it describes behavior that is not released yet.
## Summary
The "What extractNewToolResults returns" reference in the
tool-result-auditing guide did not match the SDK. It listed an `input`
field that `chat.history.extractNewToolResults()` never returns, and
marked `output` as optional when it is always present.
This corrects the block to the real `ChatNewToolResult` shape
(`toolCallId`, `toolName`, `output`, optional `errorText`). Every usage
example in the same guide already reads only those fields, so the
reference now matches both the examples and the code.
## Summary
New `/ai-chat/prompt-caching` guide covering how to cache a chat agent's
prompt prefix with Anthropic prompt caching: the system prompt, the
conversation history (a `prepareMessages` breakpoint), and how caching
interacts with compaction. It also shows how to verify cache hits via
usage and the dashboard, the prefix-stability footguns, and an "Other
providers" section (OpenAI and Google cache automatically; Amazon
Bedrock uses `cachePoint` through `systemProviderOptions`).
Registered under Features in the AI Agents nav, next to Compaction.
---------
Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: Eric Allam <ericallam@users.noreply.github.com>
## Summary
`chat.headStart` now works with the `chat.customAgent` and
`chat.createSession` backends (not just `chat.agent`), and takes a
`triggerConfig` option. These docs cover both.
The Fast starts guide gets a "Handover with custom agents" section
showing how each backend consumes the handover (`consumeHandover`
returning `{ isFinal, skipped }` for custom agents, `turn.handover` for
createSession), including threading `originalMessages` so a resumed tool
round merges into the handed-over assistant. The `chat.headStart` API
section documents `triggerConfig` (tags, queue, machine, and the rest)
on the auto-triggered run.
The reference picks up `ChatTurn.handover`, `turn.complete()` with no
source, `chat.waitForHandover`, and a new `HeadStartHandlerOptions`
table.
Docs for the SDK changes in
[#3963](https://github.com/triggerdotdev/trigger.dev/pull/3963).
## Summary
Documents the Sessions HTTP API for non-SDK and server-to-server
callers, which until now appeared only in the conceptual
[ai-chat/sessions](https://trigger.dev/docs/ai-chat/sessions) page.
## What's covered
- **Sessions API reference** —
`create`/`list`/`retrieve`/`update`/`close` added to the OpenAPI spec
and a new "Sessions API" group (`management/sessions/*`), mirroring the
Runs API.
- **Channel endpoints** — a reference page for the `.in`/`.out` realtime
HTTP endpoints (append, SSE read, records drain), the wire protocol,
`Last-Event-ID` resume, and the per-direction auth boundary (`.out`
append is secret-key only).
- **Session scopes** — `read:sessions:{id}` / `write:sessions:{id}` in
the authentication docs, with the capability boundary and the 1h token
TTL.
Cross-linked with the SDK-side `ai-chat/sessions` page. Verified by
rendering each page on the Mintlify dev server.
## Summary
Adds the 4.5.0-rc.6 entry to the AI chat changelog, covering the
chat-facing items shipping in
[#3870](https://github.com/triggerdotdev/trigger.dev/pull/3870): the
chat.agent reliability batch, the continuation boot latency fix, the
chat.headStart hydration and reasoning fixes, the chat.createSession
stop and continuation fixes, and the new trigger skills installer.
Should merge alongside the release so the changelog matches the
published version.
## Summary
Documents the two lower-level chat backend APIs and restructures the
Building agents section so it has a sane reading order.
**Custom agents page.** `chat.customAgent()` was effectively
undocumented (one passing mention) and `chat.createSession()` was buried
at the bottom of the Backend page, prompted by a customer asking whether
dropping down a level was supported at all. Both now live on one
dedicated page framed as a composition: register with `customAgent`,
then drive turns with the managed `createSession` iterator or a
hand-rolled primitives loop. The page covers the patterns the managed
lifecycle otherwise handles for you, each verified against a running
agent: seeding history on continuation runs (and why the seed must go
through the turn-0 `addIncoming`, which replaces the accumulator),
persisting the user message before streaming so a mid-stream reload
keeps it, racing `totalUsage` after a stop so the loop cannot wedge, and
the single-message wire shape.
**Backend page.** Now leads with a decision table across the three
abstraction levels and focuses on `chat.agent()`, routing to the new
page. Stale examples that read a plural `messages` field off the wire
payload are fixed (copy-pasting them broke turn accumulation), and the
ChatSessionOptions / ChatTurn reference tables gain their missing rows
(`compaction`, `pendingMessages`, usage fields, `setMessages`,
`prepareStep`).
**Anatomy page + reorder.** The Building agents group opened with the
long How it works mechanics page, a wall right after the Quick Start. A
short Anatomy page now leads the group: the three moving parts, one
annotated example where each region names the page that covers it, and a
routing table. How it works moves to the end of the group as the depth
payoff, matching where peer docs put their internals pages.
All pages visually verified against a local Mintlify build; cross-links
and anchors updated across the section.
## Summary
Two documentation improvements for the AI chat docs.
**Head-start persistence contract.** The fast starts page now documents
what your hooks can rely on across a head-start handover: one stable
assistant `messageId` for the whole turn, `onTurnComplete` as the
canonical persistence point, reasoning parts flowing into durable
history, and how Head Start composes with `hydrateMessages` (the
first-turn history arrives as `incomingMessages`, and the runtime
splices the warm partial onto the hydrated chain, deduplicated by id).
The hydrate examples on the lifecycle hooks and database persistence
pages now upsert their conversation row, since head-start first turns
run without a preload to create it.
**Sessions page.** The page opened with "a durable, task-bound,
bi-directional I/O channel pair", which reads as jargon and omitted run
orchestration entirely. It now leads with the plain mental model (a pair
of durable streams: input carries user messages, output carries
everything the agent produces) plus the Session's role orchestrating
runs, a diagram, a minimal runnable example, and a section on the
one-session-many-runs lifecycle.
Documents behavior shipping in
[#3907](https://github.com/triggerdotdev/trigger.dev/pull/3907).
## Summary
Accuracy fixes across the AI chat docs: drop the non-existent per-call
option from `transport.preload`, clarify that `onValidateMessages` only
fires on turns carrying incoming messages, soften the turn-complete
token-refresh wording (the header is optional), document the new
`onTurnComplete` `error` field and `finishReason`, and correct the
idle-timeout default to 30 seconds.
## Summary
Updates the AI-tooling docs for the new `trigger skills` installer that
shipped in #3868. The Skills page now documents `trigger skills` (skills
bundled with the CLI, version-matched to your SDK) and the four bundled
skills: `authoring-tasks`, `realtime-and-frontend`,
`authoring-chat-agent`, `chat-agent-advanced`. The old Agent Rules page
becomes a short "rules are now skills" redirect (kept because existing
redirects and the CLI link point at it), and the Building with AI
overview collapses the three-way Skills/Rules/MCP comparison into Skills
vs MCP.
Hold until the v4.5 CLI release ships, since `trigger skills` is not on
npm until then.
## Summary
Docs follow-up for #3683 (`TriggerClient` for per-instance SDK
configuration). Adds a dedicated reference page and threads the new
pattern through the existing management + preview-branches docs.
## What's in
**New page** `docs/management/multiple-clients.mdx` — when to use
`TriggerClient` vs `configure()` vs `auth.withAuth`, env-var fallback
rules, isolation contract, namespace surface, `inheritContext` opt-in,
and a when-to-use-what table.
**Updated pages**
- `docs/management/authentication.mdx` — rewrote the `auth.withAuth`
section to reflect the now-ALS-backed semantics (the prior version
warned about concurrency races and pointed at issue #3298 as a tracked
fix; that fix landed in #3683). Added `tr_preview_*` to the key prefix
list. Reframed the multi-target use case to lead with `TriggerClient`,
with `auth.withAuth` as the temporary-override helper.
- `docs/management/overview.mdx` — added a `Multiple clients in one
process` subsection.
- `docs/deployment/preview-branches.mdx` — added a `Triggering across
multiple branches from one process` example.
- `docs/triggering.mdx` — one-liner pointing at the new page for
cross-project triggering.
- `docs/docs.json` — slotted `management/multiple-clients` into the
Management API nav, right after authentication.
Paired with #3683.
## Test plan
- [ ] Mintlify preview renders cleanly
- [ ] Code samples in each updated page run as documented
- [ ] Cross-page links resolve (`/management/multiple-clients`,
`/management/authentication`)
## Problem
The item-streaming endpoint of the two-phase batch API (`POST
/api/v3/batches/:batchId/items`) processed streamed items strictly
sequentially. For a batch of many large payloads, each offloaded to
object storage inline, this serialized N object-store round-trips inside
a single request and could exceed Node's default `server.requestTimeout`
(300s). The webapp then returned `408`, which the SDK reads as `408
terminated` and retries up to 5 times, turning a slow ingest into a
failure that takes tens of minutes to surface.
## Fix
Ingest now runs through `p-map` over the NDJSON async iterable with
bounded concurrency (`STREAMING_BATCH_INGEST_CONCURRENCY`, default 10):
- `p-map` pulls lazily from the stream, so at most `concurrency` items
are read and in-flight at once. Peak memory stays bounded to roughly
`concurrency × STREAMING_BATCH_ITEM_MAXIMUM_SIZE` and request-body
backpressure is preserved.
- Set the env to `1` for fully sequential ingestion (escape hatch).
## Why this is safe (ordering and idempotency unchanged)
- Ordering derives from each item's index (enqueue `timestamp =
batch.createdAt + index`), not enqueue order.
- Dedup is atomic per index in `enqueueBatchItem`.
- The NDJSON parser now stamps oversized-item markers with their emit
position, removing the consumer's sequential `lastIndex` assumption (the
only order-dependent bit).
- The count-check and conditional-seal path is untouched.
## Scope
This speeds up every batch ingested through the streaming endpoint, not
just large-payload batches. Each item does a per-item Redis enqueue
regardless of size, and those now overlap. Large payloads benefit most
because they add an object-store offload round-trip on top of the
enqueue.
## Verification
Added an integration test (`streamBatchItems.test.ts`) that drives the
real service against Postgres + Redis + RunEngine and times a 150-item
batch at increasing concurrency. Object-store offload is modelled as a
fixed per-item latency (local round-trips are too small to compare
meaningfully):
```
runCount=150
large payloads (10ms/item offload):
concurrency=1 1739ms
concurrency=10 192ms (9.1x faster)
concurrency=50 57ms (30.7x faster)
small payloads (Redis enqueue only, no offload):
concurrency=1 90ms
concurrency=10 24ms (3.7x faster)
```
The test asserts correctness at every concurrency (all items accepted,
sealed, enqueued exactly once), that parallel ingest beats the
sequential floor, and that the small-payload case is strictly faster
than sequential, so the win is not specific to large payloads.
Also exercised end-to-end over real HTTP against a local server: a
20-item batch (12MB body) ingests and seals, a re-stream of the sealed
batch returns `sealed: true` with zero re-accepted items (idempotent
retry), and an oversized item still seals at its correct index.
Existing coverage stays green: concurrent ingest of a 100-item batch,
in-flight processing never exceeding the configured concurrency,
concurrent dedup on streaming retry, and emit-position marker indexing.
## Follow-ups (not in this PR)
- SDK pre-offload of large item payloads (send `application/store` refs
instead of raw blobs) to remove object-store work from the request hot
path and shrink the request body.
- Optional `server.requestTimeout` bump as a safety net.
## CI fix
Added `.github/workflows/codeql.yml` to replace GitHub's automatic
("dynamic") CodeQL scanning. The dynamic setup was failing to upload
SARIF results because the auto-generated `GITHUB_TOKEN` lacked the
`security-events: write` permission. The explicit workflow grants that
permission at the job level and pins all actions to commit SHAs,
consistent with the repo's security conventions.
## ✅ Checklist
- [ ] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [ ] The PR title follows the convention.
- [ ] I ran and tested the code works
---
## Testing
- Integration test (`streamBatchItems.test.ts`) validates correctness
and performance at concurrency 1, 10, and 50 for both large and small
payloads.
- End-to-end verified over real HTTP: 20-item/12MB batch ingests and
seals, idempotent retry returns `sealed: true`, oversized item seals at
correct index.
---
## Changelog
Streaming batch ingest now processes items with bounded concurrency
instead of one at a time, so batches of many large payloads ingest far
faster and no longer time out. Concurrency is configurable via
`STREAMING_BATCH_INGEST_CONCURRENCY` (default 10); set it to 1 for fully
sequential ingestion.
---
## Screenshots
_[Screenshots]_
💯
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
## Summary
A new guide for connecting a database to your tasks: where to create the
client, how to size the connection pool against your provider's limit,
when to reach for a pooler, and how to release connections at waits so
you don't hit "too many connections" or crash on resume.
It covers node-postgres, Prisma, Drizzle, and MongoDB, with researched
direct and pooled connection limits for the common Postgres providers
(Supabase, Neon, RDS, PlanetScale) and MongoDB Atlas. The page lives
under Documentation, Troubleshooting, and is linked from the chat agent
docs (overview, lifecycle hooks, chat.local, and the database
persistence pattern).
Matt Aitken