The three "dual-write" tests inherited from phase-1 were asserting
invariants that phase-3 deliberately abandoned when the mollify path
moved from "buffer.accept + engine.trigger" to "buffer.accept +
synthetic result, drainer replays later":
- `mollify action triggers dual-write` — rewritten to assert the new
contract: synthetic `MollifySyntheticResult` (run.friendlyId,
isCached:false, notice.code = "mollifier.queued"), buffer.accept
fires with the canonical engine.trigger snapshot, NO Postgres row
(the run materialises only when the drainer replays).
- `engine.trigger throwing AFTER buffer.accept` — deleted. Phase-3
never invokes engine.trigger on the mollify path, so the scenario
is structurally impossible.
- `debounce match produces an orphan buffer entry` — deleted. Phase-3's
C1 debounce bypass at the gate (returns pass_through for debounce
triggers) means the mollify branch is never entered for debounced
requests. The C1 invariant is pinned at mollifierGate.test.ts:440;
duplicating it at the trigger-task layer adds nothing.
Net: 6 mollifier integration tests → 4, all 4 passing, no coverage
gap (gate-level + drainer-handler-level tests own the deleted
scenarios' invariants).
Node's setTimeout can fire a millisecond or two early under CI load,
causing the existing `>= 500ms` lower bound to flake (saw 499ms in CI).
Loosen to `>= 450ms` — the behaviour being pinned is "stop honors the
deadline instead of waiting for the hung handler indefinitely", not
millisecond-precise timing.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
If buffer.requeue() or buffer.fail() throws during error recovery inside
processEntry, the rejection used to escape processOneFromEnv and reject
runOnce's Promise.all — discarding handler results from sibling envs in
the same tick. Wrap processEntry in try/catch so the failed env is just
counted as "failed" for the tick, matching the invariant stated in the
processOneFromEnv comment.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Brings in every phase-2 fix without regressing the phase-3 surface.
Resolutions:
* `apps/webapp/app/v3/mollifier/mollifierDrainer.server.ts` — kept
phase-3's `MollifierDrainer<MollifierSnapshot>` generic and the
`createDrainerHandler({ engine, prisma })` + `isRetryablePgError`
wiring; added phase-2's `MollifierConfigurationError` class and
switched every env read to `TRIGGER_MOLLIFIER_*`. The doc comment on
`getMollifierDrainer` now points at `mollifierDrainerWorker.server.ts`
instead of the legacy `worker.server.ts`.
* `apps/webapp/app/runEngine/services/triggerTask.server.ts` — kept
phase-3's gate inputs (`options: { debounce, oneTimeUseToken,
parentTaskRunId, resumeParentOnCompletion }` for the C1/C3/F4
bypasses) and the `#buildEngineTriggerInput` refactor; wrapped the
gate call in phase-2's `this.isMollifierGloballyEnabled()` short-
circuit so a trigger with the feature off skips the
`GateInputs` allocation + `mollifier.decisions{outcome=pass_through}`
increment. `mollifierOutcome` is `GateOutcome | null`, downstream
access uses optional chaining. Phase-2's phase-1 dual-write block is
dropped — phase-3's mollify branch already writes via
`mollifyTrigger` and returns a synthetic result.
Phase-2 fixes preserved (verified post-merge):
- `TRIGGER_MOLLIFIER_*` env-var prefix on every consumer
- `TRIGGER_MOLLIFIER_DRAINER_ENABLED` separate per-replica switch
- `MollifierConfigurationError` rethrow in the drainer bootstrap
- Factory create + bootstrap start split (`drainer.start()` only
runs after SIGTERM handlers register)
- `signalsEmitter.on` instead of `process.once`
- `MollifierDrainer.stop` deadline-timer `clearTimeout` in finally
- `vi.fn` handler spies removed from `drainer.test.ts`
- `isMollifierGloballyEnabled` DI hook on `RunEngineTriggerTaskService`
- Legacy `worker.server.ts` header
- New `mollifierDrainerWorker.server.ts` error-classification test
- Updated `.server-changes/mollifier-burst-protection.md`
Pre-existing on phase-3, NOT introduced by this merge (verified by
`git diff origin/mollifier-phase-3 -- apps/webapp/test/engine/triggerTask.test.ts`
showing only `isMollifierGloballyEnabled: () => true` additions):
- `mollifier · mollify action triggers dual-write` asserts
`prisma.taskRun.findFirst` returns a row, but phase-3's
`mollifyTrigger` writes to the buffer and returns a synthetic
result without invoking `engine.trigger`. Test name + assertions
still reflect phase-1 dual-write semantics.
- `mollifier · engine.trigger throwing AFTER buffer.accept` expects
the call to reject; phase-3 returns synthetic before
`engine.trigger` is reached, so the injected throw never fires.
- `mollifier · debounce match produces an orphan buffer entry` —
phase-3 explicitly skips the buffer branch when debounce is set
(`\!body.options?.debounce` guard), so no orphan accept happens.
These three tests describe phase-1 monitoring behaviour on phase-3
code that abandoned dual-write. Updating them is phase-3's own tech
debt, separate from this merge.
## Summary
Two papercuts new contributors hit running this repo locally:
1. Fresh clones default to v1 (Redis-only) realtime streams, so Sessions
and `chat.agent` error with `"S2 configuration is missing"`, even though
the `s2` service is already in `docker/docker-compose.yml` and pre-seeds
a `trigger-local` basin. Wire `REALTIME_STREAMS_S2_*` to it in
`.env.example` so the new-contributor flow just works. (Also drop the s2
healthcheck: the image is distroless, so the `wget` check always reports
unhealthy.)
2. Two clones can't both run `pnpm run docker` because ports, project
name, and container names are all hardcoded. Parameterize every host
port as `${VAR:-default}`, drive the project name via
`COMPOSE_PROJECT_NAME` (with a top-level `name:` field as the default),
prefix container names with `${CONTAINER_PREFIX:-}`, and pass
`--env-file .env` so compose reads the same root `.env` the webapp does.
The "Running multiple instances side by side" block in `.env.example`
lists every overridable knob.
Also split the optional services (`electric-shard-1`, `ch-ui`,
`toxiproxy`, `nginx-h2`, `otel-collector`, `prometheus`, `grafana`) into
`docker-compose.extras.yml` behind a new `pnpm run docker:full` script.
The core stack keeps everything the webapp actually needs to boot:
postgres, redis, electric, minio, clickhouse + migrator, s2-lite.
Defaults match every previous hardcoded value, so existing setups keep
working without touching `.env`.
## Test plan
- [x] `pnpm run docker` on a clean clone brings up the core services on
the standard ports under the `triggerdotdev-docker` project name.
- [x] Setting `COMPOSE_PROJECT_NAME=triggerdotdev-docker-alt` + the
`*_HOST_PORT` overrides in `.env` brings up a second stack alongside the
default one with no port or container-name clashes.
- [x] Webapp boots cleanly against the default `.env.example` values;
`/healthcheck` returns 200, no S2 errors.
- [x] s2-lite basin `trigger-local` accepts an append + read via the
same REST endpoints the webapp uses.
- [x] `pnpm run docker:full` brings up the optional services alongside
the core ones in the same project.
## Summary
Long-running chat agents were filling `session.out` forever — every
`chat.agent` turn appended to the same S2 stream with no trim, and the
Sessions dashboard re-streamed the entire history from `seq_num=0` on
every page load. After this change the agent appends an S2 `trim`
command record after each `trigger:turn-complete`, pointing back at the
previous turn-complete's seq_num. `session.out` stays roughly one turn
long at steady state, regardless of session age.
`trigger:turn-complete` and `trigger:upgrade-required` move from
`chunk.type`-shaped data records into header-form control records under
a uniform `trigger-control` namespace. Built-in transports
(`TriggerChatTransport`, `AgentChat`, the dashboard's `AgentView`)
handle the new shape transparently. Custom transports need a one-line
filter on the `trigger-control` header — see the rewritten "Records on
session.out" section in the client-protocol docs.
The Sessions detail page in the dashboard fetches the agent's per-turn
S3 snapshot via a presigned URL and seeds the transcript view, then
SSE-tails from the snapshot's `lastOutEventId`. Bandwidth and
time-to-first-render scale with unread turns instead of session
lifetime.
Resume contract is now explicit: single-turn-boundary resume always
works (the prior turn-complete is still on the stream), the S2 trim is
eventually consistent over 10-60s, and multi-turn-away resume falls back
to a snapshot reload.
## Summary
The PUT handler at `/realtime/v1/streams/:runId/:target/:streamId` ran
`taskRun.update({ realtimeStreams: { push: streamId } })` on every call,
even when the `streamId` was already present. SDK call patterns that
re-initialize the same stream key on every chunk produce a per-write row
UPDATE, duplicate entries pile up in the array, and the row-lock + TOAST
rewrite cost grows unbounded on long-running stream sessions.
## Fix
Mirror the sibling append handler: read the array first and only push
when the `streamId` isn't already present. Identical behavior for
first-time stream creation; repeat creates short-circuit to a single
indexed read. The dashboard's per-run stream listing keeps working
because the first create still records the entry.
## Test plan
- [ ] A fresh PUT for a new `(run, streamId)` adds the entry to the
array
- [ ] A repeat PUT for the same pair leaves the array unchanged
- [ ] 404 is returned when the run doesn't exist; 400 when the run is
completed
Adds the smallest DI surface to `initMollifierDrainerWorker` (`isEnabled`
and `getDrainer`, both optional, default to live env/singleton) so the
catch-block policy can be tested without manipulating module-level env:
- rethrows MollifierConfigurationError — deterministic misconfig
escapes, which is what makes the production-path crash on boot
(the call site in entry.server.tsx runs sync at module top level,
before `process.on("uncaughtException", ...)` is registered, so an
escape becomes a Node default-handler exit-1).
- rethrows when `name === "MollifierConfigurationError"` even when
`instanceof` fails — covers the Remix dev hot-reload realm edge
case where the catch holds a stale class reference.
- swallows non-configuration errors — a transient Redis blip during
buffer init shouldn't take the whole webapp down.
- no-op when disabled — the factory isn't invoked when the enabled
predicate returns false.
Also updates the existing mollifier server-changes note to: rename env
vars to TRIGGER_MOLLIFIER_* prefix, document the TRIGGER_MOLLIFIER_DRAINER_ENABLED
split for multi-replica drainer placement, and call out the new fail-loud
behaviour on drainer misconfiguration.
The bootstrap in mollifierDrainerWorker.server.ts wrapped getMollifierDrainer()
in a try/catch that logged-and-continued on any error, which absorbed the two
designed-to-crash throws in initializeMollifierDrainer():
- "MollifierDrainer initialised without a buffer" (missing buffer client)
- "TRIGGER_MOLLIFIER_DRAIN_SHUTDOWN_TIMEOUT_MS must be at least ... below
GRACEFUL_SHUTDOWN_TIMEOUT" (shutdown-timeout reconciliation)
Both are deploy-time mistakes: silently disabling the drainer means the
gate keeps writing to the buffer, the drainer never reads, and entries
TTL out in 10min. Bounded in phase 1 (monitoring-only) but customer-
visible data loss in phase 2/3 where the drainer replays into engine.trigger.
Better to fail loud now than retrofit the contract later.
Introduce MollifierConfigurationError for the two deterministic throws.
The bootstrap's catch now rethrows that class (process crashes at module
top-level → orchestrator health check fails → deploy rolls back) while
still logging-and-continuing on transient errors (Redis blip during init
shouldn't take the whole webapp down). instanceof + name fallback covers
the Remix dev hot-reload realm edge case.
## Summary
- Adds a `beforeSend` rule in `apps/webapp/sentry.server.ts` that
collapses Prisma `P1001` ("Can't reach database server") errors into a
single Sentry issue regardless of which call site threw, by setting
`event.fingerprint = ["prisma-p1001-db-unreachable"]` and tagging
`db_unreachable:true`.
- Matches both `err.code === "P1001"` (Prisma's `KnownRequestError` when
a connection drops mid-query) and `err.errorCode === "P1001"`
(`InitializationError` when the client fails to connect at startup).
- Implemented as a small extensible `FINGERPRINT_RULES` table so further
fan-out errors can be added with one entry.
## Verification
End-to-end verified locally with `debug: true` on the SDK:
- Real Prisma `P1001` thrown from a loader (DB stopped mid-request) is
captured by Sentry's Remix auto-instrumentation
- `beforeSend` fires with `originalException.code === "P1001"`, rule
matches
- `event.fingerprint = ["prisma-p1001-db-unreachable"]` and
`tags.db_unreachable = "true"` applied
- Event lands in Sentry under the new fingerprint
## Test plan
- [ ] Deploy to staging; confirm P1001 events appear under a single
`prisma-p1001-db-unreachable` issue rather than fanning out
- [ ] Confirm `db_unreachable:true` tag is filterable in Sentry
- [ ] Verify non-P1001 errors are unaffected (event passes through
`beforeSend` untouched)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The previous commit added a perf short-circuit at the call site that
read `env.TRIGGER_MOLLIFIER_ENABLED` directly. That broke three
mollifier integration tests in CI: the tests inject a custom
`evaluateGate` via the existing DI seam expecting the buffer-write
branch to be reached, but CI has no `.env` (the `apps/webapp/.env`
symlink target is absent), the Zod default `"0"` wins, the call site
short-circuits to `null` before the injected gate runs, and
`buffer.accepted` stays empty.
Make the global-enabled check itself injectable:
- New constructor opt `isMollifierGloballyEnabled?: () => boolean`,
defaulting to `() => env.TRIGGER_MOLLIFIER_ENABLED === "1"`. Each
DI hook now represents one decision (gate, buffer, global-enabled),
so a test that wants the buffer-write branch reached can inject
`isMollifierGloballyEnabled: () => true` alongside its custom gate.
- Call site now reads `this.isMollifierGloballyEnabled()` instead of
`env.TRIGGER_MOLLIFIER_ENABLED` directly. In production, with no DI
override, the default closure resolves `env` exactly once per call
just as before — same perf win when the flag is off.
- All six mollifier DI injection sites in triggerTask.test.ts now also
pass `isMollifierGloballyEnabled: () => true` so the tests' DI
surface matches the new contract regardless of CI env state.
## Summary
Local self-hosted deploys (`trigger deploy --local-build --push
--builder orbstack` or any other buildx setup using the **docker**
driver) fail at the push step with:
```
ERROR: failed to build: failed to solve:
exporter option "rewrite-timestamp" conflicts with "unpack"
```
The docker driver auto-enables `unpack=true` when pushing, and that's
incompatible with `rewrite-timestamp` (which the CLI sets for
reproducible-build hashing).
Adds a simple env-var opt-out so contributors can keep using their
default builder. The flag is only read by the local-build code path;
remote/cloud builds are unaffected.
```bash
TRIGGER_BUILD_SKIP_REWRITE_TIMESTAMP=1 \
pnpm exec trigger deploy --profile default --local-build --push --builder orbstack
```
The trade-off: skipping `rewrite-timestamp` means layer timestamps
reflect actual build time, so two identical builds produce different
layer hashes. Fine for a local-dev registry; the only real consumer of
timestamp-stability is registry-layer cache hit rates.
## Test plan
- [x] Manual: ran `trigger deploy --profile default --local-build --push
--builder orbstack` against the localhost webapp + a local Docker
registry on port 5001 — first failed with the rewrite-timestamp/unpack
error, then succeeded after setting
`TRIGGER_BUILD_SKIP_REWRITE_TIMESTAMP=1`.
- [x] Full chat.agent smoke sweep (15 tests, including suspend/resume,
deepResearch subtask, AgentChat orchestrator) against the deployed image
— all pass.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
`TriggerChatTransport` had a single `baseURL` option covering both the
`.in/append` POSTs and the long-lived `.out` SSE subscription. Customers
wanting to route the SSE through a proxy (e.g. a Cloudflare worker
capturing JA4 fingerprints for bot detection) had to send every append
through the proxy too, adding a hop to every user message.
New optional `streamBaseURL` overrides the SSE base URL only; appends
keep using `baseURL`. Falls back to `baseURL` when unset, so existing
transports are unchanged.
```ts
const transport = new TriggerChatTransport({
task: "ai-chat",
baseURL: "https://api.trigger.dev",
streamBaseURL: "https://chat-proxy.example.com",
accessToken,
startSession,
});
```
Verified with a new test in `chat.test.ts` that asserts `.in/append`
routes through `baseURL` and `.out` SSE routes through `streamBaseURL`.
All existing tests still pass.
`dorny/paths-filter` defaults to OR semantics across the pattern array,
so the leading `**` matched every file and the `!...` excludes were
no-ops. The `code` filter has been returning `true` for every PR since
#3615.
Split into two filter steps: `code` moves into its own step with
`predicate-quantifier: every` so excludes actually subtract. The two
re-include workflow files become a separate `typecheck_self` filter that
the `typecheck` job ORs into its `if:`.
Side effect: workflow-file-only PRs that don't touch `pr_checks.yml` or
`typecheck.yml` no longer trigger typecheck. Previously they did because
the filter was broken-true.
## Summary
Codify two rules for ClickHouse migration authors that came out of the
029/030 ordering incident on the TRI-9367 test cloud deploy:
1. **Number files to `max(existing) + 1`, never slot in below the
latest.** Goose runs in strict mode in the cloud deploy pipeline and
refuses to apply a missing version below the current version — slotting
a file in below an already-applied number blocks the next deploy.
2. **DDL must be idempotent** (`ADD COLUMN IF NOT EXISTS`, `DROP COLUMN
IF EXISTS`, `CREATE TABLE IF NOT EXISTS`, etc.) so a retry or
out-of-order apply (`goose up --allow-missing` for local recovery,
manual fixups) is a no-op rather than an error.
## Where the rules live
- `internal-packages/clickhouse/CLAUDE.md` — full rules + example for
migration authors (and AI agents writing migrations).
- `.claude/REVIEW.md` — added a 🔴 finding under "What makes a 🔴
Important finding" so PR reviewers flag either fault as blocking.
The existing migration files are left untouched; the idempotency
requirement applies going forward.
## Test plan
- [ ] Next ClickHouse migration PR uses `IF NOT EXISTS` / `IF EXISTS`
forms
- [ ] No new migration files numbered below an already-applied version
on test/prod
## Summary
Renumber `029_add_task_kind_to_task_runs_v2.sql` →
`031_add_task_kind_to_task_runs_v2.sql` to fix a deploy-blocking
out-of-order migration, and make the DDL idempotent with `ADD COLUMN IF
NOT EXISTS` / `DROP COLUMN IF EXISTS`.
## Root cause
- Migration `030_create_sessions_v1.sql` landed on main on 2026-04-28
(PR #3417) and was applied to test cloud ClickHouse on a subsequent
deploy. Current goose version on test ClickHouse: **30**.
- Migration `029_add_task_kind_to_task_runs_v2.sql` was authored later
on 2026-05-10 as part of the Sessions primitive PR series (`be1a6cf8`).
- The next test cloud deploy failed because goose strict-mode refused to
apply a missing version *before* the current version:
```
goose run: error: found 1 missing migrations before current version 30:
version 29: 029_add_task_kind_to_task_runs_v2.sql
```
## Fix
1. **Rename to `031_*`** (next available number after 030). Goose now
treats it as a new migration after 030 and applies it cleanly on
test/prod where the column does not yet exist.
2. **Make the DDL idempotent** (`ADD COLUMN IF NOT EXISTS`). The
original 029 may have been applied in environments that ran goose with
`--allow-missing` (e.g. some local dev databases) — those would have the
column already, and the rename causes goose to see 031 as new and
re-attempt the ADD. Idempotent DDL keeps that path safe. The `Down`
mirrors with `DROP COLUMN IF EXISTS`.
## Test plan
- [ ] Test cloud deploy (after this lands) successfully runs the
ClickHouse migration step
- [ ] `task_kind` column shows up on `trigger_dev.task_runs_v2`
post-migration
- [ ] Local environments that had previously applied 029 do not error on
the next `goose up`
evaluateGate ran on every trigger regardless of TRIGGER_MOLLIFIER_ENABLED.
With the flag off (the default everywhere it hasn't been opted in), the
gate still produced a `pass_through` decision after allocating a
GateInputs object, spreading defaultGateDependencies inside evaluateGate,
and incrementing the `mollifier.decisions{outcome=pass_through}` OTel
counter. Cheap individually, but triggerTask is the hottest code path in
the system — multiply by trigger rate and the unnecessary work compounds.
Guard the gate call with a direct env.TRIGGER_MOLLIFIER_ENABLED check at
the call site. When the flag is off, mollifierOutcome is null and the
downstream `mollifierOutcome?.action === "mollify"` branch skips the
buffer dual-write entirely — zero allocation, zero counter increment on
the disabled path. When the flag is on, behaviour is unchanged.
Lost-signal note: with mollifier off, we no longer count "pass_through"
decisions in the OTel counter (the gate never runs). That's a non-issue
— "pass_through count when feature is off" is just total trigger rate,
which is already observable via the trigger handler's own spans/counters
upstream. The gate counter remains the source of truth for the
mollify/shadow/pass_through ratio when the feature is on, which is the
load-bearing signal.
This addition was applied while phase-2 was already in review and is
out of scope for the mollifier PR. The underlying clarification is
worth landing — just not on this branch.
`process.once("SIGTERM", stopDrainer)` was the odd one out — every
other webapp service (runsReplicationInstance, llmPricingRegistry,
dynamicFlushScheduler, marqs, eventLoopMonitor) registers through
`signalsEmitter` from `~/services/signals.server`, an EventEmitter
backed by a single `process.on()` that fans out to all listeners.
Switching gets us:
- codebase consistency;
- `.on` (not `.once`) so a second SIGTERM, if the orchestrator emits
one before SIGKILL, still reaches us;
- if SIGTERM lands in the narrow gap between the listener attaching
and drainer.start() below, the first invocation no-ops (stop()
returns early because isRunning is false) but the listener stays
attached for any subsequent signal, instead of being consumed and
leaving the now-running drainer with no graceful-stop path.
The Promise.race between this.loopPromise and this.delay(timeoutMs)
discarded the timeout's underlying setTimeout handle whenever the loop
branch won. The discarded timer was still ref'd by libuv and pinned the
Node event loop alive for the remainder of `timeoutMs` — exactly the
shutdown slack the timeout was supposed to bound.
Inline the timer in stop() with a captured handle and clearTimeout() it
in a finally block, so every exit path (loop-won, timeout-won, throw)
releases the ref. The in-loop delay() calls are unchanged — they're
awaited normally and their timers fire-and-clear themselves.
## Summary
Adds `.changeset/pre.json` to put the repo into changesets pre mode with
dist-tag `rc`. After this merges, the changesets bot regenerates the
existing release PR as `chore: release v4.5.0-rc.0`. Merging that PR
publishes the first release candidate of 4.5.0 to npm under `@rc`.
The pre-mode plumbing landed in #3628. The release content (chat.agent +
sessions + ai prompts + dashboard server-changes) landed in #3629.
## What ships when the bot PR merges
Under dist-tag `rc`:
-
`@trigger.dev/{sdk,core,build,react-hooks,redis-worker,plugins,python,rsc,schema-to-json}@4.5.0-rc.0`
- `trigger.dev@4.5.0-rc.0`
Plus:
- Docker image `ghcr.io/triggerdotdev/trigger.dev:v4.5.0-rc.0`
(immutable tag only — `:v4-beta` is not touched)
- Helm chart `oci://ghcr.io/triggerdotdev/charts/trigger.dev:4.5.0-rc.0`
- GitHub release `v4.5.0-rc.0` marked as Pre-release (no Latest badge)
What does NOT happen:
- npm `latest` stays at 4.4.6
- No marketing-site changelog PR (gated on `is_prerelease != 'true'`)
- Docker `:latest` not touched (we never push it anyway in this repo)
## Iteration
For subsequent rc.N: add a regular changeset to main, bot regenerates
the release PR as `v4.5.0-rc.N`. Merge to ship.
## Exiting pre mode
When ready to ship stable: `pnpm exec changeset pre exit`, push, merge
regenerated PR. That publishes `4.5.0` under `latest` and fires the
marketing-site dispatch.
## Summary
Refocuses the v4.5.0 changeset and server-changes content on the
public-facing AI features story, replacing the pre-release-internal diff
framing that had accumulated in `.changeset/` and `.server-changes/`.
Pairs with the RC support PR — the next bot regeneration will pick up
this content.
## What's in here
### Changeset rewrites
- **`chat-agent.md` rewritten as the headline AI Agents entry** —
written from the `docs/ai-chat/` surface (not from internal pre-release
diffs). Covers useChat integration, multi-turn durability via Sessions,
lifecycle hooks, stop generation, tool approvals (HITL), pending
messages + background injection, actions, typed state primitives,
`chat.toStreamTextOptions()`, multi-tab coordination, network
resilience, and the first-turn fast path (`chat.headStart`).
- **New `ai-prompts.md`** — announces the Prompts feature publicly for
the first time. Code-defined templates, deploy-versioning, dashboard
overrides, AI SDK telemetry integration, `chat.agent` integration via
`chat.prompt.set()` + `chat.toStreamTextOptions()`, full management SDK.
- **`sessions-primitive.md` expanded** — calls out
`tasks.triggerAndSubscribe()` and `sessions.list` as standalone
primitives (not just chat.agent infrastructure).
- **`chat-agent-on-boot-hook.md` trimmed** — drops "if you previously…"
pre-release migration framing.
- **Deletes 4 changesets** that described pre-release-internal
migrations or were circular ("groundwork for the upcoming chat.agent" —
chat.agent ships in the same release).
### Server-changes rewrites (`.server-changes/`)
Five new entries for the dashboard surface of the AI feature set:
- Agents list page
- Agent Playground
- Sessions dashboard
- Prompts dashboard (list with usage sparklines + detail with template /
Generations / Metrics / Versions tabs + override UI)
- Models registry (provider-grouped catalog with cross-tenant usage
metrics)
- AI generation span inspector on run traces
- Runs list Task source filter (Standard / Scheduled / Agent)
- Run-detail Agent view (segmented control)
Each entry is 1–2 sentences, no bullets, no implementation file paths —
fits as a single bullet in a future changelog.
Three older `.server-changes/` files were merged or split into the
cleaner taxonomy above and deleted.
## Out of scope
Non-AI-feature server-changes (admin-tabs, queue-length-cap fix,
worker-deployment race, streamdown upgrade, etc.) and changesets
(idempotency-key cap, sigsegv retry, locals-key fix, plugin auth, region
filters, etc.) are untouched.
## Summary
Enables shipping `X.Y.Z-rc.N` prereleases of `@trigger.dev/*` via
changesets pre mode. RCs publish under the `rc` npm dist-tag, never
claim `latest`, and don't trigger marketing-site changelog PRs. The
plumbing is hyphen-in-version detection in `release.yml` — no separate
workflow, no opt-in flag at publish time.
Validated end-to-end against a sandbox repo (real npm publishes, Docker
builds, Helm chart pushes, GitHub releases) before porting back. Full RC
lifecycle tested: pre enter → rc.0 → iterate to rc.1 → pre exit →
stable. Plus interaction with the existing release-branch hotfix flow.
## What changes
### `release.yml`
- New `is_prerelease` output (hyphen-in-version)
- GitHub release adds `--prerelease` flag for RC publishes (Pre-release
badge, not Latest)
- `dispatch-changelog` job gated on `is_prerelease != 'true'` — no
marketing-site PR per RC
### Docker workflows
- Removes the `:v4-beta` floating tag entirely from `publish-webapp.yml`
and `publish-worker-v4.yml`. v4 is GA; the tag is a misnomer and is
already inconsistent with the npm side (npm `v4-beta` dist-tag was
frozen at 4.0.4 months ago while Docker `:v4-beta` kept bumping).
Self-hosters should pin to a versioned tag going forward — the last
value of `:v4-beta` stays frozen wherever it currently points.
### CLI version-check fix
(`packages/cli-v3/src/utilities/initialBanner.ts`)
Switches the "new version available" comparison from JavaScript
`localeCompare` to `semver.lt`. The old comparison handled `X.Y.Z-rc.N`
vs `X.Y.Z` incorrectly — a user on `4.5.0-rc.0` would never be prompted
to upgrade once `4.5.0` stable shipped (lex order put the prerelease
ahead of the bare version). Real semver gets this right.
Stable users were never affected: the check queries the `@latest`
dist-tag, which by convention never points at a prerelease.
## How an RC actually publishes after this
1. `pnpm exec changeset pre enter rc` on main, push the `pre.json`
2. Bot regenerates the release PR as `chore: release v<X.Y.Z>-rc.0`
3. Merge → `release.yml` runs `changeset publish` which reads
`pre.json.tag` and publishes under `--tag rc`. GitHub release marked
Pre-release. No marketing-site dispatch.
4. Iterate by adding changesets normally; bot bumps to `rc.1`, `rc.2`, …
5. When ready: `pnpm exec changeset pre exit`, push, merge regenerated
PR → stable ships under `latest` and the marketing-site dispatch fires.
The literal reading of "never mock anything" trips up AI reviewers
(and humans new to the repo) — they flag any `vi.mock` / `vi.fn` /
`vi.spyOn` they see, even when the usage isn't actually faking
behavior. Three patterns are fine and should NOT be flagged:
1. Module-load workarounds — vi.mock("~/db.server") at the top of a
unit test to stop prisma.$connect() firing at import. Cuts the
import graph, doesn't fake DB behavior.
2. Hand-written DI doubles where the real implementation has its own
dedicated infra-backed tests (CapturingMollifierBuffer, MockPayloadProcessor,
etc.). Unit test covers wiring, integration test covers the seam target.
3. vi.fn as a DI-seam probe — convenience for "was the seam called."
Equivalent to a closure-counter; not load-bearing on what's proven.
Still 🔴: spying on the code path under test then asserting the spy was
called (tautology), or replacing real infra with mocks in tests meant
to cover real behavior (e.g. mocking Redis in a Redis-queue test).
## Summary
Adds a Region column and Region filter (under More filters) to the runs
list dashboard, the same filter on the public runs list API
(`filter[region]`), and a matching `region` input on the MCP `list_runs`
tool. Each run's executing region is also surfaced as a new optional
`region` field on the runs list and run retrieve responses, populated
from the worker instance group's `masterQueue` identifier.
Useful when you run tasks across multiple regions and want to slice the
runs list — or your existing run-querying scripts — by where the run
actually executed.
## Design
The filter value in the URL / API is the `masterQueue` identifier (the
same string already persisted on `TaskRun` and replicated to ClickHouse
as `worker_queue`), so the query just becomes `worker_queue IN (...)`
with no server-side translation. The Region dropdown options come from a
new resource loader backed by `RegionsPresenter`, which now also exposes
`masterQueue` alongside the existing region metadata.
```ts
// public API
const runs = await runs.list({ region: ["us-east-1", "eu-west-1"] });
// each item: { id, status, ..., region?: "us-east-1" }
```
```ts
// MCP
list_runs({ environment: "prod", region: "us-east-1" })
```
All MOLLIFIER_* env vars renamed to TRIGGER_MOLLIFIER_*. The mollifier
primitive is generic — buffer + drainer + trip evaluator with no
trigger-specific assumptions at the redis-worker layer — but this
PR's webapp wiring is specifically the trigger-task mollifier, with
PII-sensitive payload handling and trigger-flow semantics. If we later
mollify another surface (deploys, schedules, etc.) those will want
their own env-var namespace; pre-prefixing now avoids a breaking
rename later.
Renames are mechanical: schema keys in env.server.ts, env.* references
across the v3/mollifier* modules, and a handful of doc-comment
mentions. The bootstrap fallback that has DRAINER_ENABLED default to
the ENABLED value is updated to read TRIGGER_MOLLIFIER_ENABLED from
process.env too. Code-side naming (classes, file names, the literal
word "mollifier") stays unchanged — the rename is env-var only.
The drainer's polling loop has been gated on WORKER_ENABLED, which
couples it to the legacy ZodWorker role. To split the drainer onto a
dedicated worker service in cloud (and keep all other replicas as
producer-only), introduce its own switch.
Semantics:
- Unset → inherits MOLLIFIER_ENABLED.
Single-container self-hosters with MOLLIFIER_ENABLED=1 get the
drainer for free, no second flag to remember.
- Explicit MOLLIFIER_DRAINER_ENABLED=0 → drainer off on this replica.
Cloud sets this everywhere except the dedicated drainer service.
- Explicit MOLLIFIER_DRAINER_ENABLED=1 → drainer on, subject to
MOLLIFIER_ENABLED still being the master kill switch (a drainer
can't construct without the gate-side buffer singleton).
The bootstrap in mollifierDrainerWorker.server.ts now gates on the new
flag instead of WORKER_ENABLED, so the drainer's lifecycle is no longer
coupled to the legacy worker role.
worker.server.ts is the original graphile-worker / ZodWorker file —
every task in its catalog is annotated "@deprecated, moved to
commonWorker.server.ts" (or similar). Adding new lifecycle wiring
there during phase-2 was a mis-routing.
Move the SIGTERM/SIGINT registration + drainer.start() call into a new
mollifierDrainerWorker.server.ts alongside the redis-worker workers,
and invoke its initMollifierDrainerWorker() from entry.server.tsx
right after Worker.init(). The drainer's own factory still validates
shutdown timeouts before constructing; the bootstrap registers signal
handlers BEFORE calling start(), preserving the create+start contract.
Also adds a header to worker.server.ts marking it legacy and pointing
new lifecycle code at the redis-worker pattern, so the next person
doesn't have to re-derive the routing rule.
Dan Sutton