Adds the smallest DI surface to `initMollifierDrainerWorker` (`isEnabled`
and `getDrainer`, both optional, default to live env/singleton) so the
catch-block policy can be tested without manipulating module-level env:
- rethrows MollifierConfigurationError — deterministic misconfig
escapes, which is what makes the production-path crash on boot
(the call site in entry.server.tsx runs sync at module top level,
before `process.on("uncaughtException", ...)` is registered, so an
escape becomes a Node default-handler exit-1).
- rethrows when `name === "MollifierConfigurationError"` even when
`instanceof` fails — covers the Remix dev hot-reload realm edge
case where the catch holds a stale class reference.
- swallows non-configuration errors — a transient Redis blip during
buffer init shouldn't take the whole webapp down.
- no-op when disabled — the factory isn't invoked when the enabled
predicate returns false.
Also updates the existing mollifier server-changes note to: rename env
vars to TRIGGER_MOLLIFIER_* prefix, document the TRIGGER_MOLLIFIER_DRAINER_ENABLED
split for multi-replica drainer placement, and call out the new fail-loud
behaviour on drainer misconfiguration.
The bootstrap in mollifierDrainerWorker.server.ts wrapped getMollifierDrainer()
in a try/catch that logged-and-continued on any error, which absorbed the two
designed-to-crash throws in initializeMollifierDrainer():
- "MollifierDrainer initialised without a buffer" (missing buffer client)
- "TRIGGER_MOLLIFIER_DRAIN_SHUTDOWN_TIMEOUT_MS must be at least ... below
GRACEFUL_SHUTDOWN_TIMEOUT" (shutdown-timeout reconciliation)
Both are deploy-time mistakes: silently disabling the drainer means the
gate keeps writing to the buffer, the drainer never reads, and entries
TTL out in 10min. Bounded in phase 1 (monitoring-only) but customer-
visible data loss in phase 2/3 where the drainer replays into engine.trigger.
Better to fail loud now than retrofit the contract later.
Introduce MollifierConfigurationError for the two deterministic throws.
The bootstrap's catch now rethrows that class (process crashes at module
top-level → orchestrator health check fails → deploy rolls back) while
still logging-and-continuing on transient errors (Redis blip during init
shouldn't take the whole webapp down). instanceof + name fallback covers
the Remix dev hot-reload realm edge case.
The previous commit added a perf short-circuit at the call site that
read `env.TRIGGER_MOLLIFIER_ENABLED` directly. That broke three
mollifier integration tests in CI: the tests inject a custom
`evaluateGate` via the existing DI seam expecting the buffer-write
branch to be reached, but CI has no `.env` (the `apps/webapp/.env`
symlink target is absent), the Zod default `"0"` wins, the call site
short-circuits to `null` before the injected gate runs, and
`buffer.accepted` stays empty.
Make the global-enabled check itself injectable:
- New constructor opt `isMollifierGloballyEnabled?: () => boolean`,
defaulting to `() => env.TRIGGER_MOLLIFIER_ENABLED === "1"`. Each
DI hook now represents one decision (gate, buffer, global-enabled),
so a test that wants the buffer-write branch reached can inject
`isMollifierGloballyEnabled: () => true` alongside its custom gate.
- Call site now reads `this.isMollifierGloballyEnabled()` instead of
`env.TRIGGER_MOLLIFIER_ENABLED` directly. In production, with no DI
override, the default closure resolves `env` exactly once per call
just as before — same perf win when the flag is off.
- All six mollifier DI injection sites in triggerTask.test.ts now also
pass `isMollifierGloballyEnabled: () => true` so the tests' DI
surface matches the new contract regardless of CI env state.
## Summary
Local self-hosted deploys (`trigger deploy --local-build --push
--builder orbstack` or any other buildx setup using the **docker**
driver) fail at the push step with:
```
ERROR: failed to build: failed to solve:
exporter option "rewrite-timestamp" conflicts with "unpack"
```
The docker driver auto-enables `unpack=true` when pushing, and that's
incompatible with `rewrite-timestamp` (which the CLI sets for
reproducible-build hashing).
Adds a simple env-var opt-out so contributors can keep using their
default builder. The flag is only read by the local-build code path;
remote/cloud builds are unaffected.
```bash
TRIGGER_BUILD_SKIP_REWRITE_TIMESTAMP=1 \
pnpm exec trigger deploy --profile default --local-build --push --builder orbstack
```
The trade-off: skipping `rewrite-timestamp` means layer timestamps
reflect actual build time, so two identical builds produce different
layer hashes. Fine for a local-dev registry; the only real consumer of
timestamp-stability is registry-layer cache hit rates.
## Test plan
- [x] Manual: ran `trigger deploy --profile default --local-build --push
--builder orbstack` against the localhost webapp + a local Docker
registry on port 5001 — first failed with the rewrite-timestamp/unpack
error, then succeeded after setting
`TRIGGER_BUILD_SKIP_REWRITE_TIMESTAMP=1`.
- [x] Full chat.agent smoke sweep (15 tests, including suspend/resume,
deepResearch subtask, AgentChat orchestrator) against the deployed image
— all pass.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
`TriggerChatTransport` had a single `baseURL` option covering both the
`.in/append` POSTs and the long-lived `.out` SSE subscription. Customers
wanting to route the SSE through a proxy (e.g. a Cloudflare worker
capturing JA4 fingerprints for bot detection) had to send every append
through the proxy too, adding a hop to every user message.
New optional `streamBaseURL` overrides the SSE base URL only; appends
keep using `baseURL`. Falls back to `baseURL` when unset, so existing
transports are unchanged.
```ts
const transport = new TriggerChatTransport({
task: "ai-chat",
baseURL: "https://api.trigger.dev",
streamBaseURL: "https://chat-proxy.example.com",
accessToken,
startSession,
});
```
Verified with a new test in `chat.test.ts` that asserts `.in/append`
routes through `baseURL` and `.out` SSE routes through `streamBaseURL`.
All existing tests still pass.
`dorny/paths-filter` defaults to OR semantics across the pattern array,
so the leading `**` matched every file and the `!...` excludes were
no-ops. The `code` filter has been returning `true` for every PR since
#3615.
Split into two filter steps: `code` moves into its own step with
`predicate-quantifier: every` so excludes actually subtract. The two
re-include workflow files become a separate `typecheck_self` filter that
the `typecheck` job ORs into its `if:`.
Side effect: workflow-file-only PRs that don't touch `pr_checks.yml` or
`typecheck.yml` no longer trigger typecheck. Previously they did because
the filter was broken-true.
## Summary
Codify two rules for ClickHouse migration authors that came out of the
029/030 ordering incident on the TRI-9367 test cloud deploy:
1. **Number files to `max(existing) + 1`, never slot in below the
latest.** Goose runs in strict mode in the cloud deploy pipeline and
refuses to apply a missing version below the current version — slotting
a file in below an already-applied number blocks the next deploy.
2. **DDL must be idempotent** (`ADD COLUMN IF NOT EXISTS`, `DROP COLUMN
IF EXISTS`, `CREATE TABLE IF NOT EXISTS`, etc.) so a retry or
out-of-order apply (`goose up --allow-missing` for local recovery,
manual fixups) is a no-op rather than an error.
## Where the rules live
- `internal-packages/clickhouse/CLAUDE.md` — full rules + example for
migration authors (and AI agents writing migrations).
- `.claude/REVIEW.md` — added a 🔴 finding under "What makes a 🔴
Important finding" so PR reviewers flag either fault as blocking.
The existing migration files are left untouched; the idempotency
requirement applies going forward.
## Test plan
- [ ] Next ClickHouse migration PR uses `IF NOT EXISTS` / `IF EXISTS`
forms
- [ ] No new migration files numbered below an already-applied version
on test/prod
## Summary
Renumber `029_add_task_kind_to_task_runs_v2.sql` →
`031_add_task_kind_to_task_runs_v2.sql` to fix a deploy-blocking
out-of-order migration, and make the DDL idempotent with `ADD COLUMN IF
NOT EXISTS` / `DROP COLUMN IF EXISTS`.
## Root cause
- Migration `030_create_sessions_v1.sql` landed on main on 2026-04-28
(PR #3417) and was applied to test cloud ClickHouse on a subsequent
deploy. Current goose version on test ClickHouse: **30**.
- Migration `029_add_task_kind_to_task_runs_v2.sql` was authored later
on 2026-05-10 as part of the Sessions primitive PR series (`be1a6cf8`).
- The next test cloud deploy failed because goose strict-mode refused to
apply a missing version *before* the current version:
```
goose run: error: found 1 missing migrations before current version 30:
version 29: 029_add_task_kind_to_task_runs_v2.sql
```
## Fix
1. **Rename to `031_*`** (next available number after 030). Goose now
treats it as a new migration after 030 and applies it cleanly on
test/prod where the column does not yet exist.
2. **Make the DDL idempotent** (`ADD COLUMN IF NOT EXISTS`). The
original 029 may have been applied in environments that ran goose with
`--allow-missing` (e.g. some local dev databases) — those would have the
column already, and the rename causes goose to see 031 as new and
re-attempt the ADD. Idempotent DDL keeps that path safe. The `Down`
mirrors with `DROP COLUMN IF EXISTS`.
## Test plan
- [ ] Test cloud deploy (after this lands) successfully runs the
ClickHouse migration step
- [ ] `task_kind` column shows up on `trigger_dev.task_runs_v2`
post-migration
- [ ] Local environments that had previously applied 029 do not error on
the next `goose up`
evaluateGate ran on every trigger regardless of TRIGGER_MOLLIFIER_ENABLED.
With the flag off (the default everywhere it hasn't been opted in), the
gate still produced a `pass_through` decision after allocating a
GateInputs object, spreading defaultGateDependencies inside evaluateGate,
and incrementing the `mollifier.decisions{outcome=pass_through}` OTel
counter. Cheap individually, but triggerTask is the hottest code path in
the system — multiply by trigger rate and the unnecessary work compounds.
Guard the gate call with a direct env.TRIGGER_MOLLIFIER_ENABLED check at
the call site. When the flag is off, mollifierOutcome is null and the
downstream `mollifierOutcome?.action === "mollify"` branch skips the
buffer dual-write entirely — zero allocation, zero counter increment on
the disabled path. When the flag is on, behaviour is unchanged.
Lost-signal note: with mollifier off, we no longer count "pass_through"
decisions in the OTel counter (the gate never runs). That's a non-issue
— "pass_through count when feature is off" is just total trigger rate,
which is already observable via the trigger handler's own spans/counters
upstream. The gate counter remains the source of truth for the
mollify/shadow/pass_through ratio when the feature is on, which is the
load-bearing signal.
This addition was applied while phase-2 was already in review and is
out of scope for the mollifier PR. The underlying clarification is
worth landing — just not on this branch.
`process.once("SIGTERM", stopDrainer)` was the odd one out — every
other webapp service (runsReplicationInstance, llmPricingRegistry,
dynamicFlushScheduler, marqs, eventLoopMonitor) registers through
`signalsEmitter` from `~/services/signals.server`, an EventEmitter
backed by a single `process.on()` that fans out to all listeners.
Switching gets us:
- codebase consistency;
- `.on` (not `.once`) so a second SIGTERM, if the orchestrator emits
one before SIGKILL, still reaches us;
- if SIGTERM lands in the narrow gap between the listener attaching
and drainer.start() below, the first invocation no-ops (stop()
returns early because isRunning is false) but the listener stays
attached for any subsequent signal, instead of being consumed and
leaving the now-running drainer with no graceful-stop path.
The Promise.race between this.loopPromise and this.delay(timeoutMs)
discarded the timeout's underlying setTimeout handle whenever the loop
branch won. The discarded timer was still ref'd by libuv and pinned the
Node event loop alive for the remainder of `timeoutMs` — exactly the
shutdown slack the timeout was supposed to bound.
Inline the timer in stop() with a captured handle and clearTimeout() it
in a finally block, so every exit path (loop-won, timeout-won, throw)
releases the ref. The in-loop delay() calls are unchanged — they're
awaited normally and their timers fire-and-clear themselves.
## Summary
Adds `.changeset/pre.json` to put the repo into changesets pre mode with
dist-tag `rc`. After this merges, the changesets bot regenerates the
existing release PR as `chore: release v4.5.0-rc.0`. Merging that PR
publishes the first release candidate of 4.5.0 to npm under `@rc`.
The pre-mode plumbing landed in #3628. The release content (chat.agent +
sessions + ai prompts + dashboard server-changes) landed in #3629.
## What ships when the bot PR merges
Under dist-tag `rc`:
-
`@trigger.dev/{sdk,core,build,react-hooks,redis-worker,plugins,python,rsc,schema-to-json}@4.5.0-rc.0`
- `trigger.dev@4.5.0-rc.0`
Plus:
- Docker image `ghcr.io/triggerdotdev/trigger.dev:v4.5.0-rc.0`
(immutable tag only — `:v4-beta` is not touched)
- Helm chart `oci://ghcr.io/triggerdotdev/charts/trigger.dev:4.5.0-rc.0`
- GitHub release `v4.5.0-rc.0` marked as Pre-release (no Latest badge)
What does NOT happen:
- npm `latest` stays at 4.4.6
- No marketing-site changelog PR (gated on `is_prerelease != 'true'`)
- Docker `:latest` not touched (we never push it anyway in this repo)
## Iteration
For subsequent rc.N: add a regular changeset to main, bot regenerates
the release PR as `v4.5.0-rc.N`. Merge to ship.
## Exiting pre mode
When ready to ship stable: `pnpm exec changeset pre exit`, push, merge
regenerated PR. That publishes `4.5.0` under `latest` and fires the
marketing-site dispatch.
## Summary
Refocuses the v4.5.0 changeset and server-changes content on the
public-facing AI features story, replacing the pre-release-internal diff
framing that had accumulated in `.changeset/` and `.server-changes/`.
Pairs with the RC support PR — the next bot regeneration will pick up
this content.
## What's in here
### Changeset rewrites
- **`chat-agent.md` rewritten as the headline AI Agents entry** —
written from the `docs/ai-chat/` surface (not from internal pre-release
diffs). Covers useChat integration, multi-turn durability via Sessions,
lifecycle hooks, stop generation, tool approvals (HITL), pending
messages + background injection, actions, typed state primitives,
`chat.toStreamTextOptions()`, multi-tab coordination, network
resilience, and the first-turn fast path (`chat.headStart`).
- **New `ai-prompts.md`** — announces the Prompts feature publicly for
the first time. Code-defined templates, deploy-versioning, dashboard
overrides, AI SDK telemetry integration, `chat.agent` integration via
`chat.prompt.set()` + `chat.toStreamTextOptions()`, full management SDK.
- **`sessions-primitive.md` expanded** — calls out
`tasks.triggerAndSubscribe()` and `sessions.list` as standalone
primitives (not just chat.agent infrastructure).
- **`chat-agent-on-boot-hook.md` trimmed** — drops "if you previously…"
pre-release migration framing.
- **Deletes 4 changesets** that described pre-release-internal
migrations or were circular ("groundwork for the upcoming chat.agent" —
chat.agent ships in the same release).
### Server-changes rewrites (`.server-changes/`)
Five new entries for the dashboard surface of the AI feature set:
- Agents list page
- Agent Playground
- Sessions dashboard
- Prompts dashboard (list with usage sparklines + detail with template /
Generations / Metrics / Versions tabs + override UI)
- Models registry (provider-grouped catalog with cross-tenant usage
metrics)
- AI generation span inspector on run traces
- Runs list Task source filter (Standard / Scheduled / Agent)
- Run-detail Agent view (segmented control)
Each entry is 1–2 sentences, no bullets, no implementation file paths —
fits as a single bullet in a future changelog.
Three older `.server-changes/` files were merged or split into the
cleaner taxonomy above and deleted.
## Out of scope
Non-AI-feature server-changes (admin-tabs, queue-length-cap fix,
worker-deployment race, streamdown upgrade, etc.) and changesets
(idempotency-key cap, sigsegv retry, locals-key fix, plugin auth, region
filters, etc.) are untouched.
## Summary
Enables shipping `X.Y.Z-rc.N` prereleases of `@trigger.dev/*` via
changesets pre mode. RCs publish under the `rc` npm dist-tag, never
claim `latest`, and don't trigger marketing-site changelog PRs. The
plumbing is hyphen-in-version detection in `release.yml` — no separate
workflow, no opt-in flag at publish time.
Validated end-to-end against a sandbox repo (real npm publishes, Docker
builds, Helm chart pushes, GitHub releases) before porting back. Full RC
lifecycle tested: pre enter → rc.0 → iterate to rc.1 → pre exit →
stable. Plus interaction with the existing release-branch hotfix flow.
## What changes
### `release.yml`
- New `is_prerelease` output (hyphen-in-version)
- GitHub release adds `--prerelease` flag for RC publishes (Pre-release
badge, not Latest)
- `dispatch-changelog` job gated on `is_prerelease != 'true'` — no
marketing-site PR per RC
### Docker workflows
- Removes the `:v4-beta` floating tag entirely from `publish-webapp.yml`
and `publish-worker-v4.yml`. v4 is GA; the tag is a misnomer and is
already inconsistent with the npm side (npm `v4-beta` dist-tag was
frozen at 4.0.4 months ago while Docker `:v4-beta` kept bumping).
Self-hosters should pin to a versioned tag going forward — the last
value of `:v4-beta` stays frozen wherever it currently points.
### CLI version-check fix
(`packages/cli-v3/src/utilities/initialBanner.ts`)
Switches the "new version available" comparison from JavaScript
`localeCompare` to `semver.lt`. The old comparison handled `X.Y.Z-rc.N`
vs `X.Y.Z` incorrectly — a user on `4.5.0-rc.0` would never be prompted
to upgrade once `4.5.0` stable shipped (lex order put the prerelease
ahead of the bare version). Real semver gets this right.
Stable users were never affected: the check queries the `@latest`
dist-tag, which by convention never points at a prerelease.
## How an RC actually publishes after this
1. `pnpm exec changeset pre enter rc` on main, push the `pre.json`
2. Bot regenerates the release PR as `chore: release v<X.Y.Z>-rc.0`
3. Merge → `release.yml` runs `changeset publish` which reads
`pre.json.tag` and publishes under `--tag rc`. GitHub release marked
Pre-release. No marketing-site dispatch.
4. Iterate by adding changesets normally; bot bumps to `rc.1`, `rc.2`, …
5. When ready: `pnpm exec changeset pre exit`, push, merge regenerated
PR → stable ships under `latest` and the marketing-site dispatch fires.
The literal reading of "never mock anything" trips up AI reviewers
(and humans new to the repo) — they flag any `vi.mock` / `vi.fn` /
`vi.spyOn` they see, even when the usage isn't actually faking
behavior. Three patterns are fine and should NOT be flagged:
1. Module-load workarounds — vi.mock("~/db.server") at the top of a
unit test to stop prisma.$connect() firing at import. Cuts the
import graph, doesn't fake DB behavior.
2. Hand-written DI doubles where the real implementation has its own
dedicated infra-backed tests (CapturingMollifierBuffer, MockPayloadProcessor,
etc.). Unit test covers wiring, integration test covers the seam target.
3. vi.fn as a DI-seam probe — convenience for "was the seam called."
Equivalent to a closure-counter; not load-bearing on what's proven.
Still 🔴: spying on the code path under test then asserting the spy was
called (tautology), or replacing real infra with mocks in tests meant
to cover real behavior (e.g. mocking Redis in a Redis-queue test).
## Summary
Adds a Region column and Region filter (under More filters) to the runs
list dashboard, the same filter on the public runs list API
(`filter[region]`), and a matching `region` input on the MCP `list_runs`
tool. Each run's executing region is also surfaced as a new optional
`region` field on the runs list and run retrieve responses, populated
from the worker instance group's `masterQueue` identifier.
Useful when you run tasks across multiple regions and want to slice the
runs list — or your existing run-querying scripts — by where the run
actually executed.
## Design
The filter value in the URL / API is the `masterQueue` identifier (the
same string already persisted on `TaskRun` and replicated to ClickHouse
as `worker_queue`), so the query just becomes `worker_queue IN (...)`
with no server-side translation. The Region dropdown options come from a
new resource loader backed by `RegionsPresenter`, which now also exposes
`masterQueue` alongside the existing region metadata.
```ts
// public API
const runs = await runs.list({ region: ["us-east-1", "eu-west-1"] });
// each item: { id, status, ..., region?: "us-east-1" }
```
```ts
// MCP
list_runs({ environment: "prod", region: "us-east-1" })
```
All MOLLIFIER_* env vars renamed to TRIGGER_MOLLIFIER_*. The mollifier
primitive is generic — buffer + drainer + trip evaluator with no
trigger-specific assumptions at the redis-worker layer — but this
PR's webapp wiring is specifically the trigger-task mollifier, with
PII-sensitive payload handling and trigger-flow semantics. If we later
mollify another surface (deploys, schedules, etc.) those will want
their own env-var namespace; pre-prefixing now avoids a breaking
rename later.
Renames are mechanical: schema keys in env.server.ts, env.* references
across the v3/mollifier* modules, and a handful of doc-comment
mentions. The bootstrap fallback that has DRAINER_ENABLED default to
the ENABLED value is updated to read TRIGGER_MOLLIFIER_ENABLED from
process.env too. Code-side naming (classes, file names, the literal
word "mollifier") stays unchanged — the rename is env-var only.
The drainer's polling loop has been gated on WORKER_ENABLED, which
couples it to the legacy ZodWorker role. To split the drainer onto a
dedicated worker service in cloud (and keep all other replicas as
producer-only), introduce its own switch.
Semantics:
- Unset → inherits MOLLIFIER_ENABLED.
Single-container self-hosters with MOLLIFIER_ENABLED=1 get the
drainer for free, no second flag to remember.
- Explicit MOLLIFIER_DRAINER_ENABLED=0 → drainer off on this replica.
Cloud sets this everywhere except the dedicated drainer service.
- Explicit MOLLIFIER_DRAINER_ENABLED=1 → drainer on, subject to
MOLLIFIER_ENABLED still being the master kill switch (a drainer
can't construct without the gate-side buffer singleton).
The bootstrap in mollifierDrainerWorker.server.ts now gates on the new
flag instead of WORKER_ENABLED, so the drainer's lifecycle is no longer
coupled to the legacy worker role.
worker.server.ts is the original graphile-worker / ZodWorker file —
every task in its catalog is annotated "@deprecated, moved to
commonWorker.server.ts" (or similar). Adding new lifecycle wiring
there during phase-2 was a mis-routing.
Move the SIGTERM/SIGINT registration + drainer.start() call into a new
mollifierDrainerWorker.server.ts alongside the redis-worker workers,
and invoke its initMollifierDrainerWorker() from entry.server.tsx
right after Worker.init(). The drainer's own factory still validates
shutdown timeouts before constructing; the bootstrap registers signal
handlers BEFORE calling start(), preserving the create+start contract.
Also adds a header to worker.server.ts marking it legacy and pointing
new lifecycle code at the redis-worker pattern, so the next person
doesn't have to re-derive the routing rule.
initializeMollifierDrainer() no longer calls drainer.start() — it
returns a configured-but-stopped drainer. worker.server.ts init() now
invokes drainer.start() AFTER the SIGTERM/SIGINT handlers are
registered, gated on the same __mollifierShutdownRegistered__ guard so
dev hot-reloads can't double-start.
Closes the residual race window between drainer.start() (previously
fired inside the singleton factory) and process.once("SIGTERM",
stopDrainer) in worker.server.ts. With construction and starting
separated, a signal landing during boot can never find the polling
loop running without a graceful-stop path.
Move the MOLLIFIER_DRAIN_SHUTDOWN_TIMEOUT_MS / GRACEFUL_SHUTDOWN_TIMEOUT
reconciliation check from worker.server.ts init() into
initializeMollifierDrainer() — BEFORE drainer.start() — so a
misconfigured deploy fails loud at module-load time instead of starting
the polling loop and then throwing back at the caller before the SIGTERM
handler can be registered.
The singleton() helper uses ??=, so a throw inside the factory leaves
the cache slot unset and the next getMollifierDrainer() call re-runs the
factory. No half-started state, no missing SIGTERM handler. The catch in
worker.server.ts init() still logs and aborts drainer registration on
either the validation error or a Redis init failure — same observable
behaviour from the caller's perspective.
The drainer was started inside the singleton factory, with the
shutdown-timeout-vs-GRACEFUL_SHUTDOWN_TIMEOUT reconciliation living in
worker.server.ts init() afterwards. If that validation threw, the polling
loop was already running and the SIGTERM handler registration below it
was never reached — the loop kept polling with no graceful-shutdown
path, and the singleton was cached in its running state (so subsequent
init() calls returned the same drainer and validation kept failing).
Move the timeout check into initializeMollifierDrainer() before
drainer.start(). singleton() uses ??=, so a throw inside the factory
leaves the cache slot unset and the next getMollifierDrainer() call
re-runs the factory — no half-started state, no missing SIGTERM
handler. The catch in worker.server.ts init() still logs and aborts
drainer registration on either the validation error or a Redis init
failure.
Replace each vi.fn(async handler) with a plain async closure that
records calls via captured counter/array variables. Assertions move
from handler.mock.* / toHaveBeenCalled* matchers to checks against
the captured state, e.g. handlerCalls.length / handlerCalls[0].
Functionally equivalent; aligns with the package convention of using
real testcontainers + closure-based probes (cf. mollifierGate.test.ts
and mollifierTripEvaluator.test.ts) rather than vitest fakes.
## Summary
The trigger-task hotpath used to early-return without a DB query when a
caller passed both a queue override and a per-trigger TTL — the hottest
configuration on the trigger API. Adding `triggerSource` to the resolver
so the runs-list "Source" filter could distinguish STANDARD / SCHEDULED
/
AGENT runs removed those early-returns, costing +2 DB queries per
trigger
on non-locked calls and +1 on locked calls.
This change caches `BackgroundWorkerTask` metadata (`ttl`,
`triggerSource`,
`queueId`, `queueName`) in Redis so the resolver can satisfy every
caller
configuration with a single `HGET` on the warm path. PG fallback on miss
back-fills the cache.
Follow-up to #3542.
## Design
Two key spaces:
- `task-meta:env:{envId}` — the "current worker" view, refreshed at
every
deploy promotion. 24h safety TTL.
- `task-meta:by-worker:{workerId}` — used for `lockToVersion` triggers.
Immutable post-create. 30d sliding TTL so historical workers age out.
Cache writes use Lua scripts via `defineCommand` so `DEL` + `HSET` +
`EXPIRE` land atomically — concurrent readers never see the empty
intermediate state of a naive pipeline. Read-path back-fill uses
single-field upserts so concurrent back-fills don't wipe each other's
siblings.
The cache lives behind its own `TASK_META_CACHE_REDIS_*` env-var prefix
that falls back to the default `REDIS_*` set, so operators can route the
cache to a dedicated Redis instance if they want.
The service/instance file split (`taskMetadataCache.server.ts` for the
pure class, `taskMetadataCacheInstance.server.ts` for the env-wired
singleton) mirrors the existing `runsReplicationService` /
`runsReplicationInstance` pattern.
## Test plan
- [ ] `pnpm run typecheck --filter webapp`
- [ ] `pnpm run test ./test/engine/triggerTask.test.ts --run` — 8
existing tests untouched + 5 new tests covering warm cache, cold
miss with back-fill, queue + ttl path, by-worker vs env keyspace,
and the promotion cache write
- [ ] End-to-end against a dev worker: registering writes both keyspaces
with the expected TTLs, and `redis-cli HGETALL
"tr:task-meta:env:<envId>"`
returns the cached entries
## Benchmark
Measured `DefaultQueueManager.resolveQueueProperties` against a real
Postgres + Redis (vitest `containerTest`, single-host docker). 500
sequential calls and 2,000 parallel calls (concurrency=50) per scenario,
request shaped as `{ taskId, queue: "bench-queue", ttl: "5m" }` — the
hot path this PR restores.
```
sequential (one in flight at a time):
[noop cache (baseline)] n=500 mean=1.423ms p50=1.394ms p95=1.735ms p99=2.629ms max=11.100ms
[redis cache, cold ] n=500 mean=1.346ms p50=1.283ms p95=1.688ms p99=2.463ms max=5.058ms
[redis cache, warm ] n=500 mean=0.084ms p50=0.078ms p95=0.105ms p99=0.156ms max=1.129ms
speedup (warm vs baseline, sequential): 16.95x
parallel (concurrency=50):
[noop cache (baseline)] n=2000 mean=10.069ms p50=8.850ms p95=14.718ms p99=31.887ms total=405ms ops/s=4,940
[redis cache, warm ] n=2000 mean=0.614ms p50=0.568ms p95=1.189ms p99=1.432ms total=25ms ops/s=80,389
throughput speedup (warm vs baseline, parallel): 16.27x
```
Read:
- **Warm cache cuts resolver latency 17×** at p50 — from ~1.4 ms to ~78
µs per call.
- **Cold cache is on par with baseline** — the extra `HGET` miss adds
<50 µs against the two Postgres queries that follow, so the worst case
is not worse than today.
- **Under burst load (50 concurrent triggers)**, the baseline's p99
jumps to ~32 ms as Postgres connections queue up; warm stays at ~1.4 ms.
The cache moves the saturation point from ~5k ops/s (PG pool) to ~80k
ops/s (single-client Redis pipelining).
Caveats: single-host docker, local Postgres + Redis, resolver-only
measurement (excludes the rest of the trigger transaction). Prod adds
region-local Redis RTT (~0.3–0.8 ms) which shifts warm absolute numbers
up but keeps the ratio intact.
The previous wording still described the in-memory env→org cache and the
"uncached envs treated as their own pseudo-org for one tick" sentinel —
both removed when the buffer started tracking `mollifier:orgs` and
`mollifier:org-envs:${orgId}` atomically. Re-describe the drainer in
terms of the current org-walk so the published changelog matches the
shipped code.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two correctness/perf fixes on top of the phase-2 drainer:
1. `runOnce` was awaiting `listEnvsForOrg` serially before any pop ran.
At the default `maxOrgsPerTick=500` and a ~1ms RTT, that's a ~500ms
per-tick latency floor before `pLimit` even sees work. `Promise.all`
over the org slice lets ioredis auto-pipeline the SMEMBERS into a
single round-trip. Order is preserved so the org→envs pairing stays
deterministic and `pickEnvForOrg` still rotates per org.
2. The SIGTERM handler is sync fire-and-forget: `drainer.stop({timeoutMs})`
returns a promise that keeps the loop alive, but in cluster mode the
primary process runs its own `GRACEFUL_SHUTDOWN_TIMEOUT` and will hit
`process.exit(0)` independently. If the drainer's deadline exceeds
the primary's, the drainer's "log a warning on timeout" turns into
"hard exit with no log". Assert at boot that
`MOLLIFIER_DRAIN_SHUTDOWN_TIMEOUT_MS <= GRACEFUL_SHUTDOWN_TIMEOUT - 1s`
so a misconfig fails loud instead of disappearing at shutdown.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`triggerTask` is the highest-throughput code path in the system and the
webapp CLAUDE.md forbids new DB queries there. The previous resolver fell
back to `flag()` (a Prisma read against `FeatureFlag`) when the org had
no `mollifierEnabled` override, which added a query to every trigger
whenever `MOLLIFIER_ENABLED=1`. The fleet-wide kill switch already lives
in `MOLLIFIER_ENABLED`; rollout is per-org via `Organization.featureFlags`
JSON, matching `canAccessAi`/`hasComputeAccess`/etc. Drop the fallback so
the resolver is purely in-memory.
Tests no longer need a postgres testcontainer or `makeFlag(prisma)`; the
per-org isolation suite now asserts directly on `Organization.featureFlags`
shape and adds a regression test for the no-override -> false contract.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary
A "Google auth conflict" Sentry alert fires whenever a user signs in via
Google whose Google account is linked to one user row but whose
Google-provided email is now on a *different* user row. The handler in
`apps/webapp/app/models/user.server.ts:236` already does the right thing
— it returns the existing auth-linked user and skips the update path so
neither row gets mutated — but it logs the situation with
`logger.error`, which routes to Sentry as an exception and pages the
on-call channel.
There's no exception to chase here: the branch is the intended outcome
for a known data shape (user changed their email on one account after
originally signing up via Google on another). Downgrading the call to
`logger.warn` keeps the diagnostic record in our logs (with all the same
context fields — email, both user IDs, authIdentifier) but stops it
firing the production error alert.
## Change
- `logger.error` → `logger.warn` for the conflict branch in
`findOrCreateGoogleUser`. Context payload is unchanged.
## Test plan
- [x] Typecheck only — there's no behavioural change to test, the log
level is the entire diff.
With the drainer walking listOrgs → listEnvsForOrg → pop, the flat
mollifier:envs SET has no consumer — `mollifier:orgs` and the per-org
`mollifier:org-envs:${orgId}` SETs cover everything the drainer needs.
Removing it drops three Lua write ops per accept/pop/requeue and one
Redis key per active env.
Changes:
- Lua: acceptMollifierEntry, popAndMarkDraining, requeueMollifierEntry
no longer touch mollifier:envs. Their KEYS arrays shrink by one.
- TS: listEnvs() method removed; only listOrgs() and listEnvsForOrg()
remain. TS bindings updated to match the new arg shapes.
- buffer.test.ts: listEnvs() assertions converted to listEnvsForOrg(
"org_1") so they verify the equivalent org-level membership. The
"stale envs SET cleanup on empty-pop" test is removed (envs SET is
gone). The "pop skips orphans" test's trailing-cleanup assertion is
updated to document the deliberate stale-tolerance in the no-runId
branch of popAndMarkDraining (can't read orgId without a popped
entry, so org-envs cleanup is skipped there).
- drainer.test.ts: stub helper moved to module scope and gains an
`eachEnvAsOwnOrg(envs)` convenience that supplies listOrgs +
listEnvsForOrg in tests where each env is its own org. Stub helpers
duplicated across describe blocks are removed in favour of the
shared one.
24/24 drainer tests pass; buffer tests pass in isolation (a few timeout
under full-suite contention against the shared redis container —
unrelated to this change).
Two prior changes are reverted:
1. MOLLIFIER_REDIS_HOST (plus _PORT/_USERNAME/_PASSWORD/_TLS_DISABLED)
regain their `.transform((v) => v ?? process.env.REDIS_*)` fallback
to the main Redis cluster, matching the convention used elsewhere in
the codebase for dedicated-cluster env vars. Operators who don't set
a dedicated mollifier Redis fall back to the main one — that's the
accepted default.
2. getMollifierBuffer() no longer degrades to disabled with a warn log
when MOLLIFIER_ENABLED=1 but MOLLIFIER_REDIS_HOST is unset. The
buffer initialises normally (falling back to the main Redis if
configured), and if that fails the pod crashes loudly. Same for the
drainer: initializeMollifierDrainer() throws "env vars inconsistent"
if the buffer comes back null, surfacing the misconfig immediately
rather than silently leaving entries un-drained.
Operationally: silent degradation hides config errors from operators
and produces "why are no triggers being mollified?" debugging sessions.
Loud failure surfaces the same misconfig at deploy time via the pod's
health checks.
Previously the drainer cached envId→orgId from popped entries and used a
sentinel pseudo-org for envs it hadn't seen yet. The sentinel polluted
the bucket map with fake org IDs and was a foreseeable source of bugs.
This commit moves org membership into the buffer's atomic Lua scripts.
New Redis keys, both maintained transactionally alongside per-env queues:
- mollifier:orgs — orgs with at least one queued env
- mollifier:org-envs:${orgId} — envs of that org with queued entries
acceptMollifierEntry SADDs into all three sets (envs + orgs + org-envs).
popAndMarkDraining cleans up envs+orgs+org-envs together when the queue
empties in the success branch (we know orgId from the popped entry). The
no-runId branch can't read orgId so it only cleans envs — stale org-envs
entries are bounded by env count and recovered on the next accept.
requeueMollifierEntry re-SADDs all three since the env may have just been
pruned.
The drainer now walks listOrgs() → listEnvsForOrg(org) → pop(env) with
two cursors: orgCursor across all active orgs and a per-org envCursor
for round-robin within each org. No client-side cache, no sentinel,
deterministic from the first tick.
Tests updated:
- multi-org-round-robin (was multi-env-round-robin): two orgs with one
and two envs respectively, asserts org_B drains its only env each
tick while org_A rotates through its two.
- concurrency-cap test spreads 12 envs across 12 orgs (otherwise one
org → one pop per tick).
- "heavy org doesn't dominate vs light org" gets explicit listOrgs /
listEnvsForOrg from the test's env→org map; assertion tightened to
0.7–1.5 ratio over 20 ticks.
- "within an org envs rotated round-robin" gets explicit listEnvsForOrg.
- "envCursor resets" → "rotation cursors reset"; cache is gone, only
orgCursor and perOrgEnvCursors reset on start().
- makeStubBuffer auto-derives listOrgs/listEnvsForOrg from listEnvs
(each env as its own org) so tests that don't care about org grouping
don't need to provide them explicitly.
24/24 drainer tests pass, 35/35 buffer tests pass (some redis-container
flakes under full-suite load; all green in isolation). Webapp typecheck
clean.
The changeset accreted across the PR's evolution and ended up reading as
three deltas ("now survives", "is now two-level", "no longer scales").
On merge this is the introduction of the feature — there's no prior
state to contrast against. Rewrite as one cohesive description of what
ships.
Previously the drainer rotated per-env: an org with N busy envs got N
scheduling slots per tick. A noisy tenant with many envs would drain
proportionally faster than a quiet tenant with one env. Switch to
hierarchical rotation: pick orgs round-robin (capped by maxOrgsPerTick),
then pick one env per picked org (also rotating).
Implementation is drainer-side only — no buffer or Lua changes. The
drainer caches envId→orgId from popped entries; envs not yet cached are
treated as their own pseudo-org for one tick, so cold start matches the
old per-env behaviour and converges to hierarchical once cache is hot
(usually within one tick). Cache and cursors reset on start() alongside
the existing cursor reset.
API change: maxEnvsPerTick → maxOrgsPerTick on MollifierDrainerOptions,
MOLLIFIER_DRAIN_MAX_ENVS_PER_TICK → MOLLIFIER_DRAIN_MAX_ORGS_PER_TICK on
the webapp env. Same default (500). Operators tune for "typical orgs
with pending entries" rather than env count.
Trade-off: total per-tick pops drop from O(envs) to O(orgs). For an org
with N envs, each env's individual drainage rate is 1/N of what it was,
but the tenant overall is bounded the same way as a single-env tenant —
which is the fairness contract.
Tests:
- Renamed maxEnvsPerTick references throughout existing tests; old
behaviour still holds at cold cache (each env = pseudo-org).
- New "heavy org with many envs does not dominate vs light org" pins
the post-warm-up ~1:1 drainage ratio between a 6-env org and a 1-env
org over a sustained 20-tick run.
- New "within an org, envs are rotated round-robin across ticks" pins
the inner env cursor's behaviour for a single multi-env org.
- Cursor-reset test renamed and now asserts cache+cursors all reset.
Also removed an outdated test-count comment in
apps/webapp/test/engine/triggerTask.test.ts that listed "four tests"
when reality has moved on.
1. start() resets envCursor to 0 — new behaviour. A stop+start cycle now
begins rotation cleanly from envs[0] rather than inheriting between-
restart cursor drift.
2. Malformed payload → non-retryable handler error path. Pins that the
deserialise failure goes terminal without invoking the handler.
3. Ack failure after handler success — documents the current behavioural
gap. ack() lives inside processEntry's try, so a Redis blip on ack
routes a successfully-handled entry through the retry/terminal path.
Phase 2's engine-replay handler will need idempotency to absorb the
re-execution, OR ack should be lifted out of the try block.
4. start() idempotency — second call is a no-op (no doubled loop).
5. stop() idempotency — safe to call when never started or twice.
6. Loop-level backoff actually grows on consecutive runOnce failures
and resets on first success. Distinct from per-entry retry attempts
already covered elsewhere; this is the consecutiveErrors counter
that drives backoffMs between ticks.
Also adds org-level fairness analogue of the existing env starvation
test: a light org (1 env, 1 entry) is not starved behind a heavy org
with many envs and many entries. The buffer doesn't track orgs as a
separate axis, so org fairness is an emergent property of env rotation
— the test pins that property explicitly.
The fairness test compared popsPerTick[0][0] vs popsPerTick[1][0]
directly. Under the redis-worker package's strict tsconfig
(noUncheckedIndexedAccess implied), array index access returns T |
undefined, which trips TS2532. Destructure into named locals and use
optional chaining — same assertion, no `\!` non-null soup.
## Summary
`LocalsKey<T>` (the type returned by `locals.create()`) was branded with
a
module-level `declare const __local: unique symbol`. Each such
declaration
is its own nominal type, and `tshy` emits separate `.d.ts` files for the
ESM and CJS outputs — each gets its own `__local` symbol. Under certain
pnpm hoisting layouts a single TypeScript compilation can resolve
`LocalsKey` from both the ESM source path and the CJS dist path within
the same call site, producing two structurally-incompatible variants of
the same type. TS surfaces this as the misleading error:
```
Argument of type 'LocalsKey<X>' is not assignable to parameter of type
'LocalsKey<X>'. Property '[__local]' is missing in type 'LocalsKey<X>'
but required in type 'BrandLocal<X>'.
```
The error has been hitting CI on PRs opened since the chat.agent stack
landed (e.g. #3625 typecheck job), but doesn't reproduce on developer
machines where the pnpm node_modules layout was built up incrementally.
## Fix
Replace the `unique symbol` brand with an optional phantom field that
carries `T` at the type level:
```ts
// before
declare const __local: unique symbol;
type BrandLocal<T> = { [__local]: T };
export type LocalsKey<T> = BrandLocal<T> & {
readonly id: string;
readonly __type: unique symbol;
};
// after
export type LocalsKey<T> = {
readonly id: string;
readonly __type: symbol;
/** Phantom carrier for the value type — never read at runtime. */
readonly __valueType?: T;
};
```
The ESM and CJS `.d.ts` outputs now produce structurally identical
types,
so cross-output resolution no longer produces a mismatch. `T` is still
carried at the type level via the optional phantom field. The runtime
shape is unchanged — `manager.ts` was already casting via `as unknown`,
which is no longer needed.
## Test plan
- [ ] `pnpm run typecheck --filter @trigger.dev/core --filter
@trigger.dev/sdk`
- [ ] `pnpm run build --filter @trigger.dev/core --filter
@trigger.dev/sdk`
(clean rebuild) — confirms the ESM and CJS dist `.d.ts` outputs
no longer carry distinct `unique symbol` declarations
- [ ] `pnpm --filter @trigger.dev/core test test/mockTaskContext.test.ts
--run`
- [ ] `pnpm --filter @trigger.dev/sdk test test/mockChatAgent.test.ts
--run`
Adds a regression test that proves a light env (single buffered entry)
is drained within (envs.length - sliceSize + 1) ticks regardless of how
many entries the heavy envs have queued. The test uses a stub buffer
whose listEnvs/pop pair mirrors the production atomic-Lua semantic: an
env disappears from listEnvs the moment its queue empties, so the light
env exits the rotation as soon as it's popped — while the heavy envs
stay in the rotation until their thousands of entries are drained.
Together with the head-of-line fairness test this pins both fairness
properties: (1) every env touches every slice slot per cycle (no
within-slice bias), and (2) no env's drainage latency depends on the
queue depth of other envs (no across-slice starvation).
The previous chunking advanced the cursor by sliceSize each tick,
producing fixed disjoint slices like [0..3], [4..7], [0..3], ... With
that pattern env_0 was always at slice position 0 (first into pLimit)
and env_3 always at position 3 (last) — reinstating the head-of-line
bias rotation was meant to prevent.
Advance the cursor by 1 instead. Slices now overlap across consecutive
ticks (e.g. [0..3], [1..4], [2..5], ...) so every env reaches every
slot position 0..sliceSize-1 across one envs.length-tick cycle.
Drainage rate per env is unchanged: each env still appears in exactly
sliceSize of every envs.length ticks. New regression test pins the
fairness property by asserting each env touches every slot at least
once per cycle.
Follow-up to #3615. The `code` filter currently fires typecheck for any
change outside `docs/`, `.changeset/`, `hosting/`, or `.github/` - so a
docs-only PR like #3623 (touching `references/ai-chat/.env.example` +
`README.md`) triggered the typecheck job. None of the `references/*`
packages declare a `typecheck` script either, so even when a real code
change lands there, `turbo run typecheck` skips them. Running the job is
pure cost.
Tightens the filter to also exclude:
- `references/**` - playground projects, none of them contribute to
`turbo run typecheck` today
- `**/*.md` - markdown anywhere
- `**/.env.example` - example env files anywhere
Two known gaps left open:
- references/ have no real CI typecheck coverage. Separate question -
either add `typecheck` scripts to each (or top-level `tsc -p`), or
accept playground status.
- `changes` job still runs (it's a path-filter step) but the dependent
jobs all skip on irrelevant PRs.
The MollifierDrainer's stop() was polling `isRunning` every 20ms until
the loop exited, which differs from the codebase's convention for
similar polling loops (FairQueue, BatchQueue both hold the loop promise
as a field and await it directly on stop).
Switch to the same pattern: store the loop promise on start(), then in
stop() race it against the timeout via Promise.race. With no timeout we
just await the loop directly. With a timeout the warn-and-return
behaviour is unchanged. No polling, no separate `isRunning` poll loop.
Behaviour is identical to the previous implementation, including the
hung-handler timeout path (covered by the existing
"stop returns after timeoutMs even if a handler is hung" test).
mollifier:envs is a Redis SET that grows with the count of envs that
currently have buffered entries. Under normal operation that's small,
but an extended drainer outage can leave entries piled up across
thousands of envs — at which point runOnce would queue one
processOneFromEnv per env through pLimit, ballooning per-tick latency
and event-loop queue depth.
Cap per-tick fan-out at MOLLIFIER_DRAIN_MAX_ENVS_PER_TICK (default 500).
When the set fits within the cap, behaviour is unchanged (take all,
rotate cursor by 1 for fairness). When the set exceeds the cap, take a
rotating slice and advance the cursor by the slice size so successive
ticks sweep through the full set.
Tests use a stub buffer to drive listEnvs() deterministically with
thousands of envs without provisioning a real Redis.
## Summary
A complete Next.js reference project that exercises `chat.agent`
end-to-end, plus the CLI MCP tools that let Claude Code, Cursor, and
similar IDE agents drive a deployed `chat.agent` task from the editor.
Builds on #3545.
## Design
`references/ai-chat` is a full Next.js app: prisma-backed persistence,
multi-chat sidebar, per-chat model picker, debug panel, tool examples
(`getCurrentTime`, `searchHackerNews`, `createGithubIssue`, PR review
helpers, code sandbox), and smoke tests. It's intended both as a
copy-paste starting point and as a place to regression-test SDK changes.
The CLI gains MCP tools (`start_agent_chat`, `send_agent_message`,
`close_agent_chat`, `list_agents`) so an IDE agent can converse with a
deployed `chat.agent` task. The dev runtime adds one-shot OOM kill on
the run controller and skills bundling in the build pipeline.
Two operational guards for misconfigured rollouts:
1. Drop the MOLLIFIER_REDIS_* fallback to the main REDIS_* cluster.
The mollifier writes to a dedicated Redis to keep burst traffic off
the engine's primary queue — silently colocating with the main Redis
when MOLLIFIER_REDIS_HOST is unset defeats the design.
2. Degrade gracefully instead of crashing the pod. If MOLLIFIER_ENABLED
was flipped on without setting MOLLIFIER_REDIS_HOST, the buffer
returns null (with a one-shot warn log per process) and the drainer
no-ops. No crash loops, no failed deploys, no traffic impact —
operators see the warn line and fix the misconfig in a follow-up
deploy.
The drainer's previously-unreachable "env vars inconsistent" throw
becomes reachable in this degraded mode; replace it with a null return
so worker.server.ts's existing null check short-circuits cleanly.
Top of the chat.agent stack: a full Next.js reference project that
exercises chat.agent end-to-end, plus the CLI MCP tools that drive
agent runs from Claude Code / Cursor / etc.
references/ai-chat:
- Full Next.js app with prisma persistence, multi-chat sidebar,
per-chat model picker, debug panel, tool examples, smoke tests
- Reference tools: getCurrentTime, searchHackerNews, createGithubIssue,
PR review helpers, code sandbox
- chat-client-test orchestrator for concurrent-send stress
- references/hello-world chatAgent + triggerAndSubscribe examples
CLI MCP tooling for chat.agent:
- mcp/tools/agentChat.ts (start_agent_chat, send_agent_message,
close_agent_chat)
- mcp/tools/agents.ts + tasks.ts (list agents, agent run details)
- dev-run-controller OOM kill + taskRunProcessPool tweaks
- dev/managed entry-point hooks for skills bundling
- buildWorker + bundleSkills (agent skills support)
Includes ai-tool-helpers + mcp-agent-chat-sessions changesets, plus
the streamdown@2 patch and pnpm-lock reconciliation.
(Will be renamed to feature/ai-chat-reference-and-cli before push.)
fix(cli): preserve lastEventId after sendMessage fallback to avoid stale turn-complete replay
Dan Sutton