Q1 ZSET-merge design lands.
redis-worker side:
- MollifierBuffer.listForEnvWithWatermark — paginated newest-first
read of buffered entries, bounded by a (createdAtMicros, runId)
watermark. ZREVRANGEBYSCORE strictly below the watermark score plus
a tied-score band scan for entries sharing the watermark's
createdAtMicros.
webapp side:
- listingMerge.server.ts: callRunListWithBufferMerge wraps
ApiRunListPresenter. Fetches a buffer page, synthesises each entry
into the presenter's ListDataItem shape (status QUEUED, timestamps
from entry hash, env slug looked up once), forwards the inner
cursor to the presenter, merges by createdAt DESC with runId DESC
tiebreak, truncates to pageSize. Compound base64-JSON cursor
{ inner, watermark, bufferExhausted } is backwards-compatible with
legacy opaque cursors.
- api.v1.runs.ts + api.v1.projects.{projectRef}.runs.ts route through
the wrapper. Project route extracts envId from filter[env]; absent
that, falls back to the bare presenter (existing behaviour).
- Buffer eligibility skips for filters that can't match buffered runs
(status not in QUEUED/PENDING/DELAYED, batch/schedule/version/
region/machine filters). Buffer outages fall open to PG-only.
- Delete RecentlyQueuedSection banner + listEntriesForEnv loader call
from dashboard runs index — buffered runs appear inline as QUEUED
rows.
Closes the last API-parity gap in the master plan.
redis-worker side:
- New casSetMetadata Lua command with optimistic lock on a
metadataVersion entry-hash field. Returns applied / version_conflict /
not_found / busy. Mirrors the PG-side UpdateMetadataService's CAS
loop so concurrent metadata.increment / metadata.set / metadata.append
calls against a buffered run never lose deltas.
- accept Lua initialises metadataVersion=0; BufferEntrySchema gains
the field.
webapp side:
- applyMetadataMutationToBufferedRun helper does the read-apply-CAS-
retry loop in JS, reusing the existing @trigger.dev/core
applyMetadataOperations function (no Lua re-implementation of the 6
operation types).
- metadata PUT route does PG-first via the existing service (which
owns the full request shape: parent/root ops, batching, validation),
then falls through to the buffer helper on PG miss. busy and
version_exhausted return 503 with retry hint; not_found returns 404.
- Parent/root operations on a buffered target are fanned out to the
snapshot's parentTaskRunId via the existing service. If the parent
is also buffered the helper recurses. Best-effort — parent/root
ingestion failures do not surface to the caller.
Tests: 3 new redis-worker tests covering CAS apply / version conflict /
not_found-busy paths. All 71 redis-worker mollifier + 68 webapp
mollifier tests green.
Buffer-side counterpart to PG's idempotency-key uniqueness, per the Q5
mollifier-idempotency design. Three changes to the buffer's atomic
Lua surface plus two new high-level methods.
acceptMollifierEntry: when the caller passes idempotencyKey + task-
Identifier, SETNX a `mollifier:idempotency:{env}:{task}:{key}` lookup
pointing at the runId. Second accepts for the same tuple return the
existing winner's runId so the loser's response can echo it as a
cached hit. accept's return shape changes from boolean to a discrim-
inated AcceptResult (accepted / duplicate_run_id / duplicate_idemp-
otency). Existing four callers that ignored the boolean continue to
work; one assertion was updated for the new shape in tests.
ackMollifierEntry: DELs the idempotency lookup atomically with
marking the entry materialised. PG becomes canonical post-mater-
ialisation; the lookup TTL is the safety net if the DEL is missed.
New lookupIdempotency: resolves a buffered run by (env, task, key)
tuple. Used by IdempotencyKeyConcern in B6b. Self-heals stale lookups
that point at expired entries.
New resetIdempotency: atomic Lua that nulls idempotencyKey +
idempotencyKeyExpiresAt on the snapshot payload, clears the
denormalised hash pointer, and DELs the lookup. Used by
ResetIdempotencyKeyService in B6b alongside the PG-side updateMany.
BufferEntrySchema gains an optional idempotencyLookupKey string field
(empty when no idempotency key was bound) so the ack Lua can DEL the
lookup without reading the payload JSON.
8 new tests cover: lookup write+TTL, duplicate_idempotency return,
lookupIdempotency hit/miss/self-heal, ack-DELs-lookup, reset clears
both stores, reset null when nothing bound.
Composes PG-first (replica) lookup, MollifierBuffer.mutateSnapshot,
and writer-side spin-wait into the Q3 wait-and-bounce flow. Returns
a discriminated outcome rather than throwing Response, so the helper
stays route-agnostic and unit-testable. Phase C mutation endpoints
(tags, metadata-put, reschedule, cancel) consume this in upcoming
commits.
Wait knobs default to safetyNetMs=2000, pollStepMs=20, pgTimeoutMs=50
per Q3. Each PG poll is bounded by pgTimeoutMs via Promise.race so
a slow query can't burn the whole safety-net budget. Abort signal is
respected between polls (callers should pass getRequestAbortSignal()
when running in a request handler).
Also exports SnapshotPatch and MutateSnapshotResult from
@trigger.dev/redis-worker so webapp consumers can type-check their
callers of mutateSnapshot.
Atomic Lua-driven snapshot mutation for the burst-buffer entry hash.
Returns one of three result codes per Q3:
- applied_to_snapshot: entry was QUEUED + not materialised; the
drainer will see the patched payload on its next pop.
- not_found: no entry hash for this runId.
- busy: entry is DRAINING / FAILED / materialised — the caller
wait-and-bounces through PG (helper lands in B5).
Four patch types:
- append_tags: union-merges into payload.tags, dedupes against
existing values.
- set_metadata: replaces metadata + metadataType (last-write-wins).
- set_delay: replaces payload.delayUntil.
- mark_cancelled: stamps cancelledAt + cancelReason; the drainer
bifurcation in Q4 reads these on next pop.
10 new tests cover: not_found, all four patch types (success +
absent-field handling), each busy state (DRAINING, FAILED,
materialised), and per-runId atomicity under 50-way concurrent
appends.
`MollifierBuffer.ack` previously deleted the entry hash. It now sets
`materialised=true` and resets the TTL to a 30s grace window via a new
atomic `ackMollifierEntry` Lua script. The entry hash persists past
materialisation as a read-fallback safety net for the brief PG replica
lag window between drainer-side write and reader-side visibility (Q1
D2).
`BufferEntrySchema` gains an optional `materialised` boolean (string
"true"/"false" in Redis → boolean in JS). Accept still refuses while
*any* entry exists for the runId — including materialised ones — as
defense-in-depth against runId reuse.
The drainer's "drains one queued entry … and acks" test now asserts
`materialised=true` instead of entry deletion. The "re-accept after
ack works" test is inverted to "accept refused while a previously-acked
entry is still inside its grace TTL".
Per-env queue `mollifier:queue:{envId}` switches from a Redis LIST
(LPUSH/RPOP) to a sorted set keyed by `createdAtMicros`. Pop semantics
are unchanged (FIFO by creation time, now via ZPOPMIN). Entry hashes
carry a new `createdAtMicros` field equal to the score.
Requeue keeps the original score — createdAt is immutable across
retries, so a retried entry continues to pop next by virtue of being
the oldest. `maxAttempts` in the drainer bounds the retry loop. The
inverted "FIFO retry" test reflects the new (correct) semantics under
the score-equals-createdAt invariant.
`listEntriesForEnv` reads via ZREVRANGE (newest-first). Orphan-handling
tests that injected via LPUSH now use ZADD; queue-depth assertions
switch from LLEN to ZCARD.
This is the substrate for the listing pagination work in Phase E and
for the snapshot-mutate work in Phase B3.
## Summary
Five hardening fixes across `@trigger.dev/sdk`, `@trigger.dev/core`, and
`@trigger.dev/build`.
- `tasks.triggerAndSubscribe` now forwards caller `requestOptions`
(custom API keys, per-request overrides) to the underlying
`apiClient.triggerTask` call instead of silently dropping them.
- `SSEStreamSubscription` no longer retries permanent client errors
forever. The default `nonRetryableStatuses` widens from `[404, 410]` to
`[400, 404, 409, 410, 422]`, so a malformed session-stream request fails
fast instead of busy-looping under bounded backoff.
- Session writer falls back to manually wiring the caller's
`AbortSignal` on Node 18, where `AbortSignal.any` is unavailable.
Caller-driven cancellation now propagates on every supported runtime.
- `TriggerChatTransport` throws immediately when a `chat.handover`
response is missing `X-Trigger-Chat-Access-Token`, instead of silently
downgrading every subsequent turn back to the handover path. `dispose()`
aborts every active `session.out` subscription before tearing the
coordinator down, so unmount/navigation no longer leaves SSE readers in
flight.
- Removed the experimental `@trigger.dev/build/extensions/secureExec`
build extension. It will return alongside the sandbox feature it was
built to support.
## Test plan
- [ ] `pnpm run build --filter @trigger.dev/sdk --filter
@trigger.dev/core --filter @trigger.dev/build`
- [ ] `pnpm --filter @trigger.dev/sdk test --run` (183 tests, including
chat / chat-server / sessions / handover)
- [ ] `pnpm --filter @trigger.dev/core test --run`
- [ ] Manually trigger a `chat.handover` whose response strips
`X-Trigger-Chat-Access-Token`, and confirm the transport throws
synchronously rather than degrading.
- [ ] Unmount a chat UI mid-stream and confirm the active `session.out`
SSE connection closes immediately.
## Summary
`TriggerChatTransport`, `AgentChat`, and `chat.createStartSessionAction`
now accept a string-or-function `baseURL` so callers can route per
endpoint — e.g. `.in/append` through a trusted edge proxy while keeping
`.out` SSE direct. The same surfaces add a `fetch` override for header
injection, custom retries, or proxy rewrites that go beyond URL routing.
SSE GETs are covered too via a new `fetchClient` option on
`SSEStreamSubscription`.
```ts
// TriggerChatTransport / AgentChat — endpoints: "in" | "out"
baseURL: ({ endpoint }) =>
endpoint === "out" ? DIRECT : PROXY,
fetch: (url, init, ctx) => {
init.headers = new Headers(init.headers);
init.headers.set("traceparent", currentTraceparent());
return globalThis.fetch(url, init);
},
// chat.createStartSessionAction — endpoints: "sessions" | "auth"
chat.createStartSessionAction("my-agent", {
baseURL: ({ endpoint }) => (endpoint === "sessions" ? PROXY : DIRECT),
});
```
`streamBaseURL` on `TriggerChatTransport` is kept as a backwards-compat
alias and continues to win for the `"out"` endpoint when set.
Plain-string `baseURL` still applies to every endpoint, matching prior
behavior.
## Summary
Hot-loop writers — `streams.writer` / `streams.pipe` on the run-scoped
side, `chat.response.write` / `chat.stream.*` on the session side — were
issuing a fresh `PUT` to mint S2 credentials for every chunk. On run
streams, each PUT also pushed the streamId onto
`TaskRun.realtimeStreams`,
so a chat-agent turn writing N chunks produced N PUTs and N duplicate
array pushes against the same row.
The SDK now caches the initialize response per cache slot: `(runId,
key)`
for run streams, the session id for session streams. First call PUTs as
before; subsequent calls reuse the cached promise. Hot-loop writers do
one PUT per slot for the lifetime of the cache.
S2 access tokens have a 1-day TTL. If a writer's `wait()` rejects (auth
error, expired token, network blip), the cache evicts the matching slot
so the next call re-PUTs and mints fresh credentials, identity-checked
so a concurrent caller's fresh promise isn't accidentally cleared.
## chat.agent guardrail
`streams.pipe / writer / append / read` called inside a `chat.agent` run
now logs a one-time warning pointing at `chat.response.write` /
`chat.stream.*` — `streams.*` is run-scoped and isn't visible on the
chat session. The ai-chat docs are updated to drop the old guidance
toward run-scoped streams.
## Summary
- Introduce the Mollifier: a Redis-backed buffer for `trigger()` API
calls during traffic spikes, with a per-env trip evaluator and a drainer
ack-loop.
- Phase 1 is dual-write monitoring — every mollified trigger is buffered
to Redis AND continues to `engine.trigger`. No customer-facing behaviour
change.
- Telemetry events: `mollifier.would_mollify`, `mollifier.buffered`,
`mollifier.drained`, plus the `mollifier.decisions` counter.
- Gated behind a feature flag (default off).
## Test plan
- [x] `pnpm run test --filter @trigger.dev/redis-worker`
- [x] `pnpm run test --filter webapp -- mollifier`
- [x] Manual: with flag off, no behaviour change vs main
- [x] Manual: with flag on + threshold lowered, observe
`mollifier.buffered` + `mollifier.drained` log pairs with matching
`runId`
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Node's setTimeout can fire a millisecond or two early under CI load,
causing the existing `>= 500ms` lower bound to flake (saw 499ms in CI).
Loosen to `>= 450ms` — the behaviour being pinned is "stop honors the
deadline instead of waiting for the hung handler indefinitely", not
millisecond-precise timing.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
If buffer.requeue() or buffer.fail() throws during error recovery inside
processEntry, the rejection used to escape processOneFromEnv and reject
runOnce's Promise.all — discarding handler results from sibling envs in
the same tick. Wrap processEntry in try/catch so the failed env is just
counted as "failed" for the tick, matching the invariant stated in the
processOneFromEnv comment.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Brings in every phase-2 fix without regressing the phase-3 surface.
Resolutions:
* `apps/webapp/app/v3/mollifier/mollifierDrainer.server.ts` — kept
phase-3's `MollifierDrainer<MollifierSnapshot>` generic and the
`createDrainerHandler({ engine, prisma })` + `isRetryablePgError`
wiring; added phase-2's `MollifierConfigurationError` class and
switched every env read to `TRIGGER_MOLLIFIER_*`. The doc comment on
`getMollifierDrainer` now points at `mollifierDrainerWorker.server.ts`
instead of the legacy `worker.server.ts`.
* `apps/webapp/app/runEngine/services/triggerTask.server.ts` — kept
phase-3's gate inputs (`options: { debounce, oneTimeUseToken,
parentTaskRunId, resumeParentOnCompletion }` for the C1/C3/F4
bypasses) and the `#buildEngineTriggerInput` refactor; wrapped the
gate call in phase-2's `this.isMollifierGloballyEnabled()` short-
circuit so a trigger with the feature off skips the
`GateInputs` allocation + `mollifier.decisions{outcome=pass_through}`
increment. `mollifierOutcome` is `GateOutcome | null`, downstream
access uses optional chaining. Phase-2's phase-1 dual-write block is
dropped — phase-3's mollify branch already writes via
`mollifyTrigger` and returns a synthetic result.
Phase-2 fixes preserved (verified post-merge):
- `TRIGGER_MOLLIFIER_*` env-var prefix on every consumer
- `TRIGGER_MOLLIFIER_DRAINER_ENABLED` separate per-replica switch
- `MollifierConfigurationError` rethrow in the drainer bootstrap
- Factory create + bootstrap start split (`drainer.start()` only
runs after SIGTERM handlers register)
- `signalsEmitter.on` instead of `process.once`
- `MollifierDrainer.stop` deadline-timer `clearTimeout` in finally
- `vi.fn` handler spies removed from `drainer.test.ts`
- `isMollifierGloballyEnabled` DI hook on `RunEngineTriggerTaskService`
- Legacy `worker.server.ts` header
- New `mollifierDrainerWorker.server.ts` error-classification test
- Updated `.server-changes/mollifier-burst-protection.md`
Pre-existing on phase-3, NOT introduced by this merge (verified by
`git diff origin/mollifier-phase-3 -- apps/webapp/test/engine/triggerTask.test.ts`
showing only `isMollifierGloballyEnabled: () => true` additions):
- `mollifier · mollify action triggers dual-write` asserts
`prisma.taskRun.findFirst` returns a row, but phase-3's
`mollifyTrigger` writes to the buffer and returns a synthetic
result without invoking `engine.trigger`. Test name + assertions
still reflect phase-1 dual-write semantics.
- `mollifier · engine.trigger throwing AFTER buffer.accept` expects
the call to reject; phase-3 returns synthetic before
`engine.trigger` is reached, so the injected throw never fires.
- `mollifier · debounce match produces an orphan buffer entry` —
phase-3 explicitly skips the buffer branch when debounce is set
(`\!body.options?.debounce` guard), so no orphan accept happens.
These three tests describe phase-1 monitoring behaviour on phase-3
code that abandoned dual-write. Updating them is phase-3's own tech
debt, separate from this merge.
## Summary
Long-running chat agents were filling `session.out` forever — every
`chat.agent` turn appended to the same S2 stream with no trim, and the
Sessions dashboard re-streamed the entire history from `seq_num=0` on
every page load. After this change the agent appends an S2 `trim`
command record after each `trigger:turn-complete`, pointing back at the
previous turn-complete's seq_num. `session.out` stays roughly one turn
long at steady state, regardless of session age.
`trigger:turn-complete` and `trigger:upgrade-required` move from
`chunk.type`-shaped data records into header-form control records under
a uniform `trigger-control` namespace. Built-in transports
(`TriggerChatTransport`, `AgentChat`, the dashboard's `AgentView`)
handle the new shape transparently. Custom transports need a one-line
filter on the `trigger-control` header — see the rewritten "Records on
session.out" section in the client-protocol docs.
The Sessions detail page in the dashboard fetches the agent's per-turn
S3 snapshot via a presigned URL and seeds the transcript view, then
SSE-tails from the snapshot's `lastOutEventId`. Bandwidth and
time-to-first-render scale with unread turns instead of session
lifetime.
Resume contract is now explicit: single-turn-boundary resume always
works (the prior turn-complete is still on the stream), the S2 trim is
eventually consistent over 10-60s, and multi-turn-away resume falls back
to a snapshot reload.
## Summary
Local self-hosted deploys (`trigger deploy --local-build --push
--builder orbstack` or any other buildx setup using the **docker**
driver) fail at the push step with:
```
ERROR: failed to build: failed to solve:
exporter option "rewrite-timestamp" conflicts with "unpack"
```
The docker driver auto-enables `unpack=true` when pushing, and that's
incompatible with `rewrite-timestamp` (which the CLI sets for
reproducible-build hashing).
Adds a simple env-var opt-out so contributors can keep using their
default builder. The flag is only read by the local-build code path;
remote/cloud builds are unaffected.
```bash
TRIGGER_BUILD_SKIP_REWRITE_TIMESTAMP=1 \
pnpm exec trigger deploy --profile default --local-build --push --builder orbstack
```
The trade-off: skipping `rewrite-timestamp` means layer timestamps
reflect actual build time, so two identical builds produce different
layer hashes. Fine for a local-dev registry; the only real consumer of
timestamp-stability is registry-layer cache hit rates.
## Test plan
- [x] Manual: ran `trigger deploy --profile default --local-build --push
--builder orbstack` against the localhost webapp + a local Docker
registry on port 5001 — first failed with the rewrite-timestamp/unpack
error, then succeeded after setting
`TRIGGER_BUILD_SKIP_REWRITE_TIMESTAMP=1`.
- [x] Full chat.agent smoke sweep (15 tests, including suspend/resume,
deepResearch subtask, AgentChat orchestrator) against the deployed image
— all pass.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
`TriggerChatTransport` had a single `baseURL` option covering both the
`.in/append` POSTs and the long-lived `.out` SSE subscription. Customers
wanting to route the SSE through a proxy (e.g. a Cloudflare worker
capturing JA4 fingerprints for bot detection) had to send every append
through the proxy too, adding a hop to every user message.
New optional `streamBaseURL` overrides the SSE base URL only; appends
keep using `baseURL`. Falls back to `baseURL` when unset, so existing
transports are unchanged.
```ts
const transport = new TriggerChatTransport({
task: "ai-chat",
baseURL: "https://api.trigger.dev",
streamBaseURL: "https://chat-proxy.example.com",
accessToken,
startSession,
});
```
Verified with a new test in `chat.test.ts` that asserts `.in/append`
routes through `baseURL` and `.out` SSE routes through `streamBaseURL`.
All existing tests still pass.
The Promise.race between this.loopPromise and this.delay(timeoutMs)
discarded the timeout's underlying setTimeout handle whenever the loop
branch won. The discarded timer was still ref'd by libuv and pinned the
Node event loop alive for the remainder of `timeoutMs` — exactly the
shutdown slack the timeout was supposed to bound.
Inline the timer in stop() with a captured handle and clearTimeout() it
in a finally block, so every exit path (loop-won, timeout-won, throw)
releases the ref. The in-loop delay() calls are unchanged — they're
awaited normally and their timers fire-and-clear themselves.
## Summary
Enables shipping `X.Y.Z-rc.N` prereleases of `@trigger.dev/*` via
changesets pre mode. RCs publish under the `rc` npm dist-tag, never
claim `latest`, and don't trigger marketing-site changelog PRs. The
plumbing is hyphen-in-version detection in `release.yml` — no separate
workflow, no opt-in flag at publish time.
Validated end-to-end against a sandbox repo (real npm publishes, Docker
builds, Helm chart pushes, GitHub releases) before porting back. Full RC
lifecycle tested: pre enter → rc.0 → iterate to rc.1 → pre exit →
stable. Plus interaction with the existing release-branch hotfix flow.
## What changes
### `release.yml`
- New `is_prerelease` output (hyphen-in-version)
- GitHub release adds `--prerelease` flag for RC publishes (Pre-release
badge, not Latest)
- `dispatch-changelog` job gated on `is_prerelease != 'true'` — no
marketing-site PR per RC
### Docker workflows
- Removes the `:v4-beta` floating tag entirely from `publish-webapp.yml`
and `publish-worker-v4.yml`. v4 is GA; the tag is a misnomer and is
already inconsistent with the npm side (npm `v4-beta` dist-tag was
frozen at 4.0.4 months ago while Docker `:v4-beta` kept bumping).
Self-hosters should pin to a versioned tag going forward — the last
value of `:v4-beta` stays frozen wherever it currently points.
### CLI version-check fix
(`packages/cli-v3/src/utilities/initialBanner.ts`)
Switches the "new version available" comparison from JavaScript
`localeCompare` to `semver.lt`. The old comparison handled `X.Y.Z-rc.N`
vs `X.Y.Z` incorrectly — a user on `4.5.0-rc.0` would never be prompted
to upgrade once `4.5.0` stable shipped (lex order put the prerelease
ahead of the bare version). Real semver gets this right.
Stable users were never affected: the check queries the `@latest`
dist-tag, which by convention never points at a prerelease.
## How an RC actually publishes after this
1. `pnpm exec changeset pre enter rc` on main, push the `pre.json`
2. Bot regenerates the release PR as `chore: release v<X.Y.Z>-rc.0`
3. Merge → `release.yml` runs `changeset publish` which reads
`pre.json.tag` and publishes under `--tag rc`. GitHub release marked
Pre-release. No marketing-site dispatch.
4. Iterate by adding changesets normally; bot bumps to `rc.1`, `rc.2`, …
5. When ready: `pnpm exec changeset pre exit`, push, merge regenerated
PR → stable ships under `latest` and the marketing-site dispatch fires.
## Summary
Adds a Region column and Region filter (under More filters) to the runs
list dashboard, the same filter on the public runs list API
(`filter[region]`), and a matching `region` input on the MCP `list_runs`
tool. Each run's executing region is also surfaced as a new optional
`region` field on the runs list and run retrieve responses, populated
from the worker instance group's `masterQueue` identifier.
Useful when you run tasks across multiple regions and want to slice the
runs list — or your existing run-querying scripts — by where the run
actually executed.
## Design
The filter value in the URL / API is the `masterQueue` identifier (the
same string already persisted on `TaskRun` and replicated to ClickHouse
as `worker_queue`), so the query just becomes `worker_queue IN (...)`
with no server-side translation. The Region dropdown options come from a
new resource loader backed by `RegionsPresenter`, which now also exposes
`masterQueue` alongside the existing region metadata.
```ts
// public API
const runs = await runs.list({ region: ["us-east-1", "eu-west-1"] });
// each item: { id, status, ..., region?: "us-east-1" }
```
```ts
// MCP
list_runs({ environment: "prod", region: "us-east-1" })
```
Replace each vi.fn(async handler) with a plain async closure that
records calls via captured counter/array variables. Assertions move
from handler.mock.* / toHaveBeenCalled* matchers to checks against
the captured state, e.g. handlerCalls.length / handlerCalls[0].
Functionally equivalent; aligns with the package convention of using
real testcontainers + closure-based probes (cf. mollifierGate.test.ts
and mollifierTripEvaluator.test.ts) rather than vitest fakes.
Two correctness/perf fixes on top of the phase-2 drainer:
1. `runOnce` was awaiting `listEnvsForOrg` serially before any pop ran.
At the default `maxOrgsPerTick=500` and a ~1ms RTT, that's a ~500ms
per-tick latency floor before `pLimit` even sees work. `Promise.all`
over the org slice lets ioredis auto-pipeline the SMEMBERS into a
single round-trip. Order is preserved so the org→envs pairing stays
deterministic and `pickEnvForOrg` still rotates per org.
2. The SIGTERM handler is sync fire-and-forget: `drainer.stop({timeoutMs})`
returns a promise that keeps the loop alive, but in cluster mode the
primary process runs its own `GRACEFUL_SHUTDOWN_TIMEOUT` and will hit
`process.exit(0)` independently. If the drainer's deadline exceeds
the primary's, the drainer's "log a warning on timeout" turns into
"hard exit with no log". Assert at boot that
`MOLLIFIER_DRAIN_SHUTDOWN_TIMEOUT_MS <= GRACEFUL_SHUTDOWN_TIMEOUT - 1s`
so a misconfig fails loud instead of disappearing at shutdown.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
With the drainer walking listOrgs → listEnvsForOrg → pop, the flat
mollifier:envs SET has no consumer — `mollifier:orgs` and the per-org
`mollifier:org-envs:${orgId}` SETs cover everything the drainer needs.
Removing it drops three Lua write ops per accept/pop/requeue and one
Redis key per active env.
Changes:
- Lua: acceptMollifierEntry, popAndMarkDraining, requeueMollifierEntry
no longer touch mollifier:envs. Their KEYS arrays shrink by one.
- TS: listEnvs() method removed; only listOrgs() and listEnvsForOrg()
remain. TS bindings updated to match the new arg shapes.
- buffer.test.ts: listEnvs() assertions converted to listEnvsForOrg(
"org_1") so they verify the equivalent org-level membership. The
"stale envs SET cleanup on empty-pop" test is removed (envs SET is
gone). The "pop skips orphans" test's trailing-cleanup assertion is
updated to document the deliberate stale-tolerance in the no-runId
branch of popAndMarkDraining (can't read orgId without a popped
entry, so org-envs cleanup is skipped there).
- drainer.test.ts: stub helper moved to module scope and gains an
`eachEnvAsOwnOrg(envs)` convenience that supplies listOrgs +
listEnvsForOrg in tests where each env is its own org. Stub helpers
duplicated across describe blocks are removed in favour of the
shared one.
24/24 drainer tests pass; buffer tests pass in isolation (a few timeout
under full-suite contention against the shared redis container —
unrelated to this change).
Previously the drainer cached envId→orgId from popped entries and used a
sentinel pseudo-org for envs it hadn't seen yet. The sentinel polluted
the bucket map with fake org IDs and was a foreseeable source of bugs.
This commit moves org membership into the buffer's atomic Lua scripts.
New Redis keys, both maintained transactionally alongside per-env queues:
- mollifier:orgs — orgs with at least one queued env
- mollifier:org-envs:${orgId} — envs of that org with queued entries
acceptMollifierEntry SADDs into all three sets (envs + orgs + org-envs).
popAndMarkDraining cleans up envs+orgs+org-envs together when the queue
empties in the success branch (we know orgId from the popped entry). The
no-runId branch can't read orgId so it only cleans envs — stale org-envs
entries are bounded by env count and recovered on the next accept.
requeueMollifierEntry re-SADDs all three since the env may have just been
pruned.
The drainer now walks listOrgs() → listEnvsForOrg(org) → pop(env) with
two cursors: orgCursor across all active orgs and a per-org envCursor
for round-robin within each org. No client-side cache, no sentinel,
deterministic from the first tick.
Tests updated:
- multi-org-round-robin (was multi-env-round-robin): two orgs with one
and two envs respectively, asserts org_B drains its only env each
tick while org_A rotates through its two.
- concurrency-cap test spreads 12 envs across 12 orgs (otherwise one
org → one pop per tick).
- "heavy org doesn't dominate vs light org" gets explicit listOrgs /
listEnvsForOrg from the test's env→org map; assertion tightened to
0.7–1.5 ratio over 20 ticks.
- "within an org envs rotated round-robin" gets explicit listEnvsForOrg.
- "envCursor resets" → "rotation cursors reset"; cache is gone, only
orgCursor and perOrgEnvCursors reset on start().
- makeStubBuffer auto-derives listOrgs/listEnvsForOrg from listEnvs
(each env as its own org) so tests that don't care about org grouping
don't need to provide them explicitly.
24/24 drainer tests pass, 35/35 buffer tests pass (some redis-container
flakes under full-suite load; all green in isolation). Webapp typecheck
clean.
Previously the drainer rotated per-env: an org with N busy envs got N
scheduling slots per tick. A noisy tenant with many envs would drain
proportionally faster than a quiet tenant with one env. Switch to
hierarchical rotation: pick orgs round-robin (capped by maxOrgsPerTick),
then pick one env per picked org (also rotating).
Implementation is drainer-side only — no buffer or Lua changes. The
drainer caches envId→orgId from popped entries; envs not yet cached are
treated as their own pseudo-org for one tick, so cold start matches the
old per-env behaviour and converges to hierarchical once cache is hot
(usually within one tick). Cache and cursors reset on start() alongside
the existing cursor reset.
API change: maxEnvsPerTick → maxOrgsPerTick on MollifierDrainerOptions,
MOLLIFIER_DRAIN_MAX_ENVS_PER_TICK → MOLLIFIER_DRAIN_MAX_ORGS_PER_TICK on
the webapp env. Same default (500). Operators tune for "typical orgs
with pending entries" rather than env count.
Trade-off: total per-tick pops drop from O(envs) to O(orgs). For an org
with N envs, each env's individual drainage rate is 1/N of what it was,
but the tenant overall is bounded the same way as a single-env tenant —
which is the fairness contract.
Tests:
- Renamed maxEnvsPerTick references throughout existing tests; old
behaviour still holds at cold cache (each env = pseudo-org).
- New "heavy org with many envs does not dominate vs light org" pins
the post-warm-up ~1:1 drainage ratio between a 6-env org and a 1-env
org over a sustained 20-tick run.
- New "within an org, envs are rotated round-robin across ticks" pins
the inner env cursor's behaviour for a single multi-env org.
- Cursor-reset test renamed and now asserts cache+cursors all reset.
Also removed an outdated test-count comment in
apps/webapp/test/engine/triggerTask.test.ts that listed "four tests"
when reality has moved on.
1. start() resets envCursor to 0 — new behaviour. A stop+start cycle now
begins rotation cleanly from envs[0] rather than inheriting between-
restart cursor drift.
2. Malformed payload → non-retryable handler error path. Pins that the
deserialise failure goes terminal without invoking the handler.
3. Ack failure after handler success — documents the current behavioural
gap. ack() lives inside processEntry's try, so a Redis blip on ack
routes a successfully-handled entry through the retry/terminal path.
Phase 2's engine-replay handler will need idempotency to absorb the
re-execution, OR ack should be lifted out of the try block.
4. start() idempotency — second call is a no-op (no doubled loop).
5. stop() idempotency — safe to call when never started or twice.
6. Loop-level backoff actually grows on consecutive runOnce failures
and resets on first success. Distinct from per-entry retry attempts
already covered elsewhere; this is the consecutiveErrors counter
that drives backoffMs between ticks.
Also adds org-level fairness analogue of the existing env starvation
test: a light org (1 env, 1 entry) is not starved behind a heavy org
with many envs and many entries. The buffer doesn't track orgs as a
separate axis, so org fairness is an emergent property of env rotation
— the test pins that property explicitly.
The fairness test compared popsPerTick[0][0] vs popsPerTick[1][0]
directly. Under the redis-worker package's strict tsconfig
(noUncheckedIndexedAccess implied), array index access returns T |
undefined, which trips TS2532. Destructure into named locals and use
optional chaining — same assertion, no `\!` non-null soup.
## Summary
`LocalsKey<T>` (the type returned by `locals.create()`) was branded with
a
module-level `declare const __local: unique symbol`. Each such
declaration
is its own nominal type, and `tshy` emits separate `.d.ts` files for the
ESM and CJS outputs — each gets its own `__local` symbol. Under certain
pnpm hoisting layouts a single TypeScript compilation can resolve
`LocalsKey` from both the ESM source path and the CJS dist path within
the same call site, producing two structurally-incompatible variants of
the same type. TS surfaces this as the misleading error:
```
Argument of type 'LocalsKey<X>' is not assignable to parameter of type
'LocalsKey<X>'. Property '[__local]' is missing in type 'LocalsKey<X>'
but required in type 'BrandLocal<X>'.
```
The error has been hitting CI on PRs opened since the chat.agent stack
landed (e.g. #3625 typecheck job), but doesn't reproduce on developer
machines where the pnpm node_modules layout was built up incrementally.
## Fix
Replace the `unique symbol` brand with an optional phantom field that
carries `T` at the type level:
```ts
// before
declare const __local: unique symbol;
type BrandLocal<T> = { [__local]: T };
export type LocalsKey<T> = BrandLocal<T> & {
readonly id: string;
readonly __type: unique symbol;
};
// after
export type LocalsKey<T> = {
readonly id: string;
readonly __type: symbol;
/** Phantom carrier for the value type — never read at runtime. */
readonly __valueType?: T;
};
```
The ESM and CJS `.d.ts` outputs now produce structurally identical
types,
so cross-output resolution no longer produces a mismatch. `T` is still
carried at the type level via the optional phantom field. The runtime
shape is unchanged — `manager.ts` was already casting via `as unknown`,
which is no longer needed.
## Test plan
- [ ] `pnpm run typecheck --filter @trigger.dev/core --filter
@trigger.dev/sdk`
- [ ] `pnpm run build --filter @trigger.dev/core --filter
@trigger.dev/sdk`
(clean rebuild) — confirms the ESM and CJS dist `.d.ts` outputs
no longer carry distinct `unique symbol` declarations
- [ ] `pnpm --filter @trigger.dev/core test test/mockTaskContext.test.ts
--run`
- [ ] `pnpm --filter @trigger.dev/sdk test test/mockChatAgent.test.ts
--run`
Adds a regression test that proves a light env (single buffered entry)
is drained within (envs.length - sliceSize + 1) ticks regardless of how
many entries the heavy envs have queued. The test uses a stub buffer
whose listEnvs/pop pair mirrors the production atomic-Lua semantic: an
env disappears from listEnvs the moment its queue empties, so the light
env exits the rotation as soon as it's popped — while the heavy envs
stay in the rotation until their thousands of entries are drained.
Together with the head-of-line fairness test this pins both fairness
properties: (1) every env touches every slice slot per cycle (no
within-slice bias), and (2) no env's drainage latency depends on the
queue depth of other envs (no across-slice starvation).
The previous chunking advanced the cursor by sliceSize each tick,
producing fixed disjoint slices like [0..3], [4..7], [0..3], ... With
that pattern env_0 was always at slice position 0 (first into pLimit)
and env_3 always at position 3 (last) — reinstating the head-of-line
bias rotation was meant to prevent.
Advance the cursor by 1 instead. Slices now overlap across consecutive
ticks (e.g. [0..3], [1..4], [2..5], ...) so every env reaches every
slot position 0..sliceSize-1 across one envs.length-tick cycle.
Drainage rate per env is unchanged: each env still appears in exactly
sliceSize of every envs.length ticks. New regression test pins the
fairness property by asserting each env touches every slot at least
once per cycle.
The MollifierDrainer's stop() was polling `isRunning` every 20ms until
the loop exited, which differs from the codebase's convention for
similar polling loops (FairQueue, BatchQueue both hold the loop promise
as a field and await it directly on stop).
Switch to the same pattern: store the loop promise on start(), then in
stop() race it against the timeout via Promise.race. With no timeout we
just await the loop directly. With a timeout the warn-and-return
behaviour is unchanged. No polling, no separate `isRunning` poll loop.
Behaviour is identical to the previous implementation, including the
hung-handler timeout path (covered by the existing
"stop returns after timeoutMs even if a handler is hung" test).
mollifier:envs is a Redis SET that grows with the count of envs that
currently have buffered entries. Under normal operation that's small,
but an extended drainer outage can leave entries piled up across
thousands of envs — at which point runOnce would queue one
processOneFromEnv per env through pLimit, ballooning per-tick latency
and event-loop queue depth.
Cap per-tick fan-out at MOLLIFIER_DRAIN_MAX_ENVS_PER_TICK (default 500).
When the set fits within the cap, behaviour is unchanged (take all,
rotate cursor by 1 for fairness). When the set exceeds the cap, take a
rotating slice and advance the cursor by the slice size so successive
ticks sweep through the full set.
Tests use a stub buffer to drive listEnvs() deterministically with
thousands of envs without provisioning a real Redis.
Top of the chat.agent stack: a full Next.js reference project that
exercises chat.agent end-to-end, plus the CLI MCP tools that drive
agent runs from Claude Code / Cursor / etc.
references/ai-chat:
- Full Next.js app with prisma persistence, multi-chat sidebar,
per-chat model picker, debug panel, tool examples, smoke tests
- Reference tools: getCurrentTime, searchHackerNews, createGithubIssue,
PR review helpers, code sandbox
- chat-client-test orchestrator for concurrent-send stress
- references/hello-world chatAgent + triggerAndSubscribe examples
CLI MCP tooling for chat.agent:
- mcp/tools/agentChat.ts (start_agent_chat, send_agent_message,
close_agent_chat)
- mcp/tools/agents.ts + tasks.ts (list agents, agent run details)
- dev-run-controller OOM kill + taskRunProcessPool tweaks
- dev/managed entry-point hooks for skills bundling
- buildWorker + bundleSkills (agent skills support)
Includes ai-tool-helpers + mcp-agent-chat-sessions changesets, plus
the streamdown@2 patch and pnpm-lock reconciliation.
(Will be renamed to feature/ai-chat-reference-and-cli before push.)
fix(cli): preserve lastEventId after sendMessage fallback to avoid stale turn-complete replay
drainer.fuzz.test.ts and evaluateTrip.fuzz.test.ts are valuable as
ongoing property checks but aren't load-bearing for the phase-1 review.
Moving them to a follow-up keeps this PR smaller without losing coverage
of the production paths (buffer.test.ts and drainer.test.ts together
cover the contract surface).
processOneFromEnv now catches buffer.pop() failures so one env's hiccup
doesn't reject Promise.all and bubble up to the loop's outer catch. The
polling loop itself wraps each runOnce in try/catch and backs off with
capped exponential delay (up to 5s) instead of exiting permanently on the
first listEnvs/pop error. Stop semantics are unchanged: only the stopping
flag breaks the loop.
Adds two regression tests using a stub buffer (no Redis container) so
fault injection is deterministic.
Phase 1 of the trigger-burst smoothing initiative. Adds the A-side trip
evaluator (atomic Lua sliding-window per env) and wires it into the trigger
hot path. When the per-org mollifierEnabled feature flag is on AND the
evaluator says divert, the canonical replay payload is buffered to Redis
(via buffer.accept) AND the trigger continues through engine.trigger —
i.e. dual-write. The drainer pops + acks (no-op handler) to prove the
dequeue mechanism works end-to-end. Operators audit by joining
mollifier.buffered (write) and mollifier.drained (consume) logs by runId.
Buffer primitives hardened:
- accept is idempotent on duplicate runId (Lua EXISTS guard)
- pop skips orphan queue references (entry HASH TTL'd while runId queued)
- fail no-ops on missing entry (no partial FAILED hash leak)
- mollifier:envs set pruned on draining pop, restored on requeue
- 16-row truth-table test enumerates the gate cascade
- BufferedTriggerPayload defines the canonical replay shape Phase 2 will
use to invoke engine.trigger
- payload hash for audit-equivalence computed off the hot path (in the
drainer) to avoid CPU during a spike
Regression tests in apps/webapp/test/engine/triggerTask.test.ts pin the
mollifier integration:
- validation throws BEFORE the gate runs (no orphan buffer write on
rejected triggers)
- mollify dual-write happy path (Postgres + Redis both reflect the run)
- pass_through path does NOT call buffer.accept
- engine.trigger throwing AFTER buffer.accept leaves an orphan
(documented behaviour — drainer auto-cleans; audit-trail surfaces it)
- idempotency-key match short-circuits BEFORE the gate is consulted
- debounce match produces an orphan (documented behaviour — Phase 2
must lift handleDebounce upfront before buffer.accept)
Behaviour with MOLLIFIER_ENABLED=0 (default) is byte-identical to main.
With MOLLIFIER_ENABLED=1 and the flag off, only mollifier.would_mollify
logs fire (no buffer state). With the flag on, dual-write activates.
Includes two opt-in *.fuzz.test.ts suites (gated on FUZZ=1) that
randomise operation sequences against evaluateTrip and the drainer to
find timing edges. They are clearly marked TEMPORARY in their headers.
Redis-backed burst-smoothing layer behind MOLLIFIER_ENABLED=0 (default).
With the kill switch off, the gate short-circuits on its first env check
and production behaviour is identical to main.
@trigger.dev/redis-worker:
- MollifierBuffer: atomic Lua-backed FIFO with accept / pop / ack /
requeue / fail + TTL. Per-env queues with HSET entry storage,
atomic RPOP + status transition, FIFO retry ordering.
- MollifierDrainer: generic round-robin worker with concurrency cap,
retry semantics, and a stop deadline to avoid livelock on a hung
handler. Phase 3 will wire the handler to engine.trigger().
- Full testcontainers-backed test suite (21 tests).
apps/webapp:
- evaluateGate cascade-check (kill switch -> org feature flag ->
shadow mode -> trip evaluator -> mollify / shadow_log / pass_through).
Dependencies injected for testability; the trip evaluator stub
returns { divert: false } in phase 1.
- Inserted into RunEngineTriggerTaskService.call() before
traceEventConcern.traceRun. The mollify branch throws (unreachable
in phase 1).
- Lazy MollifierBuffer + MollifierDrainer singletons; no Redis
connection unless MOLLIFIER_ENABLED=1.
- 12 MOLLIFIER_* env vars (all safe defaults) and a mollifierEnabled
feature flag in the global catalog.
- Drainer booted from worker.server.ts on first import.
- Read-fallback stub for phase 3.
- Gate cascade tests + .env loader so env.server validates in vitest
workers.
Phase 2 will land the real trip evaluator; phase 3 will activate the
buffer-write + drain path.
Adds the chat.agent({...}) task definition (server runtime) and the
browser-side TriggerChatTransport + AgentChat that drives it from a
React or Next.js app. The runtime sits on top of the Sessions primitive
and handles the durable conversational task lifecycle.
Server runtime:
- chat.agent({...}) — session-aware task definition
- Lifecycle hooks: onChatStart, onTurnStart, onTurnComplete, onAction,
onValidateMessages, hydrateMessages
- chat.history read primitives for HITL flows
- chat.local, chat.headStart, chat.handover, oomMachine
- Delta-only wire + S3 snapshot reconstruction at run boot
- Actions are no longer turns
Browser transport:
- TriggerChatTransport (ai-sdk Transport): delta-only wire sends,
SSE reconnection with lastEventId resume, stop/abort cleanup,
dynamic accessToken refresh
- AgentChat: direct programmatic API
- useTriggerChatTransport (React hook)
- chat-tab-coordinator: cross-tab leader election
Includes the chat-agent, chat-agent-delta-wire-snapshots,
chat-history-read-primitives, chat-head-start, chat-actions-no-turn,
chat-session-attributes, agent-skills, and mock-chat-agent-test-harness
changesets.
## Summary
A `/sessions` dashboard for inspecting durable Sessions, an `AGENT` /
`SCHEDULED` task-kind filter for the runs list, and the server-side
hardening (rate-limit exemption for packets, retry-with-backoff on
stream appends, typed too-large-chunk error) that the `chat.agent`
runtime in #3543 needs. Builds on the Sessions primitive shipped in
#3417.
## Design
The Sessions list + detail routes mirror the run inspector pattern.
`TaskTriggerSource` gains `AGENT` and `SCHEDULED` values, persisted on
`BackgroundWorker.taskKind` and `TaskRun.taskKind` (plus a matching
Clickhouse column), so the runs list can filter by kind.
New `@trigger.dev/core` modules — `sessionStreams`, `inputStreams`, a
`sessionStreamInstance` for realtime streams, and the
`realtime-streams-api` / `session-streams-api` surfaces — expose the
typed shapes that chat.agent will use to drive `session.out`.
`ChatChunkTooLargeError` lets the runtime drop oversized chunks with a
typed surface instead of failing the run. `s2Append` retries transient
failures with exponential backoff. `/api/v[12]/packets/*` is exempt from
customer rate limits so chat snapshot reads and writes don't get
throttled under load.
## Stack
Part of a 4-PR stack. Merge bottom-up.
1. **This PR** (#3542) → `main`
2. #3543 → #3542 — `chat.agent` runtime + browser transport
3. #3545 → #3543 — agent-view dashboard
4. #3546 → #3545 — ai-chat reference + MCP tooling
Replaces #3173 (closed).
<!-- GitButler Footer Boundary Top -->
---
This is **part 5 of 5 in a stack** made with GitButler:
- <kbd> 5 </kbd> #3612
- <kbd> 4 </kbd> #3546
- <kbd> 3 </kbd> #3545
- <kbd> 2 </kbd> #3543
- <kbd> 1 </kbd> #3542👈
<!-- GitButler Footer Boundary Bottom -->
`tasks.trigger`, `tasks.batchTrigger`, `batch.create`,
`wait.createToken`, `wait.forDuration`, and the input/session stream
waitpoint endpoints all accept a caller-supplied `idempotencyKey` and
store it verbatim against a composite-unique index on `TaskRun`,
`BatchTaskRun`, or `Waitpoint`. The schemas had no length cap, so a
sufficiently long high-entropy key produced an index row larger than the
underlying storage layer can hold. The insert failed at the database,
and the caller saw a generic 500 from
`RunEngineTriggerTaskService.call()` / `CreateBatchService` / waitpoint
creation, depending on the endpoint.
Keys produced by `idempotencyKeys.create()` are 64-character SHA-256
hashes and never trip this — it only manifests for direct REST callers
(or SDK callers passing a raw string they generated themselves).
Low-entropy keys also sail through, because the storage layer compresses
repeated bytes before they reach the index, which is why the failure
mode is intermittent and tied to caller-side key shape.
## Fix
Add `.max(2048, "<field> must be 2048 characters or less")` to the seven
schemas that feed an indexed `idempotencyKey` column:
- `TriggerTaskRequestBody.options.idempotencyKey`
- `BatchTriggerTaskItem.options.idempotencyKey`
- `CreateBatchRequestBody.idempotencyKey`
- `CreateWaitpointTokenRequestBody.idempotencyKey`
- `CreateInputStreamWaitpointRequestBody.idempotencyKey`
- `CreateSessionStreamWaitpointRequestBody.idempotencyKey`
- `WaitForDurationRequestBody.idempotencyKey`
Plus the `idempotency-key` HTTP header on the trigger route (and the
three batch routes that re-export `HeadersSchema`). The header schema is
lifted out of `api.v1.tasks.$taskId.trigger.ts` into
`apps/webapp/app/v3/triggerHeaders.server.ts` so it can be exercised in
tests without dragging the route's import-time side effects.
The 2048 character ceiling is chosen to sit safely under the per-row
index limit while staying generous against existing callers — keys that
fit before still fit. Oversized keys now return a structured Zod 400
instead of a generic 500.
Limit is documented under `Idempotency key` in `docs/limits.mdx` and as
a `<Note>` on `docs/idempotency.mdx`.
## Test plan
- [x] 15 schema unit tests added
(`packages/core/src/v3/schemas/idempotencyKey.test.ts`,
`apps/webapp/test/routes/triggerHeaders.test.ts`) —
rejection-with-message + boundary acceptance for each capped schema. The
webapp test exercises the extracted `TriggerHeadersSchema` directly with
no mocks.
- [x] `pnpm run build --filter @trigger.dev/core`
- [x] `pnpm run typecheck --filter webapp`
- [x] End-to-end verified locally: baseline (small key) → 200; 3000-char
high-entropy header → 400 with the expected Zod error; same key at the
2048 boundary → 200; same key with the cap reverted → the database
rejected the insert and the route returned 500 to the caller. Cap
restored.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Dan Sutton