Q1 ZSET-merge design lands.
redis-worker side:
- MollifierBuffer.listForEnvWithWatermark — paginated newest-first
read of buffered entries, bounded by a (createdAtMicros, runId)
watermark. ZREVRANGEBYSCORE strictly below the watermark score plus
a tied-score band scan for entries sharing the watermark's
createdAtMicros.
webapp side:
- listingMerge.server.ts: callRunListWithBufferMerge wraps
ApiRunListPresenter. Fetches a buffer page, synthesises each entry
into the presenter's ListDataItem shape (status QUEUED, timestamps
from entry hash, env slug looked up once), forwards the inner
cursor to the presenter, merges by createdAt DESC with runId DESC
tiebreak, truncates to pageSize. Compound base64-JSON cursor
{ inner, watermark, bufferExhausted } is backwards-compatible with
legacy opaque cursors.
- api.v1.runs.ts + api.v1.projects.{projectRef}.runs.ts route through
the wrapper. Project route extracts envId from filter[env]; absent
that, falls back to the bare presenter (existing behaviour).
- Buffer eligibility skips for filters that can't match buffered runs
(status not in QUEUED/PENDING/DELAYED, batch/schedule/version/
region/machine filters). Buffer outages fall open to PG-only.
- Delete RecentlyQueuedSection banner + listEntriesForEnv loader call
from dashboard runs index — buffered runs appear inline as QUEUED
rows.
Parallels Phase C's API-side work for the three dashboard mutation
routes.
D1 cancel — PG miss → buffer.mutateSnapshot('mark_cancelled'). Org-
membership verified against the buffered run's orgId (dashboard URL
doesn't carry an envId so the API-side env-scoped auth doesn't apply).
busy returns a "retry in a moment" message.
D2 replay — PG miss → findRunByIdWithMollifierFallback; B4-extended
SyntheticRun cast to TaskRun and fed to ReplayTaskRunService.
Project/env slugs for the redirect path looked up from the entry's
envId.
D3 idempotencyKey reset — PG miss → buffer.getEntry + readFallback to
read snapshot's idempotencyKey + taskIdentifier; org-membership
verified against entry orgId; existing ResetIdempotencyKeyService
(extended in B6b to clear both stores) handles the actual reset.
Closes the last API-parity gap in the master plan.
redis-worker side:
- New casSetMetadata Lua command with optimistic lock on a
metadataVersion entry-hash field. Returns applied / version_conflict /
not_found / busy. Mirrors the PG-side UpdateMetadataService's CAS
loop so concurrent metadata.increment / metadata.set / metadata.append
calls against a buffered run never lose deltas.
- accept Lua initialises metadataVersion=0; BufferEntrySchema gains
the field.
webapp side:
- applyMetadataMutationToBufferedRun helper does the read-apply-CAS-
retry loop in JS, reusing the existing @trigger.dev/core
applyMetadataOperations function (no Lua re-implementation of the 6
operation types).
- metadata PUT route does PG-first via the existing service (which
owns the full request shape: parent/root ops, batching, validation),
then falls through to the buffer helper on PG miss. busy and
version_exhausted return 503 with retry hint; not_found returns 404.
- Parent/root operations on a buffered target are fanned out to the
snapshot's parentTaskRunId via the existing service. If the parent
is also buffered the helper recurses. Best-effort — parent/root
ingestion failures do not surface to the caller.
Tests: 3 new redis-worker tests covering CAS apply / version conflict /
not_found-busy paths. All 71 redis-worker mollifier + 68 webapp
mollifier tests green.
Reschedule (C4): switches to mutateWithFallback. PG hits go through
the existing RescheduleTaskRunService (which enforces status ===
"DELAYED"). Buffered hits land a set_delay patch on the snapshot;
the drainer materialises the PG row with the new delayUntil. Synth-
esised response returns { id, delayUntil }.
Replay (C5): adds a read-fallback after the PG miss. The B4-extended
SyntheticRun carries every field ReplayTaskRunService reads from a
TaskRun, so the buffered case casts through and uses the existing
service unchanged. Replay creates a fresh trigger that itself
re-enters the mollifier gate — no special surge handling needed
beyond what the gate already does. Also tightens the PG lookup to
findFirst with runtimeEnvironmentId scoping (was findUnique on
friendlyId only).
Closes the live 500 the parity script flagged. The previous route did
prisma.taskRun.update after a findFirst that could miss; on buffered
runs (no PG row yet) the update raised RecordNotFound and surfaced as
a 500.
Switches to mutateWithFallback. PG hits go through the existing
select-dedupe-validate-update flow with MAX_TAGS_PER_RUN enforcement.
Buffered-QUEUED hits apply append_tags via Lua (atomic dedup against
existing snapshot tags). busy snapshots wait for drainer resolution
then update PG. 404 / 503 surface for missing / hung cases.
The MAX_TAGS_PER_RUN cap is skipped on the buffered side — the
drainer's engine.trigger doesn't enforce it either, matching the
pre-buffer trigger path. Pushing the cap into the snapshot-mutate Lua
is a possible follow-up.
Per the Q4 mollifier-cancel design — first mutation endpoint.
engine.createCancelledRun: new run-engine method that writes a CANCELED
TaskRun row directly from a buffer snapshot. Skips queue insertion,
waitpoint creation, and concurrency reservation (run never executes).
Emits runCancelled so the existing handler writes the TaskEvent
cancellation row. P2002 from double-pop is caught and returns the
existing row without re-emitting.
Drainer bifurcation: mollifierDrainerHandler routes to
createCancelledRun when snapshot.cancelledAt is set. Cancel-wins-
over-trigger — customer intent is terminal.
Cancel route: wraps the call in mutateWithFallback. PG-row hits go
through the existing CancelTaskRunService. Buffered-QUEUED hits land
a mark_cancelled patch on the snapshot via mutateSnapshot. busy
snapshots wait for drainer resolution then call the PG service
against the resulting row. 404 / 503 surface for genuine missing
or drainer-hung cases.
Known follow-up: the Q3 wait-and-bounce for cancel-of-buffered-FAILED
relies on the drainer eventually writing a SYSTEM_FAILURE PG row on
terminal materialisation failure. That drainer-side write isn't
implemented yet (the failed-drain path today only marks the buffer
entry hash FAILED). Cancel-of-state-3 will currently 503 after 2s
instead of returning the SYSTEM_FAILURE row. Acceptable rare-race
behaviour; flagged for a follow-up alongside the drainer sweeper work.
Three integration points that connect B6a's buffer-side primitives to
the customer-facing flow per Q5:
- IdempotencyKeyConcern.handleTriggerRequest falls through to
buffer.lookupIdempotency after a PG miss. Buffered hits return
isCached:true with a synthesised TaskRun via the existing
findRunByIdWithMollifierFallback. Skipped when
resumeParentOnCompletion is set: waitpoint blocking requires a PG
row that doesn't exist yet; the follow-up accept SETNX still
dedupes the trigger itself. Buffer outages fail open to "no cache
hit" so the trigger hot path is never wedged by a transient Redis
issue.
- mollifyTrigger passes idempotencyKey + taskIdentifier through to
buffer.accept. The SETNX race loser receives duplicate_idempotency
with the winner's runId; the API response echoes it with
isCached:true, matching PG-side cache-hit shape.
- ResetIdempotencyKeyService calls buffer.resetIdempotency alongside
the existing PG updateMany. 404 only fires when both stores report
nothing bound. Buffer outage during reset is logged and treated as
a miss; PG-side reset still works.
Composes PG-first (replica) lookup, MollifierBuffer.mutateSnapshot,
and writer-side spin-wait into the Q3 wait-and-bounce flow. Returns
a discriminated outcome rather than throwing Response, so the helper
stays route-agnostic and unit-testable. Phase C mutation endpoints
(tags, metadata-put, reschedule, cancel) consume this in upcoming
commits.
Wait knobs default to safetyNetMs=2000, pollStepMs=20, pgTimeoutMs=50
per Q3. Each PG poll is bounded by pgTimeoutMs via Promise.race so
a slow query can't burn the whole safety-net budget. Abort signal is
respected between polls (callers should pass getRequestAbortSignal()
when running in a request handler).
Also exports SnapshotPatch and MutateSnapshotResult from
@trigger.dev/redis-worker so webapp consumers can type-check their
callers of mutateSnapshot.
The mollifier read-fallback's SyntheticRun previously carried just
enough fields for the API retrieve/trace/spans/events/attempts/metadata
endpoints. Phase C5 (replay) needs the buffered run to be passable
where ReplayTaskRunService expects a TaskRun. Adds the missing fields:
id, runtimeEnvironmentId, engine, workerQueue, queue, concurrencyKey,
machinePreset, realtimeStreamsVersion, seedMetadata, seedMetadataType,
runTags. All populated from the engine-trigger snapshot embedded in
the buffer entry.
Also closes a pre-existing typecheck gap in
ApiRetrieveRunPresenter.synthesiseFoundRunFromBuffer — workerQueue
wasn't populated and the file had been failing tsc. Now surfaces the
buffered run's workerQueue, defaulting to "main" (the Prisma default).
Phase A2/A5/A6 of the mollifier API parity work — three more read
endpoints get the buffer fallback, plus two route-level bug fixes for
endpoints that had no GET handler.
A2 spans/{spanId}: discriminated PG vs buffered findResource (mirrors
the trace endpoint pattern from A1). For buffered runs, the only valid
spanId is the snapshot's queued spanId (recorded at gate time, reused
as the run's root spanId on materialise). That spanId returns a minimal
"span exists, no execution data yet" shape; any other spanId is a
deterministic 404.
A5 attempts: pre-existing route-bug fix. The route only had `action`
(POST creates attempt); GET hit Remix's "no loader" 400 with an internal
error message. New loader returns 200 `{ attempts: [] }` for both PG
and buffered runs. The detailed attempt list belongs on the v3 retrieve
endpoint, not here.
A6 metadata GET: same pre-existing route-bug. The route only had PUT;
GET had no handler. New loader returns
`{ metadata, metadataType }` from either the PG row or the buffer
snapshot. PG-side reads only the two fields it needs.
A3 events and A4 result need no code change — events already works via
`ApiRetrieveRunPresenter.findRun`'s existing buffer fallback (querying
events for a buffered traceId naturally returns `{ events: [] }`), and
result's 404 message "Run either doesn't exist or is not finished"
already covers both buffered-not-in-PG and PG-delayed-not-finished
cases.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase A1 of the mollifier API parity work. The trace endpoint now falls
back to the mollifier buffer when the run isn't in Postgres yet, returning
an empty trace skeleton (200) instead of a 404 for buffered runs.
`findResource` is restructured into a discriminated union — `pg` for real
TaskRun rows, `buffer` for synthesised shapes from the buffer entry. The
authorization branch handles both shapes; the handler renders an empty
`{ trace: { traceId, rootSpan: null, events: [] } }` for buffered runs so
the customer sees the same 200 contract they'd get for a freshly-triggered
PG run that hasn't had its first span recorded yet.
See _plans/2026-05-19-mollifier-api-parity.md for the full plan and
_plans/2026-05-19-mollifier-listing-design.md for the read-fallback
companion infrastructure this builds on.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary
- Dashboard loaders for runs / sessions / batches / schedule-detail
threw bare `Error("X not found")` when a slug didn't resolve. Remix
surfaces this as a 500 and Sentry captures it via auto-instrumentation,
producing ongoing noise from real users following stale preview-branch
or deleted-resource links (the URLs in those Sentry events all carry
`?_data=routes/...`, i.e. client-side revalidation, not full-page
navigation).
- Added a `throwNotFound(statusText)` helper in
`app/utils/httpErrors.ts` that throws a Response with status 404,
matching the established pattern in sibling routes (agents, alerts,
bulk-actions, etc.).
- Migrated 5 loader sites to `throwNotFound` (4× "Environment not
found", 1× "Schedule not found").
- Migrated 1 loader site (`runs._index` project branch) to
`redirectWithErrorMessage("/", request, "Project not found")` to match
the pre-existing convention used by every other dashboard route's
project-not-found branch.
- Intentionally **not** touched: bare `throw new Error("X not found")`
inside `resources.*` action routes (sit inside try/catch blocks that
already redirect with a flash message), the invariant assertion in
`vercel.connect.tsx`, and the admin config check in
`admin.api.v1.runs-replication.backfill.ts`.
## Where the fix is visible
Normal browser navigation to these URLs doesn't reach the buggy loaders
— the parent env-layout
(`_app.orgs.$organizationSlug.projects.$projectParam.env.$envParam/route.tsx`)
already filters missing envs/projects and redirects/404s before the
child loader runs. The bug fires exclusively when Remix calls a single
child loader via `?_data=routes/...`, which happens during client-side
navigation or `useRevalidator`. That matches every Sentry event URL.
## Test plan
- [x] Unit test for the new helper —
`apps/webapp/test/httpErrors.test.ts`
- [x] `pnpm run typecheck --filter webapp` clean
- [x] Manual verification via Playwright on `main` vs this branch (6
cases): main returns 500 for each defective `_data` URL; branch returns
404 or 204 + `X-Remix-Redirect` as designed
- [x] Verified user-visible 404 catch boundary on `schedules/<missing>`
(the one case reachable via normal nav)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary
The S2 access-token cache key was `${basin}:${streamPrefix}` — purely
server-derived but blind to the **scope/ops list** hardcoded one method
away. When the ops list changes in code (e.g. #3644 added `trim` so
`chat.agent`'s per-turn trim chain can issue `AppendRecord.trim()`),
pre-deploy tokens still in cache get returned to SDK callers for up to
the token's TTL (24h default), surfacing as `Operation not permitted`
403s on any op outside the old scope.
## Fix
Lift the ops list to a module constant and fold its sorted-join
fingerprint into the cache key:
```ts
const S2_TOKEN_OPS = ["append", "create-stream", "trim"] as const;
const S2_TOKEN_OPS_FINGERPRINT = [...S2_TOKEN_OPS].sort().join(",");
// in getS2AccessToken
const cacheKey = `${this.basin}:${this.streamPrefix}:${S2_TOKEN_OPS_FINGERPRINT}`;
// in s2IssueAccessToken
scope: { /* ... */ ops: [...S2_TOKEN_OPS], /* ... */ }
```
The fingerprint is derived from the single source of truth, so any
future scope change auto-invalidates without anyone remembering to bump
a literal version. The Unkey L1 (in-memory LRU) and L2 (Redis) layers
share the same key derivation, so both reset together on the next deploy
with no manual cache busting.
## Test plan
- [ ] `pnpm run typecheck --filter webapp`
- [ ] Run a multi-turn `chat.agent` chat via `references/ai-chat` and
confirm no `chat.agent: trim failed; will retry next turn` warn span
fires across turn-completes.
## Summary
Companion to #3536, which patched routes that already had a leaking
`catch (e) { return json({error: e.message}, 500) }`. That pattern can't
reach routes which have no catch in the first place — when those throw,
Remix's default error path serializes `error.message` into the response
body, and the SDK then wraps the leaked string as `TriggerApiError`.
Across 28 raw api.v1 loaders/actions plus one dashboard polling
endpoint, each handler now:
- Wraps its body in `try { ... } catch (error) { ... }`.
- Re-throws `Response` instances so auth helpers' `throw json(...)` /
`throw redirect(...)` pass through unchanged.
- Logs non-Response errors via `logger.error` so server-side visibility
is preserved.
- Returns a generic body — `{"error": "Internal Server Error"}` 500 for
raw API routes, or `{ changelogs: [] }` 200 for the polling widget
(degrade silently across transient blips; the consumer hook already
coped with empty payloads).
For six routes where #3536 left an inner try/catch covering only a
service call (`alertChannels`, `batches.results`,
`deployments.finalize`, `deployments.background-workers`,
`deployments.promote`, `projects.background-workers`): an outer
try/catch is added so auth/parsing failures are also sanitized. Inner
typed-error handling (`ServiceValidationError` → 422 with message, etc.)
is preserved exactly.
For two routes whose existing catch returned 400 + `error.message`
(`api.v1.authorization-code`, `api.v1.orgs.\$orgParam.projects` action):
the body is sanitized to a generic per-route string. **Status code stays
400** — clients that key on the 4xx/5xx distinction (and the SDK's
no-retry-on-4xx behavior) are unaffected.
## Test plan
- [x] \`pnpm run typecheck --filter webapp\`
- [x] Per-route synthetic-throw probe: inject \`throw new
Error("SYNTHETIC ...")\` at the top of each catch'd try, curl the route
with a dummy bearer, confirm the response body is the generic shape and
that the synthetic message lands server-side via \`logger.error\`. 29
routes verified.
- [x] Real-P1001 probe on the envvars loader: \`docker stop database\`
mid-flight, confirm response is generic 500 (not the leaked Prisma
message).
- [x] Sampled legitimate 4xx/2xx paths across each pattern variant
(naked-wrap, partial-expanded, 400-preserved) to confirm the wraps don't
interfere with normal control flow.
Mollified runs were materialising with `TaskRun.traceContext = {}`, so every
downstream `recordRunDebugLog` (engine QUEUED/EXECUTING/FINISHED, run:notify,
attempt events) drew a fresh traceId with null parentId. The run-detail
trace view rendered only the root span; the rest of the tree was orphaned.
The pass-through path gets traceContext for free via `traceEventConcern.traceRun`
populating the W3C traceparent. The mollifier path skips that wrapper, so seed
`traceContext.traceparent` from the queued span at the call site before
handing the snapshot to engine.trigger.
Also fixes the drainer side: wrap the `mollifier.drained` span + `engine.trigger`
call in a `context.with(parentContext, ...)` built from the snapshot's
traceId/spanId. Without this `mollifier.drained` lived in a fresh trace and
the engine instrumentation inside it inherited an empty active context.
Regression tests:
- `triggerTask.test.ts` — asserts the buffered snapshot carries a valid W3C
traceparent that references the snapshot's traceId/spanId.
- `mollifierDrainerHandler.test.ts` — captures the active traceId at the
moment engine.trigger is invoked and asserts it matches the snapshot's
traceId.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary
- Introduce the Mollifier: a Redis-backed buffer for `trigger()` API
calls during traffic spikes, with a per-env trip evaluator and a drainer
ack-loop.
- Phase 1 is dual-write monitoring — every mollified trigger is buffered
to Redis AND continues to `engine.trigger`. No customer-facing behaviour
change.
- Telemetry events: `mollifier.would_mollify`, `mollifier.buffered`,
`mollifier.drained`, plus the `mollifier.decisions` counter.
- Gated behind a feature flag (default off).
## Test plan
- [x] `pnpm run test --filter @trigger.dev/redis-worker`
- [x] `pnpm run test --filter webapp -- mollifier`
- [x] Manual: with flag off, no behaviour change vs main
- [x] Manual: with flag on + threshold lowered, observe
`mollifier.buffered` + `mollifier.drained` log pairs with matching
`runId`
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Surfaces an info-style banner above the run-detail page body when the
loader resolved the run from the mollifier buffer (`isMollified === true`,
set by the read-fallback in route.tsx when the PG row hasn't been
materialised yet). Explains the queued state to the operator and points
at `batchTrigger` as the long-term shape for high-fan-out workloads.
Dismissal is localStorage-only (`mollifier_banner_dismissed`). Plan
Task 21 leaves this an explicit choice between localStorage and a new
per-org settings endpoint; localStorage is the simpler path and avoids
adding a writeable settings endpoint just for a dismissal flag. Reads
happen in an effect (not useState's initialiser) so SSR-renders the
banner visible by default, then hides it on hydration if dismissed — no
flash-of-banner.
Styled to match the existing Callout 'info' variant (blue-400 family)
without using the Callout primitive directly because Callout's API is
oriented around link-style CTAs, not inline-dismissible banners.
The three "dual-write" tests inherited from phase-1 were asserting
invariants that phase-3 deliberately abandoned when the mollify path
moved from "buffer.accept + engine.trigger" to "buffer.accept +
synthetic result, drainer replays later":
- `mollify action triggers dual-write` — rewritten to assert the new
contract: synthetic `MollifySyntheticResult` (run.friendlyId,
isCached:false, notice.code = "mollifier.queued"), buffer.accept
fires with the canonical engine.trigger snapshot, NO Postgres row
(the run materialises only when the drainer replays).
- `engine.trigger throwing AFTER buffer.accept` — deleted. Phase-3
never invokes engine.trigger on the mollify path, so the scenario
is structurally impossible.
- `debounce match produces an orphan buffer entry` — deleted. Phase-3's
C1 debounce bypass at the gate (returns pass_through for debounce
triggers) means the mollify branch is never entered for debounced
requests. The C1 invariant is pinned at mollifierGate.test.ts:440;
duplicating it at the trigger-task layer adds nothing.
Net: 6 mollifier integration tests → 4, all 4 passing, no coverage
gap (gate-level + drainer-handler-level tests own the deleted
scenarios' invariants).
Brings in every phase-2 fix without regressing the phase-3 surface.
Resolutions:
* `apps/webapp/app/v3/mollifier/mollifierDrainer.server.ts` — kept
phase-3's `MollifierDrainer<MollifierSnapshot>` generic and the
`createDrainerHandler({ engine, prisma })` + `isRetryablePgError`
wiring; added phase-2's `MollifierConfigurationError` class and
switched every env read to `TRIGGER_MOLLIFIER_*`. The doc comment on
`getMollifierDrainer` now points at `mollifierDrainerWorker.server.ts`
instead of the legacy `worker.server.ts`.
* `apps/webapp/app/runEngine/services/triggerTask.server.ts` — kept
phase-3's gate inputs (`options: { debounce, oneTimeUseToken,
parentTaskRunId, resumeParentOnCompletion }` for the C1/C3/F4
bypasses) and the `#buildEngineTriggerInput` refactor; wrapped the
gate call in phase-2's `this.isMollifierGloballyEnabled()` short-
circuit so a trigger with the feature off skips the
`GateInputs` allocation + `mollifier.decisions{outcome=pass_through}`
increment. `mollifierOutcome` is `GateOutcome | null`, downstream
access uses optional chaining. Phase-2's phase-1 dual-write block is
dropped — phase-3's mollify branch already writes via
`mollifyTrigger` and returns a synthetic result.
Phase-2 fixes preserved (verified post-merge):
- `TRIGGER_MOLLIFIER_*` env-var prefix on every consumer
- `TRIGGER_MOLLIFIER_DRAINER_ENABLED` separate per-replica switch
- `MollifierConfigurationError` rethrow in the drainer bootstrap
- Factory create + bootstrap start split (`drainer.start()` only
runs after SIGTERM handlers register)
- `signalsEmitter.on` instead of `process.once`
- `MollifierDrainer.stop` deadline-timer `clearTimeout` in finally
- `vi.fn` handler spies removed from `drainer.test.ts`
- `isMollifierGloballyEnabled` DI hook on `RunEngineTriggerTaskService`
- Legacy `worker.server.ts` header
- New `mollifierDrainerWorker.server.ts` error-classification test
- Updated `.server-changes/mollifier-burst-protection.md`
Pre-existing on phase-3, NOT introduced by this merge (verified by
`git diff origin/mollifier-phase-3 -- apps/webapp/test/engine/triggerTask.test.ts`
showing only `isMollifierGloballyEnabled: () => true` additions):
- `mollifier · mollify action triggers dual-write` asserts
`prisma.taskRun.findFirst` returns a row, but phase-3's
`mollifyTrigger` writes to the buffer and returns a synthetic
result without invoking `engine.trigger`. Test name + assertions
still reflect phase-1 dual-write semantics.
- `mollifier · engine.trigger throwing AFTER buffer.accept` expects
the call to reject; phase-3 returns synthetic before
`engine.trigger` is reached, so the injected throw never fires.
- `mollifier · debounce match produces an orphan buffer entry` —
phase-3 explicitly skips the buffer branch when debounce is set
(`\!body.options?.debounce` guard), so no orphan accept happens.
These three tests describe phase-1 monitoring behaviour on phase-3
code that abandoned dual-write. Updating them is phase-3's own tech
debt, separate from this merge.
## Summary
Long-running chat agents were filling `session.out` forever — every
`chat.agent` turn appended to the same S2 stream with no trim, and the
Sessions dashboard re-streamed the entire history from `seq_num=0` on
every page load. After this change the agent appends an S2 `trim`
command record after each `trigger:turn-complete`, pointing back at the
previous turn-complete's seq_num. `session.out` stays roughly one turn
long at steady state, regardless of session age.
`trigger:turn-complete` and `trigger:upgrade-required` move from
`chunk.type`-shaped data records into header-form control records under
a uniform `trigger-control` namespace. Built-in transports
(`TriggerChatTransport`, `AgentChat`, the dashboard's `AgentView`)
handle the new shape transparently. Custom transports need a one-line
filter on the `trigger-control` header — see the rewritten "Records on
session.out" section in the client-protocol docs.
The Sessions detail page in the dashboard fetches the agent's per-turn
S3 snapshot via a presigned URL and seeds the transcript view, then
SSE-tails from the snapshot's `lastOutEventId`. Bandwidth and
time-to-first-render scale with unread turns instead of session
lifetime.
Resume contract is now explicit: single-turn-boundary resume always
works (the prior turn-complete is still on the stream), the S2 trim is
eventually consistent over 10-60s, and multi-turn-away resume falls back
to a snapshot reload.
## Summary
The PUT handler at `/realtime/v1/streams/:runId/:target/:streamId` ran
`taskRun.update({ realtimeStreams: { push: streamId } })` on every call,
even when the `streamId` was already present. SDK call patterns that
re-initialize the same stream key on every chunk produce a per-write row
UPDATE, duplicate entries pile up in the array, and the row-lock + TOAST
rewrite cost grows unbounded on long-running stream sessions.
## Fix
Mirror the sibling append handler: read the array first and only push
when the `streamId` isn't already present. Identical behavior for
first-time stream creation; repeat creates short-circuit to a single
indexed read. The dashboard's per-run stream listing keeps working
because the first create still records the entry.
## Test plan
- [ ] A fresh PUT for a new `(run, streamId)` adds the entry to the
array
- [ ] A repeat PUT for the same pair leaves the array unchanged
- [ ] 404 is returned when the run doesn't exist; 400 when the run is
completed
Adds the smallest DI surface to `initMollifierDrainerWorker` (`isEnabled`
and `getDrainer`, both optional, default to live env/singleton) so the
catch-block policy can be tested without manipulating module-level env:
- rethrows MollifierConfigurationError — deterministic misconfig
escapes, which is what makes the production-path crash on boot
(the call site in entry.server.tsx runs sync at module top level,
before `process.on("uncaughtException", ...)` is registered, so an
escape becomes a Node default-handler exit-1).
- rethrows when `name === "MollifierConfigurationError"` even when
`instanceof` fails — covers the Remix dev hot-reload realm edge
case where the catch holds a stale class reference.
- swallows non-configuration errors — a transient Redis blip during
buffer init shouldn't take the whole webapp down.
- no-op when disabled — the factory isn't invoked when the enabled
predicate returns false.
Also updates the existing mollifier server-changes note to: rename env
vars to TRIGGER_MOLLIFIER_* prefix, document the TRIGGER_MOLLIFIER_DRAINER_ENABLED
split for multi-replica drainer placement, and call out the new fail-loud
behaviour on drainer misconfiguration.
The bootstrap in mollifierDrainerWorker.server.ts wrapped getMollifierDrainer()
in a try/catch that logged-and-continued on any error, which absorbed the two
designed-to-crash throws in initializeMollifierDrainer():
- "MollifierDrainer initialised without a buffer" (missing buffer client)
- "TRIGGER_MOLLIFIER_DRAIN_SHUTDOWN_TIMEOUT_MS must be at least ... below
GRACEFUL_SHUTDOWN_TIMEOUT" (shutdown-timeout reconciliation)
Both are deploy-time mistakes: silently disabling the drainer means the
gate keeps writing to the buffer, the drainer never reads, and entries
TTL out in 10min. Bounded in phase 1 (monitoring-only) but customer-
visible data loss in phase 2/3 where the drainer replays into engine.trigger.
Better to fail loud now than retrofit the contract later.
Introduce MollifierConfigurationError for the two deterministic throws.
The bootstrap's catch now rethrows that class (process crashes at module
top-level → orchestrator health check fails → deploy rolls back) while
still logging-and-continuing on transient errors (Redis blip during init
shouldn't take the whole webapp down). instanceof + name fallback covers
the Remix dev hot-reload realm edge case.
## Summary
- Adds a `beforeSend` rule in `apps/webapp/sentry.server.ts` that
collapses Prisma `P1001` ("Can't reach database server") errors into a
single Sentry issue regardless of which call site threw, by setting
`event.fingerprint = ["prisma-p1001-db-unreachable"]` and tagging
`db_unreachable:true`.
- Matches both `err.code === "P1001"` (Prisma's `KnownRequestError` when
a connection drops mid-query) and `err.errorCode === "P1001"`
(`InitializationError` when the client fails to connect at startup).
- Implemented as a small extensible `FINGERPRINT_RULES` table so further
fan-out errors can be added with one entry.
## Verification
End-to-end verified locally with `debug: true` on the SDK:
- Real Prisma `P1001` thrown from a loader (DB stopped mid-request) is
captured by Sentry's Remix auto-instrumentation
- `beforeSend` fires with `originalException.code === "P1001"`, rule
matches
- `event.fingerprint = ["prisma-p1001-db-unreachable"]` and
`tags.db_unreachable = "true"` applied
- Event lands in Sentry under the new fingerprint
## Test plan
- [ ] Deploy to staging; confirm P1001 events appear under a single
`prisma-p1001-db-unreachable` issue rather than fanning out
- [ ] Confirm `db_unreachable:true` tag is filterable in Sentry
- [ ] Verify non-P1001 errors are unaffected (event passes through
`beforeSend` untouched)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The previous commit added a perf short-circuit at the call site that
read `env.TRIGGER_MOLLIFIER_ENABLED` directly. That broke three
mollifier integration tests in CI: the tests inject a custom
`evaluateGate` via the existing DI seam expecting the buffer-write
branch to be reached, but CI has no `.env` (the `apps/webapp/.env`
symlink target is absent), the Zod default `"0"` wins, the call site
short-circuits to `null` before the injected gate runs, and
`buffer.accepted` stays empty.
Make the global-enabled check itself injectable:
- New constructor opt `isMollifierGloballyEnabled?: () => boolean`,
defaulting to `() => env.TRIGGER_MOLLIFIER_ENABLED === "1"`. Each
DI hook now represents one decision (gate, buffer, global-enabled),
so a test that wants the buffer-write branch reached can inject
`isMollifierGloballyEnabled: () => true` alongside its custom gate.
- Call site now reads `this.isMollifierGloballyEnabled()` instead of
`env.TRIGGER_MOLLIFIER_ENABLED` directly. In production, with no DI
override, the default closure resolves `env` exactly once per call
just as before — same perf win when the flag is off.
- All six mollifier DI injection sites in triggerTask.test.ts now also
pass `isMollifierGloballyEnabled: () => true` so the tests' DI
surface matches the new contract regardless of CI env state.
evaluateGate ran on every trigger regardless of TRIGGER_MOLLIFIER_ENABLED.
With the flag off (the default everywhere it hasn't been opted in), the
gate still produced a `pass_through` decision after allocating a
GateInputs object, spreading defaultGateDependencies inside evaluateGate,
and incrementing the `mollifier.decisions{outcome=pass_through}` OTel
counter. Cheap individually, but triggerTask is the hottest code path in
the system — multiply by trigger rate and the unnecessary work compounds.
Guard the gate call with a direct env.TRIGGER_MOLLIFIER_ENABLED check at
the call site. When the flag is off, mollifierOutcome is null and the
downstream `mollifierOutcome?.action === "mollify"` branch skips the
buffer dual-write entirely — zero allocation, zero counter increment on
the disabled path. When the flag is on, behaviour is unchanged.
Lost-signal note: with mollifier off, we no longer count "pass_through"
decisions in the OTel counter (the gate never runs). That's a non-issue
— "pass_through count when feature is off" is just total trigger rate,
which is already observable via the trigger handler's own spans/counters
upstream. The gate counter remains the source of truth for the
mollify/shadow/pass_through ratio when the feature is on, which is the
load-bearing signal.
`process.once("SIGTERM", stopDrainer)` was the odd one out — every
other webapp service (runsReplicationInstance, llmPricingRegistry,
dynamicFlushScheduler, marqs, eventLoopMonitor) registers through
`signalsEmitter` from `~/services/signals.server`, an EventEmitter
backed by a single `process.on()` that fans out to all listeners.
Switching gets us:
- codebase consistency;
- `.on` (not `.once`) so a second SIGTERM, if the orchestrator emits
one before SIGKILL, still reaches us;
- if SIGTERM lands in the narrow gap between the listener attaching
and drainer.start() below, the first invocation no-ops (stop()
returns early because isRunning is false) but the listener stays
attached for any subsequent signal, instead of being consumed and
leaving the now-running drainer with no graceful-stop path.
## Summary
Adds a Region column and Region filter (under More filters) to the runs
list dashboard, the same filter on the public runs list API
(`filter[region]`), and a matching `region` input on the MCP `list_runs`
tool. Each run's executing region is also surfaced as a new optional
`region` field on the runs list and run retrieve responses, populated
from the worker instance group's `masterQueue` identifier.
Useful when you run tasks across multiple regions and want to slice the
runs list — or your existing run-querying scripts — by where the run
actually executed.
## Design
The filter value in the URL / API is the `masterQueue` identifier (the
same string already persisted on `TaskRun` and replicated to ClickHouse
as `worker_queue`), so the query just becomes `worker_queue IN (...)`
with no server-side translation. The Region dropdown options come from a
new resource loader backed by `RegionsPresenter`, which now also exposes
`masterQueue` alongside the existing region metadata.
```ts
// public API
const runs = await runs.list({ region: ["us-east-1", "eu-west-1"] });
// each item: { id, status, ..., region?: "us-east-1" }
```
```ts
// MCP
list_runs({ environment: "prod", region: "us-east-1" })
```
All MOLLIFIER_* env vars renamed to TRIGGER_MOLLIFIER_*. The mollifier
primitive is generic — buffer + drainer + trip evaluator with no
trigger-specific assumptions at the redis-worker layer — but this
PR's webapp wiring is specifically the trigger-task mollifier, with
PII-sensitive payload handling and trigger-flow semantics. If we later
mollify another surface (deploys, schedules, etc.) those will want
their own env-var namespace; pre-prefixing now avoids a breaking
rename later.
Renames are mechanical: schema keys in env.server.ts, env.* references
across the v3/mollifier* modules, and a handful of doc-comment
mentions. The bootstrap fallback that has DRAINER_ENABLED default to
the ENABLED value is updated to read TRIGGER_MOLLIFIER_ENABLED from
process.env too. Code-side naming (classes, file names, the literal
word "mollifier") stays unchanged — the rename is env-var only.
The drainer's polling loop has been gated on WORKER_ENABLED, which
couples it to the legacy ZodWorker role. To split the drainer onto a
dedicated worker service in cloud (and keep all other replicas as
producer-only), introduce its own switch.
Semantics:
- Unset → inherits MOLLIFIER_ENABLED.
Single-container self-hosters with MOLLIFIER_ENABLED=1 get the
drainer for free, no second flag to remember.
- Explicit MOLLIFIER_DRAINER_ENABLED=0 → drainer off on this replica.
Cloud sets this everywhere except the dedicated drainer service.
- Explicit MOLLIFIER_DRAINER_ENABLED=1 → drainer on, subject to
MOLLIFIER_ENABLED still being the master kill switch (a drainer
can't construct without the gate-side buffer singleton).
The bootstrap in mollifierDrainerWorker.server.ts now gates on the new
flag instead of WORKER_ENABLED, so the drainer's lifecycle is no longer
coupled to the legacy worker role.
worker.server.ts is the original graphile-worker / ZodWorker file —
every task in its catalog is annotated "@deprecated, moved to
commonWorker.server.ts" (or similar). Adding new lifecycle wiring
there during phase-2 was a mis-routing.
Move the SIGTERM/SIGINT registration + drainer.start() call into a new
mollifierDrainerWorker.server.ts alongside the redis-worker workers,
and invoke its initMollifierDrainerWorker() from entry.server.tsx
right after Worker.init(). The drainer's own factory still validates
shutdown timeouts before constructing; the bootstrap registers signal
handlers BEFORE calling start(), preserving the create+start contract.
Also adds a header to worker.server.ts marking it legacy and pointing
new lifecycle code at the redis-worker pattern, so the next person
doesn't have to re-derive the routing rule.
Dan Sutton