Three containerTest cases covering the novel C1 piece — the rest of
the Phase C work has unit-test coverage already.
1. Writes CANCELED PG row with snapshot fields, completedAt set to
cancelledAt, error.raw set to cancelReason, runTags / taskIdentifier
/ payload preserved.
2. Emits runCancelled with full payload (id, friendlyId, status, error,
organization / project / environment ids).
3. Idempotent on double-pop: second call after the first returns the
existing row id (P2002 caught) and does not re-emit the event.
Real PG + Redis testcontainers. ~15s total.
Adds per-endpoint contract checks beyond the status-only comparison:
- Read endpoints assert response shape (trace.traceId present;
events/attempts arrays; metadata-get { metadata, metadataType }
keys; retrieve-v3 carries id + taskIdentifier + status). The result
endpoint explicitly asserts 404 — its accidental-but-correct
pre-Phase-A behaviour is now the locked contract.
- Mutation endpoints get a read-back assertion: after PUT metadata,
re-read and confirm the snapshot reflects the patch. After POST
tags, retrieve and confirm runTags contains the new tag. Catches
the case where the API returns 200 but the snapshot didn't actually
patch.
- Replay asserts the response carries a new run_-prefixed id.
- New listing probe: hits /api/v1/runs and asserts the buffered runId
is present in the page. Locks in Phase E's listing-merge behaviour.
Script remains backwards-compatible — same exit codes, same env-var
contract. Drift count now reflects shape violations alongside status
divergences.
Q1 ZSET-merge design lands.
redis-worker side:
- MollifierBuffer.listForEnvWithWatermark — paginated newest-first
read of buffered entries, bounded by a (createdAtMicros, runId)
watermark. ZREVRANGEBYSCORE strictly below the watermark score plus
a tied-score band scan for entries sharing the watermark's
createdAtMicros.
webapp side:
- listingMerge.server.ts: callRunListWithBufferMerge wraps
ApiRunListPresenter. Fetches a buffer page, synthesises each entry
into the presenter's ListDataItem shape (status QUEUED, timestamps
from entry hash, env slug looked up once), forwards the inner
cursor to the presenter, merges by createdAt DESC with runId DESC
tiebreak, truncates to pageSize. Compound base64-JSON cursor
{ inner, watermark, bufferExhausted } is backwards-compatible with
legacy opaque cursors.
- api.v1.runs.ts + api.v1.projects.{projectRef}.runs.ts route through
the wrapper. Project route extracts envId from filter[env]; absent
that, falls back to the bare presenter (existing behaviour).
- Buffer eligibility skips for filters that can't match buffered runs
(status not in QUEUED/PENDING/DELAYED, batch/schedule/version/
region/machine filters). Buffer outages fall open to PG-only.
- Delete RecentlyQueuedSection banner + listEntriesForEnv loader call
from dashboard runs index — buffered runs appear inline as QUEUED
rows.
Parallels Phase C's API-side work for the three dashboard mutation
routes.
D1 cancel — PG miss → buffer.mutateSnapshot('mark_cancelled'). Org-
membership verified against the buffered run's orgId (dashboard URL
doesn't carry an envId so the API-side env-scoped auth doesn't apply).
busy returns a "retry in a moment" message.
D2 replay — PG miss → findRunByIdWithMollifierFallback; B4-extended
SyntheticRun cast to TaskRun and fed to ReplayTaskRunService.
Project/env slugs for the redirect path looked up from the entry's
envId.
D3 idempotencyKey reset — PG miss → buffer.getEntry + readFallback to
read snapshot's idempotencyKey + taskIdentifier; org-membership
verified against entry orgId; existing ResetIdempotencyKeyService
(extended in B6b to clear both stores) handles the actual reset.
Closes the last API-parity gap in the master plan.
redis-worker side:
- New casSetMetadata Lua command with optimistic lock on a
metadataVersion entry-hash field. Returns applied / version_conflict /
not_found / busy. Mirrors the PG-side UpdateMetadataService's CAS
loop so concurrent metadata.increment / metadata.set / metadata.append
calls against a buffered run never lose deltas.
- accept Lua initialises metadataVersion=0; BufferEntrySchema gains
the field.
webapp side:
- applyMetadataMutationToBufferedRun helper does the read-apply-CAS-
retry loop in JS, reusing the existing @trigger.dev/core
applyMetadataOperations function (no Lua re-implementation of the 6
operation types).
- metadata PUT route does PG-first via the existing service (which
owns the full request shape: parent/root ops, batching, validation),
then falls through to the buffer helper on PG miss. busy and
version_exhausted return 503 with retry hint; not_found returns 404.
- Parent/root operations on a buffered target are fanned out to the
snapshot's parentTaskRunId via the existing service. If the parent
is also buffered the helper recurses. Best-effort — parent/root
ingestion failures do not surface to the caller.
Tests: 3 new redis-worker tests covering CAS apply / version conflict /
not_found-busy paths. All 71 redis-worker mollifier + 68 webapp
mollifier tests green.
Reschedule (C4): switches to mutateWithFallback. PG hits go through
the existing RescheduleTaskRunService (which enforces status ===
"DELAYED"). Buffered hits land a set_delay patch on the snapshot;
the drainer materialises the PG row with the new delayUntil. Synth-
esised response returns { id, delayUntil }.
Replay (C5): adds a read-fallback after the PG miss. The B4-extended
SyntheticRun carries every field ReplayTaskRunService reads from a
TaskRun, so the buffered case casts through and uses the existing
service unchanged. Replay creates a fresh trigger that itself
re-enters the mollifier gate — no special surge handling needed
beyond what the gate already does. Also tightens the PG lookup to
findFirst with runtimeEnvironmentId scoping (was findUnique on
friendlyId only).
Closes the live 500 the parity script flagged. The previous route did
prisma.taskRun.update after a findFirst that could miss; on buffered
runs (no PG row yet) the update raised RecordNotFound and surfaced as
a 500.
Switches to mutateWithFallback. PG hits go through the existing
select-dedupe-validate-update flow with MAX_TAGS_PER_RUN enforcement.
Buffered-QUEUED hits apply append_tags via Lua (atomic dedup against
existing snapshot tags). busy snapshots wait for drainer resolution
then update PG. 404 / 503 surface for missing / hung cases.
The MAX_TAGS_PER_RUN cap is skipped on the buffered side — the
drainer's engine.trigger doesn't enforce it either, matching the
pre-buffer trigger path. Pushing the cap into the snapshot-mutate Lua
is a possible follow-up.
Per the Q4 mollifier-cancel design — first mutation endpoint.
engine.createCancelledRun: new run-engine method that writes a CANCELED
TaskRun row directly from a buffer snapshot. Skips queue insertion,
waitpoint creation, and concurrency reservation (run never executes).
Emits runCancelled so the existing handler writes the TaskEvent
cancellation row. P2002 from double-pop is caught and returns the
existing row without re-emitting.
Drainer bifurcation: mollifierDrainerHandler routes to
createCancelledRun when snapshot.cancelledAt is set. Cancel-wins-
over-trigger — customer intent is terminal.
Cancel route: wraps the call in mutateWithFallback. PG-row hits go
through the existing CancelTaskRunService. Buffered-QUEUED hits land
a mark_cancelled patch on the snapshot via mutateSnapshot. busy
snapshots wait for drainer resolution then call the PG service
against the resulting row. 404 / 503 surface for genuine missing
or drainer-hung cases.
Known follow-up: the Q3 wait-and-bounce for cancel-of-buffered-FAILED
relies on the drainer eventually writing a SYSTEM_FAILURE PG row on
terminal materialisation failure. That drainer-side write isn't
implemented yet (the failed-drain path today only marks the buffer
entry hash FAILED). Cancel-of-state-3 will currently 503 after 2s
instead of returning the SYSTEM_FAILURE row. Acceptable rare-race
behaviour; flagged for a follow-up alongside the drainer sweeper work.
Three integration points that connect B6a's buffer-side primitives to
the customer-facing flow per Q5:
- IdempotencyKeyConcern.handleTriggerRequest falls through to
buffer.lookupIdempotency after a PG miss. Buffered hits return
isCached:true with a synthesised TaskRun via the existing
findRunByIdWithMollifierFallback. Skipped when
resumeParentOnCompletion is set: waitpoint blocking requires a PG
row that doesn't exist yet; the follow-up accept SETNX still
dedupes the trigger itself. Buffer outages fail open to "no cache
hit" so the trigger hot path is never wedged by a transient Redis
issue.
- mollifyTrigger passes idempotencyKey + taskIdentifier through to
buffer.accept. The SETNX race loser receives duplicate_idempotency
with the winner's runId; the API response echoes it with
isCached:true, matching PG-side cache-hit shape.
- ResetIdempotencyKeyService calls buffer.resetIdempotency alongside
the existing PG updateMany. 404 only fires when both stores report
nothing bound. Buffer outage during reset is logged and treated as
a miss; PG-side reset still works.
Buffer-side counterpart to PG's idempotency-key uniqueness, per the Q5
mollifier-idempotency design. Three changes to the buffer's atomic
Lua surface plus two new high-level methods.
acceptMollifierEntry: when the caller passes idempotencyKey + task-
Identifier, SETNX a `mollifier:idempotency:{env}:{task}:{key}` lookup
pointing at the runId. Second accepts for the same tuple return the
existing winner's runId so the loser's response can echo it as a
cached hit. accept's return shape changes from boolean to a discrim-
inated AcceptResult (accepted / duplicate_run_id / duplicate_idemp-
otency). Existing four callers that ignored the boolean continue to
work; one assertion was updated for the new shape in tests.
ackMollifierEntry: DELs the idempotency lookup atomically with
marking the entry materialised. PG becomes canonical post-mater-
ialisation; the lookup TTL is the safety net if the DEL is missed.
New lookupIdempotency: resolves a buffered run by (env, task, key)
tuple. Used by IdempotencyKeyConcern in B6b. Self-heals stale lookups
that point at expired entries.
New resetIdempotency: atomic Lua that nulls idempotencyKey +
idempotencyKeyExpiresAt on the snapshot payload, clears the
denormalised hash pointer, and DELs the lookup. Used by
ResetIdempotencyKeyService in B6b alongside the PG-side updateMany.
BufferEntrySchema gains an optional idempotencyLookupKey string field
(empty when no idempotency key was bound) so the ack Lua can DEL the
lookup without reading the payload JSON.
8 new tests cover: lookup write+TTL, duplicate_idempotency return,
lookupIdempotency hit/miss/self-heal, ack-DELs-lookup, reset clears
both stores, reset null when nothing bound.
Composes PG-first (replica) lookup, MollifierBuffer.mutateSnapshot,
and writer-side spin-wait into the Q3 wait-and-bounce flow. Returns
a discriminated outcome rather than throwing Response, so the helper
stays route-agnostic and unit-testable. Phase C mutation endpoints
(tags, metadata-put, reschedule, cancel) consume this in upcoming
commits.
Wait knobs default to safetyNetMs=2000, pollStepMs=20, pgTimeoutMs=50
per Q3. Each PG poll is bounded by pgTimeoutMs via Promise.race so
a slow query can't burn the whole safety-net budget. Abort signal is
respected between polls (callers should pass getRequestAbortSignal()
when running in a request handler).
Also exports SnapshotPatch and MutateSnapshotResult from
@trigger.dev/redis-worker so webapp consumers can type-check their
callers of mutateSnapshot.
The mollifier read-fallback's SyntheticRun previously carried just
enough fields for the API retrieve/trace/spans/events/attempts/metadata
endpoints. Phase C5 (replay) needs the buffered run to be passable
where ReplayTaskRunService expects a TaskRun. Adds the missing fields:
id, runtimeEnvironmentId, engine, workerQueue, queue, concurrencyKey,
machinePreset, realtimeStreamsVersion, seedMetadata, seedMetadataType,
runTags. All populated from the engine-trigger snapshot embedded in
the buffer entry.
Also closes a pre-existing typecheck gap in
ApiRetrieveRunPresenter.synthesiseFoundRunFromBuffer — workerQueue
wasn't populated and the file had been failing tsc. Now surfaces the
buffered run's workerQueue, defaulting to "main" (the Prisma default).
Atomic Lua-driven snapshot mutation for the burst-buffer entry hash.
Returns one of three result codes per Q3:
- applied_to_snapshot: entry was QUEUED + not materialised; the
drainer will see the patched payload on its next pop.
- not_found: no entry hash for this runId.
- busy: entry is DRAINING / FAILED / materialised — the caller
wait-and-bounces through PG (helper lands in B5).
Four patch types:
- append_tags: union-merges into payload.tags, dedupes against
existing values.
- set_metadata: replaces metadata + metadataType (last-write-wins).
- set_delay: replaces payload.delayUntil.
- mark_cancelled: stamps cancelledAt + cancelReason; the drainer
bifurcation in Q4 reads these on next pop.
10 new tests cover: not_found, all four patch types (success +
absent-field handling), each busy state (DRAINING, FAILED,
materialised), and per-runId atomicity under 50-way concurrent
appends.
`MollifierBuffer.ack` previously deleted the entry hash. It now sets
`materialised=true` and resets the TTL to a 30s grace window via a new
atomic `ackMollifierEntry` Lua script. The entry hash persists past
materialisation as a read-fallback safety net for the brief PG replica
lag window between drainer-side write and reader-side visibility (Q1
D2).
`BufferEntrySchema` gains an optional `materialised` boolean (string
"true"/"false" in Redis → boolean in JS). Accept still refuses while
*any* entry exists for the runId — including materialised ones — as
defense-in-depth against runId reuse.
The drainer's "drains one queued entry … and acks" test now asserts
`materialised=true` instead of entry deletion. The "re-accept after
ack works" test is inverted to "accept refused while a previously-acked
entry is still inside its grace TTL".
Master plan Progress tracking now reflects 709d2f5af. Captures the
score-equals-createdAt invariant decision (requeue keeps original
score; createdAt is immutable across retries) so future sessions don't
relitigate the Q1 underspec.
Per-env queue `mollifier:queue:{envId}` switches from a Redis LIST
(LPUSH/RPOP) to a sorted set keyed by `createdAtMicros`. Pop semantics
are unchanged (FIFO by creation time, now via ZPOPMIN). Entry hashes
carry a new `createdAtMicros` field equal to the score.
Requeue keeps the original score — createdAt is immutable across
retries, so a retried entry continues to pop next by virtue of being
the oldest. `maxAttempts` in the drainer bounds the retry loop. The
inverted "FIFO retry" test reflects the new (correct) semantics under
the score-equals-createdAt invariant.
`listEntriesForEnv` reads via ZREVRANGE (newest-first). Orphan-handling
tests that injected via LPUSH now use ZADD; queue-depth assertions
switch from LLEN to ZCARD.
This is the substrate for the listing pagination work in Phase E and
for the snapshot-mutate work in Phase B3.
Hardens the master plan against context-loss between sessions:
- Progress table mapping each phase to commits + status.
- Phase A patterns section documenting the discriminated-union
findResource shape that A1 and A2 established (reusable for Phase
B/C/D).
- Explicit note on what SyntheticRun has today vs what Phase C's
replay work will need extended.
- Phase B's exact order spelled out (B1-B6) referencing the locked
Q-docs.
- A "resuming guidance" footer for a fresh session.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase A2/A5/A6 of the mollifier API parity work — three more read
endpoints get the buffer fallback, plus two route-level bug fixes for
endpoints that had no GET handler.
A2 spans/{spanId}: discriminated PG vs buffered findResource (mirrors
the trace endpoint pattern from A1). For buffered runs, the only valid
spanId is the snapshot's queued spanId (recorded at gate time, reused
as the run's root spanId on materialise). That spanId returns a minimal
"span exists, no execution data yet" shape; any other spanId is a
deterministic 404.
A5 attempts: pre-existing route-bug fix. The route only had `action`
(POST creates attempt); GET hit Remix's "no loader" 400 with an internal
error message. New loader returns 200 `{ attempts: [] }` for both PG
and buffered runs. The detailed attempt list belongs on the v3 retrieve
endpoint, not here.
A6 metadata GET: same pre-existing route-bug. The route only had PUT;
GET had no handler. New loader returns
`{ metadata, metadataType }` from either the PG row or the buffer
snapshot. PG-side reads only the two fields it needs.
A3 events and A4 result need no code change — events already works via
`ApiRetrieveRunPresenter.findRun`'s existing buffer fallback (querying
events for a buffered traceId naturally returns `{ events: [] }`), and
result's 404 message "Run either doesn't exist or is not finished"
already covers both buffered-not-in-PG and PG-delayed-not-finished
cases.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase A1 of the mollifier API parity work. The trace endpoint now falls
back to the mollifier buffer when the run isn't in Postgres yet, returning
an empty trace skeleton (200) instead of a 404 for buffered runs.
`findResource` is restructured into a discriminated union — `pg` for real
TaskRun rows, `buffer` for synthesised shapes from the buffer entry. The
authorization branch handles both shapes; the handler renders an empty
`{ trace: { traceId, rootSpan: null, events: [] } }` for buffered runs so
the customer sees the same 200 contract they'd get for a freshly-triggered
PG run that hasn't had its first span recorded yet.
See _plans/2026-05-19-mollifier-api-parity.md for the full plan and
_plans/2026-05-19-mollifier-listing-design.md for the read-fallback
companion infrastructure this builds on.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Master plan and five locked sub-design docs covering the API parity work:
- mollifier-api-parity.md — endpoint inventory, invariant, phased TDD plan.
- mollifier-listing-design.md (Q1) — ZSET buffer, compound cursor, no banner.
- mollifier-replay-design.md (Q2) — single code path, PG-or-buffer resolution.
- mollifier-mutation-race-design.md (Q3) — wait-and-bounce with safety net.
- mollifier-cancel-design.md (Q4) — mark_cancelled + drainer bifurcation.
- mollifier-idempotency-design.md (Q5) — keys in both stores symmetrically.
Plus the original phase-3 plan it builds on and the bash parity script
that surfaced the gaps and acts as the regression guard during
implementation.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary
- Dashboard loaders for runs / sessions / batches / schedule-detail
threw bare `Error("X not found")` when a slug didn't resolve. Remix
surfaces this as a 500 and Sentry captures it via auto-instrumentation,
producing ongoing noise from real users following stale preview-branch
or deleted-resource links (the URLs in those Sentry events all carry
`?_data=routes/...`, i.e. client-side revalidation, not full-page
navigation).
- Added a `throwNotFound(statusText)` helper in
`app/utils/httpErrors.ts` that throws a Response with status 404,
matching the established pattern in sibling routes (agents, alerts,
bulk-actions, etc.).
- Migrated 5 loader sites to `throwNotFound` (4× "Environment not
found", 1× "Schedule not found").
- Migrated 1 loader site (`runs._index` project branch) to
`redirectWithErrorMessage("/", request, "Project not found")` to match
the pre-existing convention used by every other dashboard route's
project-not-found branch.
- Intentionally **not** touched: bare `throw new Error("X not found")`
inside `resources.*` action routes (sit inside try/catch blocks that
already redirect with a flash message), the invariant assertion in
`vercel.connect.tsx`, and the admin config check in
`admin.api.v1.runs-replication.backfill.ts`.
## Where the fix is visible
Normal browser navigation to these URLs doesn't reach the buggy loaders
— the parent env-layout
(`_app.orgs.$organizationSlug.projects.$projectParam.env.$envParam/route.tsx`)
already filters missing envs/projects and redirects/404s before the
child loader runs. The bug fires exclusively when Remix calls a single
child loader via `?_data=routes/...`, which happens during client-side
navigation or `useRevalidator`. That matches every Sentry event URL.
## Test plan
- [x] Unit test for the new helper —
`apps/webapp/test/httpErrors.test.ts`
- [x] `pnpm run typecheck --filter webapp` clean
- [x] Manual verification via Playwright on `main` vs this branch (6
cases): main returns 500 for each defective `_data` URL; branch returns
404 or 204 + `X-Remix-Redirect` as designed
- [x] Verified user-visible 404 catch boundary on `schedules/<missing>`
(the one case reachable via normal nav)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary
Five hardening fixes across `@trigger.dev/sdk`, `@trigger.dev/core`, and
`@trigger.dev/build`.
- `tasks.triggerAndSubscribe` now forwards caller `requestOptions`
(custom API keys, per-request overrides) to the underlying
`apiClient.triggerTask` call instead of silently dropping them.
- `SSEStreamSubscription` no longer retries permanent client errors
forever. The default `nonRetryableStatuses` widens from `[404, 410]` to
`[400, 404, 409, 410, 422]`, so a malformed session-stream request fails
fast instead of busy-looping under bounded backoff.
- Session writer falls back to manually wiring the caller's
`AbortSignal` on Node 18, where `AbortSignal.any` is unavailable.
Caller-driven cancellation now propagates on every supported runtime.
- `TriggerChatTransport` throws immediately when a `chat.handover`
response is missing `X-Trigger-Chat-Access-Token`, instead of silently
downgrading every subsequent turn back to the handover path. `dispose()`
aborts every active `session.out` subscription before tearing the
coordinator down, so unmount/navigation no longer leaves SSE readers in
flight.
- Removed the experimental `@trigger.dev/build/extensions/secureExec`
build extension. It will return alongside the sandbox feature it was
built to support.
## Test plan
- [ ] `pnpm run build --filter @trigger.dev/sdk --filter
@trigger.dev/core --filter @trigger.dev/build`
- [ ] `pnpm --filter @trigger.dev/sdk test --run` (183 tests, including
chat / chat-server / sessions / handover)
- [ ] `pnpm --filter @trigger.dev/core test --run`
- [ ] Manually trigger a `chat.handover` whose response strips
`X-Trigger-Chat-Access-Token`, and confirm the transport throws
synchronously rather than degrading.
- [ ] Unmount a chat UI mid-stream and confirm the active `session.out`
SSE connection closes immediately.
## Summary
The S2 access-token cache key was `${basin}:${streamPrefix}` — purely
server-derived but blind to the **scope/ops list** hardcoded one method
away. When the ops list changes in code (e.g. #3644 added `trim` so
`chat.agent`'s per-turn trim chain can issue `AppendRecord.trim()`),
pre-deploy tokens still in cache get returned to SDK callers for up to
the token's TTL (24h default), surfacing as `Operation not permitted`
403s on any op outside the old scope.
## Fix
Lift the ops list to a module constant and fold its sorted-join
fingerprint into the cache key:
```ts
const S2_TOKEN_OPS = ["append", "create-stream", "trim"] as const;
const S2_TOKEN_OPS_FINGERPRINT = [...S2_TOKEN_OPS].sort().join(",");
// in getS2AccessToken
const cacheKey = `${this.basin}:${this.streamPrefix}:${S2_TOKEN_OPS_FINGERPRINT}`;
// in s2IssueAccessToken
scope: { /* ... */ ops: [...S2_TOKEN_OPS], /* ... */ }
```
The fingerprint is derived from the single source of truth, so any
future scope change auto-invalidates without anyone remembering to bump
a literal version. The Unkey L1 (in-memory LRU) and L2 (Redis) layers
share the same key derivation, so both reset together on the next deploy
with no manual cache busting.
## Test plan
- [ ] `pnpm run typecheck --filter webapp`
- [ ] Run a multi-turn `chat.agent` chat via `references/ai-chat` and
confirm no `chat.agent: trim failed; will retry next turn` warn span
fires across turn-completes.
## Summary
The prerelease (snapshot) path of the release workflow fails immediately
whenever `main` carries an active `.changeset/pre.json` (i.e. during an
in-progress RC cycle, like the current v4 RC):
```
🦋 error Snapshot release is not allowed in pre mode
🦋 To resolve this exit the pre mode by running `changeset pre exit`
```
This blocks `chat-prerelease` snapshots from main even though the
snapshots are unrelated to the RC cycle.
Adds a conditional `changeset pre exit` step right before `Snapshot
version` in the prerelease job. The job runs on a checkout with
`persist-credentials: false`, so the `pre.json` deletion stays on the
runner's working tree — main's persisted pre-mode state is untouched,
and v4 RC publishes keep working normally.
## Test plan
- [ ] Re-run the `🦋 Changesets Release` workflow with `type=prerelease`,
`ref=main`, `prerelease_tag=chat-prerelease` and confirm it gets past
the snapshot step and publishes.
- [ ] Confirm `.changeset/pre.json` on `main` is unchanged after the
run.
## Summary
Companion to #3536, which patched routes that already had a leaking
`catch (e) { return json({error: e.message}, 500) }`. That pattern can't
reach routes which have no catch in the first place — when those throw,
Remix's default error path serializes `error.message` into the response
body, and the SDK then wraps the leaked string as `TriggerApiError`.
Across 28 raw api.v1 loaders/actions plus one dashboard polling
endpoint, each handler now:
- Wraps its body in `try { ... } catch (error) { ... }`.
- Re-throws `Response` instances so auth helpers' `throw json(...)` /
`throw redirect(...)` pass through unchanged.
- Logs non-Response errors via `logger.error` so server-side visibility
is preserved.
- Returns a generic body — `{"error": "Internal Server Error"}` 500 for
raw API routes, or `{ changelogs: [] }` 200 for the polling widget
(degrade silently across transient blips; the consumer hook already
coped with empty payloads).
For six routes where #3536 left an inner try/catch covering only a
service call (`alertChannels`, `batches.results`,
`deployments.finalize`, `deployments.background-workers`,
`deployments.promote`, `projects.background-workers`): an outer
try/catch is added so auth/parsing failures are also sanitized. Inner
typed-error handling (`ServiceValidationError` → 422 with message, etc.)
is preserved exactly.
For two routes whose existing catch returned 400 + `error.message`
(`api.v1.authorization-code`, `api.v1.orgs.\$orgParam.projects` action):
the body is sanitized to a generic per-route string. **Status code stays
400** — clients that key on the 4xx/5xx distinction (and the SDK's
no-retry-on-4xx behavior) are unaffected.
## Test plan
- [x] \`pnpm run typecheck --filter webapp\`
- [x] Per-route synthetic-throw probe: inject \`throw new
Error("SYNTHETIC ...")\` at the top of each catch'd try, curl the route
with a dummy bearer, confirm the response body is the generic shape and
that the synthetic message lands server-side via \`logger.error\`. 29
routes verified.
- [x] Real-P1001 probe on the envvars loader: \`docker stop database\`
mid-flight, confirm response is generic 500 (not the leaked Prisma
message).
- [x] Sampled legitimate 4xx/2xx paths across each pattern variant
(naked-wrap, partial-expanded, 400-preserved) to confirm the wraps don't
interfere with normal control flow.
## Summary
`TriggerChatTransport`, `AgentChat`, and `chat.createStartSessionAction`
now accept a string-or-function `baseURL` so callers can route per
endpoint — e.g. `.in/append` through a trusted edge proxy while keeping
`.out` SSE direct. The same surfaces add a `fetch` override for header
injection, custom retries, or proxy rewrites that go beyond URL routing.
SSE GETs are covered too via a new `fetchClient` option on
`SSEStreamSubscription`.
```ts
// TriggerChatTransport / AgentChat — endpoints: "in" | "out"
baseURL: ({ endpoint }) =>
endpoint === "out" ? DIRECT : PROXY,
fetch: (url, init, ctx) => {
init.headers = new Headers(init.headers);
init.headers.set("traceparent", currentTraceparent());
return globalThis.fetch(url, init);
},
// chat.createStartSessionAction — endpoints: "sessions" | "auth"
chat.createStartSessionAction("my-agent", {
baseURL: ({ endpoint }) => (endpoint === "sessions" ? PROXY : DIRECT),
});
```
`streamBaseURL` on `TriggerChatTransport` is kept as a backwards-compat
alias and continues to win for the `"out"` endpoint when set.
Plain-string `baseURL` still applies to every endpoint, matching prior
behavior.
## Summary
Hot-loop writers — `streams.writer` / `streams.pipe` on the run-scoped
side, `chat.response.write` / `chat.stream.*` on the session side — were
issuing a fresh `PUT` to mint S2 credentials for every chunk. On run
streams, each PUT also pushed the streamId onto
`TaskRun.realtimeStreams`,
so a chat-agent turn writing N chunks produced N PUTs and N duplicate
array pushes against the same row.
The SDK now caches the initialize response per cache slot: `(runId,
key)`
for run streams, the session id for session streams. First call PUTs as
before; subsequent calls reuse the cached promise. Hot-loop writers do
one PUT per slot for the lifetime of the cache.
S2 access tokens have a 1-day TTL. If a writer's `wait()` rejects (auth
error, expired token, network blip), the cache evicts the matching slot
so the next call re-PUTs and mints fresh credentials, identity-checked
so a concurrent caller's fresh promise isn't accidentally cleared.
## chat.agent guardrail
`streams.pipe / writer / append / read` called inside a `chat.agent` run
now logs a one-time warning pointing at `chat.response.write` /
`chat.stream.*` — `streams.*` is run-scoped and isn't visible on the
chat session. The ai-chat docs are updated to drop the old guidance
toward run-scoped streams.
Mollified runs were materialising with `TaskRun.traceContext = {}`, so every
downstream `recordRunDebugLog` (engine QUEUED/EXECUTING/FINISHED, run:notify,
attempt events) drew a fresh traceId with null parentId. The run-detail
trace view rendered only the root span; the rest of the tree was orphaned.
The pass-through path gets traceContext for free via `traceEventConcern.traceRun`
populating the W3C traceparent. The mollifier path skips that wrapper, so seed
`traceContext.traceparent` from the queued span at the call site before
handing the snapshot to engine.trigger.
Also fixes the drainer side: wrap the `mollifier.drained` span + `engine.trigger`
call in a `context.with(parentContext, ...)` built from the snapshot's
traceId/spanId. Without this `mollifier.drained` lived in a fresh trace and
the engine instrumentation inside it inherited an empty active context.
Regression tests:
- `triggerTask.test.ts` — asserts the buffered snapshot carries a valid W3C
traceparent that references the snapshot's traceId/spanId.
- `mollifierDrainerHandler.test.ts` — captures the active traceId at the
moment engine.trigger is invoked and asserts it matches the snapshot's
traceId.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary
- Introduce the Mollifier: a Redis-backed buffer for `trigger()` API
calls during traffic spikes, with a per-env trip evaluator and a drainer
ack-loop.
- Phase 1 is dual-write monitoring — every mollified trigger is buffered
to Redis AND continues to `engine.trigger`. No customer-facing behaviour
change.
- Telemetry events: `mollifier.would_mollify`, `mollifier.buffered`,
`mollifier.drained`, plus the `mollifier.decisions` counter.
- Gated behind a feature flag (default off).
## Test plan
- [x] `pnpm run test --filter @trigger.dev/redis-worker`
- [x] `pnpm run test --filter webapp -- mollifier`
- [x] Manual: with flag off, no behaviour change vs main
- [x] Manual: with flag on + threshold lowered, observe
`mollifier.buffered` + `mollifier.drained` log pairs with matching
`runId`
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Per-org rollout procedure for turning the trigger-burst mollifier on
across the fleet. Mirrors the plan's structure (pre-rollout → test cloud
→ first customer → expansion → kill switches) but reflects the controls
that actually shipped, not the plan's original design:
- Per-ORG opt-in via `Organization.featureFlags.mollifierEnabled` JSON
(not per-env via the global FeatureFlag table — the shipped impl
deliberately keeps the trigger hot path free of an extra DB query).
- Per-replica drainer via the `TRIGGER_MOLLIFIER_DRAINER_ENABLED` env
(defaults to inherit `TRIGGER_MOLLIFIER_ENABLED`).
- `TRIGGER_MOLLIFIER_*` env-var prefix.
- `MollifierConfigurationError` fail-loud-on-boot for misconfigured
shutdown timeouts (referenced in the alarm list).
- State matrix updated to the three live controls (gate / org flag /
drainer flag) rather than the two-keyed per-env model in the plan.
Companion to `.server-changes/mollifier-phase-3-live.md` (changelog
entry) — this file is the operator runbook.
Surfaces an info-style banner above the run-detail page body when the
loader resolved the run from the mollifier buffer (`isMollified === true`,
set by the read-fallback in route.tsx when the PG row hasn't been
materialised yet). Explains the queued state to the operator and points
at `batchTrigger` as the long-term shape for high-fan-out workloads.
Dismissal is localStorage-only (`mollifier_banner_dismissed`). Plan
Task 21 leaves this an explicit choice between localStorage and a new
per-org settings endpoint; localStorage is the simpler path and avoids
adding a writeable settings endpoint just for a dismissal flag. Reads
happen in an effect (not useState's initialiser) so SSR-renders the
banner visible by default, then hides it on hydration if dismissed — no
flash-of-banner.
Styled to match the existing Callout 'info' variant (blue-400 family)
without using the Callout primitive directly because Callout's API is
oriented around link-style CTAs, not inline-dismissible banners.
The two cherry-picked example payload comments (MOLLIFIER_E2E +
MOLLIFIER_SHADOW) referenced the pre-rename `MOLLIFIER_*` env vars. They
were authored before phase-2 prefixed everything with TRIGGER_. Bring
the examples in line with the actual env-var names operators set today,
and expand the E2E example to mention the per-org `mollifierEnabled`
flag + the synthesised `mollifier.queued` response so the example
accurately describes Phase 2 (live mollify) behaviour rather than the
phase-1 dual-write semantics those comments were written against.
Dan Sutton