## Summary
When a background worker registers, the engine resolves runs that were
queued before the worker was ready (status `PENDING_VERSION`). That
lookup used to scan a Postgres status index on `TaskRun`. Move it to
ClickHouse: query candidate run ids from `task_runs_v2`, then refetch
the actual rows from Postgres by primary key with a `status =
'PENDING_VERSION'` guard for idempotency.
## Design
The lookup is a pluggable interface on the run engine
(`PendingVersionRunIdLookup`). The webapp wires a ClickHouse-backed
implementation through the org-scoped `clickhouseFactory` using a new
`"engine"` client type, configured by `RUN_ENGINE_CLICKHOUSE_*` env
vars. The URL falls back to `CLICKHOUSE_URL` when unset, so self-hosted
deployments don't need new config to keep working.
When the lookup returns no candidates, one bounded retry is scheduled
~5s later to cover ClickHouse replication lag against `task_runs_v2`.
The Postgres status guard on both the candidate refetch and the inner
`updateMany` prevents double-promotion when a retry races with a
concurrent deploy.
Tests cover three existing PENDING_VERSION cases via a small
Postgres-backed test adapter; new ClickHouse-backed integration tests
will follow.
Make the Express server's `keepAliveTimeout` configurable via
`HTTP_KEEPALIVE_TIMEOUT_MS`. Default preserved at 65000 ms — no behavior
change if unset.
Buffer entries used to EXPIRE after entryTtlSeconds (600s dev / 1h
prod). Once that window elapsed without the drainer ack'ing, the
entry just vanished — no PG row, no log, no customer signal. The
stale-entry sweep was added in the previous commit so ops gets paged
on dwell-too-long; with that signal in place, the TTL itself is now
the cause of the failure mode it was meant to mitigate.
Remove it. Buffer entries persist until the drainer ACKs (with the
existing 30s post-materialise grace TTL) or FAILs them. Idempotency
lookup keys also lose their TTL — keeping them paired to the entry
hash prevents the dedup-drift bug where a TTL'd lookup would let the
same idempotency key spawn a second buffered run while the first
still existed. `failMollifierEntry` now DELs the entry hash + lookup
because the SYSTEM_FAILURE PG row written by the drainer is the
canonical record; the buffer entry is no longer load-bearing.
Knock-on changes:
- `MollifierBufferOptions`: `entryTtlSeconds` removed (no consumers
outside this repo).
- `TRIGGER_MOLLIFIER_ENTRY_TTL_S`: removed from env.server.ts and the
example .env. The stale-sweep threshold now has its own explicit
default (5min) instead of "half of TTL".
- `MollifierBuffer.getEntryTtlSeconds`: retained — it returns the
Redis-side TTL, which is now -1 in steady state and ~30s after ack.
Used by the ack-grace-TTL test.
- Existing tests updated: TTL-related cases inverted to assert no TTL;
FAILED-state cases inverted to assert teardown; runId-reuse-after-
fail now succeeds (slot is reclaimable).
Operational alert: Redis memory pressure if the drainer is offline.
That's the same failure mode as Redis OOM in any other context, with
existing infra-level alerts. The mollifier.stale_entries.current
gauge fires first; ops should be on it long before memory becomes a
problem. See _ops/mollifier-ops.md.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary
Stamp every Sentry event with the signed-in user and the tenant (org /
project / env) the request belongs to, so "Users Impacted" counts
distinct humans and events become filterable per tenant.
**Design after review (current):**
- `user.id = real user cuid` (from `requireUser`). "Users Impacted"
counts humans, not tenants.
- Tenant context (org / project / env slugs, IDs, env type) moves
entirely onto tags: `org_slug`, `project_slug`, `env_slug`, `org_id`,
`project_id`, `project_ref`, `environment_id`, `env_type`, plus
`impersonating` when set.
- Backed by an `AsyncLocalStorage` scope established at the HTTP entry.
Each entry point fills what it knows; loaders enrich the same scope with
what they already have.
**Zero new database queries.** The middleware does a regex match only.
Dashboard loaders that already query Prisma gain a couple of extra
selected columns; nothing new round-trips.
## How it's wired
- **Express middleware (`tenantContextResolver.server.ts`)** — parses
the URL with a regex and always opens an ALS scope. Populates whatever
subset of slugs is present: `/orgs/:o` → just `orgSlug`;
`/orgs/:o/projects/:p` adds `projectSlug`; the full triple adds
`envSlug`. Non-tenant paths get an empty scope so loaders can still
enrich.
- **`_app/route.tsx`** — already calls `requireUser`. Adds
`tenantContext.enrich({ userId: user.id })` for every authenticated
dashboard request. No new query.
- **Env layout loader (`_app.orgs.$o.projects.$p.env.$e/route.tsx`)** —
its existing `prisma.project.findFirst` gains two columns in `select`
(`externalRef`, `organization.id`). After it picks an env, calls
`tenantContext.enrich({ orgId, projectId, projectRef, envId, envType
})`. Same query, +2 columns.
- **API path (`apiBuilder.server.ts`)** — wraps every handler in
`tenantContext.run(tenantContextFromAuthEnvironment(authenticationResult.environment),
…)`. The mapper pulls `userId` from `env.orgMember?.userId` (already
selected by `authIncludeBase` — no schema change). Covers
`createLoaderApiRoute`, `createActionApiRoute`, and
`createMultiMethodApiRoute`.
- **Event processor (`sentryTenantContext.server.ts`)** — registered in
`entry.server.tsx` so it lives in the Remix bundle and shares the same
`tenantContext` ALS instance as the middleware and loaders. Stamps
whatever's present; nothing forced.
## Example events from local verification
| URL | `user.id` | Tags |
|-----|-----------|------|
| `/orgs/:o/projects/:p/env/:e/...` | real user cuid | `org_slug`,
`project_slug`, `env_slug`, `org_id`, `project_id`, `project_ref`,
`environment_id`, `env_type` |
| `/orgs/:o/settings` (non-env-scoped) | real user cuid | `org_slug`
only |
| API request with `orgMember` | `orgMember.userId` | full tenant set |
| API request without `orgMember` | (unset) | full tenant set |
## Trade-offs
1. On env-scoped pages, errors that fire before the env layout loader's
enrich callback runs get slugs + `user.id` but not the tenant IDs /
`env_type`. Realistic errors deep in async work get the full set. (Same
race as before, narrower window now that slugs/`user.id` are populated
up-front by the middleware and `_app` enrich.)
2. API requests where the environment has no `orgMember` get tenant tags
but no `user.id`. Those events still show in the issue but don't
contribute to "Users Impacted".
## Out of scope (deferred)
Background workers (`redis-worker`, `schedule-engine`) and socket
handlers. Those entry points don't set `tenantContext.run` yet — their
events ship without tenant attribution until each is wired in a
follow-up.
## Tests
31 unit tests across 4 files. New tests notably cover:
- `parseTenantPath`: org-only, org+project, and full-triple URL
variants.
- `tenantContext.enrich`: in-place patch, no-op outside `run()`,
concurrent-scope isolation, empty-scope + enrich pattern (for non-tenant
pages).
- `tenantContextFromAuthEnvironment`: with and without `orgMember` —
verifies the API path's `user.id` mapping.
- `addTenantContextToEvent`: empty scope, userId-only, slugs-only, full
enrichment, conditional tag emission, preservation of prior `event.user`
fields.
## Test plan
- [ ] `pnpm run typecheck --filter webapp`
- [ ] `pnpm run test --filter webapp -- test/tenantContext.test.ts
test/sentryTenantContext.test.ts test/tenantContextResolver.test.ts
test/tenantContextFromAuthEnvironment.test.ts`
- [ ] Local manual: with `SENTRY_DSN` set, hit a dashboard URL and an
API route, confirm the captured events carry `user.id` + the expected
tag set in Sentry.
- [ ] After ship: confirm "Users Impacted" on a real Sentry issue
reflects distinct users (not tenants).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Added `OrganizationDataStore` which allows orgs to have data stored in
specific separate services.
For now this is just used for ClickHouse. When using ClickHouse we get a
client for the factory and pass in the org id.
Particular care has to be made with two hot-insert paths:
1. RunReplicationService
2. OTLPExporter
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
The mollifier.stale_entries counter from the previous commit reflects
sweep-tick events, not stable state. A single stuck entry observed
across N ticks contributes N events, so a rate() query is
proportional to (stuck-entry-count × scan-frequency), not "how many
entries are stale right now". Useful for historical views but the
wrong shape for ops alerts.
Add a companion observable gauge `mollifier.stale_entries.current`
with `{envId}` attribute. The sweep emits a per-env snapshot on each
pass (including zero counts for envs whose stale entries cleared),
and an OTel batch-observable callback exposes the latest snapshot to
the metric exporter on every scrape. Recommended alert:
mollifier_stale_entries_current{envId=...} > 0 for 5m
The snapshot replaces (not merges) so an env that paged on a
previous sweep clears when the drainer catches up, instead of
staying latched at the last stale count. Test seam captures the
snapshot to verify per-env counts and the clear-on-drain behaviour.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Without an external signal that the drainer is falling behind, a stuck
or offline drainer drives the buffer toward the entry-hash TTL line
and runs vanish silently — no PG row, no log, no dashboard
indication. Add a periodic read-only sweep over the buffer's queue
ZSETs that emits a `mollifier.stale_entries` OTel counter and a
structured `mollifier.stale_entry` warning log for each entry whose
dwell exceeds the configured threshold.
Independent of the drainer (its own gate +
`TRIGGER_MOLLIFIER_STALE_SWEEP_ENABLED`) so an entirely offline
drainer is exactly when the sweep is most useful. Defaults: interval
5min, threshold half of `entryTtlSeconds`, hard cap of 1000 entries
per env per pass.
Sweep is strictly read-only — does not remove or salvage entries.
The retention-policy question (drop the entry TTL entirely vs raise
it vs pre-TTL salvage) is intentionally deferred to a separate
change; this commit gets the signal in place first.
Tested with a real `MollifierBuffer` (testcontainers): stale entries
flagged, fresh entries left alone, multi-org scan walks every queue.
Manually verified end-to-end: with a 10s interval + 2s threshold,
each tick logs the buffered run with growing dwellMs as expected.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The previous commit's regression coverage was thin: only the log-dedup
gate was unit-tested. The load-bearing logic — synthesise a resource
when PG misses but the buffer has the run, with an `id` matching what
the drainer will eventually write — had no regression test, so a
future change that removed the buffered fallback would put the silent-
hang back into prod without anything failing in CI.
Extract the resource-resolution rules from the route's findResource
into `resolveRealtimeRunResource`, a pure function. Cover the
branching with unit tests (PG hit, PG hit during drain race, PG miss
+ buffer hit, missing taskIdentifier default, both miss) and pin the
full chain with a container-backed test that uses a real
MollifierBuffer + the real readFallback helper and asserts the
synthesised `id` matches `RunId.fromFriendlyId(friendlyId)`. That
identity is what Electric's `WHERE id='<id>'` clause depends on when
the drainer eventually INSERTs the row.
12 tests total across the three Phase-5.2 suites; one empirical probe
run after the refactor confirmed end-to-end behaviour unchanged.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Customers subscribing to a freshly-triggered run via useRealtimeRun
silently hung when the gate diverted the run into the mollifier buffer.
The route's findResource looked up the PG TaskRun by friendlyId, found
nothing, and returned 404. Electric SQL's ShapeStream treats the
initial 404 as terminal — no retry, no error surfaced to the hook,
and crucially no recovery after the drainer eventually INSERTed the
PG row. The customer's component shows the empty state indefinitely
even though the run is alive and progressing.
When the PG lookup misses but the buffer has the run, return a
synthetic resource whose `id` is derived from the friendlyId — the
same value engine.trigger will write when the drainer materialises
this run. The route then opens the Electric subscription against
`WHERE id='<id>'`, Electric streams an empty initial snapshot, and
the SDK long-polls until the drainer's INSERT propagates through.
Empirically validated end-to-end: trigger a buffered run, open the
subscription, simulate the drainer's PG INSERT + UPDATE, and the
SDK iterator yields the QUEUED and EXECUTING events in real time.
Adds a `mollifier.realtime_subscriptions.buffered` counter and a
structured log line. The observability gate fires once per cold
subscription (Electric's `handle` query param is the dedup signal),
not on every ~20s long-poll reconnect; that gate is unit-tested.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The bulk-action confirmation count is sourced from ClickHouse, so PG
rows not yet replicated to ClickHouse are silently excluded from both
the count and the processing pass. Phase 4's first-batch mollifier-
buffer scan broke that symmetry — buffered runs were processed without
being counted, so a customer confirming "Replay ~0 runs" could see N
buffered runs replayed without seeing them anywhere in the UI.
Restore the eventually-consistent contract: bulk actions only target
runs visible to ClickHouse. Buffered runs are picked up by subsequent
bulk actions once they drain into PG → ClickHouse, mirroring how
PG-not-yet-CH runs already work today.
Removes `bulkActionBuffer.server.ts` (helper) and its container-backed
test. Will reimplement once the buffered-runs UX (global status
indicator) gives the customer a way to see and confirm against the
buffered set.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Today the run span (the SERVER trace event keyed by runId) is created
inside `traceEventConcern.traceRun`, which sits *after* the mollifier
gate. When the gate diverts a trigger into the Redis buffer, the run
span is therefore not written to the event store until the drainer
replays the snapshot — buffered runs are invisible in the trace view,
parents' trace trees miss the child until drain, and alerting
pipelines can't reference the run.
Hoist the gate evaluation and mollify branch inside `traceRun` so both
paths open the run span. The mollify branch records mollifier
attributes on the same event, captures `event.traceId`/`event.spanId`
into the buffer snapshot (replacing the separately-allocated
`mollifier.queued` OTel span), and returns the synthesised result.
`traceRun` flushes the PARTIAL event to the store on callback return.
Extend the existing call-site test to assert (a) traceRun fires before
buffer.accept and (b) the snapshot's traceId/spanId match the run
span's IDs. The MockTraceEventConcern now mirrors the production
ClickhouseEventRepository shape so the `traceContext.traceparent`
assertion exercises the seeding path.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The runs list (API and dashboard) is eventually consistent — buffered
runs were creating a sandwich problem where the head of the list could
include buffered rows while in-transit rows between PG replication and
ClickHouse went missing. Drop the merge so the list returns PG/
ClickHouse rows only; buffered visibility will return via a separate
global status indicator.
Reverts the merge wiring in api.v1.runs, api.v1.projects.$projectRef
.runs, and the dashboard runs index, and deletes listingMerge.server
and dashboardListingMerge.server. The MCP list_runs tool rides through
the API and inherits the same behaviour.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Previously, a non-retryable engine.trigger failure during drain left
the buffer entry as `status: "FAILED"` in Redis with no PG row. The
customer saw the run in their SDK / dashboard listing for ~10 min
(buffer TTL) then it vanished entirely — no audit trail of the
failure. Billing was unaffected (no attempts ever ran) but
observability was zero.
Reuse the engine's existing `createFailedTaskRun` helper (the same one
batch-trigger calls when an item fails to start) — writes a terminal
SYSTEM_FAILURE TaskRun row with the engine.trigger error stored on
`error`, no attempts, P2002-idempotent on the unique constraint.
Drainer handler classifies the failure:
- Retryable PG error → rethrow so MollifierDrainer.drainOne requeues
- Non-retryable → createFailedTaskRun, swallow original error so the
buffer entry is ack'd (PG now has the audit row)
- createFailedTaskRun also fails (PG truly unreachable) → rethrow
original so drainer falls through to its existing buffer.fail
terminal-marker path
- Snapshot too malformed to construct the environment block → rethrow
(defensive — drainer falls through to buffer.fail)
Tests cover each path.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
A broader audit of every public API route's buffered branch found a
handful of schema-drift bugs the SDK would reject on existing clients:
- /api/v1/runs/{id}/spans/{spanId} returned `parentId: undefined`
(omitted in JSON). Schema declares `parentId: z.string().nullable()`
— present-but-null is required. Send `null` explicitly. Also reflect
the snapshot's cancelled state in `isPartial` / `isCancelled`.
- /api/v1/runs/{id}/reschedule's buffered branch returned a stripped
`{ id, delayUntil }`. The SDK's `rescheduleRun` validates against the
full `RetrieveRunResponse` shape. Route the buffered response through
the same ApiRetrieveRunPresenter the PG branch uses (which falls back
to the buffer for synthetic runs). Allows `synthesisedResponse` in
`mutateWithFallback` to be async.
- ApiRetrieveRunPresenter.synthesiseFoundRunFromBuffer ignored the
snapshot's `cancelledAt` and `delayUntil`. Status was hardcoded to
`PENDING` regardless of cancellation; `completedAt` and `delayUntil`
were always `null`. SDK callers (and the MCP cancel_run helper)
reported status as Queued after a successful cancel. Map the synthetic
status through a small switch so CANCELED, SYSTEM_FAILURE and PENDING
all surface correctly.
- Add `delayUntil` to SyntheticRun so set_delay reschedule patches
survive the next retrieve. Mirror it onto the dashboard SpanRun
synthesiser too.
Verified end-to-end by replaying every public-API method against a
buffered run.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two SDK schemas were drifting from what the mollifier paths emitted:
1. ListRunResponseItem declares `idempotencyKey: z.string().optional()`
(omit-or-string). The listing-merge synthesiser was emitting
`idempotencyKey: null` for buffered runs, which old SDK versions
reject with a validation error before surfacing the row.
2. RetrieveRunTraceResponseBody declares a non-nullable `rootSpan`
matching the recursive RetrieveRunTraceSpan shape. The buffered
branch of /api/v1/runs/{id}/trace returned `rootSpan: null` plus an
`events: []` field that isn't in the schema. Synthesise a real
partial span (task identifier as message, no children, isPartial:
true) from the buffer snapshot so the response satisfies the schema
the SDK validates against.
Verified end-to-end by calling the MCP server's list_runs and
get_run_details against a buffered run; both now succeed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Previous attempt wrapped the form's submit button in <DialogClose
asChild> so the dialog closed on click. That race-condition'd with
Remix's <Form>: Radix's Slot-attached onClick triggered
onOpenChange(false), the Dialog and its child Form unmounted mid-cycle,
and the button's name=value pair (carrying `redirectUrl`) was dropped
from the submitted FormData. The action then read
`submission.value.redirectUrl` as undefined and the resulting redirect
landed on `/env/dev` instead of the run-detail page.
Switch to a ControlledCancelRunDialog at the call site that owns the
Radix `open` state. The inner CancelRunDialog watches the navigation
state transitions and signals the parent to close the dialog once the
submission has captured its submitter cleanly. Submit-button name=value
is preserved; redirect resolves to the run-detail page; modal still
dismisses after submit.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The replay form loader hit `taskRun.findFirst` and threw 404 when the
run was buffered, which dumps the user back to the task list. Wire a
buffer fallback that synthesises the same loader return shape from the
snapshot, including a project-and-environments lookup scoped by the
buffer entry's orgId so the env selector renders identically. The
replay action itself already supports buffered runs via the
ReplayTaskRunService synthetic-run cast — only the form's preflight
load was broken.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Buffered runs are prepended to the runs table on the runs list page so
customers see freshly-triggered work even while the gate is diverting.
The merge uses a compound base64 cursor that wraps the PG presenter's
own cursor — page 1 can be entirely buffered (top of the list), page 2
takes the buffered overflow and transitions into the PG content, and
later pages drop the buffer scan entirely once it's been exhausted.
Filter predicates (tasks, statuses, tags, period, from/to, isTest,
runId) are evaluated against the buffer snapshot so the list reflects
the same filter scope as the PG-side query.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The cancel dialog stayed open after a successful submit because it was
uncontrolled Radix state and the action redirects to the same URL —
revalidation didn't trigger a re-mount. Wrap the submit button in
DialogClose so the click closes the dialog at the same time the form
posts.
The SyntheticRun synthesised for the run-detail page hardcoded status
PENDING regardless of whether the buffer snapshot had cancelledAt set.
Customers cancelling a buffered run saw their run still labelled Queued
until the drainer materialised it. Surface cancelledAt + cancelReason
on SyntheticRun, switch the synthesised SpanRun status to CANCELED, and
mirror the cancelled flag onto the single-span trace so the timeline
matches PG behaviour.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Synthesise the SpanRun shape from buffer snapshots so the run-detail
page's inspector panel renders identically to a PG-resident run. SSE
log stream, realtime stream resources, logs-download and debug resource
fall back to the buffer instead of 404-ing. Short-URL redirects resolve
buffered runs to the canonical dashboard URL. Bulk-cancel scans the
buffer alongside the ClickHouse selection so runs queued mid-burst are
included. Trigger response now carries the snapshot's spanId so the
dashboard's Run Test redirect opens the details panel without an extra
click.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes the PG+buffer race during the mollifier gate-transition window.
Plan: _plans/2026-05-21-mollifier-idempotency-claim.md
redis-worker:
- New MollifierBuffer methods + atomic Lua: claimIdempotency
(SETNX-with-TTL returning claimed/pending/resolved), publishClaim,
releaseClaim, readClaim. Separate key namespace mollifier:claim:*
to keep isolated from the B6a buffered-side mollifier:idempotency:*
lookup.
webapp:
- New apps/webapp/app/v3/mollifier/idempotencyClaim.server.ts wraps
the buffer primitives with a wait/poll loop. Returns claimed /
resolved / timed_out. Fail-open on buffer outage so a transient
Redis blip doesn't 500 the trigger hot path.
- IdempotencyKeyConcern.handleTriggerRequest now consults the claim
after the existing PG-findFirst + buffer.lookupIdempotency cache
checks miss. Skipped for resumeParentOnCompletion (triggerAndWait
bypasses the mollifier gate via F4 and is PG-canonical anyway). When
we own the claim, the result's new `claim` field signals the caller
to publish on success / release on failure.
- RunEngineTriggerTaskService.callV2 wraps the trigger pipeline in a
try/catch that publishes the winning runId or releases the claim
depending on outcome. The publish updates the claim key so waiters
polling for our key resolve to our runId.
Validated end-to-end:
- scripts/mollifier-challenge/04-idempotency-collision.sh runs
cold-gate (no pre-warm) with 30 concurrent same-key triggers and
converges on 1 runId / 1 isCached:false. Before this fix the same
test produced 2 race-winners.
- 13 unit tests covering claimed/resolved/pending/timed_out paths,
fail-open behaviour, abort signal, publishClaim, releaseClaim.
- All 94 webapp mollifier tests still green.
Each fix lands a focused test that fails without the fix and passes
with it.
1. Cancel route findResource (b490afe23) — extracted the PG-or-buffer
lookup into apps/webapp/app/v3/mollifier/resolveRunForMutation.server.ts
so it's unit-testable independently of the route builder.
apps/webapp/test/mollifierResolveRunForMutation.test.ts covers all
three paths: PG hit, buffer hit, both miss, plus env/org mismatch
and PG-hit-short-circuits-before-buffer.
2. createCancelledRun empty-tags (eef33e5bd) — added a containerTest
case to internal-packages/run-engine/src/engine/tests/createCancelledRun.test.ts
passing `tags: {} as unknown as string[]` (mimics the cjson decode
shape for an empty Lua table) and asserting the PG row is created
with runTags=[]. Without the defensive Array.isArray check the
Prisma create rejects with `Argument 'set' is missing`.
3. applyMetadataMutation retry budget (4e7d5d8a2) — new file
apps/webapp/test/mollifierApplyMetadataMutation.test.ts with a
stub MollifierBuffer that simulates Lua-CAS semantics in memory.
Covers: zero contention, 5/11 simulated conflicts within budget,
99 conflicts exhausting, and a 30-way concurrent-write convergence
test. Includes a regression assertion that maxRetries=3 (pre-fix
default) exhausts under 8 conflicts — confirming the regression
actually existed.
Default maxRetries was 3, matching the PG-side UpdateMetadataService.
That's fine when the only writer is the executing task itself, but
under high external-API concurrency on a single buffered run it
exhausts fast — the Phase F challenge suite saw 50-way concurrent
metadata.increment landing only 21/50 deltas with the default.
Bumps the default to 12 (covers ~50-way concurrency with sub-percent
failure) and adds small jittered backoff between retries so a
thundering herd of N retriers doesn't all re-read + re-CAS in
lockstep. Each retry is one Redis Lua call (~1ms), so the worst-case
budget is bounded.
Verified via challenge script 09: 50 concurrent increments now land
all 50 deltas, counter ends at exactly 50.
The route builder treats a null `findResource` result as a 404 BEFORE the
action handler runs (`apiBuilder.server.ts:321`). My C1 commit had
`findResource: async () => null`, which meant every cancel call —
including for valid PG-row runs — was 404'd by the builder before the
mutateWithFallback flow could resolve anything.
Fixes by mirroring the Phase A discriminated-union pattern: findResource
checks PG first, falls back to the buffer with env+org auth, returns
`null` only when neither store has the run. The action handler still
uses mutateWithFallback (slightly redundant lookup) so the wait-and-
bounce path stays intact.
Found while running the Phase F challenge suite — cancel was 404'ing
on a confirmed-buffered runId.
The old comment claimed multiple drainer replicas would "race for the
same buffer entries." That's wrong — `popAndMarkDraining` is an atomic
ZPOPMIN + status flip in a single Lua call, so only one replica can win
any given entry. Multi-replica drainers are correct, just inefficient
(polling load and per-process concurrency multiply by N). Rewrite the
comment to give the real reasons a deployment might split the drainer
onto a dedicated worker. No behaviour change.
Q1 ZSET-merge design lands.
redis-worker side:
- MollifierBuffer.listForEnvWithWatermark — paginated newest-first
read of buffered entries, bounded by a (createdAtMicros, runId)
watermark. ZREVRANGEBYSCORE strictly below the watermark score plus
a tied-score band scan for entries sharing the watermark's
createdAtMicros.
webapp side:
- listingMerge.server.ts: callRunListWithBufferMerge wraps
ApiRunListPresenter. Fetches a buffer page, synthesises each entry
into the presenter's ListDataItem shape (status QUEUED, timestamps
from entry hash, env slug looked up once), forwards the inner
cursor to the presenter, merges by createdAt DESC with runId DESC
tiebreak, truncates to pageSize. Compound base64-JSON cursor
{ inner, watermark, bufferExhausted } is backwards-compatible with
legacy opaque cursors.
- api.v1.runs.ts + api.v1.projects.{projectRef}.runs.ts route through
the wrapper. Project route extracts envId from filter[env]; absent
that, falls back to the bare presenter (existing behaviour).
- Buffer eligibility skips for filters that can't match buffered runs
(status not in QUEUED/PENDING/DELAYED, batch/schedule/version/
region/machine filters). Buffer outages fall open to PG-only.
- Delete RecentlyQueuedSection banner + listEntriesForEnv loader call
from dashboard runs index — buffered runs appear inline as QUEUED
rows.
(`OBJECT_STORE_BASE_URL`) and a named protocol provider
(`OBJECT_STORE_DEFAULT_PROTOCOL=s3`), chat.agent session snapshot writes
landed in the named provider but reads fell through to the default — so
the recovery boot couldn't find the snapshot it had just written.
After a mid-stream cancel, the missing snapshot triggered a fallback
replay path that dropped the user's follow-up message, leaving the chat
stuck in `submitted` indefinitely.
Fix:
- New `/api/v1/sessions/:id/snapshot-url` route handles PUT + GET
symmetrically — both prefix unprefixed keys with
`OBJECT_STORE_DEFAULT_PROTOCOL` so they always round-trip through the
same store.
- `Session.chatSnapshotStoragePath` persists the resolved URI on first
write so future protocol changes don't strand existing snapshots.
Reads prefer the stored URI and fall back to the computed default for
pre-column sessions.
- SDK calls `createChatSnapshotUploadUrl` / `getChatSnapshotUrl`; the
generic v1/v2 packets endpoints are unchanged.
## Test plan
- [x] Configure local with two providers (R2 default + MinIO `s3` named)
and `OBJECT_STORE_DEFAULT_PROTOCOL=s3`.
- [x] Reproduce hang: send a message, cancel mid-stream, send another —
without the fix it hangs in `submitted`; with the fix it streams.
- [x] Snapshot lands in the `s3`-protocol bucket and
`Session.chatSnapshotStoragePath` is set after first write.
- [x] SDK unit tests pass; webapp typecheck passes.
Parallels Phase C's API-side work for the three dashboard mutation
routes.
D1 cancel — PG miss → buffer.mutateSnapshot('mark_cancelled'). Org-
membership verified against the buffered run's orgId (dashboard URL
doesn't carry an envId so the API-side env-scoped auth doesn't apply).
busy returns a "retry in a moment" message.
D2 replay — PG miss → findRunByIdWithMollifierFallback; B4-extended
SyntheticRun cast to TaskRun and fed to ReplayTaskRunService.
Project/env slugs for the redirect path looked up from the entry's
envId.
D3 idempotencyKey reset — PG miss → buffer.getEntry + readFallback to
read snapshot's idempotencyKey + taskIdentifier; org-membership
verified against entry orgId; existing ResetIdempotencyKeyService
(extended in B6b to clear both stores) handles the actual reset.
Closes the last API-parity gap in the master plan.
redis-worker side:
- New casSetMetadata Lua command with optimistic lock on a
metadataVersion entry-hash field. Returns applied / version_conflict /
not_found / busy. Mirrors the PG-side UpdateMetadataService's CAS
loop so concurrent metadata.increment / metadata.set / metadata.append
calls against a buffered run never lose deltas.
- accept Lua initialises metadataVersion=0; BufferEntrySchema gains
the field.
webapp side:
- applyMetadataMutationToBufferedRun helper does the read-apply-CAS-
retry loop in JS, reusing the existing @trigger.dev/core
applyMetadataOperations function (no Lua re-implementation of the 6
operation types).
- metadata PUT route does PG-first via the existing service (which
owns the full request shape: parent/root ops, batching, validation),
then falls through to the buffer helper on PG miss. busy and
version_exhausted return 503 with retry hint; not_found returns 404.
- Parent/root operations on a buffered target are fanned out to the
snapshot's parentTaskRunId via the existing service. If the parent
is also buffered the helper recurses. Best-effort — parent/root
ingestion failures do not surface to the caller.
Tests: 3 new redis-worker tests covering CAS apply / version conflict /
not_found-busy paths. All 71 redis-worker mollifier + 68 webapp
mollifier tests green.
Reschedule (C4): switches to mutateWithFallback. PG hits go through
the existing RescheduleTaskRunService (which enforces status ===
"DELAYED"). Buffered hits land a set_delay patch on the snapshot;
the drainer materialises the PG row with the new delayUntil. Synth-
esised response returns { id, delayUntil }.
Replay (C5): adds a read-fallback after the PG miss. The B4-extended
SyntheticRun carries every field ReplayTaskRunService reads from a
TaskRun, so the buffered case casts through and uses the existing
service unchanged. Replay creates a fresh trigger that itself
re-enters the mollifier gate — no special surge handling needed
beyond what the gate already does. Also tightens the PG lookup to
findFirst with runtimeEnvironmentId scoping (was findUnique on
friendlyId only).
Closes the live 500 the parity script flagged. The previous route did
prisma.taskRun.update after a findFirst that could miss; on buffered
runs (no PG row yet) the update raised RecordNotFound and surfaced as
a 500.
Switches to mutateWithFallback. PG hits go through the existing
select-dedupe-validate-update flow with MAX_TAGS_PER_RUN enforcement.
Buffered-QUEUED hits apply append_tags via Lua (atomic dedup against
existing snapshot tags). busy snapshots wait for drainer resolution
then update PG. 404 / 503 surface for missing / hung cases.
The MAX_TAGS_PER_RUN cap is skipped on the buffered side — the
drainer's engine.trigger doesn't enforce it either, matching the
pre-buffer trigger path. Pushing the cap into the snapshot-mutate Lua
is a possible follow-up.
Per the Q4 mollifier-cancel design — first mutation endpoint.
engine.createCancelledRun: new run-engine method that writes a CANCELED
TaskRun row directly from a buffer snapshot. Skips queue insertion,
waitpoint creation, and concurrency reservation (run never executes).
Emits runCancelled so the existing handler writes the TaskEvent
cancellation row. P2002 from double-pop is caught and returns the
existing row without re-emitting.
Drainer bifurcation: mollifierDrainerHandler routes to
createCancelledRun when snapshot.cancelledAt is set. Cancel-wins-
over-trigger — customer intent is terminal.
Cancel route: wraps the call in mutateWithFallback. PG-row hits go
through the existing CancelTaskRunService. Buffered-QUEUED hits land
a mark_cancelled patch on the snapshot via mutateSnapshot. busy
snapshots wait for drainer resolution then call the PG service
against the resulting row. 404 / 503 surface for genuine missing
or drainer-hung cases.
Known follow-up: the Q3 wait-and-bounce for cancel-of-buffered-FAILED
relies on the drainer eventually writing a SYSTEM_FAILURE PG row on
terminal materialisation failure. That drainer-side write isn't
implemented yet (the failed-drain path today only marks the buffer
entry hash FAILED). Cancel-of-state-3 will currently 503 after 2s
instead of returning the SYSTEM_FAILURE row. Acceptable rare-race
behaviour; flagged for a follow-up alongside the drainer sweeper work.
Three integration points that connect B6a's buffer-side primitives to
the customer-facing flow per Q5:
- IdempotencyKeyConcern.handleTriggerRequest falls through to
buffer.lookupIdempotency after a PG miss. Buffered hits return
isCached:true with a synthesised TaskRun via the existing
findRunByIdWithMollifierFallback. Skipped when
resumeParentOnCompletion is set: waitpoint blocking requires a PG
row that doesn't exist yet; the follow-up accept SETNX still
dedupes the trigger itself. Buffer outages fail open to "no cache
hit" so the trigger hot path is never wedged by a transient Redis
issue.
- mollifyTrigger passes idempotencyKey + taskIdentifier through to
buffer.accept. The SETNX race loser receives duplicate_idempotency
with the winner's runId; the API response echoes it with
isCached:true, matching PG-side cache-hit shape.
- ResetIdempotencyKeyService calls buffer.resetIdempotency alongside
the existing PG updateMany. 404 only fires when both stores report
nothing bound. Buffer outage during reset is logged and treated as
a miss; PG-side reset still works.
Composes PG-first (replica) lookup, MollifierBuffer.mutateSnapshot,
and writer-side spin-wait into the Q3 wait-and-bounce flow. Returns
a discriminated outcome rather than throwing Response, so the helper
stays route-agnostic and unit-testable. Phase C mutation endpoints
(tags, metadata-put, reschedule, cancel) consume this in upcoming
commits.
Wait knobs default to safetyNetMs=2000, pollStepMs=20, pgTimeoutMs=50
per Q3. Each PG poll is bounded by pgTimeoutMs via Promise.race so
a slow query can't burn the whole safety-net budget. Abort signal is
respected between polls (callers should pass getRequestAbortSignal()
when running in a request handler).
Also exports SnapshotPatch and MutateSnapshotResult from
@trigger.dev/redis-worker so webapp consumers can type-check their
callers of mutateSnapshot.
The mollifier read-fallback's SyntheticRun previously carried just
enough fields for the API retrieve/trace/spans/events/attempts/metadata
endpoints. Phase C5 (replay) needs the buffered run to be passable
where ReplayTaskRunService expects a TaskRun. Adds the missing fields:
id, runtimeEnvironmentId, engine, workerQueue, queue, concurrencyKey,
machinePreset, realtimeStreamsVersion, seedMetadata, seedMetadataType,
runTags. All populated from the engine-trigger snapshot embedded in
the buffer entry.
Also closes a pre-existing typecheck gap in
ApiRetrieveRunPresenter.synthesiseFoundRunFromBuffer — workerQueue
wasn't populated and the file had been failing tsc. Now surfaces the
buffered run's workerQueue, defaulting to "main" (the Prisma default).
Phase A2/A5/A6 of the mollifier API parity work — three more read
endpoints get the buffer fallback, plus two route-level bug fixes for
endpoints that had no GET handler.
A2 spans/{spanId}: discriminated PG vs buffered findResource (mirrors
the trace endpoint pattern from A1). For buffered runs, the only valid
spanId is the snapshot's queued spanId (recorded at gate time, reused
as the run's root spanId on materialise). That spanId returns a minimal
"span exists, no execution data yet" shape; any other spanId is a
deterministic 404.
A5 attempts: pre-existing route-bug fix. The route only had `action`
(POST creates attempt); GET hit Remix's "no loader" 400 with an internal
error message. New loader returns 200 `{ attempts: [] }` for both PG
and buffered runs. The detailed attempt list belongs on the v3 retrieve
endpoint, not here.
A6 metadata GET: same pre-existing route-bug. The route only had PUT;
GET had no handler. New loader returns
`{ metadata, metadataType }` from either the PG row or the buffer
snapshot. PG-side reads only the two fields it needs.
A3 events and A4 result need no code change — events already works via
`ApiRetrieveRunPresenter.findRun`'s existing buffer fallback (querying
events for a buffered traceId naturally returns `{ events: [] }`), and
result's 404 message "Run either doesn't exist or is not finished"
already covers both buffered-not-in-PG and PG-delayed-not-finished
cases.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Env-var lookups via `GET
/api/v1/projects/:projectRef/envvars/:slug/:name` run a Prisma
`findMany` on `EnvironmentVariableValue` filtered by `environmentId` +
`isSecret`. The only existing indexes are the primary key and a unique
on `(variableId, environmentId)`, so `environmentId` is never the
leading column — the planner falls back to a Parallel Seq Scan over the
whole table to find what is, in practice, a handful of rows per
environment.
Two changes:
- Add a btree index on `EnvironmentVariableValue(environmentId)` so the
planner switches to an index scan. The composite `(variableId,
environmentId)` unique stays in place; the new index is purely additive.
- Route the `findMany` inside `getEnvironmentWithRedactedSecrets`
through the read replica via a new `replicaClient` constructor param on
the repository (defaulting to `$replica`, mirroring how `prismaClient`
defaults to `prisma`). Writes and read-after-write methods stay on the
primary.
## Test plan
- [ ] `pnpm run typecheck --filter webapp`
- [ ] Confirm `EXPLAIN` plan flips from Parallel Seq Scan to an index
scan
- [ ] Existing env-var route tests still pass
Phase A1 of the mollifier API parity work. The trace endpoint now falls
back to the mollifier buffer when the run isn't in Postgres yet, returning
an empty trace skeleton (200) instead of a 404 for buffered runs.
`findResource` is restructured into a discriminated union — `pg` for real
TaskRun rows, `buffer` for synthesised shapes from the buffer entry. The
authorization branch handles both shapes; the handler renders an empty
`{ trace: { traceId, rootSpan: null, events: [] } }` for buffered runs so
the customer sees the same 200 contract they'd get for a freshly-triggered
PG run that hasn't had its first span recorded yet.
See _plans/2026-05-19-mollifier-api-parity.md for the full plan and
_plans/2026-05-19-mollifier-listing-design.md for the read-fallback
companion infrastructure this builds on.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary
- Dashboard loaders for runs / sessions / batches / schedule-detail
threw bare `Error("X not found")` when a slug didn't resolve. Remix
surfaces this as a 500 and Sentry captures it via auto-instrumentation,
producing ongoing noise from real users following stale preview-branch
or deleted-resource links (the URLs in those Sentry events all carry
`?_data=routes/...`, i.e. client-side revalidation, not full-page
navigation).
- Added a `throwNotFound(statusText)` helper in
`app/utils/httpErrors.ts` that throws a Response with status 404,
matching the established pattern in sibling routes (agents, alerts,
bulk-actions, etc.).
- Migrated 5 loader sites to `throwNotFound` (4× "Environment not
found", 1× "Schedule not found").
- Migrated 1 loader site (`runs._index` project branch) to
`redirectWithErrorMessage("/", request, "Project not found")` to match
the pre-existing convention used by every other dashboard route's
project-not-found branch.
- Intentionally **not** touched: bare `throw new Error("X not found")`
inside `resources.*` action routes (sit inside try/catch blocks that
already redirect with a flash message), the invariant assertion in
`vercel.connect.tsx`, and the admin config check in
`admin.api.v1.runs-replication.backfill.ts`.
## Where the fix is visible
Normal browser navigation to these URLs doesn't reach the buggy loaders
— the parent env-layout
(`_app.orgs.$organizationSlug.projects.$projectParam.env.$envParam/route.tsx`)
already filters missing envs/projects and redirects/404s before the
child loader runs. The bug fires exclusively when Remix calls a single
child loader via `?_data=routes/...`, which happens during client-side
navigation or `useRevalidator`. That matches every Sentry event URL.
## Test plan
- [x] Unit test for the new helper —
`apps/webapp/test/httpErrors.test.ts`
- [x] `pnpm run typecheck --filter webapp` clean
- [x] Manual verification via Playwright on `main` vs this branch (6
cases): main returns 500 for each defective `_data` URL; branch returns
404 or 204 + `X-Remix-Redirect` as designed
- [x] Verified user-visible 404 catch boundary on `schedules/<missing>`
(the one case reachable via normal nav)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary
The S2 access-token cache key was `${basin}:${streamPrefix}` — purely
server-derived but blind to the **scope/ops list** hardcoded one method
away. When the ops list changes in code (e.g. #3644 added `trim` so
`chat.agent`'s per-turn trim chain can issue `AppendRecord.trim()`),
pre-deploy tokens still in cache get returned to SDK callers for up to
the token's TTL (24h default), surfacing as `Operation not permitted`
403s on any op outside the old scope.
## Fix
Lift the ops list to a module constant and fold its sorted-join
fingerprint into the cache key:
```ts
const S2_TOKEN_OPS = ["append", "create-stream", "trim"] as const;
const S2_TOKEN_OPS_FINGERPRINT = [...S2_TOKEN_OPS].sort().join(",");
// in getS2AccessToken
const cacheKey = `${this.basin}:${this.streamPrefix}:${S2_TOKEN_OPS_FINGERPRINT}`;
// in s2IssueAccessToken
scope: { /* ... */ ops: [...S2_TOKEN_OPS], /* ... */ }
```
The fingerprint is derived from the single source of truth, so any
future scope change auto-invalidates without anyone remembering to bump
a literal version. The Unkey L1 (in-memory LRU) and L2 (Redis) layers
share the same key derivation, so both reset together on the next deploy
with no manual cache busting.
## Test plan
- [ ] `pnpm run typecheck --filter webapp`
- [ ] Run a multi-turn `chat.agent` chat via `references/ai-chat` and
confirm no `chat.agent: trim failed; will retry next turn` warn span
fires across turn-completes.
## Summary
Companion to #3536, which patched routes that already had a leaking
`catch (e) { return json({error: e.message}, 500) }`. That pattern can't
reach routes which have no catch in the first place — when those throw,
Remix's default error path serializes `error.message` into the response
body, and the SDK then wraps the leaked string as `TriggerApiError`.
Across 28 raw api.v1 loaders/actions plus one dashboard polling
endpoint, each handler now:
- Wraps its body in `try { ... } catch (error) { ... }`.
- Re-throws `Response` instances so auth helpers' `throw json(...)` /
`throw redirect(...)` pass through unchanged.
- Logs non-Response errors via `logger.error` so server-side visibility
is preserved.
- Returns a generic body — `{"error": "Internal Server Error"}` 500 for
raw API routes, or `{ changelogs: [] }` 200 for the polling widget
(degrade silently across transient blips; the consumer hook already
coped with empty payloads).
For six routes where #3536 left an inner try/catch covering only a
service call (`alertChannels`, `batches.results`,
`deployments.finalize`, `deployments.background-workers`,
`deployments.promote`, `projects.background-workers`): an outer
try/catch is added so auth/parsing failures are also sanitized. Inner
typed-error handling (`ServiceValidationError` → 422 with message, etc.)
is preserved exactly.
For two routes whose existing catch returned 400 + `error.message`
(`api.v1.authorization-code`, `api.v1.orgs.\$orgParam.projects` action):
the body is sanitized to a generic per-route string. **Status code stays
400** — clients that key on the 4xx/5xx distinction (and the SDK's
no-retry-on-4xx behavior) are unaffected.
## Test plan
- [x] \`pnpm run typecheck --filter webapp\`
- [x] Per-route synthetic-throw probe: inject \`throw new
Error("SYNTHETIC ...")\` at the top of each catch'd try, curl the route
with a dummy bearer, confirm the response body is the generic shape and
that the synthetic message lands server-side via \`logger.error\`. 29
routes verified.
- [x] Real-P1001 probe on the envvars loader: \`docker stop database\`
mid-flight, confirm response is generic 500 (not the leaked Prisma
message).
- [x] Sampled legitimate 4xx/2xx paths across each pattern variant
(naked-wrap, partial-expanded, 400-preserved) to confirm the wraps don't
interfere with normal control flow.
Mollified runs were materialising with `TaskRun.traceContext = {}`, so every
downstream `recordRunDebugLog` (engine QUEUED/EXECUTING/FINISHED, run:notify,
attempt events) drew a fresh traceId with null parentId. The run-detail
trace view rendered only the root span; the rest of the tree was orphaned.
The pass-through path gets traceContext for free via `traceEventConcern.traceRun`
populating the W3C traceparent. The mollifier path skips that wrapper, so seed
`traceContext.traceparent` from the queued span at the call site before
handing the snapshot to engine.trigger.
Also fixes the drainer side: wrap the `mollifier.drained` span + `engine.trigger`
call in a `context.with(parentContext, ...)` built from the snapshot's
traceId/spanId. Without this `mollifier.drained` lived in a fresh trace and
the engine instrumentation inside it inherited an empty active context.
Regression tests:
- `triggerTask.test.ts` — asserts the buffered snapshot carries a valid W3C
traceparent that references the snapshot's traceId/spanId.
- `mollifierDrainerHandler.test.ts` — captures the active traceId at the
moment engine.trigger is invoked and asserts it matches the snapshot's
traceId.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary
- Introduce the Mollifier: a Redis-backed buffer for `trigger()` API
calls during traffic spikes, with a per-env trip evaluator and a drainer
ack-loop.
- Phase 1 is dual-write monitoring — every mollified trigger is buffered
to Redis AND continues to `engine.trigger`. No customer-facing behaviour
change.
- Telemetry events: `mollifier.would_mollify`, `mollifier.buffered`,
`mollifier.drained`, plus the `mollifier.decisions` counter.
- Gated behind a feature flag (default off).
## Test plan
- [x] `pnpm run test --filter @trigger.dev/redis-worker`
- [x] `pnpm run test --filter webapp -- mollifier`
- [x] Manual: with flag off, no behaviour change vs main
- [x] Manual: with flag on + threshold lowered, observe
`mollifier.buffered` + `mollifier.drained` log pairs with matching
`runId`
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Dan Sutton