Per the Q4 mollifier-cancel design — first mutation endpoint.
engine.createCancelledRun: new run-engine method that writes a CANCELED
TaskRun row directly from a buffer snapshot. Skips queue insertion,
waitpoint creation, and concurrency reservation (run never executes).
Emits runCancelled so the existing handler writes the TaskEvent
cancellation row. P2002 from double-pop is caught and returns the
existing row without re-emitting.
Drainer bifurcation: mollifierDrainerHandler routes to
createCancelledRun when snapshot.cancelledAt is set. Cancel-wins-
over-trigger — customer intent is terminal.
Cancel route: wraps the call in mutateWithFallback. PG-row hits go
through the existing CancelTaskRunService. Buffered-QUEUED hits land
a mark_cancelled patch on the snapshot via mutateSnapshot. busy
snapshots wait for drainer resolution then call the PG service
against the resulting row. 404 / 503 surface for genuine missing
or drainer-hung cases.
Known follow-up: the Q3 wait-and-bounce for cancel-of-buffered-FAILED
relies on the drainer eventually writing a SYSTEM_FAILURE PG row on
terminal materialisation failure. That drainer-side write isn't
implemented yet (the failed-drain path today only marks the buffer
entry hash FAILED). Cancel-of-state-3 will currently 503 after 2s
instead of returning the SYSTEM_FAILURE row. Acceptable rare-race
behaviour; flagged for a follow-up alongside the drainer sweeper work.
Composes PG-first (replica) lookup, MollifierBuffer.mutateSnapshot,
and writer-side spin-wait into the Q3 wait-and-bounce flow. Returns
a discriminated outcome rather than throwing Response, so the helper
stays route-agnostic and unit-testable. Phase C mutation endpoints
(tags, metadata-put, reschedule, cancel) consume this in upcoming
commits.
Wait knobs default to safetyNetMs=2000, pollStepMs=20, pgTimeoutMs=50
per Q3. Each PG poll is bounded by pgTimeoutMs via Promise.race so
a slow query can't burn the whole safety-net budget. Abort signal is
respected between polls (callers should pass getRequestAbortSignal()
when running in a request handler).
Also exports SnapshotPatch and MutateSnapshotResult from
@trigger.dev/redis-worker so webapp consumers can type-check their
callers of mutateSnapshot.
The mollifier read-fallback's SyntheticRun previously carried just
enough fields for the API retrieve/trace/spans/events/attempts/metadata
endpoints. Phase C5 (replay) needs the buffered run to be passable
where ReplayTaskRunService expects a TaskRun. Adds the missing fields:
id, runtimeEnvironmentId, engine, workerQueue, queue, concurrencyKey,
machinePreset, realtimeStreamsVersion, seedMetadata, seedMetadataType,
runTags. All populated from the engine-trigger snapshot embedded in
the buffer entry.
Also closes a pre-existing typecheck gap in
ApiRetrieveRunPresenter.synthesiseFoundRunFromBuffer — workerQueue
wasn't populated and the file had been failing tsc. Now surfaces the
buffered run's workerQueue, defaulting to "main" (the Prisma default).
## Summary
- Dashboard loaders for runs / sessions / batches / schedule-detail
threw bare `Error("X not found")` when a slug didn't resolve. Remix
surfaces this as a 500 and Sentry captures it via auto-instrumentation,
producing ongoing noise from real users following stale preview-branch
or deleted-resource links (the URLs in those Sentry events all carry
`?_data=routes/...`, i.e. client-side revalidation, not full-page
navigation).
- Added a `throwNotFound(statusText)` helper in
`app/utils/httpErrors.ts` that throws a Response with status 404,
matching the established pattern in sibling routes (agents, alerts,
bulk-actions, etc.).
- Migrated 5 loader sites to `throwNotFound` (4× "Environment not
found", 1× "Schedule not found").
- Migrated 1 loader site (`runs._index` project branch) to
`redirectWithErrorMessage("/", request, "Project not found")` to match
the pre-existing convention used by every other dashboard route's
project-not-found branch.
- Intentionally **not** touched: bare `throw new Error("X not found")`
inside `resources.*` action routes (sit inside try/catch blocks that
already redirect with a flash message), the invariant assertion in
`vercel.connect.tsx`, and the admin config check in
`admin.api.v1.runs-replication.backfill.ts`.
## Where the fix is visible
Normal browser navigation to these URLs doesn't reach the buggy loaders
— the parent env-layout
(`_app.orgs.$organizationSlug.projects.$projectParam.env.$envParam/route.tsx`)
already filters missing envs/projects and redirects/404s before the
child loader runs. The bug fires exclusively when Remix calls a single
child loader via `?_data=routes/...`, which happens during client-side
navigation or `useRevalidator`. That matches every Sentry event URL.
## Test plan
- [x] Unit test for the new helper —
`apps/webapp/test/httpErrors.test.ts`
- [x] `pnpm run typecheck --filter webapp` clean
- [x] Manual verification via Playwright on `main` vs this branch (6
cases): main returns 500 for each defective `_data` URL; branch returns
404 or 204 + `X-Remix-Redirect` as designed
- [x] Verified user-visible 404 catch boundary on `schedules/<missing>`
(the one case reachable via normal nav)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Mollified runs were materialising with `TaskRun.traceContext = {}`, so every
downstream `recordRunDebugLog` (engine QUEUED/EXECUTING/FINISHED, run:notify,
attempt events) drew a fresh traceId with null parentId. The run-detail
trace view rendered only the root span; the rest of the tree was orphaned.
The pass-through path gets traceContext for free via `traceEventConcern.traceRun`
populating the W3C traceparent. The mollifier path skips that wrapper, so seed
`traceContext.traceparent` from the queued span at the call site before
handing the snapshot to engine.trigger.
Also fixes the drainer side: wrap the `mollifier.drained` span + `engine.trigger`
call in a `context.with(parentContext, ...)` built from the snapshot's
traceId/spanId. Without this `mollifier.drained` lived in a fresh trace and
the engine instrumentation inside it inherited an empty active context.
Regression tests:
- `triggerTask.test.ts` — asserts the buffered snapshot carries a valid W3C
traceparent that references the snapshot's traceId/spanId.
- `mollifierDrainerHandler.test.ts` — captures the active traceId at the
moment engine.trigger is invoked and asserts it matches the snapshot's
traceId.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary
- Introduce the Mollifier: a Redis-backed buffer for `trigger()` API
calls during traffic spikes, with a per-env trip evaluator and a drainer
ack-loop.
- Phase 1 is dual-write monitoring — every mollified trigger is buffered
to Redis AND continues to `engine.trigger`. No customer-facing behaviour
change.
- Telemetry events: `mollifier.would_mollify`, `mollifier.buffered`,
`mollifier.drained`, plus the `mollifier.decisions` counter.
- Gated behind a feature flag (default off).
## Test plan
- [x] `pnpm run test --filter @trigger.dev/redis-worker`
- [x] `pnpm run test --filter webapp -- mollifier`
- [x] Manual: with flag off, no behaviour change vs main
- [x] Manual: with flag on + threshold lowered, observe
`mollifier.buffered` + `mollifier.drained` log pairs with matching
`runId`
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The three "dual-write" tests inherited from phase-1 were asserting
invariants that phase-3 deliberately abandoned when the mollify path
moved from "buffer.accept + engine.trigger" to "buffer.accept +
synthetic result, drainer replays later":
- `mollify action triggers dual-write` — rewritten to assert the new
contract: synthetic `MollifySyntheticResult` (run.friendlyId,
isCached:false, notice.code = "mollifier.queued"), buffer.accept
fires with the canonical engine.trigger snapshot, NO Postgres row
(the run materialises only when the drainer replays).
- `engine.trigger throwing AFTER buffer.accept` — deleted. Phase-3
never invokes engine.trigger on the mollify path, so the scenario
is structurally impossible.
- `debounce match produces an orphan buffer entry` — deleted. Phase-3's
C1 debounce bypass at the gate (returns pass_through for debounce
triggers) means the mollify branch is never entered for debounced
requests. The C1 invariant is pinned at mollifierGate.test.ts:440;
duplicating it at the trigger-task layer adds nothing.
Net: 6 mollifier integration tests → 4, all 4 passing, no coverage
gap (gate-level + drainer-handler-level tests own the deleted
scenarios' invariants).
Brings in every phase-2 fix without regressing the phase-3 surface.
Resolutions:
* `apps/webapp/app/v3/mollifier/mollifierDrainer.server.ts` — kept
phase-3's `MollifierDrainer<MollifierSnapshot>` generic and the
`createDrainerHandler({ engine, prisma })` + `isRetryablePgError`
wiring; added phase-2's `MollifierConfigurationError` class and
switched every env read to `TRIGGER_MOLLIFIER_*`. The doc comment on
`getMollifierDrainer` now points at `mollifierDrainerWorker.server.ts`
instead of the legacy `worker.server.ts`.
* `apps/webapp/app/runEngine/services/triggerTask.server.ts` — kept
phase-3's gate inputs (`options: { debounce, oneTimeUseToken,
parentTaskRunId, resumeParentOnCompletion }` for the C1/C3/F4
bypasses) and the `#buildEngineTriggerInput` refactor; wrapped the
gate call in phase-2's `this.isMollifierGloballyEnabled()` short-
circuit so a trigger with the feature off skips the
`GateInputs` allocation + `mollifier.decisions{outcome=pass_through}`
increment. `mollifierOutcome` is `GateOutcome | null`, downstream
access uses optional chaining. Phase-2's phase-1 dual-write block is
dropped — phase-3's mollify branch already writes via
`mollifyTrigger` and returns a synthetic result.
Phase-2 fixes preserved (verified post-merge):
- `TRIGGER_MOLLIFIER_*` env-var prefix on every consumer
- `TRIGGER_MOLLIFIER_DRAINER_ENABLED` separate per-replica switch
- `MollifierConfigurationError` rethrow in the drainer bootstrap
- Factory create + bootstrap start split (`drainer.start()` only
runs after SIGTERM handlers register)
- `signalsEmitter.on` instead of `process.once`
- `MollifierDrainer.stop` deadline-timer `clearTimeout` in finally
- `vi.fn` handler spies removed from `drainer.test.ts`
- `isMollifierGloballyEnabled` DI hook on `RunEngineTriggerTaskService`
- Legacy `worker.server.ts` header
- New `mollifierDrainerWorker.server.ts` error-classification test
- Updated `.server-changes/mollifier-burst-protection.md`
Pre-existing on phase-3, NOT introduced by this merge (verified by
`git diff origin/mollifier-phase-3 -- apps/webapp/test/engine/triggerTask.test.ts`
showing only `isMollifierGloballyEnabled: () => true` additions):
- `mollifier · mollify action triggers dual-write` asserts
`prisma.taskRun.findFirst` returns a row, but phase-3's
`mollifyTrigger` writes to the buffer and returns a synthetic
result without invoking `engine.trigger`. Test name + assertions
still reflect phase-1 dual-write semantics.
- `mollifier · engine.trigger throwing AFTER buffer.accept` expects
the call to reject; phase-3 returns synthetic before
`engine.trigger` is reached, so the injected throw never fires.
- `mollifier · debounce match produces an orphan buffer entry` —
phase-3 explicitly skips the buffer branch when debounce is set
(`\!body.options?.debounce` guard), so no orphan accept happens.
These three tests describe phase-1 monitoring behaviour on phase-3
code that abandoned dual-write. Updating them is phase-3's own tech
debt, separate from this merge.
## Summary
Long-running chat agents were filling `session.out` forever — every
`chat.agent` turn appended to the same S2 stream with no trim, and the
Sessions dashboard re-streamed the entire history from `seq_num=0` on
every page load. After this change the agent appends an S2 `trim`
command record after each `trigger:turn-complete`, pointing back at the
previous turn-complete's seq_num. `session.out` stays roughly one turn
long at steady state, regardless of session age.
`trigger:turn-complete` and `trigger:upgrade-required` move from
`chunk.type`-shaped data records into header-form control records under
a uniform `trigger-control` namespace. Built-in transports
(`TriggerChatTransport`, `AgentChat`, the dashboard's `AgentView`)
handle the new shape transparently. Custom transports need a one-line
filter on the `trigger-control` header — see the rewritten "Records on
session.out" section in the client-protocol docs.
The Sessions detail page in the dashboard fetches the agent's per-turn
S3 snapshot via a presigned URL and seeds the transcript view, then
SSE-tails from the snapshot's `lastOutEventId`. Bandwidth and
time-to-first-render scale with unread turns instead of session
lifetime.
Resume contract is now explicit: single-turn-boundary resume always
works (the prior turn-complete is still on the stream), the S2 trim is
eventually consistent over 10-60s, and multi-turn-away resume falls back
to a snapshot reload.
Adds the smallest DI surface to `initMollifierDrainerWorker` (`isEnabled`
and `getDrainer`, both optional, default to live env/singleton) so the
catch-block policy can be tested without manipulating module-level env:
- rethrows MollifierConfigurationError — deterministic misconfig
escapes, which is what makes the production-path crash on boot
(the call site in entry.server.tsx runs sync at module top level,
before `process.on("uncaughtException", ...)` is registered, so an
escape becomes a Node default-handler exit-1).
- rethrows when `name === "MollifierConfigurationError"` even when
`instanceof` fails — covers the Remix dev hot-reload realm edge
case where the catch holds a stale class reference.
- swallows non-configuration errors — a transient Redis blip during
buffer init shouldn't take the whole webapp down.
- no-op when disabled — the factory isn't invoked when the enabled
predicate returns false.
Also updates the existing mollifier server-changes note to: rename env
vars to TRIGGER_MOLLIFIER_* prefix, document the TRIGGER_MOLLIFIER_DRAINER_ENABLED
split for multi-replica drainer placement, and call out the new fail-loud
behaviour on drainer misconfiguration.
The previous commit added a perf short-circuit at the call site that
read `env.TRIGGER_MOLLIFIER_ENABLED` directly. That broke three
mollifier integration tests in CI: the tests inject a custom
`evaluateGate` via the existing DI seam expecting the buffer-write
branch to be reached, but CI has no `.env` (the `apps/webapp/.env`
symlink target is absent), the Zod default `"0"` wins, the call site
short-circuits to `null` before the injected gate runs, and
`buffer.accepted` stays empty.
Make the global-enabled check itself injectable:
- New constructor opt `isMollifierGloballyEnabled?: () => boolean`,
defaulting to `() => env.TRIGGER_MOLLIFIER_ENABLED === "1"`. Each
DI hook now represents one decision (gate, buffer, global-enabled),
so a test that wants the buffer-write branch reached can inject
`isMollifierGloballyEnabled: () => true` alongside its custom gate.
- Call site now reads `this.isMollifierGloballyEnabled()` instead of
`env.TRIGGER_MOLLIFIER_ENABLED` directly. In production, with no DI
override, the default closure resolves `env` exactly once per call
just as before — same perf win when the flag is off.
- All six mollifier DI injection sites in triggerTask.test.ts now also
pass `isMollifierGloballyEnabled: () => true` so the tests' DI
surface matches the new contract regardless of CI env state.
All MOLLIFIER_* env vars renamed to TRIGGER_MOLLIFIER_*. The mollifier
primitive is generic — buffer + drainer + trip evaluator with no
trigger-specific assumptions at the redis-worker layer — but this
PR's webapp wiring is specifically the trigger-task mollifier, with
PII-sensitive payload handling and trigger-flow semantics. If we later
mollify another surface (deploys, schedules, etc.) those will want
their own env-var namespace; pre-prefixing now avoids a breaking
rename later.
Renames are mechanical: schema keys in env.server.ts, env.* references
across the v3/mollifier* modules, and a handful of doc-comment
mentions. The bootstrap fallback that has DRAINER_ENABLED default to
the ENABLED value is updated to read TRIGGER_MOLLIFIER_ENABLED from
process.env too. Code-side naming (classes, file names, the literal
word "mollifier") stays unchanged — the rename is env-var only.
## Summary
The trigger-task hotpath used to early-return without a DB query when a
caller passed both a queue override and a per-trigger TTL — the hottest
configuration on the trigger API. Adding `triggerSource` to the resolver
so the runs-list "Source" filter could distinguish STANDARD / SCHEDULED
/
AGENT runs removed those early-returns, costing +2 DB queries per
trigger
on non-locked calls and +1 on locked calls.
This change caches `BackgroundWorkerTask` metadata (`ttl`,
`triggerSource`,
`queueId`, `queueName`) in Redis so the resolver can satisfy every
caller
configuration with a single `HGET` on the warm path. PG fallback on miss
back-fills the cache.
Follow-up to #3542.
## Design
Two key spaces:
- `task-meta:env:{envId}` — the "current worker" view, refreshed at
every
deploy promotion. 24h safety TTL.
- `task-meta:by-worker:{workerId}` — used for `lockToVersion` triggers.
Immutable post-create. 30d sliding TTL so historical workers age out.
Cache writes use Lua scripts via `defineCommand` so `DEL` + `HSET` +
`EXPIRE` land atomically — concurrent readers never see the empty
intermediate state of a naive pipeline. Read-path back-fill uses
single-field upserts so concurrent back-fills don't wipe each other's
siblings.
The cache lives behind its own `TASK_META_CACHE_REDIS_*` env-var prefix
that falls back to the default `REDIS_*` set, so operators can route the
cache to a dedicated Redis instance if they want.
The service/instance file split (`taskMetadataCache.server.ts` for the
pure class, `taskMetadataCacheInstance.server.ts` for the env-wired
singleton) mirrors the existing `runsReplicationService` /
`runsReplicationInstance` pattern.
## Test plan
- [ ] `pnpm run typecheck --filter webapp`
- [ ] `pnpm run test ./test/engine/triggerTask.test.ts --run` — 8
existing tests untouched + 5 new tests covering warm cache, cold
miss with back-fill, queue + ttl path, by-worker vs env keyspace,
and the promotion cache write
- [ ] End-to-end against a dev worker: registering writes both keyspaces
with the expected TTLs, and `redis-cli HGETALL
"tr:task-meta:env:<envId>"`
returns the cached entries
## Benchmark
Measured `DefaultQueueManager.resolveQueueProperties` against a real
Postgres + Redis (vitest `containerTest`, single-host docker). 500
sequential calls and 2,000 parallel calls (concurrency=50) per scenario,
request shaped as `{ taskId, queue: "bench-queue", ttl: "5m" }` — the
hot path this PR restores.
```
sequential (one in flight at a time):
[noop cache (baseline)] n=500 mean=1.423ms p50=1.394ms p95=1.735ms p99=2.629ms max=11.100ms
[redis cache, cold ] n=500 mean=1.346ms p50=1.283ms p95=1.688ms p99=2.463ms max=5.058ms
[redis cache, warm ] n=500 mean=0.084ms p50=0.078ms p95=0.105ms p99=0.156ms max=1.129ms
speedup (warm vs baseline, sequential): 16.95x
parallel (concurrency=50):
[noop cache (baseline)] n=2000 mean=10.069ms p50=8.850ms p95=14.718ms p99=31.887ms total=405ms ops/s=4,940
[redis cache, warm ] n=2000 mean=0.614ms p50=0.568ms p95=1.189ms p99=1.432ms total=25ms ops/s=80,389
throughput speedup (warm vs baseline, parallel): 16.27x
```
Read:
- **Warm cache cuts resolver latency 17×** at p50 — from ~1.4 ms to ~78
µs per call.
- **Cold cache is on par with baseline** — the extra `HGET` miss adds
<50 µs against the two Postgres queries that follow, so the worst case
is not worse than today.
- **Under burst load (50 concurrent triggers)**, the baseline's p99
jumps to ~32 ms as Postgres connections queue up; warm stays at ~1.4 ms.
The cache moves the saturation point from ~5k ops/s (PG pool) to ~80k
ops/s (single-client Redis pipelining).
Caveats: single-host docker, local Postgres + Redis, resolver-only
measurement (excludes the rest of the trigger transaction). Prod adds
region-local Redis RTT (~0.3–0.8 ms) which shifts warm absolute numbers
up but keeps the ratio intact.
`triggerTask` is the highest-throughput code path in the system and the
webapp CLAUDE.md forbids new DB queries there. The previous resolver fell
back to `flag()` (a Prisma read against `FeatureFlag`) when the org had
no `mollifierEnabled` override, which added a query to every trigger
whenever `MOLLIFIER_ENABLED=1`. The fleet-wide kill switch already lives
in `MOLLIFIER_ENABLED`; rollout is per-org via `Organization.featureFlags`
JSON, matching `canAccessAi`/`hasComputeAccess`/etc. Drop the fallback so
the resolver is purely in-memory.
Tests no longer need a postgres testcontainer or `makeFlag(prisma)`; the
per-org isolation suite now asserts directly on `Organization.featureFlags`
shape and adds a regression test for the no-override -> false contract.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Previously the drainer rotated per-env: an org with N busy envs got N
scheduling slots per tick. A noisy tenant with many envs would drain
proportionally faster than a quiet tenant with one env. Switch to
hierarchical rotation: pick orgs round-robin (capped by maxOrgsPerTick),
then pick one env per picked org (also rotating).
Implementation is drainer-side only — no buffer or Lua changes. The
drainer caches envId→orgId from popped entries; envs not yet cached are
treated as their own pseudo-org for one tick, so cold start matches the
old per-env behaviour and converges to hierarchical once cache is hot
(usually within one tick). Cache and cursors reset on start() alongside
the existing cursor reset.
API change: maxEnvsPerTick → maxOrgsPerTick on MollifierDrainerOptions,
MOLLIFIER_DRAIN_MAX_ENVS_PER_TICK → MOLLIFIER_DRAIN_MAX_ORGS_PER_TICK on
the webapp env. Same default (500). Operators tune for "typical orgs
with pending entries" rather than env count.
Trade-off: total per-tick pops drop from O(envs) to O(orgs). For an org
with N envs, each env's individual drainage rate is 1/N of what it was,
but the tenant overall is bounded the same way as a single-env tenant —
which is the fairness contract.
Tests:
- Renamed maxEnvsPerTick references throughout existing tests; old
behaviour still holds at cold cache (each env = pseudo-org).
- New "heavy org with many envs does not dominate vs light org" pins
the post-warm-up ~1:1 drainage ratio between a 6-env org and a 1-env
org over a sustained 20-tick run.
- New "within an org, envs are rotated round-robin across ticks" pins
the inner env cursor's behaviour for a single multi-env org.
- Cursor-reset test renamed and now asserts cache+cursors all reset.
Also removed an outdated test-count comment in
apps/webapp/test/engine/triggerTask.test.ts that listed "four tests"
when reality has moved on.
The TripDecision header comment claimed each webapp instance maintained
its own rate counter — wrong. evaluateTrip writes to mollifier:rate:\${envId}
with no per-instance prefix, so all replicas pointing at the same Redis
share the key. The threshold is the fleet-wide ceiling.
Also wrap d.evaluator() in evaluateGate in try/catch so a throwing
evaluator falls back to no-divert. The default createRealTripEvaluator
catches its own errors, but the contract should be symmetric with the
already-wrapped resolveOrgFlag call so a future evaluator can't break
the trigger hot path's fail-open contract.
resolveOrgFlag now checks the per-org Organization.featureFlags override
in-memory before falling back to the global flag() helper, so the common
per-org enablement path resolves without a Prisma round-trip on every
trigger call. evaluateGate also wraps the flag resolution in try/catch
and fails open to false on error, mirroring the trip evaluator.
The per-org isolation suite uses `postgresTest`, which spins up a fresh Postgres testcontainer per case. On CI the 5s vitest default regularly times out on container start before the test body runs. Match the 30s `vi.setConfig` used by other postgresTest suites in this app.
- Gate drainer init on WORKER_ENABLED so only worker replicas run the polling loop.
- Update the enqueueSystem TTL comment now that delayed/pending-version are first enqueues.
- Correct the mollifier gate docstring to describe the fixed-window counter and tripped-key rearm.
- Swap findUnique for findFirst in the trigger task test to match the webapp Prisma rule.
The unit cascade tests in mollifierGate.test.ts import the gate module,
which transitively pulls in ~/db.server. That module constructs the
prisma singleton at import time and eagerly calls $connect(), which
fails against localhost:5432 in the unit-test shard and surfaces as an
unhandled rejection that fails the whole vitest run. Mocking the module
keeps the cascade tests pure and leaves the postgresTest cases on the
testcontainer-fixture prisma untouched.
The mollifier gate's resolveOrgFlag was a global feature-flag lookup
named as if org-scoped. Phase-1 plan and design doc both intended
per-org gating; the implementation regressed because the global
flag() helper has no orgId parameter.
Adopt the existing per-org feature-flag pattern (used by canAccessAi,
canAccessPrivateConnections, compute beta gating): pass
`Organization.featureFlags` through as `flag()` overrides. Per-org
opt-in now works admin-toggleable via the existing
Organization.featureFlags JSON column — no schema migration needed.
- mollifierGate: revert resolveFlag/flagEnabled back to
resolveOrgFlag/orgFlagEnabled (the name now matches reality).
GateInputs gains `orgFeatureFlags`; the default resolver passes
them as overrides to `flag()`.
- triggerTask.server.ts: thread `environment.organization.featureFlags`
into the gate call.
- tests: three new postgresTest cases exercise the real DB-backed
resolveOrgFlag end-to-end, proving (a) per-org opt-in isolation,
(b) unrelated beta flags don't bleed across, (c) per-org overrides
take precedence over the global FeatureFlag row.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- changeset: drop "deferred" wording — phase-1 actively dual-writes + runs
the drainer ack loop.
- worker.server.ts: wrap mollifier drainer init in try/catch + register
SIGTERM/SIGINT handlers so the polling loop stops cleanly on shutdown.
- bufferedTriggerPayload: only serialise idempotencyKeyExpiresAt when an
idempotencyKey is present (avoid impossible orphan-expiry payloads).
- mollifierTelemetry: narrow recordDecision reason to DecisionReason union
to keep OTEL attribute cardinality bounded.
- mollifierGate: rename resolveOrgFlag → resolveFlag. The underlying
FeatureFlag table is global by key, so the "org" prefix was misleading;
per-org gating is out of scope for phase-1.
- tests: drop vi.fn mocks. mollifierGate now uses plain closure spies;
mollifierTripEvaluator runs against a real MollifierBuffer backed by a
redisTest container (closed client exercises the fail-open path).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase 1 of the trigger-burst smoothing initiative. Adds the A-side trip
evaluator (atomic Lua sliding-window per env) and wires it into the trigger
hot path. When the per-org mollifierEnabled feature flag is on AND the
evaluator says divert, the canonical replay payload is buffered to Redis
(via buffer.accept) AND the trigger continues through engine.trigger —
i.e. dual-write. The drainer pops + acks (no-op handler) to prove the
dequeue mechanism works end-to-end. Operators audit by joining
mollifier.buffered (write) and mollifier.drained (consume) logs by runId.
Buffer primitives hardened:
- accept is idempotent on duplicate runId (Lua EXISTS guard)
- pop skips orphan queue references (entry HASH TTL'd while runId queued)
- fail no-ops on missing entry (no partial FAILED hash leak)
- mollifier:envs set pruned on draining pop, restored on requeue
- 16-row truth-table test enumerates the gate cascade
- BufferedTriggerPayload defines the canonical replay shape Phase 2 will
use to invoke engine.trigger
- payload hash for audit-equivalence computed off the hot path (in the
drainer) to avoid CPU during a spike
Regression tests in apps/webapp/test/engine/triggerTask.test.ts pin the
mollifier integration:
- validation throws BEFORE the gate runs (no orphan buffer write on
rejected triggers)
- mollify dual-write happy path (Postgres + Redis both reflect the run)
- pass_through path does NOT call buffer.accept
- engine.trigger throwing AFTER buffer.accept leaves an orphan
(documented behaviour — drainer auto-cleans; audit-trail surfaces it)
- idempotency-key match short-circuits BEFORE the gate is consulted
- debounce match produces an orphan (documented behaviour — Phase 2
must lift handleDebounce upfront before buffer.accept)
Behaviour with MOLLIFIER_ENABLED=0 (default) is byte-identical to main.
With MOLLIFIER_ENABLED=1 and the flag off, only mollifier.would_mollify
logs fire (no buffer state). With the flag on, dual-write activates.
Includes two opt-in *.fuzz.test.ts suites (gated on FUZZ=1) that
randomise operation sequences against evaluateTrip and the drainer to
find timing edges. They are clearly marked TEMPORARY in their headers.
Redis-backed burst-smoothing layer behind MOLLIFIER_ENABLED=0 (default).
With the kill switch off, the gate short-circuits on its first env check
and production behaviour is identical to main.
@trigger.dev/redis-worker:
- MollifierBuffer: atomic Lua-backed FIFO with accept / pop / ack /
requeue / fail + TTL. Per-env queues with HSET entry storage,
atomic RPOP + status transition, FIFO retry ordering.
- MollifierDrainer: generic round-robin worker with concurrency cap,
retry semantics, and a stop deadline to avoid livelock on a hung
handler. Phase 3 will wire the handler to engine.trigger().
- Full testcontainers-backed test suite (21 tests).
apps/webapp:
- evaluateGate cascade-check (kill switch -> org feature flag ->
shadow mode -> trip evaluator -> mollify / shadow_log / pass_through).
Dependencies injected for testability; the trip evaluator stub
returns { divert: false } in phase 1.
- Inserted into RunEngineTriggerTaskService.call() before
traceEventConcern.traceRun. The mollify branch throws (unreachable
in phase 1).
- Lazy MollifierBuffer + MollifierDrainer singletons; no Redis
connection unless MOLLIFIER_ENABLED=1.
- 12 MOLLIFIER_* env vars (all safe defaults) and a mollifierEnabled
feature flag in the global catalog.
- Drainer booted from worker.server.ts on first import.
- Read-fallback stub for phase 3.
- Gate cascade tests + .env loader so env.server validates in vitest
workers.
Phase 2 will land the real trip evaluator; phase 3 will activate the
buffer-write + drain path.
Concurrent `POST /api/v1/deployments` requests for the same environment
race on the `WorkerDeployment(environmentId, version)` unique
constraint. Both requests read the same latest deployment via
`findFirst`, compute the same next version via
`calculateNextBuildVersion`, and both attempt
`prisma.workerDeployment.create()` — one wins, the other crashes with
Prisma `P2002`. The bug is a classic TOCTOU between the version read and
the version write; it's been latent since the version-assignment logic
was first added but only fires when two deploys land within milliseconds
of each other (CI matrices, retried CLI calls, webhook-triggered
redeploys).
## Approach
Extracts the version assignment + create into a small helper
`createDeploymentWithNextVersion`
(`apps/webapp/app/v3/services/initializeDeployment/createDeploymentWithNextVersion.server.ts`).
The helper retries on `P2002 (environmentId, version)` up to 5 times
with randomised 5–50ms jitter so N concurrent racers don't loop in
lockstep. Each attempt re-reads the latest version, recomputes via
`calculateNextBuildVersion`, and re-runs the caller's `buildData`
callback so version-dependent fields (image ref tag, friendlyId) are
always consistent with the version actually persisted. A `logger.warn`
fires per collision so the retry rate is observable in production logs.
When retries are exhausted, the helper throws a dedicated
`DeploymentVersionCollisionError` carrying `environmentId`, `attempts`,
and `lastAttemptedVersion`, with the original
`PrismaClientKnownRequestError` attached as `cause`. Sentry walks the
`cause` chain natively, so contention exhaustion shows up as a
distinguishable wrapper exception linked to the underlying `P2002`
rather than a generic unique-constraint violation that looks identical
to every other duplicate-key bug.
The behavioural change is limited to "catch P2002 and retry instead of
crashing." The image ref computation stays inside the builder callback
(same call site as before the refactor), so ECR / non-ECR behaviour, S2
stream creation order, and all downstream side effects are unchanged.
## Non-goals
- No new database migrations, no schema changes, no isolation-level /
locking changes. A serialisable transaction or advisory lock would also
fix this; retry-on-conflict is the smaller change that keeps the
existing version-allocation logic intact.
- Does not touch the analogous `calculateNextBuildVersion` call in
`createBackgroundWorker.server.ts`, which likely has the same race shape
against `BackgroundWorker`'s unique constraint — flagged as a follow-up.
## Test plan
- [x] `pnpm run typecheck --filter webapp` passes (no new errors in the
modified files).
- [x] Three real-Postgres tests in
`apps/webapp/test/createDeploymentWithNextVersion.test.ts` via
`containerTest`:
- 5 concurrent calls all produce distinct, persistable versions
(`Set(versions).size === concurrency`). The naive read-then-create
version of the helper fails this test with the exact same `P2002` seen
in production; the retry version passes.
- Non-`P2002` errors raised from the `buildData` callback propagate
immediately without retry, builder invoked exactly once.
- With `maxRetries: 0`, concurrent racers surface the wrapped
`DeploymentVersionCollisionError` (not a raw `P2002`); `environmentId`,
`attempts`, `lastAttemptedVersion` are populated and `error.cause.code
=== "P2002"`.
- [x] Existing `apps/webapp/test/getDeploymentImageRef.test.ts` still
green (the file was untouched in the final diff).
## Follow-ups (not in this PR)
- `createBackgroundWorker.server.ts` likely has the same TOCTOU shape
against its background-worker version unique constraint — should use the
same helper.
- Sentry visibility check: confirm `error.cause` chain renders as a
linked exception in the Sentry UI when the wrapped error fires (requires
a sandboxed triggering of the exhaustion path).
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the chat.agent({...}) task definition (server runtime) and the
browser-side TriggerChatTransport + AgentChat that drives it from a
React or Next.js app. The runtime sits on top of the Sessions primitive
and handles the durable conversational task lifecycle.
Server runtime:
- chat.agent({...}) — session-aware task definition
- Lifecycle hooks: onChatStart, onTurnStart, onTurnComplete, onAction,
onValidateMessages, hydrateMessages
- chat.history read primitives for HITL flows
- chat.local, chat.headStart, chat.handover, oomMachine
- Delta-only wire + S3 snapshot reconstruction at run boot
- Actions are no longer turns
Browser transport:
- TriggerChatTransport (ai-sdk Transport): delta-only wire sends,
SSE reconnection with lastEventId resume, stop/abort cleanup,
dynamic accessToken refresh
- AgentChat: direct programmatic API
- useTriggerChatTransport (React hook)
- chat-tab-coordinator: cross-tab leader election
Includes the chat-agent, chat-agent-delta-wire-snapshots,
chat-history-read-primitives, chat-head-start, chat-actions-no-turn,
chat-session-attributes, agent-skills, and mock-chat-agent-test-harness
changesets.
## Summary
Consolidates the webapp's authentication and authorization into a small
set of route helpers, replacing the ad-hoc `requireUser` /
`requireUserId` / `authenticatedEnvironmentForAuthentication` calls
scattered across routes. Same security model, but the per-request flow
(authenticate → authorize → load) now lives in one place per route
family.
Introduces a plugin seam (`@trigger.dev/plugins`) that lets the cloud
build install a richer RBAC implementation without touching webapp code.
The OSS fallback keeps the pre-RBAC permissive behaviour intact, so
self-hosted deployments work unchanged.
Adds a comprehensive end-to-end auth test suite that didn't exist before
— 193 `it()` blocks (vitest reports ~199 after `it.each` expansion)
covering API key, PAT and JWT auth across the public API surface, plus
dashboard session auth for admin pages.
## Changes
### Plugin contract — `@trigger.dev/plugins`
`RoleBaseAccessController` interface authoritative for both OSS
(fallback) and cloud (enterprise plugin):
- `authenticateBearer(request, { allowJWT? })` — API-key / public-JWT
auth, returns env + ability
- `authenticateSession(request, { userId, organizationId?, projectId?
})` — dashboard auth, caller resolves `userId` from the session cookie
and passes it in (no `helpers.getSessionUserId` callback — decouples the
plugin host from session-cookie code)
- `authenticatePat(request, { organizationId?, projectId? })` — PAT
auth, returns identity + `lastAccessedAt` so the host can throttle the
per-request update
- `authenticateAuthorize*` variants for the auth-and-check-in-one-call
cases
- `isUsingPlugin(): Promise<boolean>` — capability flag for UI /
branching where plugin-present-ness matters; replaces the
sentinel-string coupling that had `personalAccessToken.server` matching
`"RBAC plugin not installed"` literally
### Dashboard auth (started, partial rollout)
Admin and settings pages migrated to a unified `dashboardLoader` /
`dashboardAction` helper that authenticates the session, runs an
authorization check, and exposes the result to the route. Other
dashboard routes still on the old pattern; remaining migration tracked
in TRI-8730.
Migrated routes:
- `admin.*` (14 admin / back-office / feature-flags / LLM-models /
notifications / orgs / concurrency pages)
- `_app.orgs.$organizationSlug.settings.team`
- `_app.orgs.$organizationSlug.settings.roles`
### API / realtime / engine auth (complete for the migrated families)
71 routes migrated to a unified `apiBuilder` that centralizes Bearer /
PAT / Public-JWT authentication and applies the per-route authorization
check before the handler runs. Includes:
- `api.v1.*` and `api.v2.*` and `api.v3.*` — tasks, runs, batches,
queues, prompts, deployments, query, sessions, waitpoints, packets,
workers, idempotency keys
- `realtime.v1.*` — runs, batches, sessions, streams
- `engine.v1.*` — dev / worker-action protocols
29 routes still on the legacy `authenticateApiRequest*` helpers —
tracked as a post-deploy follow-up in TRI-9228.
Multi-resource auth direction is now explicit at the call site via
`anyResource(...)` (OR) and `everyResource(...)` (AND). Bare arrays no
longer typecheck — fixes a class of bug where a JWT scoped to one
resource could implicitly access others under OR semantics.
PAT auth path consolidated: was three DB queries per request (legacy
`authenticateApiRequestWithPersonalAccessToken` findFirst +
`rbac.authenticatePat` join + `lastAccessedAt` update). Now one query in
the steady state — plugin returns `lastAccessedAt`, host smart-skips the
update via JS-side throttle when fresh.
Side effect: action aliases preserved historic JWT scope semantics where
the new model is stricter (e.g. a `write:tasks` JWT now also satisfies
`trigger` / `batchTrigger` / `update` actions on the same resource —
matched at the auth boundary, not in the route handler).
### Backwards-compat fixes
The strict-match model regressed several real-world JWT shapes. Each
preserved via explicit `anyResource(...)` entries in the route's authz
block:
- **Batch retrieve routes** (`api.v1.batches.$batchId`, `api.v2.*`,
`realtime.v1.batches.*`) accept `read:runs` JWTs again (pre-RBAC
literal-match superScope behaviour)
- **Runs list routes** (`api.v1.runs`, `realtime.v1.runs`) accept
type-level `read:tasks` / `read:tags` on unfiltered queries (matched the
legacy `Object.keys` iteration semantic)
- **PAT/OAT auth shape** normalized through `toAuthenticated` so all
auth methods return the same slim `AuthenticatedEnvironment` (was:
API-key returned the slim shape but PAT/OAT returned raw Prisma
`Decimal` / no `orgMember`)
- **Scope `:` preservation** in resource ids — `read:tags:env:staging`
now correctly identifies the tag id as `env:staging`, not `env`
### Slim `AuthenticatedEnvironment`
Extracted to `@trigger.dev/core/v3/auth/environment` — a structural
shape independent of `@trigger.dev/database`. The plugin contract
returns this; webapp consumers import from there; the cloud plugin
(Drizzle) returns the same shape without Prisma's `Decimal` class
leaking into the public surface. Lets internal-packages (run-engine,
etc.) refer to `AuthenticatedEnvironment` without pulling Prisma in.
### Auth test suite (new — `*.e2e.full.test.ts`)
193 e2e tests run against a real spawned webapp + Postgres (no mocks).
Coverage matrix:
- **API key auth** — read / write / trigger / batchTrigger / deploy
actions across runs, batches, deployments, prompts, queues, query,
sessions, input-streams, waitpoints, tasks, idempotency keys; multi-key
resources (a run carries batch / tag / task identifiers — auth must
accept any matching scope)
- **Personal Access Token auth** — comprehensive matrix: scope match,
scope mismatch, missing scope, expired token, malformed token
- **Public JWT auth** — sub-vs-URL environment resolution, expired JWTs,
signature verification, scope checking, otu (one-time-use) token
semantics, branch-environment signing-key fallback
- **Dashboard session auth** — admin-only pages reject non-admins;
per-action gating
- **Cross-cutting edge cases** — revoked API key grace window, JWT
cross-environment isolation, MissingResource branch behaviour
### Hygiene cleanups
- Deleted dead `app/services/authorization.server.ts` (legacy
`checkAuthorization` + types — no live consumers post-migration) and its
orphaned test
- Dropped the never-populated `scopes` field from
`ApiAuthenticationResultSuccess`
- `scheduleEmail` moved out of `email.server.ts` into its own module —
breaks a `commonWorker → marqs/V1` import chain that was poisoning the
auth test graph
- OSS Roles page shows a deployment-aware empty state ("Roles aren't
available in this self-hosted deployment" vs the plan-upsell copy) via
`rbac.isUsingPlugin()`
- Team action handler: explicit per-intent ability gates
(`manage:billing` for purchase-seats, `manage:members` for set-role +
remove-member with self-leave carve-out)
### Cross-repo coordination
All public-package contract changes paired in `triggerdotdev/cloud#763`
(rbac-packages branch) — the enterprise plugin implements the same
`RoleBaseAccessController` interface against Drizzle.
## Test plan
- [x] `pnpm run typecheck --filter webapp` clean
- [x] `pnpm --filter webapp exec vitest run --config
vitest.e2e.full.config.ts` — 193/193 pass (requires Docker for
testcontainers)
- [x] Spot-check an authed API endpoint with a valid + invalid API key
against a local stack
- [x] Spot-check the migrated admin pages render and gate non-admins
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary
Stamps the active OpenTelemetry `trace_id` and `span_id` onto every
Sentry event captured from the webapp, so engineers can copy a
`trace_id` from a Sentry issue and search for the corresponding trace in
any OTel-aware backend. Also adds an `otel_sampled` tag to indicate
whether the trace was head-sampled — a cheap signal for whether the link
will resolve to span data or hit a missing trace.
## Why
Sentry and OTel were OTel-disconnected: `apps/webapp/sentry.server.ts`
initialised Sentry with `skipOpenTelemetrySetup: true`, and no
error-capture site (`logger.server.ts`, the Remix-wrapped `handleError`,
the root `ErrorBoundary`) attached OTel context to the event. With many
spans/sec across services, getting from a Sentry issue to its trace was
guesswork.
## Approach
Single global Sentry event processor, registered immediately after
`Sentry.init`. On each event it reads
`trace.getActiveSpan()?.spanContext()` via `@opentelemetry/api`, then
writes:
- `event.contexts.trace.trace_id` and `event.contexts.trace.span_id`
(Sentry's native trace context fields)
- `event.tags.otel_sampled` = `"true"` | `"false"` (derived from
`traceFlags`)
If no active span (module-load errors, scheduled timers without a
context, primary cluster process), the processor returns the event
unmodified — Sentry's default propagation context fills in.
Implementation is co-located in `apps/webapp/sentry.server.ts` (no
separate helper module — `sentry.server.ts` is built standalone by
esbuild and a separate import would have required a new bundling step).
Helper functions are exported so the unit tests can reach them without
re-running `Sentry.init`.
## Non-goals (deliberate)
- No sample rate change. ~95% of Sentry events will carry a `trace_id`
that returns no spans in the tracing backend (head-sampled out). The
`otel_sampled` tag makes that obvious at a glance. Raising find-rate is
a separate conversation with cost trade-offs.
- No user/org tags or `Sentry.setUser` (would need auth-helper +
per-request scope wiring across multiple worker entrypoints — separate
ticket).
- Webapp image only. No changes to supervisor or CLI workers.
## Test plan
- [x] Unit tests in `apps/webapp/test/sentryTraceContext.server.test.ts`
— 9 tests covering: helper returns \`undefined\` with no active span;
returns \`traceId\`/\`spanId\`/\`sampled=true\` for a recording span;
returns \`sampled=false\` for a non-recording span; processor leaves the
event unchanged with no active span; processor stamps
\`trace_id\`/\`span_id\` onto \`contexts.trace\`; preserves existing
\`contexts.trace\` fields; tags \`otel_sampled\` correctly for both
sampled and non-sampled cases; never throws if \`@opentelemetry/api\`
access throws.
- [x] \`pnpm run typecheck --filter webapp\` passes.
- [x] Manually verified end-to-end against a sandboxed Sentry project:
confirmed both sampled and non-sampled traces correctly populate
\`contexts.trace.trace_id\` matching the OTel ids logged from the
loader, and the \`otel_sampled\` tag appears with the expected value.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
<img width="2284" height="2028" alt="CleanShot 2026-05-01 at 18 53
50@2x"
src="https://github.com/user-attachments/assets/4f58cbb1-0168-40fb-a523-017f2ba625a1"
/>
## Performance
- **Per-request DB hit**: `getUserId` runs `getEffectiveSessionDuration`
(User lookup + Org `aggregate`) on *every* authenticated request,
including each fetcher poll. Consider caching the effective duration in
the session cookie with a short TTL (e.g. 60s) and revalidating in the
background.
- **Double session commit in `root.tsx`**: `getUser` already runs the
expiry check; then `commitAuthenticatedSessionLazy` commits the cookie
again. Fine, but doubles `Set-Cookie` headers on every page load — worth
a quick perf check.
## Correctness / Edge cases
- **Lazy backfill assumes a root.tsx hit first**: users whose first
post-deploy request is a fetcher/API route (`/resources/*`) skip the
backfill until they navigate to a page. Not a security hole, but
`getUserId` could backfill itself for completeness.
- **No upper bound on `Organization.maxSessionDuration`**: admin API
accepts `1` second, which would instant-logout every member on next
request. Add a `min(60)` (or `min(300)` to match the lowest user option)
to the Zod schema.
- **No clock-skew tolerance**: `isSessionExpired` is exact-millisecond.
Multi-instance deploys with skewed clocks could log users out a few
seconds early/late. Probably fine for the 5-min minimum, but worth
noting.
## Security
- **Auto-logout audit log lacks IP/orgId**: HIPAA forensics typically
wants source IP and which org context. Currently logs only `userId` +
path. IP isn't PII for audit purposes; orgIds help correlate. Add both.
- **Cookie `Max-Age` is 1 year regardless of user's setting**:
intentional (server-side `issuedAt` is the source of truth), but
reviewers will ask. Add a one-line comment on the cookie config
explaining why.
## API surface
- **`maxSessionDuration` is admin-PAT only**: no in-app UI for org
owners to set/change their own cap. If this is "Trigger staff sets it
during HIPAA onboarding", say so in the PR description; otherwise add an
org-settings UI.
- **Auto-submit dropdown has no confirmation**: misclicking "5 minutes"
immediately shortens the user's session window with no undo. Consider a
save button or 3-sec undo toast.
## Schema / migration
- **`User.sessionDuration NOT NULL DEFAULT 31556952`**: instant on PG
11+ (metadata-only), but call out in the PR description so reviewers
don't worry about a table rewrite on the User table.
- **No DB-level constraint matching `SESSION_DURATION_OPTIONS`**: if the
option list changes, existing users keep orphaned values. The dropdown's
tag-along behaviour hides this — fine for now, but if you ever drop an
option you'll need a backfill.
## UX
- **Session expiry only fires on next request**: an idle authenticated
tab keeps showing UI past the cap (until SSE/polling catches it, ~60s).
Add a client-side timer based on the user's effective duration that
triggers a fetcher to `/account` or `/logout` at expiry.
- **No "you were signed out" message on logout**: users hitting their
cap are bounced to `/` with no explanation. Was intentionally reverted
in this PR — call that out so reviewers don't request it.
## Tests
- Unit coverage on `sessionDuration.server.ts` is solid (215 lines).
Missing: integration test for `getUserId` → expired session → redirect
to `/logout`, and one for the loader's clamping fix (the most recent
bug). Add at least the second one to lock in the regression.
---------
Co-authored-by: Matt Aitken <matt@mattaitken.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary
Each successful PAT (`PersonalAccessToken`) or OAT
(`OrganizationAccessToken`) authentication issues a `prisma.X.update({
lastAccessedAt: new Date() })` to bump the timestamp. For tokens used at
high frequency (CLI clients, integrations) this generates a per-request
DB write that is mostly redundant — the `lastAccessedAt` field is only
surfaced on the settings page so users can decide which tokens to
revoke, and "within the last 5 minutes" is plenty of granularity for
that.
## Design
Replace each unconditional `update` with a conditional `updateMany`
whose `WHERE` requires the existing `lastAccessedAt` to be `NULL` or
strictly older than 5 minutes:
```ts
await prisma.personalAccessToken.updateMany({
where: {
id: personalAccessToken.id,
OR: [
{ lastAccessedAt: null },
{ lastAccessedAt: { lt: new Date(Date.now() - PAT_LAST_ACCESSED_THROTTLE_MS) } },
],
},
data: { lastAccessedAt: new Date() },
});
```
The conditional runs inside the SQL `UPDATE`, so concurrent auths can't
race into a double-write.
No schema change. No migration. No new infrastructure. Throttle is a
hardcoded constant (`5 * 60 * 1000`) — easy to revisit.
## Test plan
- [x] `pnpm run typecheck --filter webapp`
- [x] `pnpm vitest run ./test/services/personalAccessToken.test.ts
./test/services/organizationAccessToken.test.ts` — 6/6 pass, verifying
the throttle `WHERE` clause is constructed correctly and the `update` is
skipped on token-not-found / wrong-prefix paths
> ⚠️ **Not released yet.** This PR is the server-side foundation only.
The SDK changes that customers will actually use (`chat.agent`
migration, `chat.createStartSessionAction`, `useTriggerChatTransport`
updates) live on a separate branch and ship together in an upcoming
`@trigger.dev/sdk` prerelease. Until that prerelease is published, this
surface is reachable only via direct HTTP.
## What this gives Trigger.dev users
A new first-class primitive, **Session**, for durable, task-bound,
bidirectional I/O that outlives any single run. Sessions are the run
manager for `chat.agent` going forward, and they unblock anything else
that needs "one identifier, many runs over time" with a stable channel
pair the client can write to and subscribe to.
### Use cases unblocked
- **Chat agents that persist across many runs.** One session per chat
(keyed on your own `chatId` via `externalId`), turns 1..N attach to the
same Session, the UI subscribes once and keeps receiving output as new
runs take over.
- **Approval loops and long-running tasks with user feedback.** The task
waits on `.in`, the client writes to `.in`, the server enforces
no-writes-after-close.
- **Workflow progress streams that live past the run.** Subscribe to
`.out` after the task finishes to replay history.
- **Resume-next-day flows.** A session is a durable row, not a transient
stream. Send a message a day later and the server triggers a fresh run
on the same session.
### How it works (Session-as-run-manager)
A Session row is task-bound (`taskIdentifier` + `triggerConfig` are
required) and owns its current run via `currentRunId` +
`currentRunVersion` for optimistic claim. Three trigger paths:
1. **Session create** — `POST /api/v1/sessions` creates the row and
triggers the first run synchronously.
2. **Append-time probe** — `POST
/realtime/v1/sessions/:session/in/append` checks if the current run is
alive; if it has terminated (idle exit, crash, etc.), the server
triggers a new run before processing the append.
3. **End-and-continue handoff** — `POST
/api/v1/sessions/:session/end-and-continue`, called by the running
agent, triggers a fresh run and atomically swaps `currentRunId`. Used by
`chat.requestUpgrade()` for version handoffs.
Every triggered run is recorded in the `SessionRun` audit table with a
reason (`initial`, `continuation`, `upgrade`, `manual`).
## Public API surface
### Control plane
- `POST /api/v1/sessions` — create. Idempotent on `(env, externalId)`.
Triggers the first run, returns the session and a session-scoped public
access token. Returns 409 if the upserted row is already closed.
- `GET /api/v1/sessions/:session` — retrieve by friendlyId
(`session_abc...`) or by your own externalId (server disambiguates by
prefix).
- `GET /api/v1/sessions` — list with filters (`type`, `tag`,
`taskIdentifier`, `externalId`, derived `status` ACTIVE/CLOSED/EXPIRED,
created-at range) and cursor pagination. Backed by ClickHouse.
- `PATCH /api/v1/sessions/:session` — update tags / metadata /
externalId.
- `POST /api/v1/sessions/:session/close` — terminate. Idempotent,
hard-blocks new server-brokered writes.
- `POST /api/v1/sessions/:session/end-and-continue` — agent-only handoff
to a fresh run.
### Realtime
- `PUT /realtime/v1/sessions/:session/:io` — initialize a channel.
Returns S2 credentials in headers so high-throughput clients can write
direct to S2.
- `GET /realtime/v1/sessions/:session/:io` — SSE subscribe. Supports
Last-Event-ID resume and an opt-in `X-Peek-Settled: 1` header that
fast-closes the stream when the upstream is already settled
(`trigger:turn-complete`), eliminating long-poll wait on
reconnect-on-reload paths.
- `POST /realtime/v1/sessions/:session/:io/append` — server-side
appends.
- `POST /api/v1/runs/:runFriendlyId/session-streams/wait` — runs wait on
a session stream as a waitpoint, with a race-check to avoid suspending
if data already landed.
### Auth scopes
`sessions` is a new resource type. `read:sessions:{id}`,
`write:sessions:{id}`, `admin:sessions:{id}` flow through the existing
JWT validator. Session-scoped public access tokens minted by the server
replace browser-held trigger-task tokens for chat-style flows — the
browser never sees a run identifier or a run-scoped token in steady
state.
## What's coming after this PR
- **SDK + chat.agent migration**: separate branch, separate PR, ships in
the next `@trigger.dev/sdk` prerelease alongside this server deploy.
Customers using the prerelease `chat.agent` will follow the [upgrade
guide](https://github.com/triggerdotdev/trigger.dev/blob/docs/tri-7532-ai-sdk-chat-transport-and-chat-task-system/docs/ai-chat/upgrade-guide.mdx).
- **Dashboard surfaces**: dedicated agent list, agent playground, agent
view on the run dashboard. Tracking separately.
## Implementation notes
- **Postgres `Session` table**: scalar scoping columns (`projectId`,
`runtimeEnvironmentId`, `environmentType`, `organizationId`) without
FKs, matching the January TaskRun FK-removal decision. Point-lookup
indexes only — list queries go to ClickHouse. Terminal markers
(`closedAt`, `expiresAt`) are write-once.
- **ClickHouse `sessions_v1`**: ReplacingMergeTree, partitioned by
month, ordered by `(org_id, project_id, environment_id, created_at,
session_id)`. Tags indexed via `tokenbf_v1` skip index.
- **`SessionsReplicationService`**: mirrors `RunsReplicationService`
exactly — leader-locked logical replication consumer,
`ConcurrentFlushScheduler`, retry with exponential backoff + jitter,
identical metric shape. Dedicated slot + publication so the two consume
independently.
- **S2 keys**: `sessions/{addressingKey}/{out|in}`. The existing
`runs/{runId}/{streamId}` key format for run-scoped streams is
untouched.
- **Optimistic claim**: `ensureRunForSession` triggers a run upfront
(cheap to cancel if it loses the race), then attempts an `updateMany`
keyed on `currentRunVersion`. Loser cancels its triggered run and reuses
the winner's. No DB lock held across the trigger.
### What did NOT change
Run-scoped `streams.pipe` / `streams.input` and the existing
`/realtime/v1/streams/{runId}/...` routes are unchanged. Sessions are
net-new — not a reshaping of the current streams API.
## Deploy notes
- Set `SESSION_REPLICATION_CLICKHOUSE_URL` and
`SESSION_REPLICATION_ENABLED=1` to enable the replication consumer.
- The `Session` table needs `REPLICA IDENTITY FULL` set on the prod
source DB before the publication is created (same one-time DDL we did
for `TaskRun`). Required for delete events to carry full column values.
- Cross-form authorization on the `GET /api/v1/sessions/:session` loader
(a JWT minted for either form authorizes both URL forms). Action routes
are URL-form-specific, matching how the SDK mints PATs.
## Verification
- Webapp typecheck clean (10/10).
- `apps/webapp/test/sessionsReplicationService.test.ts` — round-trip
tests for insert/update/delete through Postgres logical replication into
ClickHouse via testcontainers.
- Live end-to-end against local dev: create + retrieve (both forms) +
update + close, `.out.initialize` + `.out.append` x2 + `.in.send` +
`.out.subscribe` over SSE, list with all filter combinations +
pagination, `end-and-continue` swap, `X-Peek-Settled` fast-close
(verified in browser via reconnect-on-reload and via curl). Replicated
row lands in ClickHouse within ~1s.
- Multi-round Devin + CodeRabbit review feedback addressed
(read-after-write paths use `prisma` writer, info-leak on auth-routes
masked as 403, peek-settled discriminator parsing fix, etc.).
## Test plan
- [ ] `pnpm run typecheck --filter webapp`
- [ ] `pnpm run test --filter webapp
./test/sessionsReplicationService.test.ts --run`
- [ ] Start the webapp with `SESSION_REPLICATION_CLICKHOUSE_URL` and
`SESSION_REPLICATION_ENABLED=1`. Confirm the slot and publication
auto-create on boot.
- [ ] `POST /api/v1/sessions` and verify the row replicates to
`trigger_dev.sessions_v1` within a couple of seconds.
- [ ] `POST /api/v1/sessions/:id/close`, then confirm `POST
/realtime/v1/sessions/:id/out/append` returns 400.
- [ ] Reuse a closed session's `externalId` on `POST /api/v1/sessions`
and confirm 409.
- [ ] `GET /realtime/v1/sessions/:id/out` with `X-Peek-Settled: 1` after
a turn completes and confirm `X-Session-Settled: true` response header +
immediate close.
Adds a minimal end-to-end test harness that spawns the compiled webapp
as a child
process against a throwaway Postgres container, plus a baseline of 8
auth-behaviour
tests. These tests will be used as a regression check before and after
the upcoming
apiBuilder RBAC migration to confirm auth behaviour is unchanged.
## What's included
**`internal-packages/testcontainers/src/webapp.ts`** (new)
Spawns `build/server.js` with a dynamically allocated port, polls
`/healthcheck`,
and exposes `WebappInstance` and `startTestServer()` (postgres container
+ webapp +
PrismaClient in one call). Key details:
- Uses `process.execPath` so the correct Node binary is found in forked
test processes
- Sets `NODE_PATH` to `node_modules/.pnpm/node_modules` so pnpm-hoisted
transitive
deps (e.g. `eventsource-parser`) resolve correctly inside the subprocess
- Overrides both `PORT` and `REMIX_APP_PORT` so Vite's automatic `.env`
loading
doesn't override the dynamically allocated port
**`internal-packages/testcontainers/package.json`**
Adds `./webapp` sub-path export so tests can `import from
"@internal/testcontainers/webapp"`.
**`internal-packages/testcontainers/src/index.ts`**
Exports `createPostgresContainer` (used internally by `webapp.ts`).
**`apps/webapp/test/helpers/seedTestEnvironment.ts`** (new)
Creates a minimal org → project → environment row set with random
suffixes.
**`apps/webapp/test/api-auth.e2e.test.ts`** (new)
8 tests across two suites:
- API-key bearer: valid key (auth passes, 404), missing header (401),
invalid key (401), error body shape
- JWT bearer: valid JWT on JWT-enabled route (passes), valid JWT on
non-JWT route (401), empty-scope JWT (403), wrong signing key (401)
## How to run
```bash
# Build required first (one-time)
pnpm run build --filter webapp
cd apps/webapp && pnpm exec vitest run test/api-auth.e2e.test.ts
```
## Test plan
- [x] All 8 tests pass against the current webapp build
- [x] Webapp healthcheck returns 200 on startup
- [ ] CI passes
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
## Problem
When `batchTrigger()` is called with large payloads, each item's payload
is uploaded to R2 server-side during the streaming loop before being
enqueued. This makes the loop slow — around 3 seconds per item. Workers
pick up and execute each item as it's enqueued, running concurrently
with the ongoing stream.
For the last item in the batch, a race exists between the streaming loop
finishing and the batch completion cleanup:
1. The loop enqueues the last item and returns from `enqueueBatchItem()`
2. A waiting worker picks up the item almost instantly and executes it
3. `recordSuccess()` fires, `processedCount` hits the expected total,
`finalizeBatch()` runs
4. `cleanup()` deletes all Redis keys for the batch, including
`enqueuedItemsKey`
5. The streaming loop exits and calls `getBatchEnqueuedCount()` — reads
the now-deleted key — returns 0
The count check finds `enqueuedCount (0) !== batch.runCount`, falls
through to a Postgres fallback, but the fallback only checked `sealed`.
The BatchQueue completion path sets `status = COMPLETED` in Postgres
without setting `sealed = true` (that's the streaming endpoint's job),
so the fallback misses it too.
This causes the endpoint to return `sealed: false`. The SDK treats this
as retryable and retries up to 5 times with exponential backoff. Each
retry calls `enqueueBatchItem()`, which reads the batch meta key from
Redis — also deleted by `cleanup()` — and throws "Batch not found or not
initialized" (500). The final retry gets a 422 because the batch is
already COMPLETED, which the SDK does not retry, causing an `ApiError`
to be thrown from `await batchTrigger()` in the parent run — even though
all child runs completed successfully.
## Fix
In the Postgres fallback inside `StreamBatchItemsService`, also check
`status === "COMPLETED"` alongside `sealed`. This covers the
fast-completion path where the BatchQueue finishes all runs before the
streaming endpoint gets to seal the batch normally.
Also switches `findUnique` to `findFirst` per webapp convention.
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Replace the expensive DISTINCT query for task filter dropdowns with a
dedicated TaskIdentifier registry table backed by Redis. Environments
migrate automatically on their next deploy, with a transparent fallback
to the legacy query for unmigrated environments. Also fixes duplicate
dropdown entries when a task changes trigger source, and adds
active/archived grouping for removed tasks. Moves BackgroundWorkerTask
reads in the trigger hot path to the read replica.
A single "fetch failed" from the object store was aborting the entire
batch stream with no retry. Added p-retry (3 attempts, 500ms-2s backoff)
around ploadPacketToObjectStore so transient network errors self-heal
server-side instead of propagating to the SDK.
This allows seamless migration to different object storage.
Existing runs that have offloaded payloads/outputs will continue to use
the default object store (configured using `OBJECT_STORE_*` env vars).
You can add additional stores by setting new env vars:
- `OBJECT_STORE_DEFAULT_PROTOCOL` this determines where new run large
payloads will get stored.
- If you set that you need to set new env vars for that protocol.
Example:
```
OBJECT_STORE_DEFAULT_PROTOCOL=“s3"
OBJECT_STORE_S3_BASE_URL=https://s3.us-east-1.amazonaws.com
OBJECT_STORE_S3_ACCESS_KEY_ID=<val>
OBJECT_STORE_S3_SECRET_ACCESS_KEY=<val>
OBJECT_STORE_S3_REGION=us-east-1
OBJECT_STORE_S3_SERVICE=s3
```
---------
Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
- Added versions filtering on the Errors list and page
- Added errors stacked bars to the graph on the individual error page
---------
Co-authored-by: James Ritchie <james@trigger.dev>
Adds three new top-level columns to the ClickHouse task_runs_v2 table
primarily for analytics:
- `trigger_source` / `root_trigger_source` - extracted from the existing
TaskRun.annotations JSON during WAL
replication
- `is_warm_start` - new nullable boolean on TaskRun in Postgres, set in
the existing taskRun.update() at attempt
start (no additional write). null until the first attempt starts.
Run region is already available via the existing `worker_queue` column
in ClickHouse.
- Automatic LLM cost enrichment for AI SDK spans (streamText,
generateText, generateObject) or any other spans that use semantic
gen_ai attributes with support for 145+ models
- New AI span inspector sidebar showing model, tokens, cost, messages,
tool calls, and response text
- LLM metrics dual-write to ClickHouse `llm_metrics_v1` table for
analytics
- LLM metrics built-in dashboard (unlinked at the moment)
- Provider cost fallback — uses gateway/OpenRouter reported costs from
`providerMetadata` when registry pricing is unavailable
- Prefix-stripping for gateway/OpenRouter model names (e.g.
`mistral/mistral-large-3` matches `mistral-large-3` pricing)
- Admin dashboard for managing LLM model pricing (list, create, edit,
delete, search, test pattern matching)
- Missing models detection page — queries ClickHouse for unpriced models
with sample spans and Claude Code-ready prompts for adding pricing
- AI span seed script (`pnpm run db:seed:ai-spans`) with 51 spans across
12 provider systems for local dev testing
- UI fixes: `completionTokens`/`promptTokens` aliases,
`ai.response.object` display for generateObject, cache read/write token
breakdown
## Screenshots:
<img width="1030" height="104" alt="CleanShot 2026-03-17 at 16 48 54@2x"
src="https://github.com/user-attachments/assets/bc8fccda-e48b-4d0c-bfb1-e620064e5979"
/>
<img width="1094" height="1512" alt="CleanShot 2026-03-17 at 16 49
23@2x"
src="https://github.com/user-attachments/assets/c2424569-d07e-4d67-a436-e8250043a1ee"
/>
<img width="1074" height="1412" alt="CleanShot 2026-03-17 at 16 49
18@2x"
src="https://github.com/user-attachments/assets/22342ac4-4769-45d1-a328-a24fb9a82a50"
/>
<img width="1012" height="2292" alt="CleanShot 2026-03-17 at 16 39
01@2x"
src="https://github.com/user-attachments/assets/59e327d1-6652-4293-8be0-bb8326e5fbc5"
/>
<img width="3680" height="2392" alt="CleanShot 2026-03-15 at 08 29
38@2x"
src="https://github.com/user-attachments/assets/1f77beb8-de67-495b-b890-bcdb8d7f1fe8"
/>
---------
Co-authored-by: James Ritchie <james@trigger.dev>
Dan Sutton