fix/core-api-request-timeout
287 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Eric Allam
|
f1bd11a7ef |
feat(webapp): gracefully shut down the v3 engine behind a flag (#4017)
## Summary Adds a single env flag, `DEPRECATE_V3_ENABLED` (default off), that gracefully winds down the v3 engine (`RunEngineVersion.V1`). While it's off nothing changes, so self-hosted instances still on v3 keep working. When it's on: - Triggers that resolve to v3 are rejected with a clear, actionable error pointing at the [v4 migration guide](https://trigger.dev/docs/migrating-from-v3), instead of silently creating runs that never execute. This covers single triggers, batches, scheduled fires, replays, and `triggerAndWait`, which all funnel through one place. - The legacy `trigger dev` websocket used by v3 CLIs is closed with an upgrade message (v4 CLIs use a different dev transport). - The v3 shared-queue consumer refuses to start, so no deployed v3 runs are dequeued. - The v3 run-lifecycle background jobs (heartbeat timeout, TTL expiry, retry, resume batch/dependency, delayed-run enqueue, and scheduled fires) become no-ops, so abandoned v3 runs stop generating database load. This builds on the existing deploy deprecation flag, which already rejects v3 CLI deploys. ## Design Enforcement is read through one helper, `isV3Disabled()`. Every gate combines it with a per-run or per-project engine check (`isV3Disabled() && engine === "V1"`), so a v4 run that happens to reach a shared service behaves exactly as before. v4 (V2) is never affected. The flag is a hard switch, not a drain: when it's on, in-flight v3 runs are abandoned in place rather than failed or expired, which is the intended behaviour for the final shutdown. |
||
|
Katia Bulatova
|
b1987dc090 |
feat(webapp): billing limits — pause, reject, recovery, and settings UI (#3996)
## Summary Adds Billing Limits to the webapp. Customers can set a monthly spend cap. When usage crosses the limit, billable environments enter a grace period. If the limit is not resolved before grace expires, new triggers are rejected until the organization increases or removes the limit. |
||
|
Chris Arderne
|
4fde283e76 |
chore: format and lint webapp also (#4056)
#3977 added formatting and linting everywhere else. This extends it to the webapp. |
||
|
Eric Allam
|
9ef5cf0055 |
fix(webapp): stop showing the in-dashboard agent to admins by default (#4050)
## Summary The in-dashboard agent button was rendered for all admins and impersonators regardless of the `hasDashboardAgentAccess` flag, so it appeared even where the agent is disabled (for example, floating over the run inspector controls). It is now gated by the flag for everyone, so it stays hidden until the flag is turned on. ## Rollout Both levers default off, so nothing changes for users until deliberately enabled: - **Per-org:** set `hasDashboardAgentAccess` on an org's feature flags to enable the agent for just that org. - **All admins:** set `DASHBOARD_AGENT_ADMIN_PREVIEW=1` to give admins and impersonators an everywhere-preview, independent of the per-org flag. Previously admins bypassed the flag unconditionally, which is why the button showed up before the agent was ready to ship. |
||
|
nicktrn
|
bc605eedaf |
fix(webapp): verify deployment image exists before finalizing (#4049)
A deployment could be marked deployed and promoted to current without its image ever landing in the registry. Finalize trusted the CLI: the v1 path never pushed or checked, and the v2/v3 path skips its own push when the CLI sends `skipPushToRegistry` - which the local-build path always does. In the happy path the CLI pushes the image itself, so this stayed latent. But any deviation - `--no-push`/`--load`, a push that lands in a different registry, or an old CLI - promoted a version whose image can't be pulled, so every run failed at pull time while the deploy itself reported success. This adds a registry existence check after push and before finalize. If the image isn't there, the deploy fails loudly instead of promoting a version that can't start. The check is ECR-only (a no-op for other registries, so self-hosted setups are unaffected) and uses `BatchGetImage`, which the deploy role already allows. It fails open on an ambiguous registry error so the check can't itself turn into a deploy outage. The image reference is the platform-generated value and the lookup is bound to the configured registry host; the CLI-supplied digest is validated before use. Can be turned off with `DEPLOY_IMAGE_VERIFICATION_ENABLED=0` for setups that push images out of band (e.g. an air-gapped registry the platform can't reach). refs TRI-11243 |
||
|
Eric Allam
|
2fa84ea124 |
feat(webapp): gate worker dequeues by worker queue via env var (#4030)
## Summary Adds a `RUN_ENGINE_DEQUEUE_DISABLED_WORKER_QUEUES` setting that refuses worker dequeue requests for the listed worker queues (or base regions), so their runs stay queued instead of being handed to workers that can't run them. Blocked dequeues are counted via a `run_engine.dequeue.blocked` OTel counter (labeled by `worker_queue` and `region`). |
||
|
Eric Allam
|
8890d7a258 |
feat(run-engine,webapp): always report worker queue length metrics (#4029)
## Summary The `runqueue.workerQueue.length` gauge only reported a worker queue's depth while runs were being dequeued from it. When dequeues stop, the metric goes stale or missing, so a queue that has backed up because nothing is draining it can't be alerted on. This adds a small observer that refreshes the observed set of worker queues from the `WorkerInstanceGroup` records on an interval, so every active worker queue (and its scheduled split variant) keeps reporting its length regardless of dequeue activity. The observer is off by default and enabled per service via `RUN_ENGINE_WORKER_QUEUE_OBSERVER_ENABLED`, reads from the read replica, and skips a configurable set of cloud providers (`RUN_ENGINE_WORKER_QUEUE_OBSERVER_EXCLUDED_CLOUD_PROVIDERS`, default `digitalocean`). When enabled it is the source of truth for the observed set, so the per-dequeue registration is skipped on that instance, and it groups by worker queue so the per-instance duplicates collapse to the true depth. Also removes the unused `GET`/`POST /api/v1/workers` endpoints. Their only consumer was a CLI command group that is no longer registered. ## Verification Verified end to end against a local stack: the gauge reports each worker queue's length with no dequeues happening, excludes the configured providers, includes hidden groups, and the removed endpoints return as if they never existed. Added a run-engine test (`workerQueueObservation.test.ts`). |
||
|
Eric Allam
|
c06005b353 |
feat(webapp,sdk): in-dashboard AI agent (#4018)
## Summary Adds an in-dashboard AI agent: a chat panel, reachable from any environment page, that answers questions about your runs, errors, tasks, and analytics, diagnoses why a run failed, charts your data, reads your connected repo's source, and answers product and how-to questions. It is gated behind the `hasDashboardAgentAccess` feature flag (global or per-org, default off), so this PR ships disabled: the launcher is hidden unless the flag is enabled. ## Design The agent runs as a standalone `chat.agent` Trigger task in its own internal package, with no access to the webapp database, Prisma, or ClickHouse. It reads the user's data over the public API, acting as the user via a short-lived delegated user-actor token minted server-side each turn (never in the browser), building on [#3997](https://github.com/triggerdotdev/trigger.dev/pull/3997). The error and analytics tools use [#4005](https://github.com/triggerdotdev/trigger.dev/pull/4005) and the TRQL query API. The first turn of a new chat streams from a warm webapp route (Head Start) while the durable agent boots in parallel. Structured answers (a run-failure diagnosis card, a live chart) render through a small typed view catalog rather than arbitrary markup. A knowledge lane forwards product and how-to questions to the support assistant. Conversation history lives in a separate Drizzle-backed store on its own Postgres schema, kept as a display read-model so it can never corrupt the agent's model context. The SDK changes add an `apiClient` option to `chat.createStartSessionAction` and `chat.headStart`, and keep the Head Start tool-approval tail intact across a custom `prepareMessages` hook so prompt caching and Head Start compose. |
||
|
Eric Allam
|
c6f0769299 |
fix(webapp): bound logs search memory and fix pagination at scale (#4012)
## Summary The logs search page (behind a feature flag) ran ClickHouse out of memory when browsing back over long time ranges. This keeps it within bounded memory and fixes a pagination bug that could skip or duplicate rows at a page boundary. ## Fix Memory: the list query reads in sort-key order, which opens one read stream per part in the window, and on object storage those per-part read buffers dominate peak memory, so it scaled with the number of parts scanned. Two changes bound it: - The logs ClickHouse client caps the per-part read buffers via new env-tunable settings. The object-storage-only setting is opt-in, so it is never sent to a ClickHouse version that lacks it. - Recent-first window narrowing: rows come back newest first, so the presenter probes the most recent window and only widens toward the full requested range when a page is short. A busy environment fills a page from a few recent parts instead of scanning the whole range; a quiet one still returns every row in a couple of cheap reads. Correctness: the keyset cursor ordered on (triggered_timestamp, trace_id), which is not unique because the spans of a trace share both, so rows at a tie could be skipped or duplicated across pages. The cursor and ORDER BY now include span_id, and the cursor is versioned so stale cursors reset to the first page. Guards: the effective page size is capped, and the existing per-query memory limit lets a pathological wide browse fail with an error instead of taking the node down. ## ClickHouse 26.2 The memory fix relies on lazy materialization deferring the wide attributes column to the output rows, which only holds on 26.x. Cloud already runs 26.2, so this moves the dev stack, testcontainers, and CI to match. The ClickHouse test suite passes on 26.2. 🤖 Generated with [Claude Code](https://claude.com/claude-code) |
||
|
Oskar Otwinowski
|
e98a547e6c | feat(sso): SAML/OIDC single sign-on (#3911) | ||
|
Iss
|
c97d246197 |
feat(webapp): sync new orgs + users to Attio CRM on signup (#3896)
Pushes new organizations and users into the Attio CRM at signup time, for Customer Success (TRI-10431). - Orgs → Attio `workspaces`, users → Attio `users`, keyed on Attio's built-in unique `workspace_id` / `user_id` so writes are idempotent upserts. - Runs on the common Redis worker (not inline), so a slow or unavailable Attio never blocks the signup path; failures retry (3 attempts). - Hooks: user-created (alongside the existing Loops call) and org-created (`createOrganization`). - Gated behind `ATTIO_API_KEY`, no key means the sync is skipped entirely, so OSS / self-hosted installs are unaffected. Only creation is covered here (the record "shell"); spend, runs, plan changes, churn, and role/relationship linking are populated by the scheduled full sync, tracked separately. **Deploy note:** requires an Attio API key set as `ATTIO_API_KEY` in the webapp env, with scopes **Records (read-write)** + **Object Configuration (read)**, the assert/upsert endpoint reads object config to resolve the matching attribute. Without the key the sync no-ops. --------- Co-authored-by: Matt Aitken <matt@mattaitken.com> Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
nicktrn
|
7aa871f37b |
feat(webapp): plan-aware compute migration (#3957)
Adds an opt-in mechanism to route a configurable percentage of organizations onto the compute (MicroVM) backing of their region at trigger time, without changing their stored region settings. Routing is gated by three global feature flags - `computeMigrationEnabled`, `computeMigrationFreePercentage`, `computeMigrationPaidPercentage` - plus a per-org `computeMigrationEnabled` override that wins in both directions. A region's compute backing is resolved from a new `WorkerInstanceGroup.region` column: a container group and its MicroVM group share one geo `region`, so the migration swaps the resolved worker queue to the backing group's queue. Orgs are bucketed deterministically by id, so ramping a percentage down keeps a strict subset rather than reshuffling, and a region with no compute backing is never touched. Everything is off by default - behaviour is unchanged unless the flags are set. The flags and the worker-region groups are read on the trigger hot path from in-memory snapshots rather than the database: a small `createReloadingRegistry` helper loads each at startup and refreshes them on an interval, so no per-trigger query is added and a percentage or kill-switch change propagates within the reload interval. A cold replica whose snapshot hasn't loaded yet reads as not-migrated (the container path) and self-corrects on the next load - the same cold-start contract as the datastore / LLM-pricing registries, with a `reloading_registry_loaded` metric so a never-loaded registry is alertable. The same migration decision is consulted at deploy-time template creation so a migrated org gets a compute template built ahead of its first run. This runs in shadow mode (best-effort, never fails the deploy) by default, or - when the `computeMigrationRequireTemplate` flag is on - in required mode, built synchronously at deploy so the first run never builds on-demand and template errors surface at deploy time. So operators keep "which runs ran where" while customers only see geography: the run's actual worker queue is stored raw, and the geo region is stamped separately on `TaskRun.region` (and a new ClickHouse `region` column) at trigger time. Read surfaces - the dashboard, the API, and the Query/Logs page - show the geo region, falling back to the worker queue for runs written before the column existed. Minor follow-ups left out of scope: the percentage flags render as text inputs on the admin flags page (the catalog UI has no numeric control type yet), and `createReloadingRegistry` could later gain pub/sub for sub-second cross-replica propagation if the reload interval proves too slow. |
||
|
Matt Aitken
|
f48c89752c |
perf(webapp): parallelize streaming batch-item ingest (#3777)
## Problem
The item-streaming endpoint of the two-phase batch API (`POST
/api/v3/batches/:batchId/items`) processed streamed items strictly
sequentially. For a batch of many large payloads, each offloaded to
object storage inline, this serialized N object-store round-trips inside
a single request and could exceed Node's default `server.requestTimeout`
(300s). The webapp then returned `408`, which the SDK reads as `408
terminated` and retries up to 5 times, turning a slow ingest into a
failure that takes tens of minutes to surface.
## Fix
Ingest now runs through `p-map` over the NDJSON async iterable with
bounded concurrency (`STREAMING_BATCH_INGEST_CONCURRENCY`, default 10):
- `p-map` pulls lazily from the stream, so at most `concurrency` items
are read and in-flight at once. Peak memory stays bounded to roughly
`concurrency × STREAMING_BATCH_ITEM_MAXIMUM_SIZE` and request-body
backpressure is preserved.
- Set the env to `1` for fully sequential ingestion (escape hatch).
## Why this is safe (ordering and idempotency unchanged)
- Ordering derives from each item's index (enqueue `timestamp =
batch.createdAt + index`), not enqueue order.
- Dedup is atomic per index in `enqueueBatchItem`.
- The NDJSON parser now stamps oversized-item markers with their emit
position, removing the consumer's sequential `lastIndex` assumption (the
only order-dependent bit).
- The count-check and conditional-seal path is untouched.
## Scope
This speeds up every batch ingested through the streaming endpoint, not
just large-payload batches. Each item does a per-item Redis enqueue
regardless of size, and those now overlap. Large payloads benefit most
because they add an object-store offload round-trip on top of the
enqueue.
## Verification
Added an integration test (`streamBatchItems.test.ts`) that drives the
real service against Postgres + Redis + RunEngine and times a 150-item
batch at increasing concurrency. Object-store offload is modelled as a
fixed per-item latency (local round-trips are too small to compare
meaningfully):
```
runCount=150
large payloads (10ms/item offload):
concurrency=1 1739ms
concurrency=10 192ms (9.1x faster)
concurrency=50 57ms (30.7x faster)
small payloads (Redis enqueue only, no offload):
concurrency=1 90ms
concurrency=10 24ms (3.7x faster)
```
The test asserts correctness at every concurrency (all items accepted,
sealed, enqueued exactly once), that parallel ingest beats the
sequential floor, and that the small-payload case is strictly faster
than sequential, so the win is not specific to large payloads.
Also exercised end-to-end over real HTTP against a local server: a
20-item batch (12MB body) ingests and seals, a re-stream of the sealed
batch returns `sealed: true` with zero re-accepted items (idempotent
retry), and an oversized item still seals at its correct index.
Existing coverage stays green: concurrent ingest of a 100-item batch,
in-flight processing never exceeding the configured concurrency,
concurrent dedup on streaming retry, and emit-position marker indexing.
## Follow-ups (not in this PR)
- SDK pre-offload of large item payloads (send `application/store` refs
instead of raw blobs) to remove object-store work from the request hot
path and shrink the request body.
- Optional `server.requestTimeout` bump as a safety net.
## CI fix
Added `.github/workflows/codeql.yml` to replace GitHub's automatic
("dynamic") CodeQL scanning. The dynamic setup was failing to upload
SARIF results because the auto-generated `GITHUB_TOKEN` lacked the
`security-events: write` permission. The explicit workflow grants that
permission at the job level and pins all actions to commit SHAs,
consistent with the repo's security conventions.
## ✅ Checklist
- [ ] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [ ] The PR title follows the convention.
- [ ] I ran and tested the code works
---
## Testing
- Integration test (`streamBatchItems.test.ts`) validates correctness
and performance at concurrency 1, 10, and 50 for both large and small
payloads.
- End-to-end verified over real HTTP: 20-item/12MB batch ingests and
seals, idempotent retry returns `sealed: true`, oversized item seals at
correct index.
---
## Changelog
Streaming batch ingest now processes items with bounded concurrency
instead of one at a time, so batches of many large payloads ingest far
faster and no longer time out. Concurrency is configurable via
`STREAMING_BATCH_INGEST_CONCURRENCY` (default 10); set it to 1 for fully
sequential ingestion.
---
## Screenshots
_[Screenshots]_
💯
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
|
||
|
Eric Allam
|
954ee5c572 |
fix(webapp): deliver realtime changes with current content when the read replica lags (#3910)
## Summary When the realtime runs feed (the backend behind the `realtimeBackend` feature flag) hydrates a change from a Postgres read replica, the read can race the replica's apply of the very write that triggered it. The delivered row then carries the previous change's content, and an isolated final change (for example a last `metadata.set` before a run goes quiet) is not corrected until the roughly 20 second backstop poll. Measured against a replica with deliberate apply delay, every delivery trailed exactly one change behind and a final change stranded for the full backstop interval. ## Fix Publishers stamp each change record with the committed row's `updatedAt`, taken from writes they already perform, so the stamp costs no extra queries. The router delays its wake hydrate until the replica's measured lag has passed, anchored to that timestamp: a record that has already spent longer than the lag in transit is hydrated immediately, so only the racing leading edge ever waits. After hydrating, a tripwire compares each row against its record's watermark. Still-stale rows are withheld and retried briefly, and each detection feeds the lag estimate. If retries run out, the rows are delivered anyway (liveness over freshness) and follow-up re-hydrates emit the fresh version through the normal working-set diff once the replica catches up, with the backstop as the terminal net. Replica lag is sampled reader-side only, and only while feeds are active. Aurora reports live lag via `aurora_replica_status()`; vanilla Postgres can only report "caught up or not" (mid-apply lag is not honestly measurable from a replica), so tripwire observations floor the estimate there. Deployments without a replica resolve to zero lag and skip the gate entirely. Tunables live under `REALTIME_BACKEND_NATIVE_REPLICA_LAG_*`, and `realtime_native.stale_hydrates` plus `realtime_native.replica_lag_estimate_ms` make replica health observable. Two adjacent fixes: a metadata update that writes nothing no longer publishes a change record, and buffered parent and root metadata operations now publish when the flusher writes them, so those changes wake live feeds instead of waiting for the backstop. For local testing, `docker-compose` gains an opt-in `database-replica` service (compose profile `replica`) with a configurable `recovery_min_apply_delay`, which reproduces replica-lag behavior deterministically. With the gate disabled this rig reproduces the one-change-behind delivery exactly; with it enabled, deliveries arrive with current content at roughly the true replica lag, across write rates faster and slower than the lag itself. |
||
|
Eric Allam
|
187c0476c3 |
perf(webapp): shrink run trace loader payload and add trace span cap controls (#3906)
## Summary The run trace page loader serialized every span's raw OTel events (with full properties) into the response, even though the tree UI only renders the derived `timelineEvents` and the span detail panel refetches what it needs. On event-heavy traces that inflated both the loader payload and the server-side heap copies built per request. This PR keeps raw span events server-side and pairs that with a few related trace-view improvements: - A new optional `TRACE_VIEW_EMERGENCY_SPAN_CAP` env var (unset by default) clamps the trace summary and detailed trace summary span limits on both event store paths, including the public run trace endpoint, so operators can bound trace query sizes in one place without retuning the per-store limits. - The TreeView virtualizer resolved every rendered row with a linear scan over the whole tree (and `getNodeProps` did the same via `findIndex`); rows now resolve through memoized id lookup maps, which matters once traces reach tens of thousands of spans. - The run stream SSE lookup now applies the same organization membership scoping as the rest of the run page presenters, for consistency. Behavior is unchanged by default: the trace tree renders from the same `timelineEvents` it always has, and the new cap only takes effect when set. |
||
|
Eric Allam
|
f9d57d3bd5 |
feat(webapp): add a new backend for the realtime runs feed (#3864)
## Summary Adds a second backend for the realtime runs feed (`useRealtimeRun`, `subscribeToRunsWithTag`, `subscribeToBatch`), built to stay healthy when a single busy environment has many subscribers watching many runs at once. It is gated behind a feature flag with the existing backend as the default, so nothing changes for users until it is enabled per environment. ## Design A run change is published once, as a small self-describing record, to a single per-environment channel. Every feed is then a predicate over that one stream rather than owning a channel: - A per-instance router indexes the currently-held feeds by run, tag, and batch. When a run changes it hydrates the affected rows once and serializes them once, then fans the result to every matching feed. One hot shared tag watched by many subscribers costs a single database query and serialize, not one per subscriber. - Feeds that don't match a change are never woken, wake delivery per environment is coalesced on a leading edge (250ms default) so a burst of changes costs one wake, and cold reads coalesce onto a single short-TTL-cached resolve. - An admission gate bounds how many cold ClickHouse resolves run concurrently, so a mass reconnect across many distinct filters queues instead of stampeding the database. - Changes that land while a client is between long-polls are delivered on its next poll instead of waiting for the periodic backstop: each environment buffers its recent change records, subscriptions linger briefly after the last feed closes, and a newly-armed poll replays exactly the connection's gap. - The per-connection replay cursors behind that are shared across instances via Redis (a single timestamp each), so a poll landing on a different instance behind the load balancer still reads the connection's true gap instead of falling back to a cold resolve. Cursor reads have a bounded deadline and degrade to the cold-read path on any Redis trouble. - Tag subscriptions with multiple tags match runs carrying all of the tags, mirroring the existing backend's filter semantics, and live long-polls hold for about 20 seconds to match its cadence. - The per-environment channel supports Redis Cluster sharded pub/sub, so the wake path scales horizontally across shards by environment. - The backend reports its health through OpenTelemetry metrics (delivery lag, poll resolution paths, backstop outcomes, replay and cursor-store activity), with a provisioned Grafana dashboard for local development. Everything is behind the feature flag and tunable via env vars; the existing backend remains the default. |
||
|
Daniel Sutton
|
6afc9bfa4c |
fix(run-engine): retry getSnapshotsSince on the replica then primary when the read replica lags (#3889)
## Summary
When `RUN_ENGINE_READ_REPLICA_SNAPSHOTS_SINCE_ENABLED` is on,
`RunEngine.getSnapshotsSince` reads from the read replica. During write
spikes the replica can briefly lag, so the snapshot id a runner just
learned from the writer isn't visible there yet: the lookup threw, the
worker route returned a 500, and the runner waited for its next poll —
turning sub-second snapshot notifications into poll-interval latency
exactly when things are busiest. This PR makes the flag safe to enable:
a replica miss of the since snapshot gets one jittered retry on the
replica (most lag windows are shorter than the ~50–200ms wait, so the
writer is never touched), then falls back to the primary, observed via a
new `run_engine.snapshots_since.replica_miss` counter with an `outcome`
attribute (`replica_retry` vs `primary`). Only genuine misses — absent
on the primary too — remain errors.
## Design
- `getExecutionSnapshotsSince` now throws a typed
`ExecutionSnapshotNotFoundError` so the engine can distinguish the
expected lag miss from real failures. The message string is unchanged
and the error never leaves the engine.
- The recovery path only engages when the flag is on, a distinct replica
client is configured, and no transaction client was passed. With the
flag off, the path is behaviorally identical to before.
- Retry delay bounds are configurable
(`RUN_ENGINE_SNAPSHOTS_SINCE_REPLICA_RETRY_MIN_MS`/`MAX_MS`, default
50/200; `MAX_MS=0` skips the replica retry and goes straight to the
primary).
- The warn log fires only when the primary serves the read (the writer
spill is the operationally interesting event); replica-retry recoveries
are counted but quiet. A permanently-missing snapshot id stays an
error-level failure with a `failedDuring` field, so lag metrics aren't
polluted by bogus ids.
- Stale-tail lag (replica has the since snapshot but not newer rows)
deliberately still returns the replica's view; the next poll catches up.
- The since-snapshot anchor lookup is now scoped to the polled run
(`where: { id, runId }`), so a snapshot id from a different run raises
not-found instead of silently anchoring a too-wide window of the run's
snapshots.
## Test plan
All vitest + testcontainers, no mocks. A new `schemaOnlyPrisma` fixture
(migrated-but-empty clone database) simulates a replica that hasn't
caught up, and a real in-memory OTel meter pins the counter semantics
per outcome.
- [x] Replica catches up during the jittered retry window → served by
the replica, `outcome=replica_retry` = 1, primary never consulted
- [x] Replica permanently missing the since snapshot → served by the
primary, `outcome=primary` = 1
- [x] Snapshot missing on both replica and primary → null, counter = 0
- [x] Replica has the since snapshot but lags by one → the replica's
view is served, no fallback (verified discriminating power: the test
fails if reads secretly hit the primary)
- [x] Flag off with a replica configured → primary serves the read
- [x] Transaction client provided → bypasses the replica entirely
- [x] Since snapshot belonging to a different run → null
- [x] Existing getSnapshotsSince + waitpoints suites green; run-engine,
testcontainers, and webapp typechecks pass
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
|
||
|
Eric Allam
|
85886b96da |
feat(webapp,supervisor): isolate scheduled runs on a dedicated worker queue (#3839)
## Summary Scheduled runs and their descendants can now be routed to a dedicated per-region worker queue, processed by a separate worker fleet, so a burst of scheduled crons no longer competes with standard and agent runs for the same queue and inflates their startup latency. It is off by default and enabled per organization via a feature flag (with a global default), so nothing changes until it is turned on. ## Design At trigger time, any run whose lineage originates from a schedule (`rootTriggerSource === "schedule"`, which already propagates from a scheduled run down to all of its children) gets its worker queue suffixed with `:scheduled`. The worker queue name is an opaque string persisted on the run and used verbatim by enqueue and dequeue, so this needs no Lua, message-envelope, or concurrency changes. Concurrency stays keyed by environment and queue, not by worker queue. On the consumer side, the dequeue endpoint gains an optional `queueClass` selector. A supervisor sends `queueClass: "scheduled"` and the server derives the actual queue from the worker's own group, so a token can only ever reach its own region's queues. A fleet picks its class with the `TRIGGER_WORKER_QUEUE_CLASS` env var (`default` or `scheduled`), so a dedicated scheduled fleet can run alongside the standard one. Verified end to end against a local managed-worker setup: scheduled runs route to the dedicated queue, are drained only by the scheduled fleet, and standard runs are left untouched. |
||
|
Daniel Sutton
|
4ea3ef138f |
chore(webapp,redis-worker): make mollifier constants configurable (#3822)
## Summary The mollifier had ~21 behavioural constants baked in as hardcoded values — the buffer's ack-grace TTL and Redis retry/reconnect tuning, the drainer's poll interval and backoff envelope, the pre-gate idempotency claim TTL/wait/poll, the buffered-run mutate-with-fallback wait loop, the metadata CAS retry budget and backoff, the stale-sweep scan bounds, and the draining-gauge interval. None could be adjusted without a code change, which makes tuning the system under production load impossible. This exposes all of them as `TRIGGER_MOLLIFIER_*` environment variables, each defaulting to its previous hardcoded value. Behaviour is identical unless an operator sets a var, so it's a safe no-op deploy. ## Design The package-level classes (`MollifierBuffer`, `MollifierDrainer` in `@trigger.dev/redis-worker`) gain optional constructor options defaulting to the old constants — backward compatible, hence a patch changeset. The webapp factories and worker bootstraps read the env and pass them through. The route- and concern-level pure helpers (mutate-with-fallback, metadata mutation, idempotency claim, stale-sweep state) keep their existing `?? DEFAULT` option fallbacks and are fed env values at their call sites, so they stay unit-testable without importing `env.server`. ## Test plan - [x] `@trigger.dev/redis-worker` builds - [x] webapp typecheck passes - [x] mollifier buffer + drainer testcontainer suites pass (modulo a couple of pre-existing flaky timing tests) - [x] Reviewer: confirm the `TRIGGER_MOLLIFIER_*` env var names match ops conventions --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> |
||
|
Daniel Sutton
|
139eede41c |
feat(redis-worker): batched pop in MollifierDrainer for fast single-env drains (#3797)
## Summary Adds `drainBatchSize` to `MollifierDrainer` (default `1` — preserves existing behaviour) and wires `TRIGGER_MOLLIFIER_DRAIN_BATCH_SIZE` through the webapp (default `50`). Each tick the drainer now pops up to `drainBatchSize` from each chosen env, then dispatches every popped entry through the shared `concurrency`-bounded `pLimit`. Per-org/per-env fairness is unchanged — only the in-env pop count grows. Pre-existing behaviour was one pop per env per tick. For a single-env burst that single-flighted the drain at the per-tick floor of `pop + engine.trigger ≈ 50–60 ms`. With buffer entries piling up under a real-world tenant burst that's tens of minutes of tail latency to fully materialise — even though PG itself could comfortably sustain the writes. ## Why this matters — heavy-tail illustration Scenario: 100 customers in one window — 94 fire 20 triggers each, 5 fire 100, 1 fires 1000. Gate at `THRESHOLD=10/s`, `HOLD_MS=500`. First 10 of each burst hit PG directly; the rest buffer. | Customers | Triggers each | PG direct | Buffered each | Total buffered | |---|---|---|---|---| | 94 small | 20 | 10 | 10 | 940 | | 5 medium | 100 | 10 | 90 | 450 | | 1 heavy | 1000 | 10 | 990 | 990 | **With `DRAIN_BATCH_SIZE=50`, `DRAIN_CONCURRENCY=50`, ~50 ms `engine.trigger`:** | Tick | Pops | Dispatch waves | Wall-clock | |---|---|---|---| | 1 | 94×10 + 5×50 + 1×50 = 1 240 | 25 × 50 ms | ~1 300 ms (94 smalls done) | | 2 | 5×40 + 1×50 = 250 | 5 × 50 ms | ~300 ms (5 mediums done) | | 3–20 | heavy alone, 50/tick | 1 × 50 ms | ~100 ms each | | Customer class | Buffered fully drained | |---|---| | 94 small | **~1.3 s** | | 5 medium | **~1.6 s** | | 1 heavy | **~3.4 s** | **Without batching (one pop per env per tick — current behaviour):** | Customer class | Buffered fully drained | |---|---| | 94 small | ~500 ms | | 5 medium | ~4.5 s | | 1 heavy | **~49 s** | So the heavy single-tenant tail drops from ~49 s to ~3.4 s (~14× faster) without changing PG load characteristics. Smalls go up slightly in this scenario (500 ms → 1.3 s) because all 100 envs share one tick's dispatch queue — that's the trade we accept for the heavy tail; the worst-case small wait is still inside one tick. PG load is identical either way (50 concurrent inserts at a time, capped by `DRAIN_CONCURRENCY`). ## What changed **`packages/redis-worker`** - New `drainBatchSize` option (default 1 — full backward compat). - `runOnce()` refactored to pop per-env batches in parallel, then dispatch all popped entries through the existing global `pLimit`. Mid-batch pop failure aborts only that env's batch and counts as one failure (same semantic as the old per-env path). - Removed the now-unused `processOneFromEnv` helper. **`apps/webapp`** - `TRIGGER_MOLLIFIER_DRAIN_BATCH_SIZE` env var (default 50, matching `DRAIN_CONCURRENCY`). - Wired into `mollifierDrainer.server.ts`. **Test cloud config** (separate cloud PR): `TRIGGER_MOLLIFIER_DRAIN_BATCH_SIZE="50"` on the worker service. Production rollout deferred until we've watched it on test cloud. ## Test plan - [x] All 25 stub-based drainer tests pass (18 pre-existing + 7 new). 7 new tests under `MollifierDrainer.drainBatchSize`: - pops up to `drainBatchSize` across ticks - global `concurrency` cap still holds when batch > concurrency - mid-batch pop failure isolation - multi-env batch fan-out in one tick - **hierarchical org fairness preserved at `drainBatchSize > 1`** (load-bearing — guards against future regressions to per-env-instead-of-per-org rotation) - mixed success/failure accounting in a batched tick - bounded pops on empty queue (no Lua spam past `drainBatchSize`) - [x] All pre-existing tests still pass unchanged at default `drainBatchSize=1` → backward-compat locked. - [x] `pnpm run build --filter @trigger.dev/redis-worker` clean. - [x] `pnpm run typecheck --filter webapp` clean. - [x] `redisTest` block (real Redis via testcontainers) — couldn't run locally on this branch due to testcontainers runtime discovery; will validate in CI. - [ ] Test-cloud smoke after cloud PR lands: fire `burst 50` against a flagged env and confirm the 50th entry's drain time drops from ~2.5 s to <200 ms. ## Notes - Per-tick memory bound: `maxOrgsPerTick × drainBatchSize` entries can sit in the JS pLimit queue between pop and dispatch. At defaults that's `500 × 50 = 25 000` × ~5 KB snapshot ≈ ~125 MB worst case per worker — well within headroom. - The pre-batch model's strict per-env throughput cap of `1/tick` is documented as the fairness baseline elsewhere. Org-level fairness is what callers actually rely on; this change does not weaken that. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Daniel Sutton
|
4745754a7a |
feat(webapp,run-engine): mollifier drainer replay + stale sweep + cancelled-run engine API (#3754)
## Summary The replay side of the mollifier: - `DrainerHandler`: reads buffered snapshots and replays them through `engine.trigger` to materialise PG rows. - `RunEngine.createCancelledRun`: new public method the handler uses to write CANCELED rows directly from snapshots (bypass queue + waitpoint, emit `runCancelled`). Tolerates the cjson empty-table tags edge case found during validation. - Drainer fairness: org → env rotation so a heavy env doesn't starve light ones in the same org. - Stale-entry sweep + telemetry + alertable gauge so a stuck/offline drainer surfaces in alerts. Both the drainer and sweep default-off; nothing fires unless flagged on (`TRIGGER_MOLLIFIER_DRAINER_ENABLED`, `TRIGGER_MOLLIFIER_STALE_SWEEP_ENABLED`). Stacked on the trigger-time decisions PR. ## Test plan - [x] \`pnpm run typecheck --filter webapp\` passes - [x] \`pnpm run test --filter webapp test/mollifierDrainerHandler.test.ts\` passes - [x] \`pnpm run test --filter webapp test/mollifierStaleSweep.test.ts\` passes - [x] \`pnpm run test --filter @internal/run-engine src/engine/tests/createCancelledRun.test.ts\` passes - [x] \`pnpm run test --filter @trigger.dev/redis-worker packages/redis-worker/src/mollifier/drainer.test.ts\` passes --- ## Ship-gate follow-up fix **Drainer writes SYSTEM_FAILURE on max-attempts exhaustion.** Adds an `onTerminalFailure` callback on `MollifierDrainerOptions` so the customer's run lands a SYSTEM_FAILURE PG row even when the drainer exhausts `MAX_ATTEMPTS` on a retryable PG error (previously `buffer.fail()` was called with no row written → silent data loss). The callback runs before `buffer.fail()` on every terminal path (non-retryable AND max-attempts-exhausted), and re-throwing a retryable error from the callback causes the drainer to requeue rather than fail. Bumps `@trigger.dev/redis-worker` to a **minor** changeset (additive option + new exported types). Includes 5 unit tests covering both terminal causes plus the requeue-on-retryable-callback-failure path and no-callback back-compat. --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Daniel Sutton
|
3924aa4adb |
feat(redis-worker,webapp): mollifier buffer extensions + snapshot type (#3752)
## Summary Buffer-side data layer used by the rest of the mollifier phase-3 stack. - `buffer.ts` gains entry inspection (`getEntry`), idempotency lookup (`lookupIdempotency`), in-place snapshot mutation (`mutateSnapshot`), and dwell tracking. All atomic via Lua. - `mollifierSnapshot.server.ts`: shared `MollifierSnapshot` type plus (de)serialise helpers. - Drops the entry-TTL config and its env var. The drainer is the recovery mechanism; an entry that survives the drainer should surface as a stale-sweep alert, not silently TTL away. Adds methods to the buffer interface; nothing consumes them yet. Subsequent PRs in the stack wire trigger-time mollify, read-fallback, and mutation paths against this surface. ## Test plan - [x] \`pnpm run typecheck --filter webapp\` passes - [x] \`pnpm run test --filter @trigger.dev/redis-worker packages/redis-worker/src/mollifier/buffer.test.ts\` passes --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Eric Allam
|
37eeaa3690 |
fix(webapp): skip V1 WAITING_FOR_DEPLOY drain on V2 promotes (#3742)
## Summary Stops the legacy V1 `WAITING_FOR_DEPLOY` drain from running on every V2 deployment promotion. The drain queries `TaskRun` by `status='WAITING_FOR_DEPLOY'`, which only V1-engine runs ever have — V2 runs use `PENDING_VERSION` and are handled out of band. Calling the drain on V2 promotes produced empty queries against the status index and unnecessary reader-DB load. ## Fix Two layers: 1. Gate the enqueue at the call site in `ChangeCurrentDeploymentService` so it only fires when the deployment's worker is on engine V1. 2. Add a `LEGACY_RUN_ENGINE_WAITING_FOR_DEPLOY_DISABLED` env var (default `0`). When set to `1`, the service returns immediately from `call()` — neuters any jobs already sitting in the worker queue from before the deploy lands. V1 customers see no change; V2 promotes no longer trigger the drain. |
||
|
Eric Allam
|
61ca40b4b1 |
perf(run-engine,webapp): look up PENDING_VERSION runs via ClickHouse (#3707)
## Summary When a background worker registers, the engine resolves runs that were queued before the worker was ready (status `PENDING_VERSION`). That lookup used to scan a Postgres status index on `TaskRun`. Move it to ClickHouse: query candidate run ids from `task_runs_v2`, then refetch the actual rows from Postgres by primary key with a `status = 'PENDING_VERSION'` guard for idempotency. ## Design The lookup is a pluggable interface on the run engine (`PendingVersionRunIdLookup`). The webapp wires a ClickHouse-backed implementation through the org-scoped `clickhouseFactory` using a new `"engine"` client type, configured by `RUN_ENGINE_CLICKHOUSE_*` env vars. The URL falls back to `CLICKHOUSE_URL` when unset, so self-hosted deployments don't need new config to keep working. When the lookup returns no candidates, one bounded retry is scheduled ~5s later to cover ClickHouse replication lag against `task_runs_v2`. The Postgres status guard on both the candidate refetch and the inner `updateMany` prevents double-promotion when a retry races with a concurrent deploy. Tests cover three existing PENDING_VERSION cases via a small Postgres-backed test adapter; new ClickHouse-backed integration tests will follow. |
||
|
Daniel Sutton
|
1015876b98 |
feat(webapp): user-based Sentry attribution with tenant tags (#3678)
## Summary
Stamp every Sentry event with the signed-in user and the tenant (org /
project / env) the request belongs to, so "Users Impacted" counts
distinct humans and events become filterable per tenant.
**Design after review (current):**
- `user.id = real user cuid` (from `requireUser`). "Users Impacted"
counts humans, not tenants.
- Tenant context (org / project / env slugs, IDs, env type) moves
entirely onto tags: `org_slug`, `project_slug`, `env_slug`, `org_id`,
`project_id`, `project_ref`, `environment_id`, `env_type`, plus
`impersonating` when set.
- Backed by an `AsyncLocalStorage` scope established at the HTTP entry.
Each entry point fills what it knows; loaders enrich the same scope with
what they already have.
**Zero new database queries.** The middleware does a regex match only.
Dashboard loaders that already query Prisma gain a couple of extra
selected columns; nothing new round-trips.
## How it's wired
- **Express middleware (`tenantContextResolver.server.ts`)** — parses
the URL with a regex and always opens an ALS scope. Populates whatever
subset of slugs is present: `/orgs/:o` → just `orgSlug`;
`/orgs/:o/projects/:p` adds `projectSlug`; the full triple adds
`envSlug`. Non-tenant paths get an empty scope so loaders can still
enrich.
- **`_app/route.tsx`** — already calls `requireUser`. Adds
`tenantContext.enrich({ userId: user.id })` for every authenticated
dashboard request. No new query.
- **Env layout loader (`_app.orgs.$o.projects.$p.env.$e/route.tsx`)** —
its existing `prisma.project.findFirst` gains two columns in `select`
(`externalRef`, `organization.id`). After it picks an env, calls
`tenantContext.enrich({ orgId, projectId, projectRef, envId, envType
})`. Same query, +2 columns.
- **API path (`apiBuilder.server.ts`)** — wraps every handler in
`tenantContext.run(tenantContextFromAuthEnvironment(authenticationResult.environment),
…)`. The mapper pulls `userId` from `env.orgMember?.userId` (already
selected by `authIncludeBase` — no schema change). Covers
`createLoaderApiRoute`, `createActionApiRoute`, and
`createMultiMethodApiRoute`.
- **Event processor (`sentryTenantContext.server.ts`)** — registered in
`entry.server.tsx` so it lives in the Remix bundle and shares the same
`tenantContext` ALS instance as the middleware and loaders. Stamps
whatever's present; nothing forced.
## Example events from local verification
| URL | `user.id` | Tags |
|-----|-----------|------|
| `/orgs/:o/projects/:p/env/:e/...` | real user cuid | `org_slug`,
`project_slug`, `env_slug`, `org_id`, `project_id`, `project_ref`,
`environment_id`, `env_type` |
| `/orgs/:o/settings` (non-env-scoped) | real user cuid | `org_slug`
only |
| API request with `orgMember` | `orgMember.userId` | full tenant set |
| API request without `orgMember` | (unset) | full tenant set |
## Trade-offs
1. On env-scoped pages, errors that fire before the env layout loader's
enrich callback runs get slugs + `user.id` but not the tenant IDs /
`env_type`. Realistic errors deep in async work get the full set. (Same
race as before, narrower window now that slugs/`user.id` are populated
up-front by the middleware and `_app` enrich.)
2. API requests where the environment has no `orgMember` get tenant tags
but no `user.id`. Those events still show in the issue but don't
contribute to "Users Impacted".
## Out of scope (deferred)
Background workers (`redis-worker`, `schedule-engine`) and socket
handlers. Those entry points don't set `tenantContext.run` yet — their
events ship without tenant attribution until each is wired in a
follow-up.
## Tests
31 unit tests across 4 files. New tests notably cover:
- `parseTenantPath`: org-only, org+project, and full-triple URL
variants.
- `tenantContext.enrich`: in-place patch, no-op outside `run()`,
concurrent-scope isolation, empty-scope + enrich pattern (for non-tenant
pages).
- `tenantContextFromAuthEnvironment`: with and without `orgMember` —
verifies the API path's `user.id` mapping.
- `addTenantContextToEvent`: empty scope, userId-only, slugs-only, full
enrichment, conditional tag emission, preservation of prior `event.user`
fields.
## Test plan
- [ ] `pnpm run typecheck --filter webapp`
- [ ] `pnpm run test --filter webapp -- test/tenantContext.test.ts
test/sentryTenantContext.test.ts test/tenantContextResolver.test.ts
test/tenantContextFromAuthEnvironment.test.ts`
- [ ] Local manual: with `SENTRY_DSN` set, hit a dashboard URL and an
API route, confirm the captured events carry `user.id` + the expected
tag set in Sentry.
- [ ] After ship: confirm "Users Impacted" on a real Sentry issue
reflects distinct users (not tenants).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
|
||
|
Matt Aitken
|
71d98b4e6b |
Support for org-scoped ClickHouse (#3333)
Added `OrganizationDataStore` which allows orgs to have data stored in specific separate services. For now this is just used for ClickHouse. When using ClickHouse we get a client for the factory and pass in the org id. Particular care has to be made with two hot-insert paths: 1. RunReplicationService 2. OTLPExporter --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Daniel Sutton
|
906d5fafb6 |
feat(mollifier): trigger burst smoothing — Phase 1 (monitoring) (#3614)
## Summary - Introduce the Mollifier: a Redis-backed buffer for `trigger()` API calls during traffic spikes, with a per-env trip evaluator and a drainer ack-loop. - Phase 1 is dual-write monitoring — every mollified trigger is buffered to Redis AND continues to `engine.trigger`. No customer-facing behaviour change. - Telemetry events: `mollifier.would_mollify`, `mollifier.buffered`, `mollifier.drained`, plus the `mollifier.decisions` counter. - Gated behind a feature flag (default off). ## Test plan - [x] `pnpm run test --filter @trigger.dev/redis-worker` - [x] `pnpm run test --filter webapp -- mollifier` - [x] Manual: with flag off, no behaviour change vs main - [x] Manual: with flag on + threshold lowered, observe `mollifier.buffered` + `mollifier.drained` log pairs with matching `runId` --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Eric Allam
|
454f0c949a |
perf(webapp): cache task metadata in Redis for the trigger hotpath (#3625)
## Summary The trigger-task hotpath used to early-return without a DB query when a caller passed both a queue override and a per-trigger TTL — the hottest configuration on the trigger API. Adding `triggerSource` to the resolver so the runs-list "Source" filter could distinguish STANDARD / SCHEDULED / AGENT runs removed those early-returns, costing +2 DB queries per trigger on non-locked calls and +1 on locked calls. This change caches `BackgroundWorkerTask` metadata (`ttl`, `triggerSource`, `queueId`, `queueName`) in Redis so the resolver can satisfy every caller configuration with a single `HGET` on the warm path. PG fallback on miss back-fills the cache. Follow-up to #3542. ## Design Two key spaces: - `task-meta:env:{envId}` — the "current worker" view, refreshed at every deploy promotion. 24h safety TTL. - `task-meta:by-worker:{workerId}` — used for `lockToVersion` triggers. Immutable post-create. 30d sliding TTL so historical workers age out. Cache writes use Lua scripts via `defineCommand` so `DEL` + `HSET` + `EXPIRE` land atomically — concurrent readers never see the empty intermediate state of a naive pipeline. Read-path back-fill uses single-field upserts so concurrent back-fills don't wipe each other's siblings. The cache lives behind its own `TASK_META_CACHE_REDIS_*` env-var prefix that falls back to the default `REDIS_*` set, so operators can route the cache to a dedicated Redis instance if they want. The service/instance file split (`taskMetadataCache.server.ts` for the pure class, `taskMetadataCacheInstance.server.ts` for the env-wired singleton) mirrors the existing `runsReplicationService` / `runsReplicationInstance` pattern. ## Test plan - [ ] `pnpm run typecheck --filter webapp` - [ ] `pnpm run test ./test/engine/triggerTask.test.ts --run` — 8 existing tests untouched + 5 new tests covering warm cache, cold miss with back-fill, queue + ttl path, by-worker vs env keyspace, and the promotion cache write - [ ] End-to-end against a dev worker: registering writes both keyspaces with the expected TTLs, and `redis-cli HGETALL "tr:task-meta:env:<envId>"` returns the cached entries ## Benchmark Measured `DefaultQueueManager.resolveQueueProperties` against a real Postgres + Redis (vitest `containerTest`, single-host docker). 500 sequential calls and 2,000 parallel calls (concurrency=50) per scenario, request shaped as `{ taskId, queue: "bench-queue", ttl: "5m" }` — the hot path this PR restores. ``` sequential (one in flight at a time): [noop cache (baseline)] n=500 mean=1.423ms p50=1.394ms p95=1.735ms p99=2.629ms max=11.100ms [redis cache, cold ] n=500 mean=1.346ms p50=1.283ms p95=1.688ms p99=2.463ms max=5.058ms [redis cache, warm ] n=500 mean=0.084ms p50=0.078ms p95=0.105ms p99=0.156ms max=1.129ms speedup (warm vs baseline, sequential): 16.95x parallel (concurrency=50): [noop cache (baseline)] n=2000 mean=10.069ms p50=8.850ms p95=14.718ms p99=31.887ms total=405ms ops/s=4,940 [redis cache, warm ] n=2000 mean=0.614ms p50=0.568ms p95=1.189ms p99=1.432ms total=25ms ops/s=80,389 throughput speedup (warm vs baseline, parallel): 16.27x ``` Read: - **Warm cache cuts resolver latency 17×** at p50 — from ~1.4 ms to ~78 µs per call. - **Cold cache is on par with baseline** — the extra `HGET` miss adds <50 µs against the two Postgres queries that follow, so the worst case is not worse than today. - **Under burst load (50 concurrent triggers)**, the baseline's p99 jumps to ~32 ms as Postgres connections queue up; warm stays at ~1.4 ms. The cache moves the saturation point from ~5k ops/s (PG pool) to ~80k ops/s (single-client Redis pipelining). Caveats: single-host docker, local Postgres + Redis, resolver-only measurement (excludes the rest of the trigger transaction). Prod adds region-local Redis RTT (~0.3–0.8 ms) which shifts warm absolute numbers up but keeps the ratio intact. |
||
|
Matt Aitken
|
e4981d1b11 |
feat(webapp): consolidate auth path + add comprehensive auth tests (#3499)
## Summary
Consolidates the webapp's authentication and authorization into a small
set of route helpers, replacing the ad-hoc `requireUser` /
`requireUserId` / `authenticatedEnvironmentForAuthentication` calls
scattered across routes. Same security model, but the per-request flow
(authenticate → authorize → load) now lives in one place per route
family.
Introduces a plugin seam (`@trigger.dev/plugins`) that lets the cloud
build install a richer RBAC implementation without touching webapp code.
The OSS fallback keeps the pre-RBAC permissive behaviour intact, so
self-hosted deployments work unchanged.
Adds a comprehensive end-to-end auth test suite that didn't exist before
— 193 `it()` blocks (vitest reports ~199 after `it.each` expansion)
covering API key, PAT and JWT auth across the public API surface, plus
dashboard session auth for admin pages.
## Changes
### Plugin contract — `@trigger.dev/plugins`
`RoleBaseAccessController` interface authoritative for both OSS
(fallback) and cloud (enterprise plugin):
- `authenticateBearer(request, { allowJWT? })` — API-key / public-JWT
auth, returns env + ability
- `authenticateSession(request, { userId, organizationId?, projectId?
})` — dashboard auth, caller resolves `userId` from the session cookie
and passes it in (no `helpers.getSessionUserId` callback — decouples the
plugin host from session-cookie code)
- `authenticatePat(request, { organizationId?, projectId? })` — PAT
auth, returns identity + `lastAccessedAt` so the host can throttle the
per-request update
- `authenticateAuthorize*` variants for the auth-and-check-in-one-call
cases
- `isUsingPlugin(): Promise<boolean>` — capability flag for UI /
branching where plugin-present-ness matters; replaces the
sentinel-string coupling that had `personalAccessToken.server` matching
`"RBAC plugin not installed"` literally
### Dashboard auth (started, partial rollout)
Admin and settings pages migrated to a unified `dashboardLoader` /
`dashboardAction` helper that authenticates the session, runs an
authorization check, and exposes the result to the route. Other
dashboard routes still on the old pattern; remaining migration tracked
in TRI-8730.
Migrated routes:
- `admin.*` (14 admin / back-office / feature-flags / LLM-models /
notifications / orgs / concurrency pages)
- `_app.orgs.$organizationSlug.settings.team`
- `_app.orgs.$organizationSlug.settings.roles`
### API / realtime / engine auth (complete for the migrated families)
71 routes migrated to a unified `apiBuilder` that centralizes Bearer /
PAT / Public-JWT authentication and applies the per-route authorization
check before the handler runs. Includes:
- `api.v1.*` and `api.v2.*` and `api.v3.*` — tasks, runs, batches,
queues, prompts, deployments, query, sessions, waitpoints, packets,
workers, idempotency keys
- `realtime.v1.*` — runs, batches, sessions, streams
- `engine.v1.*` — dev / worker-action protocols
29 routes still on the legacy `authenticateApiRequest*` helpers —
tracked as a post-deploy follow-up in TRI-9228.
Multi-resource auth direction is now explicit at the call site via
`anyResource(...)` (OR) and `everyResource(...)` (AND). Bare arrays no
longer typecheck — fixes a class of bug where a JWT scoped to one
resource could implicitly access others under OR semantics.
PAT auth path consolidated: was three DB queries per request (legacy
`authenticateApiRequestWithPersonalAccessToken` findFirst +
`rbac.authenticatePat` join + `lastAccessedAt` update). Now one query in
the steady state — plugin returns `lastAccessedAt`, host smart-skips the
update via JS-side throttle when fresh.
Side effect: action aliases preserved historic JWT scope semantics where
the new model is stricter (e.g. a `write:tasks` JWT now also satisfies
`trigger` / `batchTrigger` / `update` actions on the same resource —
matched at the auth boundary, not in the route handler).
### Backwards-compat fixes
The strict-match model regressed several real-world JWT shapes. Each
preserved via explicit `anyResource(...)` entries in the route's authz
block:
- **Batch retrieve routes** (`api.v1.batches.$batchId`, `api.v2.*`,
`realtime.v1.batches.*`) accept `read:runs` JWTs again (pre-RBAC
literal-match superScope behaviour)
- **Runs list routes** (`api.v1.runs`, `realtime.v1.runs`) accept
type-level `read:tasks` / `read:tags` on unfiltered queries (matched the
legacy `Object.keys` iteration semantic)
- **PAT/OAT auth shape** normalized through `toAuthenticated` so all
auth methods return the same slim `AuthenticatedEnvironment` (was:
API-key returned the slim shape but PAT/OAT returned raw Prisma
`Decimal` / no `orgMember`)
- **Scope `:` preservation** in resource ids — `read:tags:env:staging`
now correctly identifies the tag id as `env:staging`, not `env`
### Slim `AuthenticatedEnvironment`
Extracted to `@trigger.dev/core/v3/auth/environment` — a structural
shape independent of `@trigger.dev/database`. The plugin contract
returns this; webapp consumers import from there; the cloud plugin
(Drizzle) returns the same shape without Prisma's `Decimal` class
leaking into the public surface. Lets internal-packages (run-engine,
etc.) refer to `AuthenticatedEnvironment` without pulling Prisma in.
### Auth test suite (new — `*.e2e.full.test.ts`)
193 e2e tests run against a real spawned webapp + Postgres (no mocks).
Coverage matrix:
- **API key auth** — read / write / trigger / batchTrigger / deploy
actions across runs, batches, deployments, prompts, queues, query,
sessions, input-streams, waitpoints, tasks, idempotency keys; multi-key
resources (a run carries batch / tag / task identifiers — auth must
accept any matching scope)
- **Personal Access Token auth** — comprehensive matrix: scope match,
scope mismatch, missing scope, expired token, malformed token
- **Public JWT auth** — sub-vs-URL environment resolution, expired JWTs,
signature verification, scope checking, otu (one-time-use) token
semantics, branch-environment signing-key fallback
- **Dashboard session auth** — admin-only pages reject non-admins;
per-action gating
- **Cross-cutting edge cases** — revoked API key grace window, JWT
cross-environment isolation, MissingResource branch behaviour
### Hygiene cleanups
- Deleted dead `app/services/authorization.server.ts` (legacy
`checkAuthorization` + types — no live consumers post-migration) and its
orphaned test
- Dropped the never-populated `scopes` field from
`ApiAuthenticationResultSuccess`
- `scheduleEmail` moved out of `email.server.ts` into its own module —
breaks a `commonWorker → marqs/V1` import chain that was poisoning the
auth test graph
- OSS Roles page shows a deployment-aware empty state ("Roles aren't
available in this self-hosted deployment" vs the plan-upsell copy) via
`rbac.isUsingPlugin()`
- Team action handler: explicit per-intent ability gates
(`manage:billing` for purchase-seats, `manage:members` for set-role +
remove-member with self-leave carve-out)
### Cross-repo coordination
All public-package contract changes paired in `triggerdotdev/cloud#763`
(rbac-packages branch) — the enterprise plugin implements the same
`RoleBaseAccessController` interface against Drizzle.
## Test plan
- [x] `pnpm run typecheck --filter webapp` clean
- [x] `pnpm --filter webapp exec vitest run --config
vitest.e2e.full.config.ts` — 193/193 pass (requires Docker for
testcontainers)
- [x] Spot-check an authed API endpoint with a valid + invalid API key
against a local stack
- [x] Spot-check the migrated admin pages render and gate non-admins
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Eric Allam
|
ead1e5a53d |
feat(webapp): reload LLM pricing registry on Redis pub/sub (#3534)
## Summary Adds a Redis pub/sub reload path to the webapp's in-memory LLM pricing registry. When enabled on a process, the registry reloads from the database whenever a publish lands on the configured channel — instead of waiting for the existing 5-minute interval. Lets pricing/model changes propagate to cost enrichment within seconds. Subscription is **off by default** and opt-in per process. Only OTel-ingesting services need real-time freshness; dashboard and worker services run fine on the periodic interval and shouldn't pile onto each publish with a full-table reload. ## Design When `LLM_PRICING_RELOAD_PUBSUB_ENABLED=true`, subscribes via `createRedisClient` against `COMMON_WORKER_REDIS_*` and listens on `LLM_PRICING_RELOAD_CHANNEL` (default `llm-registry:reload`). The 5-minute periodic reload stays as a backstop, and a SIGTERM/SIGINT handler closes the subscription cleanly. The publisher side lives outside this PR — any process running in the same Redis namespace can trigger a reload by `PUBLISH llm-registry:reload <anything>`. Includes a `.server-changes/` note for the changelog. ### Debounced reload Bursts of publishes are coalesced. The first publish schedules a reload at T+`LLM_PRICING_RELOAD_DEBOUNCE_MS` (default 1s); subsequent publishes during that window are no-ops because the trailing reload picks up everything when it queries the DB. Bounds reload rate to at most 1 per debounce window regardless of publisher chattiness, so a runaway upstream publisher can't fan out into a flood of full-table-scan reloads. ## Test plan - [ ] With `LLM_PRICING_RELOAD_PUBSUB_ENABLED=false` (default): `redis-cli PUBSUB NUMSUB llm-registry:reload` returns `0` while the webapp is up - [ ] With it set to `true`: returns `>= 1` - [ ] `redis-cli PUBLISH llm-registry:reload test` returns `1` (one subscriber received) on a subscribed process - [ ] Mutate an `LlmModel` row externally, publish on the channel, observe the registry's match() picks up the change without waiting for the 5-min tick - [ ] Publish 100x in rapid succession; confirm only one reload fires within the debounce window |
||
|
Iss
|
f8ddb766fa | feat: Plain customer cards (#2933) | ||
|
Eric Allam
|
386b4f65ff |
feat(webapp): per-org S2 basin migration (#3516)
## Summary Move from a single shared S2 basin to **per-org basins** with retention tied to the org's billing plan. Stops S2 from deleting streams out from under live chat sessions when basin retention fires before the chat ends, and unlocks per-org cost attribution. OSS / s2-lite installs are unaffected: provisioning is gated by `REALTIME_STREAMS_PER_ORG_BASINS_ENABLED` (default `false`), and the read precedence falls back to the global basin env var when an entity has no stamped basin. ``` basin = run.streamBasinName ?? session.streamBasinName ?? env.REALTIME_STREAMS_S2_BASIN ``` ## Design Three nullable `streamBasinName` columns (`Organization`, `TaskRun`, `Session`) plus a provisioner that idempotently creates the basin and reconfigures retention on plan changes. The trigger and session-create paths stamp the org's basin onto new rows; the realtime read path picks the basin from the entity context. Admin routes back-fill existing orgs and force-reconfigure a single org. ## Test plan - [x] `pnpm run typecheck --filter webapp --filter @internal/run-engine` - [x] Backfill admin route end-to-end (provision + DB stamp + S2 basin config). - [x] Reconfigure on plan change (all retention tiers). - [x] chat.agent multi-turn drives streams into the per-org basin. - [x] Legacy fallback when entity has no stamped basin. - [x] Provisioner is a no-op when the flag is off. |
||
|
nicktrn
|
ee3887a321 |
feat(webapp): configurable deploy template machine presets (#3492)
The webapp's compute template creation hardcoded a single machine preset (`small-1x`) at deploy time, regardless of which presets a project actually uses. Tasks running on any other preset paid full cold-snapshot creation cost on first run. Two new env vars: - `COMPUTE_TEMPLATE_MACHINE_PRESETS` - CSV of preset names to build boot snapshots for during deploy. Defaults to `small-1x` so existing deploys don't change behavior. - `COMPUTE_TEMPLATE_MACHINE_PRESETS_REQUIRED` - CSV of presets whose failure fails a required-mode deploy. Defaults to the full `PRESETS` list. Optional preset failures are logged but don't block the deploy. The compute client now sends the multi-config request shape; the service evaluates per-preset outcomes against the required set and surfaces a combined failure message when a required preset fails. Both env vars are validated at boot via the env schema - unknown preset names or `_REQUIRED` entries that aren't a subset of `_PRESETS` fail loudly at startup rather than silently per-deploy. |
||
|
ThullyoCunha
|
f1736595cd |
feat(webapp): apply default repository policy on ECR repo creation (#3467)
🚀 Publish Trigger.dev Docker / units (push) Failing after 13m3s
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 13m3s
🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker-v4 (push) Has been skipped
## Summary
Self-hosters that operate the webapp's ECR account separately from the
account running the EKS workers (e.g., a shared platform account that
hosts the registry plus per-team accounts that host clusters) currently
hit a 403 Forbidden the first time **any** project is deployed:
```
Failed to pull image "<acct-A>.dkr.ecr.<region>.amazonaws.com/<namespace>/proj_…:…":
unexpected status from HEAD request to .../v2/.../manifests/sha256:…: 403 Forbidden
```
`ensureEcrRepositoryExists` in
`apps/webapp/app/v3/getDeploymentImageRef.server.ts` calls
`CreateRepository` and `PutLifecyclePolicy`, but never
`SetRepositoryPolicy` — so the new repo inherits the AWS default (only
the registry-owner account can read/pull). Workers in the cluster
account get 403 every single deploy. The only workarounds today are
running a one-off post-create script or pre-creating every repo by hand.
## Proposed change
Add an optional env var:
```
DEPLOY_REGISTRY_ECR_DEFAULT_REPOSITORY_POLICY (V4 mirror: V4_DEPLOY_REGISTRY_ECR_DEFAULT_REPOSITORY_POLICY)
```
Raw IAM policy JSON. When set, the webapp calls `SetRepositoryPolicy`
immediately after `CreateRepository` so every new repo carries that
policy from creation. Operators control the principal/actions; we don't
bake in any opinions about cross-account boundaries.
Example value (for the typical self-host case — grant pull to the
cluster account):
```json
{
"Version": "2012-10-17",
"Statement": [{
"Sid": "AllowClusterAccountPull",
"Effect": "Allow",
"Principal": {"AWS": "arn:aws:iam::<cluster-account-id>:root"},
"Action": [
"ecr:GetDownloadUrlForLayer",
"ecr:BatchGetImage",
"ecr:BatchCheckLayerAvailability"
]
}]
}
```
## Why env var (not a chart-level field)
- Mirrors the shape of the sibling vars (`DEPLOY_REGISTRY_ECR_TAGS`,
`DEPLOY_REGISTRY_ECR_ASSUME_ROLE_ARN`, etc.) which are already
operator-supplied via `webapp.extraEnvVars` in self-host setups.
- Cloud is unaffected — the env var is optional, unset by default;
existing behavior unchanged.
- Existing repos are unaffected — only newly-created repos get the
policy.
- `RepositoryCreationTemplate` from the AWS provider isn't an
alternative here: it only applies to repos created via
pull-through-cache or replication, not to `ecr:CreateRepository` API
calls.
## Implementation
- `apps/webapp/app/env.server.ts` — declare
`DEPLOY_REGISTRY_ECR_DEFAULT_REPOSITORY_POLICY` and the V4 fallback.
- `apps/webapp/app/v3/registryConfig.server.ts` — propagate
`ecrDefaultRepositoryPolicy` to `RegistryConfig`.
- `apps/webapp/app/v3/getDeploymentImageRef.server.ts` —
`createEcrRepository` accepts the policy; if set, calls
`SetRepositoryPolicy` after `PutLifecyclePolicy`.
- `docs/self-hosting/env/webapp.mdx` — documentation row added under
**Deploy & Registry**.
## Verification
Verified end-to-end against a self-hosted Trigger.dev on EKS where the
ECR account is separate from the cluster account:
- **Without the env var** (current `main`): the new project's first run
pod stays in `ImagePullBackOff` with `403 Forbidden`.
- **With the env var set** to a JSON granting
`ecr:BatchGetImage`/`GetDownloadUrlForLayer`/`BatchCheckLayerAvailability`
to the cluster account: a fresh `trigger.dev deploy --env prod` followed
by a `hello-world` run completes in ~5s end-to-end on the first try.
Manually also confirmed that existing repos are untouched (the call only
fires inside `createEcrRepository`, which only runs when
`DescribeRepositories` returned `RepositoryNotFoundException`).
## Out of scope
- Chart values surface for this — operators already pass the existing
ECR vars via `webapp.extraEnvVars`, so this follows the same pattern.
Happy to add a first-class chart field in a follow-up if that's the
preferred direction.
- IAM-policy validation in the webapp — we forward the JSON verbatim to
AWS and surface AWS's error messages on misuse, matching how
`DEPLOY_REGISTRY_ECR_TAGS` is handled today.
This is a draft pending CI / CodeRabbit pass — happy to iterate on
direction (e.g., split into per-action env vars, or extend the chart
values schema) if any of the above choices feels off.
---------
Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
|
||
|
Eric Allam
|
c69e939c34 |
feat: Sessions - bidirectional durable agent streams (#3417)
> ⚠️ **Not released yet.** This PR is the server-side foundation only. The SDK changes that customers will actually use (`chat.agent` migration, `chat.createStartSessionAction`, `useTriggerChatTransport` updates) live on a separate branch and ship together in an upcoming `@trigger.dev/sdk` prerelease. Until that prerelease is published, this surface is reachable only via direct HTTP. ## What this gives Trigger.dev users A new first-class primitive, **Session**, for durable, task-bound, bidirectional I/O that outlives any single run. Sessions are the run manager for `chat.agent` going forward, and they unblock anything else that needs "one identifier, many runs over time" with a stable channel pair the client can write to and subscribe to. ### Use cases unblocked - **Chat agents that persist across many runs.** One session per chat (keyed on your own `chatId` via `externalId`), turns 1..N attach to the same Session, the UI subscribes once and keeps receiving output as new runs take over. - **Approval loops and long-running tasks with user feedback.** The task waits on `.in`, the client writes to `.in`, the server enforces no-writes-after-close. - **Workflow progress streams that live past the run.** Subscribe to `.out` after the task finishes to replay history. - **Resume-next-day flows.** A session is a durable row, not a transient stream. Send a message a day later and the server triggers a fresh run on the same session. ### How it works (Session-as-run-manager) A Session row is task-bound (`taskIdentifier` + `triggerConfig` are required) and owns its current run via `currentRunId` + `currentRunVersion` for optimistic claim. Three trigger paths: 1. **Session create** — `POST /api/v1/sessions` creates the row and triggers the first run synchronously. 2. **Append-time probe** — `POST /realtime/v1/sessions/:session/in/append` checks if the current run is alive; if it has terminated (idle exit, crash, etc.), the server triggers a new run before processing the append. 3. **End-and-continue handoff** — `POST /api/v1/sessions/:session/end-and-continue`, called by the running agent, triggers a fresh run and atomically swaps `currentRunId`. Used by `chat.requestUpgrade()` for version handoffs. Every triggered run is recorded in the `SessionRun` audit table with a reason (`initial`, `continuation`, `upgrade`, `manual`). ## Public API surface ### Control plane - `POST /api/v1/sessions` — create. Idempotent on `(env, externalId)`. Triggers the first run, returns the session and a session-scoped public access token. Returns 409 if the upserted row is already closed. - `GET /api/v1/sessions/:session` — retrieve by friendlyId (`session_abc...`) or by your own externalId (server disambiguates by prefix). - `GET /api/v1/sessions` — list with filters (`type`, `tag`, `taskIdentifier`, `externalId`, derived `status` ACTIVE/CLOSED/EXPIRED, created-at range) and cursor pagination. Backed by ClickHouse. - `PATCH /api/v1/sessions/:session` — update tags / metadata / externalId. - `POST /api/v1/sessions/:session/close` — terminate. Idempotent, hard-blocks new server-brokered writes. - `POST /api/v1/sessions/:session/end-and-continue` — agent-only handoff to a fresh run. ### Realtime - `PUT /realtime/v1/sessions/:session/:io` — initialize a channel. Returns S2 credentials in headers so high-throughput clients can write direct to S2. - `GET /realtime/v1/sessions/:session/:io` — SSE subscribe. Supports Last-Event-ID resume and an opt-in `X-Peek-Settled: 1` header that fast-closes the stream when the upstream is already settled (`trigger:turn-complete`), eliminating long-poll wait on reconnect-on-reload paths. - `POST /realtime/v1/sessions/:session/:io/append` — server-side appends. - `POST /api/v1/runs/:runFriendlyId/session-streams/wait` — runs wait on a session stream as a waitpoint, with a race-check to avoid suspending if data already landed. ### Auth scopes `sessions` is a new resource type. `read:sessions:{id}`, `write:sessions:{id}`, `admin:sessions:{id}` flow through the existing JWT validator. Session-scoped public access tokens minted by the server replace browser-held trigger-task tokens for chat-style flows — the browser never sees a run identifier or a run-scoped token in steady state. ## What's coming after this PR - **SDK + chat.agent migration**: separate branch, separate PR, ships in the next `@trigger.dev/sdk` prerelease alongside this server deploy. Customers using the prerelease `chat.agent` will follow the [upgrade guide](https://github.com/triggerdotdev/trigger.dev/blob/docs/tri-7532-ai-sdk-chat-transport-and-chat-task-system/docs/ai-chat/upgrade-guide.mdx). - **Dashboard surfaces**: dedicated agent list, agent playground, agent view on the run dashboard. Tracking separately. ## Implementation notes - **Postgres `Session` table**: scalar scoping columns (`projectId`, `runtimeEnvironmentId`, `environmentType`, `organizationId`) without FKs, matching the January TaskRun FK-removal decision. Point-lookup indexes only — list queries go to ClickHouse. Terminal markers (`closedAt`, `expiresAt`) are write-once. - **ClickHouse `sessions_v1`**: ReplacingMergeTree, partitioned by month, ordered by `(org_id, project_id, environment_id, created_at, session_id)`. Tags indexed via `tokenbf_v1` skip index. - **`SessionsReplicationService`**: mirrors `RunsReplicationService` exactly — leader-locked logical replication consumer, `ConcurrentFlushScheduler`, retry with exponential backoff + jitter, identical metric shape. Dedicated slot + publication so the two consume independently. - **S2 keys**: `sessions/{addressingKey}/{out|in}`. The existing `runs/{runId}/{streamId}` key format for run-scoped streams is untouched. - **Optimistic claim**: `ensureRunForSession` triggers a run upfront (cheap to cancel if it loses the race), then attempts an `updateMany` keyed on `currentRunVersion`. Loser cancels its triggered run and reuses the winner's. No DB lock held across the trigger. ### What did NOT change Run-scoped `streams.pipe` / `streams.input` and the existing `/realtime/v1/streams/{runId}/...` routes are unchanged. Sessions are net-new — not a reshaping of the current streams API. ## Deploy notes - Set `SESSION_REPLICATION_CLICKHOUSE_URL` and `SESSION_REPLICATION_ENABLED=1` to enable the replication consumer. - The `Session` table needs `REPLICA IDENTITY FULL` set on the prod source DB before the publication is created (same one-time DDL we did for `TaskRun`). Required for delete events to carry full column values. - Cross-form authorization on the `GET /api/v1/sessions/:session` loader (a JWT minted for either form authorizes both URL forms). Action routes are URL-form-specific, matching how the SDK mints PATs. ## Verification - Webapp typecheck clean (10/10). - `apps/webapp/test/sessionsReplicationService.test.ts` — round-trip tests for insert/update/delete through Postgres logical replication into ClickHouse via testcontainers. - Live end-to-end against local dev: create + retrieve (both forms) + update + close, `.out.initialize` + `.out.append` x2 + `.in.send` + `.out.subscribe` over SSE, list with all filter combinations + pagination, `end-and-continue` swap, `X-Peek-Settled` fast-close (verified in browser via reconnect-on-reload and via curl). Replicated row lands in ClickHouse within ~1s. - Multi-round Devin + CodeRabbit review feedback addressed (read-after-write paths use `prisma` writer, info-leak on auth-routes masked as 403, peek-settled discriminator parsing fix, etc.). ## Test plan - [ ] `pnpm run typecheck --filter webapp` - [ ] `pnpm run test --filter webapp ./test/sessionsReplicationService.test.ts --run` - [ ] Start the webapp with `SESSION_REPLICATION_CLICKHOUSE_URL` and `SESSION_REPLICATION_ENABLED=1`. Confirm the slot and publication auto-create on boot. - [ ] `POST /api/v1/sessions` and verify the row replicates to `trigger_dev.sessions_v1` within a couple of seconds. - [ ] `POST /api/v1/sessions/:id/close`, then confirm `POST /realtime/v1/sessions/:id/out/append` returns 400. - [ ] Reuse a closed session's `externalId` on `POST /api/v1/sessions` and confirm 409. - [ ] `GET /realtime/v1/sessions/:id/out` with `X-Peek-Settled: 1` after a turn completes and confirm `X-Session-Settled: true` response header + immediate close. |
||
|
Eric Allam
|
e134da7306 |
fix(run-engine): debounce hot-key lock contention and 5xx feedback loop (#3453)
## Changes
Three changes in
`internal-packages/run-engine/src/engine/systems/debounceSystem.ts`, in
order of impact:
1. **Fast-path skip before the lock.** In `handleExistingRun`, do an
unlocked read of `delayUntil` (and `createdAt` for the max-duration
check) from the run row before entering `runLock.lock("handleDebounce",
...)`. If `newDelayUntil <= currentDelayUntil` and the run is still
within its max-duration window, return the existing run immediately
without taking the lock. Safe because debounce is monotonic-forward only
— a stale read either matches reality or undershoots, both of which
decay correctly (re-checked properly inside the lock by whichever caller
is actually pushing forward). Trailing-mode triggers carrying
`updateData` still take the lock so the data update is applied.
2. **Quantize `newDelayUntil`.** Round the computed `newDelayUntil` to
1-second buckets (configurable via `quantizeNewDelayUntilMs`, set to 0
to disable). Without quantization, every call has a slightly larger
`newDelayUntil` than the last and they all pass the fast-path check.
With it, concurrent callers on the same key share a target time and ~95%
short-circuit. User-visible effect: a debounced run might fire up to 1s
earlier than the strict spec — non-issue for typical debounce use cases
(chat summarization, batched notifications, etc.).
3. **Graceful lock-contention fallback.** Wrap the `runLock.lock(...)`
call so `LockAcquisitionTimeoutError` and Redlock `ExecutionError` /
`ResourceLockedError` return the existing run id with success instead of
propagating a 5xx. Debounce is best-effort: if we can't take the lock,
the herd is already updating it for us; fall in line. This kills the 5xx
→ SDK-retry feedback loop. With (1)+(2) this rarely fires; without them
it's the difference between 5xx and 200.
Defaults preserve current behaviour aside from quantization (1s) and
fast-path (on). Both are configurable via `RunEngineOptions.debounce`.
## ✅ Checklist
- [x] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [x] The PR title follows the convention.
- [x] I ran and tested the code works
---
## Changelog
Reduce 5xx feedback loops on hot debounce keys by quantizing
`delayUntil`, adding an unlocked fast-path skip before the redlock, and
gracefully handling redlock contention in `handleDebounce` so the SDK no
longer retries into a herd.
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
|
||
|
Eric Allam
|
486f49791d |
fix(webapp): eliminate SSE abort-signal memory leak (#3430)
## Summary Fixes a server-side memory leak in the webapp's SSE helper. Every aborted SSE connection (client tab close, navigation, timeout) was pinning its full request/response graph indefinitely on Node 20, so any long-running webapp process accumulated retained memory proportional to streaming-request churn. ## Root cause `apps/webapp/app/utils/sse.ts` combined four abort signals via `AbortSignal.any([requestAbortSignal, timeoutSignal, internalController.signal])`. The composite signal tracks its source signals in an internal `Set<WeakRef>` registered against a `FinalizationRegistry`; under sustained traffic those entries accumulate faster than they're cleaned up, pinning every source signal (and its listeners, and anything those listeners close over) until the parent signal itself is GC'd or aborts. This is a long-standing Node issue with multiple open reports: - [nodejs/node#54614](https://github.com/nodejs/node/issues/54614) — original report, still open. A [follow-up from ChainSafe](https://github.com/nodejs/node/issues/54614#issuecomment-4055656572) describes the exact same shape in a Lodestar production workload (req + timeout signals composed per request accumulating in long-running worker) and the same mitigation: drop `AbortSignal.any`, compose manually. - [nodejs/node#55351](https://github.com/nodejs/node/issues/55351) — mechanism confirmed by Node member @jasnell: *"the set of dependent signals known to the AbortSignal are kept in an internal Set using WeakRefs. The AbortSignals are being properly gc'd but the Set is never cleaned out of the WeakRefs making those leak."* Partially fixed by [PR #55354](https://github.com/nodejs/node/pull/55354), shipped in Node 22.12.0 — but only covers the tight-loop case, not long-lived parent signals. - [nodejs/node#57584](https://github.com/nodejs/node/issues/57584) — circular-dependency variant, still open. - [nodejs/node#62363](https://github.com/nodejs/node/issues/62363) — regression in Node 24/25 from an unrelated V8 change ("Don't pretenure WeakCells"). Different root cause, same symptom. A separate issue in `apps/webapp/app/entry.server.tsx` — `setTimeout(abort, ABORT_DELAY)` with no `clearTimeout` on success paths — kept the React render tree + `remixContext` alive for 30s per successful HTML request. Same pattern fixed upstream in React Router templates ([react-router#14200](https://github.com/remix-run/react-router/pull/14200)), never backported to Remix v2. ## What changed - **`apps/webapp/app/utils/sse.ts`** — single-signal abort chain. `AbortSignal.any` removed; `AbortSignal.timeout` replaced by a plain `setTimeout` cleared when the controller aborts; named sentinel constants used as stackless abort reasons; request-abort handler explicitly removed on cleanup. - **`apps/webapp/app/entry.server.tsx`** — clears the `setTimeout(abort, ABORT_DELAY)` timer in `onShellReady` / `onAllReady` / `onShellError`. - **`apps/webapp/app/v3/tracer.server.ts` + `env.server.ts`** — gates OpenTelemetry `HttpInstrumentation` and `ExpressInstrumentation` behind `DISABLE_HTTP_INSTRUMENTATION=true` as an escape hatch for future OTel-listener retention patterns. Defaults to enabled. - **`apps/webapp/app/presenters/v3/RunStreamPresenter.server.ts`** — uses the shared `ABORT_REASON_SEND_ERROR` sentinel. ## Verification ### Full-app reproduction (memlab) Isolated local harness, 500 abrupt SSE disconnects against a dev-presence route, GC between passes, heap snapshot diff with [memlab](https://facebook.github.io/memlab/): | Run | Heap delta after 500 conns + GC | memlab retained leaks | | --- | --- | --- | | Before | +16.0 MB (linear with request count) | 158 clusters; 250 `ServerResponse`, 1000 `AbortController`, 250 `SpanImpl` retained | | After | **+3.3 MB (noise)** | **0 app-code leaks** | ### Standalone mechanism isolation To confirm *which* axis of the change is load-bearing, a separate standalone Node script (`/tmp/abort-leak-test.mjs`) ran 2000 requests × 200 KB payload per variant: | Variant | Heap delta after GC | | --- | --- | | baseline (no signal machinery) | 0 MB | | V1: `AbortSignal.any` + string abort reason | **+9.1 MB** | | V2: `AbortSignal.any` only (no reason) | **+10.8 MB** | | V3: string reason only (no `AbortSignal.any`) | 0 MB | | V4: neither (the fix) | 0 MB | | V5: `AbortSignal.any` with no listener on the composite | **+10.2 MB** | This proves `AbortSignal.any` is the sole mechanism. The reason type (`.abort()` vs `.abort("string")`) is irrelevant for retention — V3 is clean, V5 leaks even without a listener on the composite. ## Risk - `sse.ts` is used by the dev-presence routes. Behaviour is equivalent — timeouts and client disconnects still abort the stream. `signal.reason` is now a named string sentinel (`"timeout"`, `"request_aborted"`, etc.) instead of the previous string arg or default `AbortError`. No in-tree reader of `signal.reason` exists. - `entry.server.tsx` change is a standard cleanup of an abort timer, matches upstream React Router guidance. - `tracer.server.ts` change is env-gated and defaults to current behaviour. - Three other webapp `AbortSignal.timeout()` callsites (alert delivery, remote-build status) are fire-and-forget passed directly to `fetch` — not composed with anything long-lived, no retention risk, untouched. ## Test plan - [ ] Existing SSE integration tests pass - [ ] Dev-presence SSE behaves normally across tab open/close cycles - [ ] No heap growth under sustained aborted-connection traffic (heap snapshot diff) ## Follow-up The same `AbortSignal.any([userSignal, internalSignal])` pattern exists in several SDK/core callsites that ship to customers (`packages/core/src/v3/realtimeStreams/manager.ts`, `packages/trigger-sdk/src/v3/{ai,chat,chat-client,sessions}.ts`, `packages/core/src/v3/workers/warmStartClient.ts`). Whether those leak in practice depends on the user passing a long-lived signal. Tracked separately. |
||
|
Eric Allam
|
2d3b2e82e6 |
feat(run-engine): flag to route getSnapshotsSince through read replica (#3423)
## Summary Adds `RUN_ENGINE_READ_REPLICA_SNAPSHOTS_SINCE_ENABLED` (default `"0"`). When enabled, the Prisma reads inside `RunEngine.getSnapshotsSince` run against the read-only replica client instead of the primary. Offloads the snapshot-polling queries fired by every running task runner off the writer. ## Why `getSnapshotsSince` is called from the managed runner's fetch-and-process loop (once per poll interval, plus on every snapshot-change notification). It runs four sequential reads per call — one `findFirst` by snapshot id, one `findMany` on snapshots with `createdAt > X`, one raw SQL against `_completedWaitpoints`, and chunked `findMany` on `waitpoint`. Per concurrent run, every few seconds. It's read-only, tolerates a small amount of staleness, and is an obvious candidate for the replica. ## Replica-lag considerations - **Step 1 "since snapshot not found"**: if the runner just received a snapshot id from the primary and asks the replica before it replicates, the function throws and the caller treats the response as an error (runner falls back to a metadata refresh). Self-correcting, not silent. - **Step 2 missing newly-created snapshots**: the next poll's `createdAt > sinceSnapshot.createdAt` filter still picks them up once the replica catches up. - **Waitpoint junction race**: the riskiest path — if a latest snapshot is replicated but its `_completedWaitpoints` join rows aren't yet, the runner could advance past that snapshot with `completedWaitpoints: []`. WAL/storage-level replication replays commits in order, so in practice both should appear atomically on the reader, but the race window is why the flag ships disabled. Aurora reader shrinks all three windows to single-digit ms in typical conditions, and its storage-level replication gives atomic visibility of committed transactions on the reader. ## Test plan - [ ] Flip the flag on in a non-prod environment, confirm snapshot polling behaves normally and `getSnapshotsSince` errors in Sentry stay flat. - [ ] Verify writer query volume drops and reader query volume rises on the snapshot-polling queries. - [ ] Keep an eye on `AuroraReplicaLag` (or equivalent) during rollout. |
||
|
Eric Allam
|
881288c615 |
feat(webapp): deprecate v3 CLI deploys server-side (#3415)
## ✅ Checklist - [x] I have followed every step in the [contributing guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md) - [x] The PR title follows the convention. - [x] I ran and tested the code works --- ## Summary Adds a server-side gate that detects deploy attempts from v3 CLI versions (i.e. `trigger.dev@3.x`) at the `POST /api/v1/deployments` entry point and, when enabled, rejects them with a clear upgrade message. v4 CLI deploys are completely unaffected. The last 3.x CLI release was `3.3.7`, which we can't update. This approach short-circuits the deploy before any DB writes, image-ref generation, S2 stream creation, or queue enqueue — no side effects in either mode. ## How v3 vs v4 are distinguished I pulled the published CLI tarballs for `trigger.dev@3.3.7`, `4.0.0`, `4.0.1`, `4.0.5`, `4.1.0`, `4.2.0`, and the current `4.4.4` in the repo. The cleanest, most reliable signal is the request body to `POST /api/v1/deployments`: | Field on initialize | v3.3.7 CLI | v4.x CLI | |---|---|---| | `type` | **never sent** | always sent — `"MANAGED"` (run_engine_v2) or `"V1"` | | `isNativeBuild` / `gitMeta` / `triggeredVia` / `runtime` | not sent | sent | | `registryHost` / `namespace` | sent (v3-only; stripped by current Zod schema) | not sent | Every v4 call site I inspected sets `type: features.run_engine_v2 ? "MANAGED" : "V1"` unconditionally. `payload.type` is `undefined` if and only if the client is a 3.x CLI. ## Behavior - Detection always runs and emits `logger.warn("Detected deploy from deprecated v3 CLI", { environmentId, projectId, organizationId, enforced })`, which lets us watch how many v3 deploys are still happening before enforcement is flipped. - Enforcement is gated behind `DEPRECATE_V3_CLI_DEPLOYS_ENABLED` (default `"0"`, off). When `"1"`, the server returns `400` with: > The trigger.dev CLI v3 is no longer supported for deployments. Please upgrade your project to v4: https://trigger.dev/docs/migrating-from-v3 The v3 CLI surfaces this verbatim as `Failed to start deployment: <message>` because `zodfetch` throws `ApiError` for non-retryable 4xx (400/422) and `deploy.js` in 3.3.7 prints `error.message`. ## Out of scope (intentionally) - `api.v1.deployments.$deploymentId.finalize.ts` / `FinalizeDeploymentService` / `createDeploymentBackgroundWorkerV3.server.ts` are V1-engine paths, not the v3 CLI gate. Leaving them alone per review. - Container-side `createDeploymentBackgroundWorker` call in `managed-index-controller.ts` is still used by v4's in-image indexer. Not touched. - v3 `trigger dev` flow (different code path) — separate deprecation if/when needed. ## Testing - Ran `pnpm run typecheck --filter webapp` locally — passes. - Verified v4 tarballs (4.0.0, 4.0.1, 4.0.5, 4.1.0, 4.2.0, 4.4.4) all include `type:` in the `initializeDeployment` call site, so none will be accidentally blocked. - Verified v3.3.7 tarball's `initializeDeployment` payload has no `type` field. Rollout plan after merge: 1. Deploy with `DEPRECATE_V3_CLI_DEPLOYS_ENABLED` unset → watch `Detected deploy from deprecated v3 CLI` log volume. 2. When comfortable, set `DEPRECATE_V3_CLI_DEPLOYS_ENABLED=1` to enforce. --- ## Changelog Detect v3 CLI deploys on `/api/v1/deployments` and, when `DEPRECATE_V3_CLI_DEPLOYS_ENABLED=1`, reject them with an upgrade message pointing at https://trigger.dev/docs/migrating-from-v3. v4 CLI deploys are unaffected. Link to Devin session: https://app.devin.ai/sessions/b242c11bd86e4099aeec8b59bab62143 Requested by: @ericallam |
||
|
Eric Allam
|
f1f1d02f1d | feat(dashboard): a few tweaks to the AI models page (#3315) | ||
|
Matt Aitken
|
68e88d0d71 |
Object Storage seamless migration (#3275)
This allows seamless migration to different object storage. Existing runs that have offloaded payloads/outputs will continue to use the default object store (configured using `OBJECT_STORE_*` env vars). You can add additional stores by setting new env vars: - `OBJECT_STORE_DEFAULT_PROTOCOL` this determines where new run large payloads will get stored. - If you set that you need to set new env vars for that protocol. Example: ``` OBJECT_STORE_DEFAULT_PROTOCOL=“s3" OBJECT_STORE_S3_BASE_URL=https://s3.us-east-1.amazonaws.com OBJECT_STORE_S3_ACCESS_KEY_ID=<val> OBJECT_STORE_S3_SECRET_ACCESS_KEY=<val> OBJECT_STORE_S3_REGION=us-east-1 OBJECT_STORE_S3_SERVICE=s3 ``` --------- Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com> |
||
|
nicktrn
|
9cb3dcb07c |
feat(supervisor): compute workload manager (#3114)
Adds the `ComputeWorkloadManager` for routing task execution through the compute gateway, including full checkpoint/restore support, OTel trace integration, and template pre-warming. ## Changes **Compute workload manager** (`apps/supervisor/src/workloadManager/compute.ts`) - Routes instance create, snapshot, delete, and restore through the compute gateway API - Wide event logging on create with full timing and context - Configurable gateway timeout, auth token, image digest stripping **Compute snapshot service** (`apps/supervisor/src/services/computeSnapshotService.ts`) - Timer wheel for delayed snapshot dispatch (avoids wasted work on short-lived waitpoints) - Configurable dispatch concurrency limit (`COMPUTE_SNAPSHOT_DISPATCH_LIMIT`) - Snapshot-complete callback handler with suspend completion reporting - Trace context management and OTel span emission for snapshot operations **OTel trace service** (`apps/supervisor/src/services/otlpTraceService.ts`) - Fire-and-forget OTLP span emission for compute operations (provision, restore, snapshot) - BigInt nanosecond conversion preserving sub-ms precision for span ordering **Template creation** (`apps/webapp/app/v3/services/computeTemplateCreation.server.ts`) - Three-mode rollout: required (MICROVM projects), shadow (feature flag / percentage), skip - Integrated into deploy finalize flow **Shared compute package** (`internal-packages/compute/`) - Gateway client with namespace-based API (instances, templates, snapshots) - Zod schemas for all gateway request/response types **Database** - `COMPUTE` variant added to `TaskRunCheckpointType` enum - `WorkloadType` enum and column on `WorkerInstanceGroup` - `hasComputeAccess` feature flag **Env / config** - Compute gateway URL, auth token, timeout - Snapshot enable flag, delay, dispatch limit - Dedicated OTLP endpoint for compute spans (`COMPUTE_TRACE_OTLP_ENDPOINT`) |
||
|
Oskar Otwinowski
|
efe24f9c2a |
feat(private-link): Add private links UI (#3264)
⚒️ Publish Worker (v4) / build (supervisor) (push) Has been cancelled
|
||
|
Eric Allam
|
1a6481a579 |
feat: add Model Registry feature with catalog pipeline, dashboard pages, and TSQL schema (#3270)
- Add llm-model-catalog package (renamed from llm-pricing) with Claude CLI research pipeline - Add Prisma schema: catalog columns + baseModelName on LlmModel - Add ClickHouse: llm_model_aggregates MV + base_response_model column - Add TSQL llm_models schema for query page integration - Add ModelRegistryPresenter with catalog, metrics, and comparison queries - Add 3 dashboard pages: catalog (cards+table+filters), detail (overview+metrics+cost estimator), compare - Add sidebar navigation under AI section with hasAiAccess feature flag - Add admin dashboard sync/seed for catalog metadata - Add model variant grouping (dated snapshots under base models) - Add shared formatters and design system component usage refs TRI-7941 |
||
|
Eric Allam
|
1cfc296c6b |
feat(ai): LLM metrics tracking and AI span inspector (#3213)
- Automatic LLM cost enrichment for AI SDK spans (streamText, generateText, generateObject) or any other spans that use semantic gen_ai attributes with support for 145+ models - New AI span inspector sidebar showing model, tokens, cost, messages, tool calls, and response text - LLM metrics dual-write to ClickHouse `llm_metrics_v1` table for analytics - LLM metrics built-in dashboard (unlinked at the moment) - Provider cost fallback — uses gateway/OpenRouter reported costs from `providerMetadata` when registry pricing is unavailable - Prefix-stripping for gateway/OpenRouter model names (e.g. `mistral/mistral-large-3` matches `mistral-large-3` pricing) - Admin dashboard for managing LLM model pricing (list, create, edit, delete, search, test pattern matching) - Missing models detection page — queries ClickHouse for unpriced models with sample spans and Claude Code-ready prompts for adding pricing - AI span seed script (`pnpm run db:seed:ai-spans`) with 51 spans across 12 provider systems for local dev testing - UI fixes: `completionTokens`/`promptTokens` aliases, `ai.response.object` display for generateObject, cache read/write token breakdown ## Screenshots: <img width="1030" height="104" alt="CleanShot 2026-03-17 at 16 48 54@2x" src="https://github.com/user-attachments/assets/bc8fccda-e48b-4d0c-bfb1-e620064e5979" /> <img width="1094" height="1512" alt="CleanShot 2026-03-17 at 16 49 23@2x" src="https://github.com/user-attachments/assets/c2424569-d07e-4d67-a436-e8250043a1ee" /> <img width="1074" height="1412" alt="CleanShot 2026-03-17 at 16 49 18@2x" src="https://github.com/user-attachments/assets/22342ac4-4769-45d1-a328-a24fb9a82a50" /> <img width="1012" height="2292" alt="CleanShot 2026-03-17 at 16 39 01@2x" src="https://github.com/user-attachments/assets/59e327d1-6652-4293-8be0-bb8326e5fbc5" /> <img width="3680" height="2392" alt="CleanShot 2026-03-15 at 08 29 38@2x" src="https://github.com/user-attachments/assets/1f77beb8-de67-495b-b890-bcdb8d7f1fe8" /> --------- Co-authored-by: James Ritchie <james@trigger.dev> |
||
|
Matt Aitken
|
5f359be286 |
feature: Errors page (#3172)
A top-level Errors page that aggregates errors from failed runs with occurrences metrics. https://github.com/user-attachments/assets/8f0ef55e-90dd-4faa-9051-59f4665181e4 Errors are “fingerprinted” so similar errors are grouped together (e.g. has an ID in the error message). You can view an individual error to view a timeline of when it fired, the runs, and bulk replay them. |
||
|
Eric Allam
|
dee6f1d09e |
fix(batch): move batch queue global rate limiter to worker consumer level (#3166)
The global rate limiter was being applied at the FairQueue claim phase,
consuming 1 token per queue-claim-attempt rather than per item
processed.
With many small queues (each batch is its own queue), consumers burned
through tokens on empty or single-item queues, causing aggressive
throttling well below the intended items/sec limit.
Changes:
- Move rate limiter from FairQueue claim phase to BatchQueue worker
queue
consumer loop (before blockingPop), so each token = 1 item processed
- Replace the FairQueue rate limiter with a worker queue depth cap to
prevent unbounded growth that could cause visibility timeouts
- Add BATCH_QUEUE_WORKER_QUEUE_MAX_DEPTH env var (optional, disabled by
default)
|
||
|
Eric Allam
|
540e1c86a4 |
feat: Input Streams - Bidirectional task communication (#3146)
Input streams enable sending typed data to executing tasks from external
callers — backends, frontends, or other tasks. This unlocks interactive
use cases like approval UIs, cancel buttons, chat interfaces, and
human-in-the-loop AI workflows where the task needs to receive data
while running.
Three consumption patterns inside a task:
* `.wait()` — Suspend the task until data arrives (process freed, most
efficient)
* `.once()` — Wait for the next message (process stays alive)
* `.on()` — Subscribe to a continuous stream of messages
One send pattern from outside:
* `.send(runId, data)` — Send typed data to a specific run's input
stream
## User-facing API
### Define a typed input stream
```ts
import { streams, task } from "@trigger.dev/sdk";
const approval = streams.input<{ approved: boolean; reviewer: string }>({ id: "approval" });
```
### Consume inside a task
```ts
export const myTask = task({
id: "my-task",
run: async () => {
// Pattern 1: Suspend until data arrives (most efficient — frees the process)
const result = await approval.wait({ timeout: "5m" });
// Pattern 2: Wait for next message (process stays alive)
const data = await approval.once().unwrap();
// Pattern 3: Subscribe to multiple messages
approval.on((data) => { /* handle each message */ });
},
});
```
### Send from outside
```ts
// From a backend (using secret API key)
await approval.send(runId, { approved: true, reviewer: "alice" });
// From a frontend (using public JWT token from trigger response)
const { send } = useInputStreamSend("approval", runId, { accessToken });
send({ approved: true, reviewer: "alice" });
```
---------
Co-authored-by: Claude <noreply@anthropic.com>
|
||
|
Eric Allam
|
bed3789c31 |
fix(batch-queue): speed up batch queue processing by disabling cooloff and fixing retry race (#3079)
Fix slow fair queue processing by removing spurious cooloff on concurrency blocks and fixing a race condition where retry attempt counts were not atomically updated during message re-queue. Removed cooloff entirely from the batch queue |
||
|
Eric Allam
|
6409fea6ac | fix(engine): allow disabling the ttl system consumers independently from the whole system (#3115) |