296 Commits

Author	SHA1	Message	Date
Eric Allam	1a0198cc5e	perf(webapp,clickhouse): move runs empty-state check to ClickHouse (#4202) ... ## Summary The runs page's empty-state check (whether an environment has ever had a run, which decides between the "getting started" and "no runs match your filters" states) ran a `findFirst` against the Postgres `TaskRun` table. This moves it to ClickHouse, the same store the runs list itself reads from, so the check no longer queries `TaskRun`. ## Design Only the runs list triggers the check now (via an `includeHasAnyRuns` flag); the other presenters that reuse `NextRunListPresenter` (API, schedule detail, waitpoint detail, error group) no longer issue it. When the list is empty it runs `SELECT 1 FROM task_runs_v2 ... LIMIT 1` filtered on the full `(organization_id, project_id, environment_id)` sort-key prefix with a configurable `created_at` lower bound (`RUN_LIST_HAS_RUNS_LOOKBACK_DAYS`, default 30), so it hits the primary index and reads minimal granules. Results are cached in a tiered memory + Redis SWR cache. Only positive ("has runs") results are cached, so an environment with no runs is always re-checked and its first run shows up immediately.	2026-07-09 15:24:31 +01:00
DKP	00ee0751ec	feat(webapp): proxy PostHog through a same-origin /ph path (#4183) ... ## Summary posthog-js sent product analytics to PostHog Cloud directly from the browser. This points `api_host` at a same-origin `/ph` path that forwards to PostHog Cloud EU server-side, following PostHog's standard first-party reverse-proxy setup. ## How it works A resource route forwards each request server-side, splitting by path: `/ph/static/*` and `/ph/array/*` go to the asset host, everything else (analytics events, feature flags) goes to the ingest host. It rewrites the `Host` header, strips the `/ph` prefix, and streams the response back. Only PostHog's own cookies are forwarded, so the app session cookie stays first-party. Upstream hosts default to PostHog Cloud EU, overridable via `POSTHOG_INGEST_HOST` / `POSTHOG_ASSETS_HOST`. It also sets `cross_subdomain_cookie` so a single PostHog session is shared across the marketing site and app. Verified locally: static assets return 200 from the EU asset host, and analytics events return 200 through the ingest host.	2026-07-08 11:35:04 +01:00
Chris Arderne	aa74e68c71	feat(sdk): add bulk replay to api and sdk (#4105) ... ## Summary Adds SDK and API support for run bulk actions. You can now create bulk cancel or replay actions from `@trigger.dev/sdk` using run IDs or the same filters as `runs.list()`, then retrieve, list, poll, or abort the action by its `bulk_` handle. Tests, docs, changesets added. ## Design The dashboard bulk action service now accepts structured filters instead of reading directly from a dashboard request, so the dashboard and API share the same creation path. Replay actions created through the API are attributed with the existing `api` trigger source, while dashboard-created actions keep `dashboard`. The SDK exposes the new surface under `runs.bulk.*`, including `targetRegion` for replay region overrides and cursor pagination for listing bulk actions. ## Filters and runIds Nuance on filters. If `filter` is provided, it MUST have at least one key. This is to remove the footgun of passing no filter and selecting all runs. ```typescript { action: "cancel", runIds: ["run_1"] } // valid { action: "cancel", runIds: [] } // invalid, min(1) { action: "cancel", filter: { status: "FAILED" } } // valid { action: "cancel", filter: {} } // invalid { action: "cancel", filter: {}, runIds: ["run_1"] } // invalid ```	2026-07-07 15:43:30 +01:00
Daniel Sutton	712c7c3b1a	feat(webapp): add RUN_REPLICATION_RUN_OPS_DATABASE_URL for the runs-replication source (#4160) ... ## Summary The run-ops runs-replication source now takes its connection URL from `RUN_REPLICATION_RUN_OPS_DATABASE_URL`, required whenever the run-ops split is enabled. The runs replicator speaks the Postgres streaming replication protocol, which cannot run through a transaction pooler, so it needs its own direct endpoint separate from the app's `RUN_OPS_DATABASE_URL` (which may point at a pooler). When the split is on and this is unset, boot fails via `SplitReplicationMisconfiguredError` rather than silently falling back to a wrong endpoint.	2026-07-05 13:29:22 +00:00
Daniel Sutton	092b9ef07a	fix(run-ops): DNS-safe, sortable base32hex run id (replace base62 KSUID) (#4154) ... ## Problem The run-ops split mints NEW-store run ids as **27-char base62 KSUIDs**. The supervisor writes the run id into the Kubernetes pod name (`runner-<id>`), and pod names must be DNS-1123 labels (lowercase `[a-z0-9-]`) — so uppercase base62 ids make k8s reject the pod (422) and **those runs never launch** (they loop in `PENDING_EXECUTING` until the heartbeat-stall handler nacks them, forever). `.toLowerCase()` can't fix it: base62 has both `A`(10) and `a`(36) as distinct symbols, so folding collides distinct ids and destroys sort order. ## Fix: change the encoding, not the structure Mint a **26-char lowercase base32hex** run id: ``` run_<24-char base32hex core><region char><version char> [ 6-byte ms timestamp ][ 9 CSPRNG bytes ] ``` - **base32hex** (RFC 4648 §7, alphabet `0-9a-v`): lowercase, order-preserving, DNS-safe; 15 bytes → exactly 24 chars, no padding. Hand-rolled encode/decode (no new dependency). - **48-bit ms timestamp** in the leading bytes → plain string sort == creation order at millisecond resolution. - **72 bits CSPRNG** entropy; PK unique constraint is the backstop (no retry loop). - **region / version** are raw positional chars (read via one `charAt` before decoding/routing), version = `"1"`. DNS-safe from birth and hyphen-free, so **firekeeper is unchanged** — `runner-<id>-attempt-N` → strip `runner-`, cut at first hyphen still recovers the exact id incl. region+version. ## Residency discriminator: length → version char `classifyKind`/`classifyResidency` (`runOpsResidency.ts`) previously distinguished NEW vs LEGACY by **id length**. That gets ambiguous with a third format. It now discriminates on the **version char at a fixed position** (`isRunOpsIdBody`: 26 chars, `[25] === "1"`, base32hex alphabet) → NEW; everything else → LEGACY. Total, never throws. The `Residency` (NEW/LEGACY) contract the routing store consumes is unchanged; the `"ksuid"` `ResidencyKind` label is retained only because it's the persisted `runOpsMintKsuid` feature-flag value. ## Scope / verification - Generator + discriminator in `@trigger.dev/core` isomorphic; mint path + all id-shape call sites swept (~40 webapp files); changeset added (`@trigger.dev/core` patch). - Core unit tests (encode/decode round-trip + property, generator shape, ms sort-order incl. intra-second, parse partitioned-vs-legacy, firekeeper round-trip): **24 pass**. `@trigger.dev/core` builds; webapp typechecks; format/lint clean. ## Open decisions (flagged, not silently chosen) 1. **Backward-compat**: existing 27-char base62 KSUID runs now classify LEGACY. On test cloud these are the broken/looping runs that never completed, so this is acceptable — but worth a conscious call before prod. No transitional length-recognition added (keeps the discriminator clean). 2. **Storage collation**: the sort guarantee is byte-order — if the run-ops id column is `TEXT` with default locale collation it's silently not honored. Confirm whether `COLLATE "C"` / `BYTEA` is needed on the run-ops schema. 3. **Region sourcing** wiring — see `regionCharForRegion` / `REGION_CODES`. --- ## ⚠️ Required migration — deploy in lockstep This PR renames a persisted feature-flag key/value and an env var. These are **not** changed by the code alone and must be migrated when this deploys, or affected orgs silently fall back to `cuid` minting (no crash — `defaultValue: "cuid"`): 1. **Env var** (terraform): `RUN_OPS_MINT_KSUID_ENABLED` → `RUN_OPS_MINT_ENABLED` (carry the value over). 2. **DB** `organization.featureFlags`: migrate both the key and value together: - key `runOpsMintKsuid` → `runOpsMintKind` - value `"ksuid"` → `"runOpsId"` Until an org's flag row is migrated, its `runOpsMintKind` lookup misses and it mints `cuid` (legacy) — so no NEW-store ids for that org until the data lands. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-05 10:05:54 +01:00
Daniel Sutton	962bc48738	feat(run-ops): automatically migrate the dedicated run-ops database (#4150) ... ## What Adds the ability to **automatically migrate the dedicated run-ops database** (the NEW DB in the run-ops split), matching how every other database in the system is migrated. Follow-up to the run-ops split activation. ## Changes - **Migrate runner** — new `internal-packages/run-ops-database/scripts/migrate.mjs`, exposed as `db:migrate:deploy` / `db:migrate:status`. Connects via `RUN_OPS_DATABASE_URL` (the same var the app uses) and expands `${VAR}` refs like Prisma's dotenv. - **Self-host** — `docker/scripts/entrypoint.sh` runs the run-ops migration on boot when the DB is configured, gated by `SKIP_RUN_OPS_MIGRATIONS`. Single-DB installs never set the URL, so it's a clean no-op. - **Single env-var family** — the run-ops DB is now addressed by one canonical `RUN_OPS_*` family, connect path and migrations resolving the identical URL: - `RUN_OPS_DATABASE_URL` (writer) — replaces `TASK_RUN_DATABASE_URL` - `RUN_OPS_LEGACY_DATABASE_URL` — replaces `TASK_RUN_LEGACY_DATABASE_URL` - `RUN_OPS_DATABASE_READ_REPLICA_URL` — replaces `TASK_RUN_DATABASE_READ_REPLICA_URL` - the old `TASK_RUN_*` aliases, the `??` coalesce, the `runOpsNewDatabaseUrl` indirection, and the migrate-only `directUrl` are all removed (consumers read `env.RUN_OPS_DATABASE_URL` directly). `directUrl` was dropped because it was only ever used by `prisma migrate` (never the app runtime) to bypass a pooler for advisory locks — premature here since the run-ops connection isn't wired to the app yet. If a pooler is later introduced for the app, a direct URL can be reintroduced then. ## Safety - **Pure rename** — nothing deployed sets any `TASK_RUN_*` var (the split isn't activated anywhere yet; `.env.example`, docker-compose, and cloud already use `RUN_OPS_*`), so there is no config migration. - **Single-DB / self-host** — no new required env var; entrypoint and migrate are no-ops when `RUN_OPS_DATABASE_URL` is unset. - **Cloud** — runs migrations as pre-deploy ECS tasks (companion cloud PR), calling these same `db:migrate:deploy` / `db:migrate:status` commands. ## Verification - Live migration against a fresh scratch DB with only `RUN_OPS_DATABASE_URL` set: both migrations applied, no `P1012`/`P1013`; `${VAR}` expansion, idempotent re-run, `status`, and no-op skip all pass. - Schema parity 4/4; `typecheck --filter webapp` 18/18; affected split/replication tests 34/34. ## Scope This delivers automatic migrations only. Enabling the app to *use* the new DB (setting `RUN_OPS_DATABASE_URL` + `RUN_OPS_SPLIT_ENABLED` on the service) is a separate activation step. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-04 15:44:47 +00:00
Daniel Sutton	8465ac5ac3	feat(run-ops): webapp db topology, flags, and split-mode resolver wiring (#4117) ... ## What Wires the run-ops split into the webapp: database topology, environment flags, split-mode gating, and the control-plane resolver/cache layer that the run-store and run-engine seams from the previous PR plug into. - **DB topology & env** (`apps/webapp/app/db.server.ts`, `env.server.ts`, `entry.server.tsx`): adds the run-ops database clients/topology and the environment variables that configure and gate the split. - **runOpsMigration module** (new `apps/webapp/app/v3/runOpsMigration/`): the webapp-side machinery — `splitMode.server.ts`, `controlPlaneResolver.server.ts` + `controlPlaneCache.server.ts`, `readThrough.server.ts`, `crossSeamGuard.server.ts`, `distinctDbSentinel.server.ts`, id-minting helpers (`mintBatchFriendlyId`, `runOpsMintKind`, `resolveInheritedMintKind`), `runOpsCascadeCleanup.server.ts`, the split read gate, and route/unblock catalogs. - **Store/engine wiring** (`app/v3/runStore.server.ts`, `runEngine.server.ts`, `runEngineHandlers.server.ts` + new `runEngineHandlersShared.server.ts`): points the webapp's store/engine construction at the resolver, and factors shared handler logic out so both seams use one path. - **Read-path touch-ups**: `runtimeEnvironment.server.ts`, `eventRepository/index.server.ts`, `taskRunHeartbeatFailed.server.ts`, `engineVersion.server.ts` route their run/environment lookups read-through the resolver. - `413a94511` — interlocks split mode against the native realtime backend so the two aren't enabled in an incompatible combination (see `.server-changes/run-ops-split-realtime-interlock.md`). - `dc74c57fd` — drops the earlier "known-migrated" read layer; residency is determined by id-shape only. ## Why PR5 of the run-ops split stack. This is the webapp foundation layer: it stands up the DB topology, flags, and resolver/cache the rest of the stack depends on, and repoints webapp read paths through the resolver. Additive when the split is not enabled (existing single-DB behavior preserved behind flags); behavior-changing on the read-through paths and the realtime interlock. ## Tests New vitest coverage across `apps/webapp/test/` and colocated `*.server.test.ts` files: db topology, split mode, split read gate, cross-seam guard, mint cutover / flip latency, control-plane cache, control-plane resolver, distinct-db sentinel, read-through loaders (route loaders, run-detail loaders, `findEnvironmentFromRun`), and the run-engine handlers. Testcontainers-backed; no mocks. `pnpm-lock.yaml` synced for the two new webapp deps. ## Notes Draft, **stacked on #4116** (`runops/pr04-store-engine`). Review that first; this diff is against it. Server-change / changeset note to be added at stack-assembly time. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-03 18:02:22 +01:00
Eric Allam	ea9d6f7430	feat(webapp): pin the dashboard agent to a deployed version (#4128) ... ## Summary The dashboard agent runs as a `chat.agent` in its own Trigger.dev project, deployed separately from the app that starts its sessions. This adds independent, non-disruptive deploys for it: a new workflow deploys the agent with `--skip-promotion` (so a deploy never becomes the current version on its own), and the app pins its sessions to a chosen version. ## How it works `DASHBOARD_AGENT_VERSION` (unset by default) pins agent sessions to a specific deployed version; when unset, sessions run on the project environment's current version. The pin is passed on session start and head start and is forwarded to every continuation run, so a pinned session stays on its version for its whole life. Cutting over to a new build becomes a config change (set the version) rather than a redeploy, and rollback is flipping it back. The workflow (`dashboard-agent-deploy.yml`) runs a leg per environment (staging and prod), each gated by its own environment, and triggers on pushes to `main` that touch the agent or its store (also available via manual dispatch). Deploy versions are per-environment, so each environment pins to its own leg's version.	2026-07-03 10:19:38 +01:00
Oskar Otwinowski	c7f6ed501c	feat(cli,core): add opt-in dev-only telnet log streaming (#4110) ... Stream dev logs over a local telnet/TCP socket. `trigger dev` mirrors its terminal output on port 6767 by default (override with --telnet-logs-port or TRIGGER_DEV_TELNET_LOGS_PORT, 0 disables). webapp, supervisor, and coordinator each expose an opt-in stream gated on a per-service *_TELNET_LOGS_PORT env var. New @trigger.dev/core/v3/telnetLogServer module (localhost-only, backpressure-safe, plain-text) plus optional static Logger.onLog / SimpleStructuredLogger.onLog sinks. Then you (or your agent) can use `nc` to connect and filter out the stream. <img width="1103" height="239" alt="image" src="https://github.com/user-attachments/assets/b4d47efc-8a57-4185-a159-10f2806627ae" />	2026-07-02 19:16:14 +00:00
Eric Allam	f1bd11a7ef	feat(webapp): gracefully shut down the v3 engine behind a flag (#4017) ... ## Summary Adds a single env flag, `DEPRECATE_V3_ENABLED` (default off), that gracefully winds down the v3 engine (`RunEngineVersion.V1`). While it's off nothing changes, so self-hosted instances still on v3 keep working. When it's on: - Triggers that resolve to v3 are rejected with a clear, actionable error pointing at the [v4 migration guide](https://trigger.dev/docs/migrating-from-v3), instead of silently creating runs that never execute. This covers single triggers, batches, scheduled fires, replays, and `triggerAndWait`, which all funnel through one place. - The legacy `trigger dev` websocket used by v3 CLIs is closed with an upgrade message (v4 CLIs use a different dev transport). - The v3 shared-queue consumer refuses to start, so no deployed v3 runs are dequeued. - The v3 run-lifecycle background jobs (heartbeat timeout, TTL expiry, retry, resume batch/dependency, delayed-run enqueue, and scheduled fires) become no-ops, so abandoned v3 runs stop generating database load. This builds on the existing deploy deprecation flag, which already rejects v3 CLI deploys. ## Design Enforcement is read through one helper, `isV3Disabled()`. Every gate combines it with a per-run or per-project engine check (`isV3Disabled() && engine === "V1"`), so a v4 run that happens to reach a shared service behaves exactly as before. v4 (V2) is never affected. The flag is a hard switch, not a drain: when it's on, in-flight v3 runs are abandoned in place rather than failed or expired, which is the intended behaviour for the final shutdown.	2026-06-27 15:51:18 +01:00
Katia Bulatova	b1987dc090	feat(webapp): billing limits — pause, reject, recovery, and settings UI (#3996) ... ## Summary Adds Billing Limits to the webapp. Customers can set a monthly spend cap. When usage crosses the limit, billable environments enter a grace period. If the limit is not resolved before grace expires, new triggers are rejected until the organization increases or removes the limit.	2026-06-26 17:12:53 +02:00
Chris Arderne	4fde283e76	chore: format and lint webapp also (#4056) ... #3977 added formatting and linting everywhere else. This extends it to the webapp.	2026-06-26 13:02:53 +01:00
Eric Allam	9ef5cf0055	fix(webapp): stop showing the in-dashboard agent to admins by default (#4050) ... ## Summary The in-dashboard agent button was rendered for all admins and impersonators regardless of the `hasDashboardAgentAccess` flag, so it appeared even where the agent is disabled (for example, floating over the run inspector controls). It is now gated by the flag for everyone, so it stays hidden until the flag is turned on. ## Rollout Both levers default off, so nothing changes for users until deliberately enabled: - **Per-org:** set `hasDashboardAgentAccess` on an org's feature flags to enable the agent for just that org. - **All admins:** set `DASHBOARD_AGENT_ADMIN_PREVIEW=1` to give admins and impersonators an everywhere-preview, independent of the per-org flag. Previously admins bypassed the flag unconditionally, which is why the button showed up before the agent was ready to ship.	2026-06-26 11:09:14 +01:00
nicktrn	bc605eedaf	fix(webapp): verify deployment image exists before finalizing (#4049) ... A deployment could be marked deployed and promoted to current without its image ever landing in the registry. Finalize trusted the CLI: the v1 path never pushed or checked, and the v2/v3 path skips its own push when the CLI sends `skipPushToRegistry` - which the local-build path always does. In the happy path the CLI pushes the image itself, so this stayed latent. But any deviation - `--no-push`/`--load`, a push that lands in a different registry, or an old CLI - promoted a version whose image can't be pulled, so every run failed at pull time while the deploy itself reported success. This adds a registry existence check after push and before finalize. If the image isn't there, the deploy fails loudly instead of promoting a version that can't start. The check is ECR-only (a no-op for other registries, so self-hosted setups are unaffected) and uses `BatchGetImage`, which the deploy role already allows. It fails open on an ambiguous registry error so the check can't itself turn into a deploy outage. The image reference is the platform-generated value and the lookup is bound to the configured registry host; the CLI-supplied digest is validated before use. Can be turned off with `DEPLOY_IMAGE_VERIFICATION_ENABLED=0` for setups that push images out of band (e.g. an air-gapped registry the platform can't reach). refs TRI-11243	2026-06-25 21:49:56 +00:00
Eric Allam	2fa84ea124	feat(webapp): gate worker dequeues by worker queue via env var (#4030) ... ## Summary Adds a `RUN_ENGINE_DEQUEUE_DISABLED_WORKER_QUEUES` setting that refuses worker dequeue requests for the listed worker queues (or base regions), so their runs stay queued instead of being handed to workers that can't run them. Blocked dequeues are counted via a `run_engine.dequeue.blocked` OTel counter (labeled by `worker_queue` and `region`).	2026-06-24 19:07:43 +01:00
Eric Allam	8890d7a258	feat(run-engine,webapp): always report worker queue length metrics (#4029) ... ## Summary The `runqueue.workerQueue.length` gauge only reported a worker queue's depth while runs were being dequeued from it. When dequeues stop, the metric goes stale or missing, so a queue that has backed up because nothing is draining it can't be alerted on. This adds a small observer that refreshes the observed set of worker queues from the `WorkerInstanceGroup` records on an interval, so every active worker queue (and its scheduled split variant) keeps reporting its length regardless of dequeue activity. The observer is off by default and enabled per service via `RUN_ENGINE_WORKER_QUEUE_OBSERVER_ENABLED`, reads from the read replica, and skips a configurable set of cloud providers (`RUN_ENGINE_WORKER_QUEUE_OBSERVER_EXCLUDED_CLOUD_PROVIDERS`, default `digitalocean`). When enabled it is the source of truth for the observed set, so the per-dequeue registration is skipped on that instance, and it groups by worker queue so the per-instance duplicates collapse to the true depth. Also removes the unused `GET`/`POST /api/v1/workers` endpoints. Their only consumer was a CLI command group that is no longer registered. ## Verification Verified end to end against a local stack: the gauge reports each worker queue's length with no dequeues happening, excludes the configured providers, includes hidden groups, and the removed endpoints return as if they never existed. Added a run-engine test (`workerQueueObservation.test.ts`).	2026-06-24 19:04:38 +01:00
Eric Allam	c06005b353	feat(webapp,sdk): in-dashboard AI agent (#4018) ... ## Summary Adds an in-dashboard AI agent: a chat panel, reachable from any environment page, that answers questions about your runs, errors, tasks, and analytics, diagnoses why a run failed, charts your data, reads your connected repo's source, and answers product and how-to questions. It is gated behind the `hasDashboardAgentAccess` feature flag (global or per-org, default off), so this PR ships disabled: the launcher is hidden unless the flag is enabled. ## Design The agent runs as a standalone `chat.agent` Trigger task in its own internal package, with no access to the webapp database, Prisma, or ClickHouse. It reads the user's data over the public API, acting as the user via a short-lived delegated user-actor token minted server-side each turn (never in the browser), building on [#3997](https://github.com/triggerdotdev/trigger.dev/pull/3997). The error and analytics tools use [#4005](https://github.com/triggerdotdev/trigger.dev/pull/4005) and the TRQL query API. The first turn of a new chat streams from a warm webapp route (Head Start) while the durable agent boots in parallel. Structured answers (a run-failure diagnosis card, a live chart) render through a small typed view catalog rather than arbitrary markup. A knowledge lane forwards product and how-to questions to the support assistant. Conversation history lives in a separate Drizzle-backed store on its own Postgres schema, kept as a display read-model so it can never corrupt the agent's model context. The SDK changes add an `apiClient` option to `chat.createStartSessionAction` and `chat.headStart`, and keep the Head Start tool-approval tail intact across a custom `prepareMessages` hook so prompt caching and Head Start compose.	2026-06-24 19:04:28 +01:00
Eric Allam	c6f0769299	fix(webapp): bound logs search memory and fix pagination at scale (#4012) ... ## Summary The logs search page (behind a feature flag) ran ClickHouse out of memory when browsing back over long time ranges. This keeps it within bounded memory and fixes a pagination bug that could skip or duplicate rows at a page boundary. ## Fix Memory: the list query reads in sort-key order, which opens one read stream per part in the window, and on object storage those per-part read buffers dominate peak memory, so it scaled with the number of parts scanned. Two changes bound it: - The logs ClickHouse client caps the per-part read buffers via new env-tunable settings. The object-storage-only setting is opt-in, so it is never sent to a ClickHouse version that lacks it. - Recent-first window narrowing: rows come back newest first, so the presenter probes the most recent window and only widens toward the full requested range when a page is short. A busy environment fills a page from a few recent parts instead of scanning the whole range; a quiet one still returns every row in a couple of cheap reads. Correctness: the keyset cursor ordered on (triggered_timestamp, trace_id), which is not unique because the spans of a trace share both, so rows at a tie could be skipped or duplicated across pages. The cursor and ORDER BY now include span_id, and the cursor is versioned so stale cursors reset to the first page. Guards: the effective page size is capped, and the existing per-query memory limit lets a pathological wide browse fail with an error instead of taking the node down. ## ClickHouse 26.2 The memory fix relies on lazy materialization deferring the wide attributes column to the output rows, which only holds on 26.x. Cloud already runs 26.2, so this moves the dev stack, testcontainers, and CI to match. The ClickHouse test suite passes on 26.2. 🤖 Generated with [Claude Code](https://claude.com/claude-code)	2026-06-22 13:48:27 +01:00
Oskar Otwinowski	e98a547e6c	feat(sso): SAML/OIDC single sign-on (#3911)	2026-06-19 09:40:20 +01:00
Iss	c97d246197	feat(webapp): sync new orgs + users to Attio CRM on signup (#3896) ... Pushes new organizations and users into the Attio CRM at signup time, for Customer Success (TRI-10431). - Orgs → Attio `workspaces`, users → Attio `users`, keyed on Attio's built-in unique `workspace_id` / `user_id` so writes are idempotent upserts. - Runs on the common Redis worker (not inline), so a slow or unavailable Attio never blocks the signup path; failures retry (3 attempts). - Hooks: user-created (alongside the existing Loops call) and org-created (`createOrganization`). - Gated behind `ATTIO_API_KEY`, no key means the sync is skipped entirely, so OSS / self-hosted installs are unaffected. Only creation is covered here (the record "shell"); spend, runs, plan changes, churn, and role/relationship linking are populated by the scheduled full sync, tracked separately. **Deploy note:** requires an Attio API key set as `ATTIO_API_KEY` in the webapp env, with scopes **Records (read-write)** + **Object Configuration (read)**, the assert/upsert endpoint reads object config to resolve the matching attribute. Without the key the sync no-ops. --------- Co-authored-by: Matt Aitken <matt@mattaitken.com> Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-06-18 14:46:37 +01:00
nicktrn	7aa871f37b	feat(webapp): plan-aware compute migration (#3957) ... Adds an opt-in mechanism to route a configurable percentage of organizations onto the compute (MicroVM) backing of their region at trigger time, without changing their stored region settings. Routing is gated by three global feature flags - `computeMigrationEnabled`, `computeMigrationFreePercentage`, `computeMigrationPaidPercentage` - plus a per-org `computeMigrationEnabled` override that wins in both directions. A region's compute backing is resolved from a new `WorkerInstanceGroup.region` column: a container group and its MicroVM group share one geo `region`, so the migration swaps the resolved worker queue to the backing group's queue. Orgs are bucketed deterministically by id, so ramping a percentage down keeps a strict subset rather than reshuffling, and a region with no compute backing is never touched. Everything is off by default - behaviour is unchanged unless the flags are set. The flags and the worker-region groups are read on the trigger hot path from in-memory snapshots rather than the database: a small `createReloadingRegistry` helper loads each at startup and refreshes them on an interval, so no per-trigger query is added and a percentage or kill-switch change propagates within the reload interval. A cold replica whose snapshot hasn't loaded yet reads as not-migrated (the container path) and self-corrects on the next load - the same cold-start contract as the datastore / LLM-pricing registries, with a `reloading_registry_loaded` metric so a never-loaded registry is alertable. The same migration decision is consulted at deploy-time template creation so a migrated org gets a compute template built ahead of its first run. This runs in shadow mode (best-effort, never fails the deploy) by default, or - when the `computeMigrationRequireTemplate` flag is on - in required mode, built synchronously at deploy so the first run never builds on-demand and template errors surface at deploy time. So operators keep "which runs ran where" while customers only see geography: the run's actual worker queue is stored raw, and the geo region is stamped separately on `TaskRun.region` (and a new ClickHouse `region` column) at trigger time. Read surfaces - the dashboard, the API, and the Query/Logs page - show the geo region, falling back to the worker queue for runs written before the column existed. Minor follow-ups left out of scope: the percentage flags render as text inputs on the admin flags page (the catalog UI has no numeric control type yet), and `createReloadingRegistry` could later gain pub/sub for sub-second cross-replica propagation if the reload interval proves too slow.	2026-06-17 08:28:15 +01:00
Matt Aitken	f48c89752c	perf(webapp): parallelize streaming batch-item ingest (#3777) ... ## Problem The item-streaming endpoint of the two-phase batch API (`POST /api/v3/batches/:batchId/items`) processed streamed items strictly sequentially. For a batch of many large payloads, each offloaded to object storage inline, this serialized N object-store round-trips inside a single request and could exceed Node's default `server.requestTimeout` (300s). The webapp then returned `408`, which the SDK reads as `408 terminated` and retries up to 5 times, turning a slow ingest into a failure that takes tens of minutes to surface. ## Fix Ingest now runs through `p-map` over the NDJSON async iterable with bounded concurrency (`STREAMING_BATCH_INGEST_CONCURRENCY`, default 10): - `p-map` pulls lazily from the stream, so at most `concurrency` items are read and in-flight at once. Peak memory stays bounded to roughly `concurrency × STREAMING_BATCH_ITEM_MAXIMUM_SIZE` and request-body backpressure is preserved. - Set the env to `1` for fully sequential ingestion (escape hatch). ## Why this is safe (ordering and idempotency unchanged) - Ordering derives from each item's index (enqueue `timestamp = batch.createdAt + index`), not enqueue order. - Dedup is atomic per index in `enqueueBatchItem`. - The NDJSON parser now stamps oversized-item markers with their emit position, removing the consumer's sequential `lastIndex` assumption (the only order-dependent bit). - The count-check and conditional-seal path is untouched. ## Scope This speeds up every batch ingested through the streaming endpoint, not just large-payload batches. Each item does a per-item Redis enqueue regardless of size, and those now overlap. Large payloads benefit most because they add an object-store offload round-trip on top of the enqueue. ## Verification Added an integration test (`streamBatchItems.test.ts`) that drives the real service against Postgres + Redis + RunEngine and times a 150-item batch at increasing concurrency. Object-store offload is modelled as a fixed per-item latency (local round-trips are too small to compare meaningfully): ``` runCount=150 large payloads (10ms/item offload): concurrency=1 1739ms concurrency=10 192ms (9.1x faster) concurrency=50 57ms (30.7x faster) small payloads (Redis enqueue only, no offload): concurrency=1 90ms concurrency=10 24ms (3.7x faster) ``` The test asserts correctness at every concurrency (all items accepted, sealed, enqueued exactly once), that parallel ingest beats the sequential floor, and that the small-payload case is strictly faster than sequential, so the win is not specific to large payloads. Also exercised end-to-end over real HTTP against a local server: a 20-item batch (12MB body) ingests and seals, a re-stream of the sealed batch returns `sealed: true` with zero re-accepted items (idempotent retry), and an oversized item still seals at its correct index. Existing coverage stays green: concurrent ingest of a 100-item batch, in-flight processing never exceeding the configured concurrency, concurrent dedup on streaming retry, and emit-position marker indexing. ## Follow-ups (not in this PR) - SDK pre-offload of large item payloads (send `application/store` refs instead of raw blobs) to remove object-store work from the request hot path and shrink the request body. - Optional `server.requestTimeout` bump as a safety net. ## CI fix Added `.github/workflows/codeql.yml` to replace GitHub's automatic ("dynamic") CodeQL scanning. The dynamic setup was failing to upload SARIF results because the auto-generated `GITHUB_TOKEN` lacked the `security-events: write` permission. The explicit workflow grants that permission at the job level and pins all actions to commit SHAs, consistent with the repo's security conventions. ## ✅ Checklist - [ ] I have followed every step in the [contributing guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md) - [ ] The PR title follows the convention. - [ ] I ran and tested the code works --- ## Testing - Integration test (`streamBatchItems.test.ts`) validates correctness and performance at concurrency 1, 10, and 50 for both large and small payloads. - End-to-end verified over real HTTP: 20-item/12MB batch ingests and seals, idempotent retry returns `sealed: true`, oversized item seals at correct index. --- ## Changelog Streaming batch ingest now processes items with bounded concurrency instead of one at a time, so batches of many large payloads ingest far faster and no longer time out. Concurrency is configurable via `STREAMING_BATCH_INGEST_CONCURRENCY` (default 10); set it to 1 for fully sequential ingestion. --- ## Screenshots _[Screenshots]_ 💯 --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-12 13:01:29 +01:00
Eric Allam	954ee5c572	fix(webapp): deliver realtime changes with current content when the read replica lags (#3910) ... ## Summary When the realtime runs feed (the backend behind the `realtimeBackend` feature flag) hydrates a change from a Postgres read replica, the read can race the replica's apply of the very write that triggered it. The delivered row then carries the previous change's content, and an isolated final change (for example a last `metadata.set` before a run goes quiet) is not corrected until the roughly 20 second backstop poll. Measured against a replica with deliberate apply delay, every delivery trailed exactly one change behind and a final change stranded for the full backstop interval. ## Fix Publishers stamp each change record with the committed row's `updatedAt`, taken from writes they already perform, so the stamp costs no extra queries. The router delays its wake hydrate until the replica's measured lag has passed, anchored to that timestamp: a record that has already spent longer than the lag in transit is hydrated immediately, so only the racing leading edge ever waits. After hydrating, a tripwire compares each row against its record's watermark. Still-stale rows are withheld and retried briefly, and each detection feeds the lag estimate. If retries run out, the rows are delivered anyway (liveness over freshness) and follow-up re-hydrates emit the fresh version through the normal working-set diff once the replica catches up, with the backstop as the terminal net. Replica lag is sampled reader-side only, and only while feeds are active. Aurora reports live lag via `aurora_replica_status()`; vanilla Postgres can only report "caught up or not" (mid-apply lag is not honestly measurable from a replica), so tripwire observations floor the estimate there. Deployments without a replica resolve to zero lag and skip the gate entirely. Tunables live under `REALTIME_BACKEND_NATIVE_REPLICA_LAG_*`, and `realtime_native.stale_hydrates` plus `realtime_native.replica_lag_estimate_ms` make replica health observable. Two adjacent fixes: a metadata update that writes nothing no longer publishes a change record, and buffered parent and root metadata operations now publish when the flusher writes them, so those changes wake live feeds instead of waiting for the backstop. For local testing, `docker-compose` gains an opt-in `database-replica` service (compose profile `replica`) with a configurable `recovery_min_apply_delay`, which reproduces replica-lag behavior deterministically. With the gate disabled this rig reproduces the one-change-behind delivery exactly; with it enabled, deliveries arrive with current content at roughly the true replica lag, across write rates faster and slower than the lag itself.	2026-06-12 07:34:50 +01:00
Eric Allam	187c0476c3	perf(webapp): shrink run trace loader payload and add trace span cap controls (#3906) ... ## Summary The run trace page loader serialized every span's raw OTel events (with full properties) into the response, even though the tree UI only renders the derived `timelineEvents` and the span detail panel refetches what it needs. On event-heavy traces that inflated both the loader payload and the server-side heap copies built per request. This PR keeps raw span events server-side and pairs that with a few related trace-view improvements: - A new optional `TRACE_VIEW_EMERGENCY_SPAN_CAP` env var (unset by default) clamps the trace summary and detailed trace summary span limits on both event store paths, including the public run trace endpoint, so operators can bound trace query sizes in one place without retuning the per-store limits. - The TreeView virtualizer resolved every rendered row with a linear scan over the whole tree (and `getNodeProps` did the same via `findIndex`); rows now resolve through memoized id lookup maps, which matters once traces reach tens of thousands of spans. - The run stream SSE lookup now applies the same organization membership scoping as the rest of the run page presenters, for consistency. Behavior is unchanged by default: the trace tree renders from the same `timelineEvents` it always has, and the new cap only takes effect when set.	2026-06-11 18:49:05 +01:00
Eric Allam	f9d57d3bd5	feat(webapp): add a new backend for the realtime runs feed (#3864) ... ## Summary Adds a second backend for the realtime runs feed (`useRealtimeRun`, `subscribeToRunsWithTag`, `subscribeToBatch`), built to stay healthy when a single busy environment has many subscribers watching many runs at once. It is gated behind a feature flag with the existing backend as the default, so nothing changes for users until it is enabled per environment. ## Design A run change is published once, as a small self-describing record, to a single per-environment channel. Every feed is then a predicate over that one stream rather than owning a channel: - A per-instance router indexes the currently-held feeds by run, tag, and batch. When a run changes it hydrates the affected rows once and serializes them once, then fans the result to every matching feed. One hot shared tag watched by many subscribers costs a single database query and serialize, not one per subscriber. - Feeds that don't match a change are never woken, wake delivery per environment is coalesced on a leading edge (250ms default) so a burst of changes costs one wake, and cold reads coalesce onto a single short-TTL-cached resolve. - An admission gate bounds how many cold ClickHouse resolves run concurrently, so a mass reconnect across many distinct filters queues instead of stampeding the database. - Changes that land while a client is between long-polls are delivered on its next poll instead of waiting for the periodic backstop: each environment buffers its recent change records, subscriptions linger briefly after the last feed closes, and a newly-armed poll replays exactly the connection's gap. - The per-connection replay cursors behind that are shared across instances via Redis (a single timestamp each), so a poll landing on a different instance behind the load balancer still reads the connection's true gap instead of falling back to a cold resolve. Cursor reads have a bounded deadline and degrade to the cold-read path on any Redis trouble. - Tag subscriptions with multiple tags match runs carrying all of the tags, mirroring the existing backend's filter semantics, and live long-polls hold for about 20 seconds to match its cadence. - The per-environment channel supports Redis Cluster sharded pub/sub, so the wake path scales horizontally across shards by environment. - The backend reports its health through OpenTelemetry metrics (delivery lag, poll resolution paths, backstop outcomes, replay and cursor-store activity), with a provisioned Grafana dashboard for local development. Everything is behind the feature flag and tunable via env vars; the existing backend remains the default.	2026-06-11 07:56:10 +01:00
Daniel Sutton	6afc9bfa4c	fix(run-engine): retry getSnapshotsSince on the replica then primary when the read replica lags (#3889) ... ## Summary When `RUN_ENGINE_READ_REPLICA_SNAPSHOTS_SINCE_ENABLED` is on, `RunEngine.getSnapshotsSince` reads from the read replica. During write spikes the replica can briefly lag, so the snapshot id a runner just learned from the writer isn't visible there yet: the lookup threw, the worker route returned a 500, and the runner waited for its next poll — turning sub-second snapshot notifications into poll-interval latency exactly when things are busiest. This PR makes the flag safe to enable: a replica miss of the since snapshot gets one jittered retry on the replica (most lag windows are shorter than the ~50–200ms wait, so the writer is never touched), then falls back to the primary, observed via a new `run_engine.snapshots_since.replica_miss` counter with an `outcome` attribute (`replica_retry` vs `primary`). Only genuine misses — absent on the primary too — remain errors. ## Design - `getExecutionSnapshotsSince` now throws a typed `ExecutionSnapshotNotFoundError` so the engine can distinguish the expected lag miss from real failures. The message string is unchanged and the error never leaves the engine. - The recovery path only engages when the flag is on, a distinct replica client is configured, and no transaction client was passed. With the flag off, the path is behaviorally identical to before. - Retry delay bounds are configurable (`RUN_ENGINE_SNAPSHOTS_SINCE_REPLICA_RETRY_MIN_MS`/`MAX_MS`, default 50/200; `MAX_MS=0` skips the replica retry and goes straight to the primary). - The warn log fires only when the primary serves the read (the writer spill is the operationally interesting event); replica-retry recoveries are counted but quiet. A permanently-missing snapshot id stays an error-level failure with a `failedDuring` field, so lag metrics aren't polluted by bogus ids. - Stale-tail lag (replica has the since snapshot but not newer rows) deliberately still returns the replica's view; the next poll catches up. - The since-snapshot anchor lookup is now scoped to the polled run (`where: { id, runId }`), so a snapshot id from a different run raises not-found instead of silently anchoring a too-wide window of the run's snapshots. ## Test plan All vitest + testcontainers, no mocks. A new `schemaOnlyPrisma` fixture (migrated-but-empty clone database) simulates a replica that hasn't caught up, and a real in-memory OTel meter pins the counter semantics per outcome. - [x] Replica catches up during the jittered retry window → served by the replica, `outcome=replica_retry` = 1, primary never consulted - [x] Replica permanently missing the since snapshot → served by the primary, `outcome=primary` = 1 - [x] Snapshot missing on both replica and primary → null, counter = 0 - [x] Replica has the since snapshot but lags by one → the replica's view is served, no fallback (verified discriminating power: the test fails if reads secretly hit the primary) - [x] Flag off with a replica configured → primary serves the read - [x] Transaction client provided → bypasses the replica entirely - [x] Since snapshot belonging to a different run → null - [x] Existing getSnapshotsSince + waitpoints suites green; run-engine, testcontainers, and webapp typechecks pass 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>	2026-06-10 16:54:03 +00:00
Eric Allam	85886b96da	feat(webapp,supervisor): isolate scheduled runs on a dedicated worker queue (#3839) ... ## Summary Scheduled runs and their descendants can now be routed to a dedicated per-region worker queue, processed by a separate worker fleet, so a burst of scheduled crons no longer competes with standard and agent runs for the same queue and inflates their startup latency. It is off by default and enabled per organization via a feature flag (with a global default), so nothing changes until it is turned on. ## Design At trigger time, any run whose lineage originates from a schedule (`rootTriggerSource === "schedule"`, which already propagates from a scheduled run down to all of its children) gets its worker queue suffixed with `:scheduled`. The worker queue name is an opaque string persisted on the run and used verbatim by enqueue and dequeue, so this needs no Lua, message-envelope, or concurrency changes. Concurrency stays keyed by environment and queue, not by worker queue. On the consumer side, the dequeue endpoint gains an optional `queueClass` selector. A supervisor sends `queueClass: "scheduled"` and the server derives the actual queue from the worker's own group, so a token can only ever reach its own region's queues. A fleet picks its class with the `TRIGGER_WORKER_QUEUE_CLASS` env var (`default` or `scheduled`), so a dedicated scheduled fleet can run alongside the standard one. Verified end to end against a local managed-worker setup: scheduled runs route to the dedicated queue, are drained only by the scheduled fleet, and standard runs are left untouched.	2026-06-05 09:41:57 +01:00
Daniel Sutton	4ea3ef138f	chore(webapp,redis-worker): make mollifier constants configurable (#3822) ... ## Summary The mollifier had ~21 behavioural constants baked in as hardcoded values — the buffer's ack-grace TTL and Redis retry/reconnect tuning, the drainer's poll interval and backoff envelope, the pre-gate idempotency claim TTL/wait/poll, the buffered-run mutate-with-fallback wait loop, the metadata CAS retry budget and backoff, the stale-sweep scan bounds, and the draining-gauge interval. None could be adjusted without a code change, which makes tuning the system under production load impossible. This exposes all of them as `TRIGGER_MOLLIFIER_*` environment variables, each defaulting to its previous hardcoded value. Behaviour is identical unless an operator sets a var, so it's a safe no-op deploy. ## Design The package-level classes (`MollifierBuffer`, `MollifierDrainer` in `@trigger.dev/redis-worker`) gain optional constructor options defaulting to the old constants — backward compatible, hence a patch changeset. The webapp factories and worker bootstraps read the env and pass them through. The route- and concern-level pure helpers (mutate-with-fallback, metadata mutation, idempotency claim, stale-sweep state) keep their existing `?? DEFAULT` option fallbacks and are fed env values at their call sites, so they stay unit-testable without importing `env.server`. ## Test plan - [x] `@trigger.dev/redis-worker` builds - [x] webapp typecheck passes - [x] mollifier buffer + drainer testcontainer suites pass (modulo a couple of pre-existing flaky timing tests) - [x] Reviewer: confirm the `TRIGGER_MOLLIFIER_*` env var names match ops conventions --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>	2026-06-04 09:36:34 +00:00
Daniel Sutton	139eede41c	feat(redis-worker): batched pop in MollifierDrainer for fast single-env drains (#3797) ... ## Summary Adds `drainBatchSize` to `MollifierDrainer` (default `1` — preserves existing behaviour) and wires `TRIGGER_MOLLIFIER_DRAIN_BATCH_SIZE` through the webapp (default `50`). Each tick the drainer now pops up to `drainBatchSize` from each chosen env, then dispatches every popped entry through the shared `concurrency`-bounded `pLimit`. Per-org/per-env fairness is unchanged — only the in-env pop count grows. Pre-existing behaviour was one pop per env per tick. For a single-env burst that single-flighted the drain at the per-tick floor of `pop + engine.trigger ≈ 50–60 ms`. With buffer entries piling up under a real-world tenant burst that's tens of minutes of tail latency to fully materialise — even though PG itself could comfortably sustain the writes. ## Why this matters — heavy-tail illustration Scenario: 100 customers in one window — 94 fire 20 triggers each, 5 fire 100, 1 fires 1000. Gate at `THRESHOLD=10/s`, `HOLD_MS=500`. First 10 of each burst hit PG directly; the rest buffer. | Customers | Triggers each | PG direct | Buffered each | Total buffered | |---|---|---|---|---| | 94 small | 20 | 10 | 10 | 940 | | 5 medium | 100 | 10 | 90 | 450 | | 1 heavy | 1000 | 10 | 990 | 990 | **With `DRAIN_BATCH_SIZE=50`, `DRAIN_CONCURRENCY=50`, ~50 ms `engine.trigger`:** | Tick | Pops | Dispatch waves | Wall-clock | |---|---|---|---| | 1 | 94×10 + 5×50 + 1×50 = 1 240 | 25 × 50 ms | ~1 300 ms (94 smalls done) | | 2 | 5×40 + 1×50 = 250 | 5 × 50 ms | ~300 ms (5 mediums done) | | 3–20 | heavy alone, 50/tick | 1 × 50 ms | ~100 ms each | | Customer class | Buffered fully drained | |---|---| | 94 small | **~1.3 s** | | 5 medium | **~1.6 s** | | 1 heavy | **~3.4 s** | **Without batching (one pop per env per tick — current behaviour):** | Customer class | Buffered fully drained | |---|---| | 94 small | ~500 ms | | 5 medium | ~4.5 s | | 1 heavy | **~49 s** | So the heavy single-tenant tail drops from ~49 s to ~3.4 s (~14× faster) without changing PG load characteristics. Smalls go up slightly in this scenario (500 ms → 1.3 s) because all 100 envs share one tick's dispatch queue — that's the trade we accept for the heavy tail; the worst-case small wait is still inside one tick. PG load is identical either way (50 concurrent inserts at a time, capped by `DRAIN_CONCURRENCY`). ## What changed **`packages/redis-worker`** - New `drainBatchSize` option (default 1 — full backward compat). - `runOnce()` refactored to pop per-env batches in parallel, then dispatch all popped entries through the existing global `pLimit`. Mid-batch pop failure aborts only that env's batch and counts as one failure (same semantic as the old per-env path). - Removed the now-unused `processOneFromEnv` helper. **`apps/webapp`** - `TRIGGER_MOLLIFIER_DRAIN_BATCH_SIZE` env var (default 50, matching `DRAIN_CONCURRENCY`). - Wired into `mollifierDrainer.server.ts`. **Test cloud config** (separate cloud PR): `TRIGGER_MOLLIFIER_DRAIN_BATCH_SIZE="50"` on the worker service. Production rollout deferred until we've watched it on test cloud. ## Test plan - [x] All 25 stub-based drainer tests pass (18 pre-existing + 7 new). 7 new tests under `MollifierDrainer.drainBatchSize`: - pops up to `drainBatchSize` across ticks - global `concurrency` cap still holds when batch > concurrency - mid-batch pop failure isolation - multi-env batch fan-out in one tick - **hierarchical org fairness preserved at `drainBatchSize > 1`** (load-bearing — guards against future regressions to per-env-instead-of-per-org rotation) - mixed success/failure accounting in a batched tick - bounded pops on empty queue (no Lua spam past `drainBatchSize`) - [x] All pre-existing tests still pass unchanged at default `drainBatchSize=1` → backward-compat locked. - [x] `pnpm run build --filter @trigger.dev/redis-worker` clean. - [x] `pnpm run typecheck --filter webapp` clean. - [x] `redisTest` block (real Redis via testcontainers) — couldn't run locally on this branch due to testcontainers runtime discovery; will validate in CI. - [ ] Test-cloud smoke after cloud PR lands: fire `burst 50` against a flagged env and confirm the 50th entry's drain time drops from ~2.5 s to <200 ms. ## Notes - Per-tick memory bound: `maxOrgsPerTick × drainBatchSize` entries can sit in the JS pLimit queue between pop and dispatch. At defaults that's `500 × 50 = 25 000` × ~5 KB snapshot ≈ ~125 MB worst case per worker — well within headroom. - The pre-batch model's strict per-env throughput cap of `1/tick` is documented as the fairness baseline elsewhere. Org-level fairness is what callers actually rely on; this change does not weaken that. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-06-02 14:21:37 +01:00
Daniel Sutton	4745754a7a	feat(webapp,run-engine): mollifier drainer replay + stale sweep + cancelled-run engine API (#3754) ... ## Summary The replay side of the mollifier: - `DrainerHandler`: reads buffered snapshots and replays them through `engine.trigger` to materialise PG rows. - `RunEngine.createCancelledRun`: new public method the handler uses to write CANCELED rows directly from snapshots (bypass queue + waitpoint, emit `runCancelled`). Tolerates the cjson empty-table tags edge case found during validation. - Drainer fairness: org → env rotation so a heavy env doesn't starve light ones in the same org. - Stale-entry sweep + telemetry + alertable gauge so a stuck/offline drainer surfaces in alerts. Both the drainer and sweep default-off; nothing fires unless flagged on (`TRIGGER_MOLLIFIER_DRAINER_ENABLED`, `TRIGGER_MOLLIFIER_STALE_SWEEP_ENABLED`). Stacked on the trigger-time decisions PR. ## Test plan - [x] \`pnpm run typecheck --filter webapp\` passes - [x] \`pnpm run test --filter webapp test/mollifierDrainerHandler.test.ts\` passes - [x] \`pnpm run test --filter webapp test/mollifierStaleSweep.test.ts\` passes - [x] \`pnpm run test --filter @internal/run-engine src/engine/tests/createCancelledRun.test.ts\` passes - [x] \`pnpm run test --filter @trigger.dev/redis-worker packages/redis-worker/src/mollifier/drainer.test.ts\` passes --- ## Ship-gate follow-up fix **Drainer writes SYSTEM_FAILURE on max-attempts exhaustion.** Adds an `onTerminalFailure` callback on `MollifierDrainerOptions` so the customer's run lands a SYSTEM_FAILURE PG row even when the drainer exhausts `MAX_ATTEMPTS` on a retryable PG error (previously `buffer.fail()` was called with no row written → silent data loss). The callback runs before `buffer.fail()` on every terminal path (non-retryable AND max-attempts-exhausted), and re-throwing a retryable error from the callback causes the drainer to requeue rather than fail. Bumps `@trigger.dev/redis-worker` to a **minor** changeset (additive option + new exported types). Includes 5 unit tests covering both terminal causes plus the requeue-on-retryable-callback-failure path and no-callback back-compat. --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-06-01 13:20:05 +01:00
Daniel Sutton	3924aa4adb	feat(redis-worker,webapp): mollifier buffer extensions + snapshot type (#3752) ... ## Summary Buffer-side data layer used by the rest of the mollifier phase-3 stack. - `buffer.ts` gains entry inspection (`getEntry`), idempotency lookup (`lookupIdempotency`), in-place snapshot mutation (`mutateSnapshot`), and dwell tracking. All atomic via Lua. - `mollifierSnapshot.server.ts`: shared `MollifierSnapshot` type plus (de)serialise helpers. - Drops the entry-TTL config and its env var. The drainer is the recovery mechanism; an entry that survives the drainer should surface as a stale-sweep alert, not silently TTL away. Adds methods to the buffer interface; nothing consumes them yet. Subsequent PRs in the stack wire trigger-time mollify, read-fallback, and mutation paths against this surface. ## Test plan - [x] \`pnpm run typecheck --filter webapp\` passes - [x] \`pnpm run test --filter @trigger.dev/redis-worker packages/redis-worker/src/mollifier/buffer.test.ts\` passes --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-06-01 11:42:55 +01:00
Eric Allam	37eeaa3690	fix(webapp): skip V1 WAITING_FOR_DEPLOY drain on V2 promotes (#3742) ... ## Summary Stops the legacy V1 `WAITING_FOR_DEPLOY` drain from running on every V2 deployment promotion. The drain queries `TaskRun` by `status='WAITING_FOR_DEPLOY'`, which only V1-engine runs ever have — V2 runs use `PENDING_VERSION` and are handled out of band. Calling the drain on V2 promotes produced empty queries against the status index and unnecessary reader-DB load. ## Fix Two layers: 1. Gate the enqueue at the call site in `ChangeCurrentDeploymentService` so it only fires when the deployment's worker is on engine V1. 2. Add a `LEGACY_RUN_ENGINE_WAITING_FOR_DEPLOY_DISABLED` env var (default `0`). When set to `1`, the service returns immediately from `call()` — neuters any jobs already sitting in the worker queue from before the deploy lands. V1 customers see no change; V2 promotes no longer trigger the drain.	2026-05-25 07:10:13 +01:00
Eric Allam	61ca40b4b1	perf(run-engine,webapp): look up PENDING_VERSION runs via ClickHouse (#3707) ... ## Summary When a background worker registers, the engine resolves runs that were queued before the worker was ready (status `PENDING_VERSION`). That lookup used to scan a Postgres status index on `TaskRun`. Move it to ClickHouse: query candidate run ids from `task_runs_v2`, then refetch the actual rows from Postgres by primary key with a `status = 'PENDING_VERSION'` guard for idempotency. ## Design The lookup is a pluggable interface on the run engine (`PendingVersionRunIdLookup`). The webapp wires a ClickHouse-backed implementation through the org-scoped `clickhouseFactory` using a new `"engine"` client type, configured by `RUN_ENGINE_CLICKHOUSE_*` env vars. The URL falls back to `CLICKHOUSE_URL` when unset, so self-hosted deployments don't need new config to keep working. When the lookup returns no candidates, one bounded retry is scheduled ~5s later to cover ClickHouse replication lag against `task_runs_v2`. The Postgres status guard on both the candidate refetch and the inner `updateMany` prevents double-promotion when a retry races with a concurrent deploy. Tests cover three existing PENDING_VERSION cases via a small Postgres-backed test adapter; new ClickHouse-backed integration tests will follow.	2026-05-22 17:18:41 +01:00
Daniel Sutton	1015876b98	feat(webapp): user-based Sentry attribution with tenant tags (#3678) ... ## Summary Stamp every Sentry event with the signed-in user and the tenant (org / project / env) the request belongs to, so "Users Impacted" counts distinct humans and events become filterable per tenant. **Design after review (current):** - `user.id = real user cuid` (from `requireUser`). "Users Impacted" counts humans, not tenants. - Tenant context (org / project / env slugs, IDs, env type) moves entirely onto tags: `org_slug`, `project_slug`, `env_slug`, `org_id`, `project_id`, `project_ref`, `environment_id`, `env_type`, plus `impersonating` when set. - Backed by an `AsyncLocalStorage` scope established at the HTTP entry. Each entry point fills what it knows; loaders enrich the same scope with what they already have. **Zero new database queries.** The middleware does a regex match only. Dashboard loaders that already query Prisma gain a couple of extra selected columns; nothing new round-trips. ## How it's wired - **Express middleware (`tenantContextResolver.server.ts`)** — parses the URL with a regex and always opens an ALS scope. Populates whatever subset of slugs is present: `/orgs/:o` → just `orgSlug`; `/orgs/:o/projects/:p` adds `projectSlug`; the full triple adds `envSlug`. Non-tenant paths get an empty scope so loaders can still enrich. - **`_app/route.tsx`** — already calls `requireUser`. Adds `tenantContext.enrich({ userId: user.id })` for every authenticated dashboard request. No new query. - **Env layout loader (`_app.orgs.$o.projects.$p.env.$e/route.tsx`)** — its existing `prisma.project.findFirst` gains two columns in `select` (`externalRef`, `organization.id`). After it picks an env, calls `tenantContext.enrich({ orgId, projectId, projectRef, envId, envType })`. Same query, +2 columns. - **API path (`apiBuilder.server.ts`)** — wraps every handler in `tenantContext.run(tenantContextFromAuthEnvironment(authenticationResult.environment), …)`. The mapper pulls `userId` from `env.orgMember?.userId` (already selected by `authIncludeBase` — no schema change). Covers `createLoaderApiRoute`, `createActionApiRoute`, and `createMultiMethodApiRoute`. - **Event processor (`sentryTenantContext.server.ts`)** — registered in `entry.server.tsx` so it lives in the Remix bundle and shares the same `tenantContext` ALS instance as the middleware and loaders. Stamps whatever's present; nothing forced. ## Example events from local verification | URL | `user.id` | Tags | |-----|-----------|------| | `/orgs/:o/projects/:p/env/:e/...` | real user cuid | `org_slug`, `project_slug`, `env_slug`, `org_id`, `project_id`, `project_ref`, `environment_id`, `env_type` | | `/orgs/:o/settings` (non-env-scoped) | real user cuid | `org_slug` only | | API request with `orgMember` | `orgMember.userId` | full tenant set | | API request without `orgMember` | (unset) | full tenant set | ## Trade-offs 1. On env-scoped pages, errors that fire before the env layout loader's enrich callback runs get slugs + `user.id` but not the tenant IDs / `env_type`. Realistic errors deep in async work get the full set. (Same race as before, narrower window now that slugs/`user.id` are populated up-front by the middleware and `_app` enrich.) 2. API requests where the environment has no `orgMember` get tenant tags but no `user.id`. Those events still show in the issue but don't contribute to "Users Impacted". ## Out of scope (deferred) Background workers (`redis-worker`, `schedule-engine`) and socket handlers. Those entry points don't set `tenantContext.run` yet — their events ship without tenant attribution until each is wired in a follow-up. ## Tests 31 unit tests across 4 files. New tests notably cover: - `parseTenantPath`: org-only, org+project, and full-triple URL variants. - `tenantContext.enrich`: in-place patch, no-op outside `run()`, concurrent-scope isolation, empty-scope + enrich pattern (for non-tenant pages). - `tenantContextFromAuthEnvironment`: with and without `orgMember` — verifies the API path's `user.id` mapping. - `addTenantContextToEvent`: empty scope, userId-only, slugs-only, full enrichment, conditional tag emission, preservation of prior `event.user` fields. ## Test plan - [ ] `pnpm run typecheck --filter webapp` - [ ] `pnpm run test --filter webapp -- test/tenantContext.test.ts test/sentryTenantContext.test.ts test/tenantContextResolver.test.ts test/tenantContextFromAuthEnvironment.test.ts` - [ ] Local manual: with `SENTRY_DSN` set, hit a dashboard URL and an API route, confirm the captured events carry `user.id` + the expected tag set in Sentry. - [ ] After ship: confirm "Users Impacted" on a real Sentry issue reflects distinct users (not tenants). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-05-22 15:10:41 +01:00
Matt Aitken	71d98b4e6b	Support for org-scoped ClickHouse (#3333) ... Added `OrganizationDataStore` which allows orgs to have data stored in specific separate services. For now this is just used for ClickHouse. When using ClickHouse we get a client for the factory and pass in the org id. Particular care has to be made with two hot-insert paths: 1. RunReplicationService 2. OTLPExporter --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Claude <noreply@anthropic.com>	2026-05-22 14:43:03 +01:00
Daniel Sutton	906d5fafb6	feat(mollifier): trigger burst smoothing — Phase 1 (monitoring) (#3614) ... ## Summary - Introduce the Mollifier: a Redis-backed buffer for `trigger()` API calls during traffic spikes, with a per-env trip evaluator and a drainer ack-loop. - Phase 1 is dual-write monitoring — every mollified trigger is buffered to Redis AND continues to `engine.trigger`. No customer-facing behaviour change. - Telemetry events: `mollifier.would_mollify`, `mollifier.buffered`, `mollifier.drained`, plus the `mollifier.decisions` counter. - Gated behind a feature flag (default off). ## Test plan - [x] `pnpm run test --filter @trigger.dev/redis-worker` - [x] `pnpm run test --filter webapp -- mollifier` - [x] Manual: with flag off, no behaviour change vs main - [x] Manual: with flag on + threshold lowered, observe `mollifier.buffered` + `mollifier.drained` log pairs with matching `runId` --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-18 14:58:38 +01:00
Eric Allam	454f0c949a	perf(webapp): cache task metadata in Redis for the trigger hotpath (#3625) ... ## Summary The trigger-task hotpath used to early-return without a DB query when a caller passed both a queue override and a per-trigger TTL — the hottest configuration on the trigger API. Adding `triggerSource` to the resolver so the runs-list "Source" filter could distinguish STANDARD / SCHEDULED / AGENT runs removed those early-returns, costing +2 DB queries per trigger on non-locked calls and +1 on locked calls. This change caches `BackgroundWorkerTask` metadata (`ttl`, `triggerSource`, `queueId`, `queueName`) in Redis so the resolver can satisfy every caller configuration with a single `HGET` on the warm path. PG fallback on miss back-fills the cache. Follow-up to #3542. ## Design Two key spaces: - `task-meta:env:{envId}` — the "current worker" view, refreshed at every deploy promotion. 24h safety TTL. - `task-meta:by-worker:{workerId}` — used for `lockToVersion` triggers. Immutable post-create. 30d sliding TTL so historical workers age out. Cache writes use Lua scripts via `defineCommand` so `DEL` + `HSET` + `EXPIRE` land atomically — concurrent readers never see the empty intermediate state of a naive pipeline. Read-path back-fill uses single-field upserts so concurrent back-fills don't wipe each other's siblings. The cache lives behind its own `TASK_META_CACHE_REDIS_*` env-var prefix that falls back to the default `REDIS_*` set, so operators can route the cache to a dedicated Redis instance if they want. The service/instance file split (`taskMetadataCache.server.ts` for the pure class, `taskMetadataCacheInstance.server.ts` for the env-wired singleton) mirrors the existing `runsReplicationService` / `runsReplicationInstance` pattern. ## Test plan - [ ] `pnpm run typecheck --filter webapp` - [ ] `pnpm run test ./test/engine/triggerTask.test.ts --run` — 8 existing tests untouched + 5 new tests covering warm cache, cold miss with back-fill, queue + ttl path, by-worker vs env keyspace, and the promotion cache write - [ ] End-to-end against a dev worker: registering writes both keyspaces with the expected TTLs, and `redis-cli HGETALL "tr:task-meta:env:<envId>"` returns the cached entries ## Benchmark Measured `DefaultQueueManager.resolveQueueProperties` against a real Postgres + Redis (vitest `containerTest`, single-host docker). 500 sequential calls and 2,000 parallel calls (concurrency=50) per scenario, request shaped as `{ taskId, queue: "bench-queue", ttl: "5m" }` — the hot path this PR restores. ``` sequential (one in flight at a time): [noop cache (baseline)] n=500 mean=1.423ms p50=1.394ms p95=1.735ms p99=2.629ms max=11.100ms [redis cache, cold ] n=500 mean=1.346ms p50=1.283ms p95=1.688ms p99=2.463ms max=5.058ms [redis cache, warm ] n=500 mean=0.084ms p50=0.078ms p95=0.105ms p99=0.156ms max=1.129ms speedup (warm vs baseline, sequential): 16.95x parallel (concurrency=50): [noop cache (baseline)] n=2000 mean=10.069ms p50=8.850ms p95=14.718ms p99=31.887ms total=405ms ops/s=4,940 [redis cache, warm ] n=2000 mean=0.614ms p50=0.568ms p95=1.189ms p99=1.432ms total=25ms ops/s=80,389 throughput speedup (warm vs baseline, parallel): 16.27x ``` Read: - **Warm cache cuts resolver latency 17×** at p50 — from ~1.4 ms to ~78 µs per call. - **Cold cache is on par with baseline** — the extra `HGET` miss adds <50 µs against the two Postgres queries that follow, so the worst case is not worse than today. - **Under burst load (50 concurrent triggers)**, the baseline's p99 jumps to ~32 ms as Postgres connections queue up; warm stays at ~1.4 ms. The cache moves the saturation point from ~5k ops/s (PG pool) to ~80k ops/s (single-client Redis pipelining). Caveats: single-host docker, local Postgres + Redis, resolver-only measurement (excludes the rest of the trigger transaction). Prod adds region-local Redis RTT (~0.3–0.8 ms) which shifts warm absolute numbers up but keeps the ratio intact.	2026-05-15 11:52:53 +01:00
Matt Aitken	e4981d1b11	feat(webapp): consolidate auth path + add comprehensive auth tests (#3499) ... ## Summary Consolidates the webapp's authentication and authorization into a small set of route helpers, replacing the ad-hoc `requireUser` / `requireUserId` / `authenticatedEnvironmentForAuthentication` calls scattered across routes. Same security model, but the per-request flow (authenticate → authorize → load) now lives in one place per route family. Introduces a plugin seam (`@trigger.dev/plugins`) that lets the cloud build install a richer RBAC implementation without touching webapp code. The OSS fallback keeps the pre-RBAC permissive behaviour intact, so self-hosted deployments work unchanged. Adds a comprehensive end-to-end auth test suite that didn't exist before — 193 `it()` blocks (vitest reports ~199 after `it.each` expansion) covering API key, PAT and JWT auth across the public API surface, plus dashboard session auth for admin pages. ## Changes ### Plugin contract — `@trigger.dev/plugins` `RoleBaseAccessController` interface authoritative for both OSS (fallback) and cloud (enterprise plugin): - `authenticateBearer(request, { allowJWT? })` — API-key / public-JWT auth, returns env + ability - `authenticateSession(request, { userId, organizationId?, projectId? })` — dashboard auth, caller resolves `userId` from the session cookie and passes it in (no `helpers.getSessionUserId` callback — decouples the plugin host from session-cookie code) - `authenticatePat(request, { organizationId?, projectId? })` — PAT auth, returns identity + `lastAccessedAt` so the host can throttle the per-request update - `authenticateAuthorize*` variants for the auth-and-check-in-one-call cases - `isUsingPlugin(): Promise<boolean>` — capability flag for UI / branching where plugin-present-ness matters; replaces the sentinel-string coupling that had `personalAccessToken.server` matching `"RBAC plugin not installed"` literally ### Dashboard auth (started, partial rollout) Admin and settings pages migrated to a unified `dashboardLoader` / `dashboardAction` helper that authenticates the session, runs an authorization check, and exposes the result to the route. Other dashboard routes still on the old pattern; remaining migration tracked in TRI-8730. Migrated routes: - `admin.*` (14 admin / back-office / feature-flags / LLM-models / notifications / orgs / concurrency pages) - `_app.orgs.$organizationSlug.settings.team` - `_app.orgs.$organizationSlug.settings.roles` ### API / realtime / engine auth (complete for the migrated families) 71 routes migrated to a unified `apiBuilder` that centralizes Bearer / PAT / Public-JWT authentication and applies the per-route authorization check before the handler runs. Includes: - `api.v1.*` and `api.v2.*` and `api.v3.*` — tasks, runs, batches, queues, prompts, deployments, query, sessions, waitpoints, packets, workers, idempotency keys - `realtime.v1.*` — runs, batches, sessions, streams - `engine.v1.*` — dev / worker-action protocols 29 routes still on the legacy `authenticateApiRequest*` helpers — tracked as a post-deploy follow-up in TRI-9228. Multi-resource auth direction is now explicit at the call site via `anyResource(...)` (OR) and `everyResource(...)` (AND). Bare arrays no longer typecheck — fixes a class of bug where a JWT scoped to one resource could implicitly access others under OR semantics. PAT auth path consolidated: was three DB queries per request (legacy `authenticateApiRequestWithPersonalAccessToken` findFirst + `rbac.authenticatePat` join + `lastAccessedAt` update). Now one query in the steady state — plugin returns `lastAccessedAt`, host smart-skips the update via JS-side throttle when fresh. Side effect: action aliases preserved historic JWT scope semantics where the new model is stricter (e.g. a `write:tasks` JWT now also satisfies `trigger` / `batchTrigger` / `update` actions on the same resource — matched at the auth boundary, not in the route handler). ### Backwards-compat fixes The strict-match model regressed several real-world JWT shapes. Each preserved via explicit `anyResource(...)` entries in the route's authz block: - **Batch retrieve routes** (`api.v1.batches.$batchId`, `api.v2.*`, `realtime.v1.batches.*`) accept `read:runs` JWTs again (pre-RBAC literal-match superScope behaviour) - **Runs list routes** (`api.v1.runs`, `realtime.v1.runs`) accept type-level `read:tasks` / `read:tags` on unfiltered queries (matched the legacy `Object.keys` iteration semantic) - **PAT/OAT auth shape** normalized through `toAuthenticated` so all auth methods return the same slim `AuthenticatedEnvironment` (was: API-key returned the slim shape but PAT/OAT returned raw Prisma `Decimal` / no `orgMember`) - **Scope `:` preservation** in resource ids — `read:tags:env:staging` now correctly identifies the tag id as `env:staging`, not `env` ### Slim `AuthenticatedEnvironment` Extracted to `@trigger.dev/core/v3/auth/environment` — a structural shape independent of `@trigger.dev/database`. The plugin contract returns this; webapp consumers import from there; the cloud plugin (Drizzle) returns the same shape without Prisma's `Decimal` class leaking into the public surface. Lets internal-packages (run-engine, etc.) refer to `AuthenticatedEnvironment` without pulling Prisma in. ### Auth test suite (new — `*.e2e.full.test.ts`) 193 e2e tests run against a real spawned webapp + Postgres (no mocks). Coverage matrix: - **API key auth** — read / write / trigger / batchTrigger / deploy actions across runs, batches, deployments, prompts, queues, query, sessions, input-streams, waitpoints, tasks, idempotency keys; multi-key resources (a run carries batch / tag / task identifiers — auth must accept any matching scope) - **Personal Access Token auth** — comprehensive matrix: scope match, scope mismatch, missing scope, expired token, malformed token - **Public JWT auth** — sub-vs-URL environment resolution, expired JWTs, signature verification, scope checking, otu (one-time-use) token semantics, branch-environment signing-key fallback - **Dashboard session auth** — admin-only pages reject non-admins; per-action gating - **Cross-cutting edge cases** — revoked API key grace window, JWT cross-environment isolation, MissingResource branch behaviour ### Hygiene cleanups - Deleted dead `app/services/authorization.server.ts` (legacy `checkAuthorization` + types — no live consumers post-migration) and its orphaned test - Dropped the never-populated `scopes` field from `ApiAuthenticationResultSuccess` - `scheduleEmail` moved out of `email.server.ts` into its own module — breaks a `commonWorker → marqs/V1` import chain that was poisoning the auth test graph - OSS Roles page shows a deployment-aware empty state ("Roles aren't available in this self-hosted deployment" vs the plan-upsell copy) via `rbac.isUsingPlugin()` - Team action handler: explicit per-intent ability gates (`manage:billing` for purchase-seats, `manage:members` for set-role + remove-member with self-leave carve-out) ### Cross-repo coordination All public-package contract changes paired in `triggerdotdev/cloud#763` (rbac-packages branch) — the enterprise plugin implements the same `RoleBaseAccessController` interface against Drizzle. ## Test plan - [x] `pnpm run typecheck --filter webapp` clean - [x] `pnpm --filter webapp exec vitest run --config vitest.e2e.full.config.ts` — 193/193 pass (requires Docker for testcontainers) - [x] Spot-check an authed API endpoint with a valid + invalid API key against a local stack - [x] Spot-check the migrated admin pages render and gate non-admins --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-12 17:16:20 +01:00
Eric Allam	ead1e5a53d	feat(webapp): reload LLM pricing registry on Redis pub/sub (#3534) ... ## Summary Adds a Redis pub/sub reload path to the webapp's in-memory LLM pricing registry. When enabled on a process, the registry reloads from the database whenever a publish lands on the configured channel — instead of waiting for the existing 5-minute interval. Lets pricing/model changes propagate to cost enrichment within seconds. Subscription is **off by default** and opt-in per process. Only OTel-ingesting services need real-time freshness; dashboard and worker services run fine on the periodic interval and shouldn't pile onto each publish with a full-table reload. ## Design When `LLM_PRICING_RELOAD_PUBSUB_ENABLED=true`, subscribes via `createRedisClient` against `COMMON_WORKER_REDIS_*` and listens on `LLM_PRICING_RELOAD_CHANNEL` (default `llm-registry:reload`). The 5-minute periodic reload stays as a backstop, and a SIGTERM/SIGINT handler closes the subscription cleanly. The publisher side lives outside this PR — any process running in the same Redis namespace can trigger a reload by `PUBLISH llm-registry:reload <anything>`. Includes a `.server-changes/` note for the changelog. ### Debounced reload Bursts of publishes are coalesced. The first publish schedules a reload at T+`LLM_PRICING_RELOAD_DEBOUNCE_MS` (default 1s); subsequent publishes during that window are no-ops because the trailing reload picks up everything when it queries the DB. Bounds reload rate to at most 1 per debounce window regardless of publisher chattiness, so a runaway upstream publisher can't fan out into a flood of full-table-scan reloads. ## Test plan - [ ] With `LLM_PRICING_RELOAD_PUBSUB_ENABLED=false` (default): `redis-cli PUBSUB NUMSUB llm-registry:reload` returns `0` while the webapp is up - [ ] With it set to `true`: returns `>= 1` - [ ] `redis-cli PUBLISH llm-registry:reload test` returns `1` (one subscriber received) on a subscribed process - [ ] Mutate an `LlmModel` row externally, publish on the channel, observe the registry's match() picks up the change without waiting for the 5-min tick - [ ] Publish 100x in rapid succession; confirm only one reload fires within the debounce window	2026-05-09 08:59:40 +01:00
Iss	f8ddb766fa	feat: Plain customer cards (#2933)	2026-05-08 11:51:57 -04:00
Eric Allam	386b4f65ff	feat(webapp): per-org S2 basin migration (#3516) ... ## Summary Move from a single shared S2 basin to **per-org basins** with retention tied to the org's billing plan. Stops S2 from deleting streams out from under live chat sessions when basin retention fires before the chat ends, and unlocks per-org cost attribution. OSS / s2-lite installs are unaffected: provisioning is gated by `REALTIME_STREAMS_PER_ORG_BASINS_ENABLED` (default `false`), and the read precedence falls back to the global basin env var when an entity has no stamped basin. ``` basin = run.streamBasinName ?? session.streamBasinName ?? env.REALTIME_STREAMS_S2_BASIN ``` ## Design Three nullable `streamBasinName` columns (`Organization`, `TaskRun`, `Session`) plus a provisioner that idempotently creates the basin and reconfigures retention on plan changes. The trigger and session-create paths stamp the org's basin onto new rows; the realtime read path picks the basin from the entity context. Admin routes back-fill existing orgs and force-reconfigure a single org. ## Test plan - [x] `pnpm run typecheck --filter webapp --filter @internal/run-engine` - [x] Backfill admin route end-to-end (provision + DB stamp + S2 basin config). - [x] Reconfigure on plan change (all retention tiers). - [x] chat.agent multi-turn drives streams into the per-org basin. - [x] Legacy fallback when entity has no stamped basin. - [x] Provisioner is a no-op when the flag is off.	2026-05-05 10:06:58 +01:00
nicktrn	ee3887a321	feat(webapp): configurable deploy template machine presets (#3492) ... The webapp's compute template creation hardcoded a single machine preset (`small-1x`) at deploy time, regardless of which presets a project actually uses. Tasks running on any other preset paid full cold-snapshot creation cost on first run. Two new env vars: - `COMPUTE_TEMPLATE_MACHINE_PRESETS` - CSV of preset names to build boot snapshots for during deploy. Defaults to `small-1x` so existing deploys don't change behavior. - `COMPUTE_TEMPLATE_MACHINE_PRESETS_REQUIRED` - CSV of presets whose failure fails a required-mode deploy. Defaults to the full `PRESETS` list. Optional preset failures are logged but don't block the deploy. The compute client now sends the multi-config request shape; the service evaluates per-preset outcomes against the required set and surfaces a combined failure message when a required preset fails. Both env vars are validated at boot via the env schema - unknown preset names or `_REQUIRED` entries that aren't a subset of `_PRESETS` fail loudly at startup rather than silently per-deploy.	2026-05-01 15:10:26 +01:00
ThullyoCunha	f1736595cd	feat(webapp): apply default repository policy on ECR repo creation (#3467) ... ## Summary Self-hosters that operate the webapp's ECR account separately from the account running the EKS workers (e.g., a shared platform account that hosts the registry plus per-team accounts that host clusters) currently hit a 403 Forbidden the first time **any** project is deployed: ``` Failed to pull image "<acct-A>.dkr.ecr.<region>.amazonaws.com/<namespace>/proj_…:…": unexpected status from HEAD request to .../v2/.../manifests/sha256:…: 403 Forbidden ``` `ensureEcrRepositoryExists` in `apps/webapp/app/v3/getDeploymentImageRef.server.ts` calls `CreateRepository` and `PutLifecyclePolicy`, but never `SetRepositoryPolicy` — so the new repo inherits the AWS default (only the registry-owner account can read/pull). Workers in the cluster account get 403 every single deploy. The only workarounds today are running a one-off post-create script or pre-creating every repo by hand. ## Proposed change Add an optional env var: ``` DEPLOY_REGISTRY_ECR_DEFAULT_REPOSITORY_POLICY (V4 mirror: V4_DEPLOY_REGISTRY_ECR_DEFAULT_REPOSITORY_POLICY) ``` Raw IAM policy JSON. When set, the webapp calls `SetRepositoryPolicy` immediately after `CreateRepository` so every new repo carries that policy from creation. Operators control the principal/actions; we don't bake in any opinions about cross-account boundaries. Example value (for the typical self-host case — grant pull to the cluster account): ```json { "Version": "2012-10-17", "Statement": [{ "Sid": "AllowClusterAccountPull", "Effect": "Allow", "Principal": {"AWS": "arn:aws:iam::<cluster-account-id>:root"}, "Action": [ "ecr:GetDownloadUrlForLayer", "ecr:BatchGetImage", "ecr:BatchCheckLayerAvailability" ] }] } ``` ## Why env var (not a chart-level field) - Mirrors the shape of the sibling vars (`DEPLOY_REGISTRY_ECR_TAGS`, `DEPLOY_REGISTRY_ECR_ASSUME_ROLE_ARN`, etc.) which are already operator-supplied via `webapp.extraEnvVars` in self-host setups. - Cloud is unaffected — the env var is optional, unset by default; existing behavior unchanged. - Existing repos are unaffected — only newly-created repos get the policy. - `RepositoryCreationTemplate` from the AWS provider isn't an alternative here: it only applies to repos created via pull-through-cache or replication, not to `ecr:CreateRepository` API calls. ## Implementation - `apps/webapp/app/env.server.ts` — declare `DEPLOY_REGISTRY_ECR_DEFAULT_REPOSITORY_POLICY` and the V4 fallback. - `apps/webapp/app/v3/registryConfig.server.ts` — propagate `ecrDefaultRepositoryPolicy` to `RegistryConfig`. - `apps/webapp/app/v3/getDeploymentImageRef.server.ts` — `createEcrRepository` accepts the policy; if set, calls `SetRepositoryPolicy` after `PutLifecyclePolicy`. - `docs/self-hosting/env/webapp.mdx` — documentation row added under **Deploy & Registry**. ## Verification Verified end-to-end against a self-hosted Trigger.dev on EKS where the ECR account is separate from the cluster account: - **Without the env var** (current `main`): the new project's first run pod stays in `ImagePullBackOff` with `403 Forbidden`. - **With the env var set** to a JSON granting `ecr:BatchGetImage`/`GetDownloadUrlForLayer`/`BatchCheckLayerAvailability` to the cluster account: a fresh `trigger.dev deploy --env prod` followed by a `hello-world` run completes in ~5s end-to-end on the first try. Manually also confirmed that existing repos are untouched (the call only fires inside `createEcrRepository`, which only runs when `DescribeRepositories` returned `RepositoryNotFoundException`). ## Out of scope - Chart values surface for this — operators already pass the existing ECR vars via `webapp.extraEnvVars`, so this follows the same pattern. Happy to add a first-class chart field in a follow-up if that's the preferred direction. - IAM-policy validation in the webapp — we forward the JSON verbatim to AWS and surface AWS's error messages on misuse, matching how `DEPLOY_REGISTRY_ECR_TAGS` is handled today. This is a draft pending CI / CodeRabbit pass — happy to iterate on direction (e.g., split into per-action env vars, or extend the chart values schema) if any of the above choices feels off. --------- Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>	2026-04-29 15:17:23 +01:00
Eric Allam	c69e939c34	feat: Sessions - bidirectional durable agent streams (#3417) ... > ⚠️ **Not released yet.** This PR is the server-side foundation only. The SDK changes that customers will actually use (`chat.agent` migration, `chat.createStartSessionAction`, `useTriggerChatTransport` updates) live on a separate branch and ship together in an upcoming `@trigger.dev/sdk` prerelease. Until that prerelease is published, this surface is reachable only via direct HTTP. ## What this gives Trigger.dev users A new first-class primitive, **Session**, for durable, task-bound, bidirectional I/O that outlives any single run. Sessions are the run manager for `chat.agent` going forward, and they unblock anything else that needs "one identifier, many runs over time" with a stable channel pair the client can write to and subscribe to. ### Use cases unblocked - **Chat agents that persist across many runs.** One session per chat (keyed on your own `chatId` via `externalId`), turns 1..N attach to the same Session, the UI subscribes once and keeps receiving output as new runs take over. - **Approval loops and long-running tasks with user feedback.** The task waits on `.in`, the client writes to `.in`, the server enforces no-writes-after-close. - **Workflow progress streams that live past the run.** Subscribe to `.out` after the task finishes to replay history. - **Resume-next-day flows.** A session is a durable row, not a transient stream. Send a message a day later and the server triggers a fresh run on the same session. ### How it works (Session-as-run-manager) A Session row is task-bound (`taskIdentifier` + `triggerConfig` are required) and owns its current run via `currentRunId` + `currentRunVersion` for optimistic claim. Three trigger paths: 1. **Session create** — `POST /api/v1/sessions` creates the row and triggers the first run synchronously. 2. **Append-time probe** — `POST /realtime/v1/sessions/:session/in/append` checks if the current run is alive; if it has terminated (idle exit, crash, etc.), the server triggers a new run before processing the append. 3. **End-and-continue handoff** — `POST /api/v1/sessions/:session/end-and-continue`, called by the running agent, triggers a fresh run and atomically swaps `currentRunId`. Used by `chat.requestUpgrade()` for version handoffs. Every triggered run is recorded in the `SessionRun` audit table with a reason (`initial`, `continuation`, `upgrade`, `manual`). ## Public API surface ### Control plane - `POST /api/v1/sessions` — create. Idempotent on `(env, externalId)`. Triggers the first run, returns the session and a session-scoped public access token. Returns 409 if the upserted row is already closed. - `GET /api/v1/sessions/:session` — retrieve by friendlyId (`session_abc...`) or by your own externalId (server disambiguates by prefix). - `GET /api/v1/sessions` — list with filters (`type`, `tag`, `taskIdentifier`, `externalId`, derived `status` ACTIVE/CLOSED/EXPIRED, created-at range) and cursor pagination. Backed by ClickHouse. - `PATCH /api/v1/sessions/:session` — update tags / metadata / externalId. - `POST /api/v1/sessions/:session/close` — terminate. Idempotent, hard-blocks new server-brokered writes. - `POST /api/v1/sessions/:session/end-and-continue` — agent-only handoff to a fresh run. ### Realtime - `PUT /realtime/v1/sessions/:session/:io` — initialize a channel. Returns S2 credentials in headers so high-throughput clients can write direct to S2. - `GET /realtime/v1/sessions/:session/:io` — SSE subscribe. Supports Last-Event-ID resume and an opt-in `X-Peek-Settled: 1` header that fast-closes the stream when the upstream is already settled (`trigger:turn-complete`), eliminating long-poll wait on reconnect-on-reload paths. - `POST /realtime/v1/sessions/:session/:io/append` — server-side appends. - `POST /api/v1/runs/:runFriendlyId/session-streams/wait` — runs wait on a session stream as a waitpoint, with a race-check to avoid suspending if data already landed. ### Auth scopes `sessions` is a new resource type. `read:sessions:{id}`, `write:sessions:{id}`, `admin:sessions:{id}` flow through the existing JWT validator. Session-scoped public access tokens minted by the server replace browser-held trigger-task tokens for chat-style flows — the browser never sees a run identifier or a run-scoped token in steady state. ## What's coming after this PR - **SDK + chat.agent migration**: separate branch, separate PR, ships in the next `@trigger.dev/sdk` prerelease alongside this server deploy. Customers using the prerelease `chat.agent` will follow the [upgrade guide](https://github.com/triggerdotdev/trigger.dev/blob/docs/tri-7532-ai-sdk-chat-transport-and-chat-task-system/docs/ai-chat/upgrade-guide.mdx). - **Dashboard surfaces**: dedicated agent list, agent playground, agent view on the run dashboard. Tracking separately. ## Implementation notes - **Postgres `Session` table**: scalar scoping columns (`projectId`, `runtimeEnvironmentId`, `environmentType`, `organizationId`) without FKs, matching the January TaskRun FK-removal decision. Point-lookup indexes only — list queries go to ClickHouse. Terminal markers (`closedAt`, `expiresAt`) are write-once. - **ClickHouse `sessions_v1`**: ReplacingMergeTree, partitioned by month, ordered by `(org_id, project_id, environment_id, created_at, session_id)`. Tags indexed via `tokenbf_v1` skip index. - **`SessionsReplicationService`**: mirrors `RunsReplicationService` exactly — leader-locked logical replication consumer, `ConcurrentFlushScheduler`, retry with exponential backoff + jitter, identical metric shape. Dedicated slot + publication so the two consume independently. - **S2 keys**: `sessions/{addressingKey}/{out|in}`. The existing `runs/{runId}/{streamId}` key format for run-scoped streams is untouched. - **Optimistic claim**: `ensureRunForSession` triggers a run upfront (cheap to cancel if it loses the race), then attempts an `updateMany` keyed on `currentRunVersion`. Loser cancels its triggered run and reuses the winner's. No DB lock held across the trigger. ### What did NOT change Run-scoped `streams.pipe` / `streams.input` and the existing `/realtime/v1/streams/{runId}/...` routes are unchanged. Sessions are net-new — not a reshaping of the current streams API. ## Deploy notes - Set `SESSION_REPLICATION_CLICKHOUSE_URL` and `SESSION_REPLICATION_ENABLED=1` to enable the replication consumer. - The `Session` table needs `REPLICA IDENTITY FULL` set on the prod source DB before the publication is created (same one-time DDL we did for `TaskRun`). Required for delete events to carry full column values. - Cross-form authorization on the `GET /api/v1/sessions/:session` loader (a JWT minted for either form authorizes both URL forms). Action routes are URL-form-specific, matching how the SDK mints PATs. ## Verification - Webapp typecheck clean (10/10). - `apps/webapp/test/sessionsReplicationService.test.ts` — round-trip tests for insert/update/delete through Postgres logical replication into ClickHouse via testcontainers. - Live end-to-end against local dev: create + retrieve (both forms) + update + close, `.out.initialize` + `.out.append` x2 + `.in.send` + `.out.subscribe` over SSE, list with all filter combinations + pagination, `end-and-continue` swap, `X-Peek-Settled` fast-close (verified in browser via reconnect-on-reload and via curl). Replicated row lands in ClickHouse within ~1s. - Multi-round Devin + CodeRabbit review feedback addressed (read-after-write paths use `prisma` writer, info-leak on auth-routes masked as 403, peek-settled discriminator parsing fix, etc.). ## Test plan - [ ] `pnpm run typecheck --filter webapp` - [ ] `pnpm run test --filter webapp ./test/sessionsReplicationService.test.ts --run` - [ ] Start the webapp with `SESSION_REPLICATION_CLICKHOUSE_URL` and `SESSION_REPLICATION_ENABLED=1`. Confirm the slot and publication auto-create on boot. - [ ] `POST /api/v1/sessions` and verify the row replicates to `trigger_dev.sessions_v1` within a couple of seconds. - [ ] `POST /api/v1/sessions/:id/close`, then confirm `POST /realtime/v1/sessions/:id/out/append` returns 400. - [ ] Reuse a closed session's `externalId` on `POST /api/v1/sessions` and confirm 409. - [ ] `GET /realtime/v1/sessions/:id/out` with `X-Peek-Settled: 1` after a turn completes and confirm `X-Session-Settled: true` response header + immediate close.	2026-04-28 12:35:55 +01:00
Eric Allam	e134da7306	fix(run-engine): debounce hot-key lock contention and 5xx feedback loop (#3453) ... ## Changes Three changes in `internal-packages/run-engine/src/engine/systems/debounceSystem.ts`, in order of impact: 1. **Fast-path skip before the lock.** In `handleExistingRun`, do an unlocked read of `delayUntil` (and `createdAt` for the max-duration check) from the run row before entering `runLock.lock("handleDebounce", ...)`. If `newDelayUntil <= currentDelayUntil` and the run is still within its max-duration window, return the existing run immediately without taking the lock. Safe because debounce is monotonic-forward only — a stale read either matches reality or undershoots, both of which decay correctly (re-checked properly inside the lock by whichever caller is actually pushing forward). Trailing-mode triggers carrying `updateData` still take the lock so the data update is applied. 2. **Quantize `newDelayUntil`.** Round the computed `newDelayUntil` to 1-second buckets (configurable via `quantizeNewDelayUntilMs`, set to 0 to disable). Without quantization, every call has a slightly larger `newDelayUntil` than the last and they all pass the fast-path check. With it, concurrent callers on the same key share a target time and ~95% short-circuit. User-visible effect: a debounced run might fire up to 1s earlier than the strict spec — non-issue for typical debounce use cases (chat summarization, batched notifications, etc.). 3. **Graceful lock-contention fallback.** Wrap the `runLock.lock(...)` call so `LockAcquisitionTimeoutError` and Redlock `ExecutionError` / `ResourceLockedError` return the existing run id with success instead of propagating a 5xx. Debounce is best-effort: if we can't take the lock, the herd is already updating it for us; fall in line. This kills the 5xx → SDK-retry feedback loop. With (1)+(2) this rarely fires; without them it's the difference between 5xx and 200. Defaults preserve current behaviour aside from quantization (1s) and fast-path (on). Both are configurable via `RunEngineOptions.debounce`. ## ✅ Checklist - [x] I have followed every step in the [contributing guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md) - [x] The PR title follows the convention. - [x] I ran and tested the code works --- ## Changelog Reduce 5xx feedback loops on hot debounce keys by quantizing `delayUntil`, adding an unlocked fast-path skip before the redlock, and gracefully handling redlock contention in `handleDebounce` so the SDK no longer retries into a herd. --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-04-28 11:22:00 +01:00
Eric Allam	486f49791d	fix(webapp): eliminate SSE abort-signal memory leak (#3430) ... ## Summary Fixes a server-side memory leak in the webapp's SSE helper. Every aborted SSE connection (client tab close, navigation, timeout) was pinning its full request/response graph indefinitely on Node 20, so any long-running webapp process accumulated retained memory proportional to streaming-request churn. ## Root cause `apps/webapp/app/utils/sse.ts` combined four abort signals via `AbortSignal.any([requestAbortSignal, timeoutSignal, internalController.signal])`. The composite signal tracks its source signals in an internal `Set<WeakRef>` registered against a `FinalizationRegistry`; under sustained traffic those entries accumulate faster than they're cleaned up, pinning every source signal (and its listeners, and anything those listeners close over) until the parent signal itself is GC'd or aborts. This is a long-standing Node issue with multiple open reports: - [nodejs/node#54614](https://github.com/nodejs/node/issues/54614) — original report, still open. A [follow-up from ChainSafe](https://github.com/nodejs/node/issues/54614#issuecomment-4055656572) describes the exact same shape in a Lodestar production workload (req + timeout signals composed per request accumulating in long-running worker) and the same mitigation: drop `AbortSignal.any`, compose manually. - [nodejs/node#55351](https://github.com/nodejs/node/issues/55351) — mechanism confirmed by Node member @jasnell: *"the set of dependent signals known to the AbortSignal are kept in an internal Set using WeakRefs. The AbortSignals are being properly gc'd but the Set is never cleaned out of the WeakRefs making those leak."* Partially fixed by [PR #55354](https://github.com/nodejs/node/pull/55354), shipped in Node 22.12.0 — but only covers the tight-loop case, not long-lived parent signals. - [nodejs/node#57584](https://github.com/nodejs/node/issues/57584) — circular-dependency variant, still open. - [nodejs/node#62363](https://github.com/nodejs/node/issues/62363) — regression in Node 24/25 from an unrelated V8 change ("Don't pretenure WeakCells"). Different root cause, same symptom. A separate issue in `apps/webapp/app/entry.server.tsx` — `setTimeout(abort, ABORT_DELAY)` with no `clearTimeout` on success paths — kept the React render tree + `remixContext` alive for 30s per successful HTML request. Same pattern fixed upstream in React Router templates ([react-router#14200](https://github.com/remix-run/react-router/pull/14200)), never backported to Remix v2. ## What changed - **`apps/webapp/app/utils/sse.ts`** — single-signal abort chain. `AbortSignal.any` removed; `AbortSignal.timeout` replaced by a plain `setTimeout` cleared when the controller aborts; named sentinel constants used as stackless abort reasons; request-abort handler explicitly removed on cleanup. - **`apps/webapp/app/entry.server.tsx`** — clears the `setTimeout(abort, ABORT_DELAY)` timer in `onShellReady` / `onAllReady` / `onShellError`. - **`apps/webapp/app/v3/tracer.server.ts` + `env.server.ts`** — gates OpenTelemetry `HttpInstrumentation` and `ExpressInstrumentation` behind `DISABLE_HTTP_INSTRUMENTATION=true` as an escape hatch for future OTel-listener retention patterns. Defaults to enabled. - **`apps/webapp/app/presenters/v3/RunStreamPresenter.server.ts`** — uses the shared `ABORT_REASON_SEND_ERROR` sentinel. ## Verification ### Full-app reproduction (memlab) Isolated local harness, 500 abrupt SSE disconnects against a dev-presence route, GC between passes, heap snapshot diff with [memlab](https://facebook.github.io/memlab/): | Run | Heap delta after 500 conns + GC | memlab retained leaks | | --- | --- | --- | | Before | +16.0 MB (linear with request count) | 158 clusters; 250 `ServerResponse`, 1000 `AbortController`, 250 `SpanImpl` retained | | After | **+3.3 MB (noise)** | **0 app-code leaks** | ### Standalone mechanism isolation To confirm *which* axis of the change is load-bearing, a separate standalone Node script (`/tmp/abort-leak-test.mjs`) ran 2000 requests × 200 KB payload per variant: | Variant | Heap delta after GC | | --- | --- | | baseline (no signal machinery) | 0 MB | | V1: `AbortSignal.any` + string abort reason | **+9.1 MB** | | V2: `AbortSignal.any` only (no reason) | **+10.8 MB** | | V3: string reason only (no `AbortSignal.any`) | 0 MB | | V4: neither (the fix) | 0 MB | | V5: `AbortSignal.any` with no listener on the composite | **+10.2 MB** | This proves `AbortSignal.any` is the sole mechanism. The reason type (`.abort()` vs `.abort("string")`) is irrelevant for retention — V3 is clean, V5 leaks even without a listener on the composite. ## Risk - `sse.ts` is used by the dev-presence routes. Behaviour is equivalent — timeouts and client disconnects still abort the stream. `signal.reason` is now a named string sentinel (`"timeout"`, `"request_aborted"`, etc.) instead of the previous string arg or default `AbortError`. No in-tree reader of `signal.reason` exists. - `entry.server.tsx` change is a standard cleanup of an abort timer, matches upstream React Router guidance. - `tracer.server.ts` change is env-gated and defaults to current behaviour. - Three other webapp `AbortSignal.timeout()` callsites (alert delivery, remote-build status) are fire-and-forget passed directly to `fetch` — not composed with anything long-lived, no retention risk, untouched. ## Test plan - [ ] Existing SSE integration tests pass - [ ] Dev-presence SSE behaves normally across tab open/close cycles - [ ] No heap growth under sustained aborted-connection traffic (heap snapshot diff) ## Follow-up The same `AbortSignal.any([userSignal, internalSignal])` pattern exists in several SDK/core callsites that ship to customers (`packages/core/src/v3/realtimeStreams/manager.ts`, `packages/trigger-sdk/src/v3/{ai,chat,chat-client,sessions}.ts`, `packages/core/src/v3/workers/warmStartClient.ts`). Whether those leak in practice depends on the user passing a long-lived signal. Tracked separately.	2026-04-23 15:54:05 +02:00
Eric Allam	2d3b2e82e6	feat(run-engine): flag to route getSnapshotsSince through read replica (#3423) ... ## Summary Adds `RUN_ENGINE_READ_REPLICA_SNAPSHOTS_SINCE_ENABLED` (default `"0"`). When enabled, the Prisma reads inside `RunEngine.getSnapshotsSince` run against the read-only replica client instead of the primary. Offloads the snapshot-polling queries fired by every running task runner off the writer. ## Why `getSnapshotsSince` is called from the managed runner's fetch-and-process loop (once per poll interval, plus on every snapshot-change notification). It runs four sequential reads per call — one `findFirst` by snapshot id, one `findMany` on snapshots with `createdAt > X`, one raw SQL against `_completedWaitpoints`, and chunked `findMany` on `waitpoint`. Per concurrent run, every few seconds. It's read-only, tolerates a small amount of staleness, and is an obvious candidate for the replica. ## Replica-lag considerations - **Step 1 "since snapshot not found"**: if the runner just received a snapshot id from the primary and asks the replica before it replicates, the function throws and the caller treats the response as an error (runner falls back to a metadata refresh). Self-correcting, not silent. - **Step 2 missing newly-created snapshots**: the next poll's `createdAt > sinceSnapshot.createdAt` filter still picks them up once the replica catches up. - **Waitpoint junction race**: the riskiest path — if a latest snapshot is replicated but its `_completedWaitpoints` join rows aren't yet, the runner could advance past that snapshot with `completedWaitpoints: []`. WAL/storage-level replication replays commits in order, so in practice both should appear atomically on the reader, but the race window is why the flag ships disabled. Aurora reader shrinks all three windows to single-digit ms in typical conditions, and its storage-level replication gives atomic visibility of committed transactions on the reader. ## Test plan - [ ] Flip the flag on in a non-prod environment, confirm snapshot polling behaves normally and `getSnapshotsSince` errors in Sentry stay flat. - [ ] Verify writer query volume drops and reader query volume rises on the snapshot-polling queries. - [ ] Keep an eye on `AuroraReplicaLag` (or equivalent) during rollout.	2026-04-22 11:48:04 +01:00
Eric Allam	881288c615	feat(webapp): deprecate v3 CLI deploys server-side (#3415) ... ## ✅ Checklist - [x] I have followed every step in the [contributing guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md) - [x] The PR title follows the convention. - [x] I ran and tested the code works --- ## Summary Adds a server-side gate that detects deploy attempts from v3 CLI versions (i.e. `trigger.dev@3.x`) at the `POST /api/v1/deployments` entry point and, when enabled, rejects them with a clear upgrade message. v4 CLI deploys are completely unaffected. The last 3.x CLI release was `3.3.7`, which we can't update. This approach short-circuits the deploy before any DB writes, image-ref generation, S2 stream creation, or queue enqueue — no side effects in either mode. ## How v3 vs v4 are distinguished I pulled the published CLI tarballs for `trigger.dev@3.3.7`, `4.0.0`, `4.0.1`, `4.0.5`, `4.1.0`, `4.2.0`, and the current `4.4.4` in the repo. The cleanest, most reliable signal is the request body to `POST /api/v1/deployments`: | Field on initialize | v3.3.7 CLI | v4.x CLI | |---|---|---| | `type` | **never sent** | always sent — `"MANAGED"` (run_engine_v2) or `"V1"` | | `isNativeBuild` / `gitMeta` / `triggeredVia` / `runtime` | not sent | sent | | `registryHost` / `namespace` | sent (v3-only; stripped by current Zod schema) | not sent | Every v4 call site I inspected sets `type: features.run_engine_v2 ? "MANAGED" : "V1"` unconditionally. `payload.type` is `undefined` if and only if the client is a 3.x CLI. ## Behavior - Detection always runs and emits `logger.warn("Detected deploy from deprecated v3 CLI", { environmentId, projectId, organizationId, enforced })`, which lets us watch how many v3 deploys are still happening before enforcement is flipped. - Enforcement is gated behind `DEPRECATE_V3_CLI_DEPLOYS_ENABLED` (default `"0"`, off). When `"1"`, the server returns `400` with: > The trigger.dev CLI v3 is no longer supported for deployments. Please upgrade your project to v4: https://trigger.dev/docs/migrating-from-v3 The v3 CLI surfaces this verbatim as `Failed to start deployment: <message>` because `zodfetch` throws `ApiError` for non-retryable 4xx (400/422) and `deploy.js` in 3.3.7 prints `error.message`. ## Out of scope (intentionally) - `api.v1.deployments.$deploymentId.finalize.ts` / `FinalizeDeploymentService` / `createDeploymentBackgroundWorkerV3.server.ts` are V1-engine paths, not the v3 CLI gate. Leaving them alone per review. - Container-side `createDeploymentBackgroundWorker` call in `managed-index-controller.ts` is still used by v4's in-image indexer. Not touched. - v3 `trigger dev` flow (different code path) — separate deprecation if/when needed. ## Testing - Ran `pnpm run typecheck --filter webapp` locally — passes. - Verified v4 tarballs (4.0.0, 4.0.1, 4.0.5, 4.1.0, 4.2.0, 4.4.4) all include `type:` in the `initializeDeployment` call site, so none will be accidentally blocked. - Verified v3.3.7 tarball's `initializeDeployment` payload has no `type` field. Rollout plan after merge: 1. Deploy with `DEPRECATE_V3_CLI_DEPLOYS_ENABLED` unset → watch `Detected deploy from deprecated v3 CLI` log volume. 2. When comfortable, set `DEPRECATE_V3_CLI_DEPLOYS_ENABLED=1` to enforce. --- ## Changelog Detect v3 CLI deploys on `/api/v1/deployments` and, when `DEPRECATE_V3_CLI_DEPLOYS_ENABLED=1`, reject them with an upgrade message pointing at https://trigger.dev/docs/migrating-from-v3. v4 CLI deploys are unaffected. Link to Devin session: https://app.devin.ai/sessions/b242c11bd86e4099aeec8b59bab62143 Requested by: @ericallam	2026-04-20 15:26:57 +01:00
Eric Allam	f1f1d02f1d	feat(dashboard): a few tweaks to the AI models page (#3315)	2026-04-02 16:48:36 +01:00
Matt Aitken	68e88d0d71	Object Storage seamless migration (#3275) ... This allows seamless migration to different object storage. Existing runs that have offloaded payloads/outputs will continue to use the default object store (configured using `OBJECT_STORE_*` env vars). You can add additional stores by setting new env vars: - `OBJECT_STORE_DEFAULT_PROTOCOL` this determines where new run large payloads will get stored. - If you set that you need to set new env vars for that protocol. Example: ``` OBJECT_STORE_DEFAULT_PROTOCOL=“s3" OBJECT_STORE_S3_BASE_URL=https://s3.us-east-1.amazonaws.com OBJECT_STORE_S3_ACCESS_KEY_ID=<val> OBJECT_STORE_S3_SECRET_ACCESS_KEY=<val> OBJECT_STORE_S3_REGION=us-east-1 OBJECT_STORE_S3_SERVICE=s3 ``` --------- Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>	2026-04-01 10:06:12 +01:00