triggerdotdev--trigger.dev

github-featured/triggerdotdev--trigger.dev

Author	SHA1	Message	Date
Eric Allam	c0b84595a3	feat(webapp): hosted webhook ingress, delivery pipeline, and dashboard (#4344 ) ## Summary The server half of hosted webhooks: the public ingress endpoint, signature verification, the delivery pipeline (Postgres partitioned storage + ClickHouse for ordering), the in-app partition manager, the HTTP API, and the dashboard (Deliveries, Endpoints, and the in-app test console). The public SDK and docs half is #4537. That PR carries the user-facing API (`webhook()`, `chat.event` / `chat.channels`, the `@trigger.dev/slack` connector) and builds on the shared `@trigger.dev/core` schemas that ship here. ## Shipping behind a flag A `WEBHOOK_ENABLED` env var (default off) gates the public ingress route and the engine worker plus partition cron, so merging and deploying this changes nothing in production until it is flipped on per environment. The dashboard is separately gated per org by the `hasWebhooksAccess` feature flag. ## Note on packages This PR includes the `@trigger.dev/core` schema additions the server compiles against, but carries no changeset. Core is not consumed independently of the SDK, so it is released together with the SDK via #4537. Keeping its changeset off `main` means no release cut from `main` publishes it early.	2026-08-16 14:33:42 +01:00
Katia Bulatova	4569657923	feat(webapp): dashboard agent — chat, reports, investigate (#4418 ) ## What & why This is the system behind the Dashboard Agent — an assistant that answers questions about a project's runs, errors, queues, deploys and health, and can investigate failures end to end. The agent runs as a chat.agent task in its own Trigger project. It has no access to the main database or ClickHouse; all platform data is read through the public API using a delegated, read-only user token. Everything here is behind `canAccessDashboardAgent` and inert with the flag off. The UI that mounts the panel lands in #4529. ## Stack `#4418` (this, base) ← `#4529` UI ← `#4525` Watch ← `#4516` storybook gallery. The scenario/contract reference for the whole stack is `internal-packages/dashboard-agent/GUIDEBOOK.md` (it lands on the Watch branch): it states, per feature, what makes each thing happen and where that is decided. ## What's inside Agent runtime and tools — `internal-packages/dashboard-agent`: prompt, tool set (API reads, TRQL query, docs, navigation, evidence/investigations, repo source), conversation compaction, a prompt-prefix token budget pinned by snapshot test, and sampled LLM-judged turn evals. The package cannot import webapp server code, which is what makes the "no DB access" claim structural rather than a convention. Contracts — `internal-packages/dashboard-agent-contracts`: `trigger://` URIs, intents, and the block envelope every rendered card travels in. Conversation store — `internal-packages/dashboard-agent-db`: drizzle over postgres-js in its own `trigger_dashboard_agent` Postgres schema, plus one additive migration. Auth boundary — the user-actor token gains an optional environment claim; one guard (`userActorEnvironment.server.ts`) enforces it so routes don't each re-derive the rule. Token minting, cap ceiling, and the RBAC fallback path for self-hosted. Transport — webapp resource routes that mint the token and proxy each turn, and SDK-side mid-turn reconnect. Public API the agent reads through — orgs, projects, environments, runs, queue metrics, workers, a run's commit metadata, repo snapshot, reports, and `POST /api/v1/query`. Reports — the health report's layout is declared once and shared by the card, the markdown surface and the JSON/MCP surface, so the same report reads the same in the dashboard, the terminal and an editor. Block renderers — the report and investigation cards the flows above already emit (`app/components/dashboard-agent/`). The panel that hosts them, and the rest of the chat UI, is #4529. Query safety and CSP — see below. ## Key decisions - The agent is a separate Trigger project, not webapp code. It reads platform data over the public API with a delegated user-actor token whose `cap` ceilings it to read scopes. No Prisma, no ClickHouse, no webapp imports. - The PAT-only auth helper now refuses user-actor tokens. This is an intentional behavioral change: its callers consume only a bare userId and do not enforce delegated-token capabilities. Actor-aware routes continue through the scoped route builders instead. - RBAC fallback builds a delegated token's ability from its own cap, never the blanket ability a PAT gets (read-only when the token declares none). Without this, the agent's read-only cap would buy a write JWT on self-hosted. - Org creation checks RBAC only for user-actor tokens, and only after the env gate, so an install with `ORG_CREATION_API_ENABLED` off returns 404 rather than 403, and an ordinary PAT never consults an ability the route has no org to scope. Both orderings are pinned by test. - The query path is read-only in depth. TRQL rejects write statements at the grammar level (they don't parse, rather than being filtered), ClickHouse runs with `readonly=1`, and the org/project/env filters are injected server-side from the credential — the request body cannot widen scope. An unparseable query denies instead of falling through to the permissive resource. - Document-wide img-src CSP. Remote images are an outbound-request/exfiltration surface, so the policy permits only own-origin/data/blob, the required SSO avatar hosts, and the favicon endpoint. Operators can add exact origins through CSP_IMG_SRC_ALLOWLIST; wildcard hosts and bare schemes are intentionally not allowed. - The chat transport reconnects on a mid-turn EOF (`@trigger.dev/sdk`). A body that ends without a turn-complete is terminal only when the server says `X-Session-Settled: true`; otherwise the transport resubscribes from `lastEventId` with bounded backoff, and any record re-earns the budget. Previously a closed long-poll window or a proxy restart left the reply stuck as if still generating. - Conversations live in their own datastore, schema-scoped and foreign-key-free (it references `organizationId`/`userId` by id, because in cloud it is a different database). It is a display read-model for the History tab and transport resume; `chat.agent`'s object-store snapshot remains the model's source of truth. - Deterministic first. Reports and health checks contain no LLM — they are computed from the same data the dashboard shows, and the model only narrates and links them. That is what makes a number in an answer auditable. ## Testing - 63 new test files, run with `pnpm run test --filter webapp` and per-package vitest. Heaviest coverage on the auth boundary (`userActorPatOnlyBoundary`, `userActorTokenClaimsAndScopes`, `contextlessPatRoutes`, `rbacFallbackBranch`), TRQL read-only, the report layout, and the SDK reconnect. - The agent package has a separate eval lane (`pnpm run test:evals`, `vitest.eval.config.ts`) that hits the real model, so it never runs in `pnpm test`. - Live-tested against a local stack scenario by scenario; the GUIDEBOOK lists the condition each behaviour is expected under, which is what those runs were checked against. ## Changelog `.server-changes/dashboard-agent.md`, plus changesets for `@trigger.dev/core` (report schemas), `@trigger.dev/sdk` (chat reconnect) and the CLI's `mint-token` help text.	2026-08-11 18:56:14 +02:00
Saadi Myftija	02de2e693f	feat(api): separate rate limit budget for deployment endpoints (#4565 ) Most deploy-flow API calls shared the general per-environment rate limit bucket with all of that environment's runtime traffic, so an org with heavy API usage could intermittently 429 its own deploys; the `/api/v/deployments` endpoints themselves were fully exempt from rate limits as a stopgap ([#2774](https://github.com/triggerdotdev/trigger.dev/pull/2774)), which promised a dedicated limiter as the follow-up. This is that follow-up: the whole deploy-flow group now runs on its own budget, separate from runtime API limits. ### Design A new `deploymentRateLimiter` covers every endpoint the deploy flow depends on: the `/api/v/deployments` group, the env API key exchange (`/api/v1/projects/:ref/:env`), build-time env var resolution and sync (`/envvars`, `/envvars/:slug/import`), preview branches, `/api/v1/remote-build-provider-status` and `/api/v1/artifacts`. The general API limiter whitelists the same shared path list, so exactly one limiter applies to each path and the two can't drift apart. Buckets are keyed per environment for environment API keys and per token for the PAT-authenticated phase of a CLI deploy (whoami, key exchange, branches). The deploy budget is controlled via the `DEPLOYMENT_RATE_LIMIT_*` env vars.	2026-08-11 17:51:31 +02:00
Eric Allam	4eb9292cbe	feat(webapp,run-engine): queue metrics and health dashboard (#4131 ) ## Summary Three related changes, each independently gated: Queue metrics and health. Per-queue depth, throughput (enqueued, started, completed), concurrency, whether a queue is throttled, and scheduling delay (how long a run waits between becoming eligible and actually starting), plus a per concurrency-key breakdown for keyed queues. Collected from inside the run queue itself, stored in ClickHouse, and surfaced on the Queues list, a new per-queue detail page, the task pages, and the run inspector. The question it answers is "does this queue have enough concurrency to keep up, and if not, which key or which limit is the constraint". Percent-based queue concurrency limits. A queue's concurrency override can now be expressed as a percentage of the environment limit, stored as the source of truth and re-materialized whenever the environment limit changes. Absolute overrides above the environment limit are now rejected with a 400 instead of being silently capped, which is a behavior change on `POST /api/v1/queues/:queue/concurrency/override`. The `health` report. A server-computed verdict on whether work is flowing, whether the runs that do start are healthy, and whether telemetry is fresh, rendered as text with sparklines. Available as `GET /api/v1/reports/:key`, `trigger report`, and the `get_report` MCP tool (plus a `report` MCP prompt, which shows up as a slash command in hosts that support prompts). With the flags off, the Queues page renders the pre-metrics component verbatim, nothing is emitted, and nothing is written to ClickHouse. ## Configuration Two independent gates, on purpose. Emission is global so data accrues for everyone before anyone can look at it; the view is per organization so it can be turned on for one org at a time without a deploy. Runtime flags (no restart) \| Flag \| Store \| Gates \| \| --- \| --- \| --- \| \| `queue_metrics:enabled` \| run-queue Redis key (`"1"`/`"0"`, off by default) \| All emission, gauges and counters. Cached in-process for 10s with stale-while-revalidate, warmed eagerly at boot so the first op after a deploy is not dropped. \| \| `queue_metrics:gauge_sample_rate` \| run-queue Redis key, `0..1` \| Fraction of queue ops that emit a gauge. Counters are never sampled, so throughput stays exact at any rate. \| \| `queueMetricsUiEnabled` \| feature-flag catalog: global `FeatureFlag` row, per-org `Organization.featureFlags` override wins \| Whether an org sees the metrics view at all: the Queues list variant, the queue detail route, the built-in Queues dashboard, the concurrency-keys endpoint, and the metrics blocks on task pages and the run inspector. Off by default; a gated org gets a 404 on the detail route rather than an empty page. \| Both Redis keys are readable and writable from `/admin/queue-metrics` (super-admin UI, with a live per-shard stream-health table) and `GET`/`POST /admin/api/v1/queue-metrics` (admin PAT). The admin surface uses its own Redis client, so it works on any instance regardless of whether that instance runs the emitter or the consumer. Environment variables (boot time) \| Variable \| Default \| Notes \| \| --- \| --- \| --- \| \| `QUEUE_METRICS_EMIT_ENABLED` \| `0` \| Constructs the emitter and injects it into the run engine. Without it the run queue has no emitter at all. \| \| `QUEUE_METRICS_CONSUMER_ENABLED` \| `0` \| Boots the stream consumer on this instance. Independent of emission, so consumers can be sized separately from the API. \| \| `QUEUE_METRICS_STREAM_SHARD_COUNT` \| `4` \| Stream shards, hashed per queue. \| \| `QUEUE_METRICS_CONSUMER_BATCH_SIZE` \| `1000` \| Poll batch equals insert batch, so an ack can never outrun a write. \| \| `QUEUE_METRICS_REDIS_{HOST,PORT,USERNAME,PASSWORD,TLS_DISABLED}` \| falls back to the run-queue Redis \| Set `HOST` to move the metrics stream onto a dedicated instance so a metrics backlog cannot compete with the run queue for memory. Self-hosters can leave it unset and get a single-Redis deployment. \| \| `QUEUE_METRICS_COUNTER_STREAM_MAXLEN` \| `2000000` shared, `8000000` dedicated \| Bound on how much a stalled consumer can hold. The default is deliberately lower when the stream shares the queue-critical Redis. \| \| `QUEUE_METRICS_COUNTER_ODOMETER_TTL_SECONDS` \| `604800` \| TTL on the per-queue cumulative counter key, refreshed on every write, so only queues idle for the whole window are purged. \| \| `QUEUE_METRICS_MAX_QUEUE_NAMES_PER_ENV` \| `1000` \| Distinct queue names tracked per environment; overflow collapses into `__overflow__`. \| \| `QUEUE_METRICS_MAX_CONCURRENCY_KEYS_PER_QUEUE` \| `10000` \| Same idea one level down, per queue. \| \| `QUEUE_METRICS_GAUGE_SAMPLE_RATE` \| `1` \| Default for the live sample-rate key above. \| \| `QUEUE_METRICS_QUERY_TABLES_VISIBLE` \| `0` \| Lists the queue-metrics tables in the Query page, its schema docs, the schema API and the AI query context. Off keeps them unlisted while the feature is dark; a query naming them still runs either way. \| \| `QUEUE_METRICS_CLICKHOUSE_URL` \| falls back to the shared wiring \| Runs queue metrics on their own ClickHouse service: the consumer's inserts and every queue-metrics read go through it, so a metrics-heavy chart refresh never competes with runs-list or trace reads. Unset reproduces the previous split exactly (inserts on `CLICKHOUSE_URL`, reads on the query pool). \| \| `QUEUE_METRICS_CLICKHOUSE_READER_URL` \| the write URL \| Reader split, so the consumer's inserts can never land on a read endpoint. \| \| `QUEUE_METRICS_CLICKHOUSE_{KEEP_ALIVE_ENABLED,KEEP_ALIVE_IDLE_SOCKET_TTL_MS,MAX_OPEN_CONNECTIONS,LOG_LEVEL,COMPRESSION_REQUEST}` \| `1`, unset, `10`, `info`, `1` \| Pool tuning, matching the other per-workload ClickHouse clients. \| Migrations to apply: ClickHouse `036_create_queue_metrics_v1.sql`, and a Postgres migration adding the nullable `TaskQueue.concurrencyLimitOverridePercent`. Both are additive. ## How collection works Queue operations produce two kinds of signal, and they have opposite failure modes, so they are handled differently. Gauges (queued, running, queue limit, env queued, env running, env limit, throttled, plus keys-with-backlog and worst-key wait on keyed queues) are read inside the same Redis script that performs the enqueue or dequeue, so the reading is atomic with the operation it describes rather than a racy follow-up read. The script returns them on its reply and the app forwards them to the stream. Gauges are sampled and drop-tolerant: they are aggregated with `max`, so a lost reading costs resolution, never correctness. Counters (enqueued, started, completed, plus nack and dead-lettered) are cumulative odometers. Each event increments a per-queue key on the metrics Redis and emits the absolute total, and ClickHouse takes the difference across buckets at read time. This is the important property of the design: a summed-delta counter undercounts permanently on any lost event, while a cumulative one self-heals, because the next surviving reading restates the whole total. Only bucket granularity can be lost, never the total. A queue returning after its odometer TTL expired restarts at 1 and reset detection handles it, which is safe precisely because expiry only spans a window with no activity. Both land on one sharded Redis stream. A consumer reads it with a consumer group, reclaims stale pending entries on a 15s interval rather than on every poll, maps one entry to one or two ClickHouse rows (whole-queue and, for keyed queues, per-key), and acks only after the insert lands. Each batch carries a dedup token derived from its stream-entry ids, and the target tables set `non_replicated_deduplication_window`, so a retried batch cannot double-count either the raw rows or the aggregates that hang off them. Consumer and emitter both emit OTel metrics (`queue_metrics.emitter.emitted`, `queue_metrics.consumer.{entries,rows_inserted,insert_errors,insert_duration,stream_depth,group_lag,pending,lag_unknown}`); stream depth and group lag are the two worth alerting on, and `lag_unknown` exists because Redis can report a null lag after a trim, which must not be read as zero. ## Storage and read path `queue_metrics_raw_v1` is a short landing table with a 6 hour TTL. Four aggregate tiers are materialized straight from raw, never cascaded off each other, each with a 30 day TTL: - `queue_metrics_v1`, 10 second buckets per queue, the default read path - `queue_metrics_5m_v1`, 5 minute buckets per queue, for wide ranges and cross-queue ranking - `env_metrics_v1`, 10 second buckets per environment, queue-independent so it stays cheap at any range - `queue_metrics_ck_v1`, 10 second buckets per concurrency key Every tier is an MV from raw because the counter states do not survive a cascade: their merge is order sensitive, so a `-MergeState` chain off the 10s table inflates the result, and the same property means an aggregate state may only be merged inside one queue. That constraint is now enforced by the query engine rather than by reviewer discipline: a column can declare a `mergeGroupKey`, and any query that references it without grouping by, or pinning to a single value of, every named key fails to compile with an actionable message. On the read side, TRQL gains three tables (`queue_metrics`, `env_metrics`, and a `queue_metrics_by_key` that is hidden from the editor, schema docs and schema API but still queryable, so per-key rows can never silently merge into a plain per-queue query), plus `deltaSumTimestampMerge` and `quantilesTDigestMerge`. Two schema-level optimizations ride along: a table can declare coarser rollups, so a query whose bucket interval is 5 minutes or wider is routed to the 5m table with no change to the query itself, and it can opt into the ClickHouse query cache with time bounds floored to a fixed grid, so the auto-refreshing dashboards actually share cache entries instead of missing on every tick. Both are caller-side substitutions, so the printer stays unaware of physical layout. All of this can also live on its own ClickHouse service. A table declares the pool its reads run on, the three queue-metrics tables name the dedicated one, and the ingestion consumer writes through the same client, so both directions move together with one env var and nothing else routes differently. The other engine change is opt-in gap filling: charts can request rows for empty buckets, where counters zero-fill and gauges carry forward. Grouped gauge series are densified per group and carried inside a partition, so a quiet queue's line holds its last value without bleeding another queue's value into it. ## Queue concurrency limits `concurrencyLimitOverridePercent` on `TaskQueue` is the source of truth when an override is set as a percentage; the absolute `concurrencyLimit` is materialized from it (floored, clamped to at least 1 so a percentage can never act as a pause, and never above the environment limit). Every path that changes an environment limit now recalculates the environment's percent-based overrides afterwards, outside the transaction, and pushes changed limits to the engine. The push is attempted even when the stored value did not change, so a previously failed sync self-heals rather than leaving the database and the engine diverged; paused queues are skipped so a recalculation cannot effectively unpause one. The API accepts exactly one of `concurrencyLimit` or `percent`, and the reject-instead-of-clamp change above means a request asking for more than the environment allows now fails loudly. The percent bound (greater than 0, at most 100) is defined once and shared by the zod schema, the dashboard mutation handler and the service, so the three cannot drift. The concurrency-keys table on a queue is now paginated against the ClickHouse per-key tier, ranked by peak backlog with the total on every row from a single scan, and only the keys on the current page are enriched with live counts from Redis. That replaces a hard top-50 cap with something whose cost is a function of page size rather than key cardinality. ## The health report `GET /api/v1/reports/:key?period=&format=markdown\|ansi\|json`. The verdict is computed on the server and is deterministic, not model-generated. Three independent analyzers run over one input snapshot: flow (is work moving, and if not, is the cause a limit, throttling, one bad queue, or dead-lettering), execution (are the runs that start succeeding, and at what latency), and liveness (how fresh is the telemetry). When telemetry is genuinely stale, the first two are forced to unknown and every actionable field is stripped, so no surface ever advises action off stale data. Authorization is per query table rather than a blanket query grant: a JWT must be scoped to every table the report reads (`runs`, `env_metrics`, `queue_metrics`), so a narrowly scoped token cannot pull a report that reads more than it was granted. `period` is validated as a shorthand with a 90 day ceiling at the edge. The report catalog is a registry of `{ load, interpret }` entries, so the next report is a new entry and no change to the route, the view model, the renderers, the CLI or the MCP tool. `trigger mcp` no longer launches the install wizard when stdout is a TTY, which fixed a real failure: hosts spawn the server over a PTY, so the wizard would open and the client would time out waiting for a server that never started. The wizard now needs `trigger mcp --install`. ## The part that is live regardless of every flag The enqueue and dequeue scripts now return a 2-tuple so a gauge reading can ride back on the reply. Every return site in the eight affected scripts is wrapped, and a `nil` original is converted to `false` on the way out, because a raw `nil` in the first slot would make Lua truncate the multi-bulk reply and silently drop the gauge on the throttled and empty-queue paths. The reply shape and the destructuring on the app side are exercised on every queue operation whether or not metrics are enabled, so that is the part of `run-engine` worth the closest review. One behavior fix in the same area: the scheduling-delay anchor is set only on a run's first entry into the queue. Anchoring it to trigger time on re-enqueues made waitpoint and checkpoint resumes report the entire wait as scheduling delay. Queue ordering is untouched, so a re-enqueued run keeps its position, and nacks deliberately keep the original anchor because a rolled-back dequeue is the same continuous wait. A pending-version promotion still anchors to trigger time, on purpose: that promotion is the run's first real entry into the queue, since the trigger deliberately held it back waiting for a worker version, and the TTL is armed at the same point for the same reason. The consequence is worth naming, because it is a judgement call: a run that waits on a deployment reports that wait as scheduling delay on its queue, which is time unrelated to queue capacity. ## Verification Unit and integration suites across the new package, the run queue, the mapping layer, the query engine and ClickHouse (including a test that applies migration 036 through the same splitter CI uses, and a regression test that inserts the same batch three times to prove the aggregates do not inflate). Beyond that, the whole path was driven end to end against a live stack with real runs: emitter to Redis stream to consumer to ClickHouse to the dashboards, for both the local dev path and the deployed path where a supervisor drives the dequeue, with assertions on exact counter reconstruction per queue and per concurrency key, throttling, environment saturation, scheduling delay, and a deliberate mid-stream reading drop to confirm the cumulative counters still reconstruct the correct total. The gated-off state was checked on every touched surface. The dedicated ClickHouse service was verified against a second, separately-schema'd instance: with it configured, the driven counters reconstruct exactly on the dedicated instance, the shared instance gains no rows for that window, a read through the query API returns the value that exists only on the dedicated instance, and a `runs` query still succeeds (it would fail outright if it were mis-routed to a service without that table). With the variable unset, the full suite passes unchanged. --------- Co-authored-by: Katia Bulatova <katia@trigger.dev> Co-authored-by: Katia Bulatova <katherine.bulatova@gmail.com> Co-authored-by: James Ritchie <james@trigger.dev>	2026-07-29 16:45:24 +01:00
Chris Arderne	6997aeb05e	fix: security release 2026-07-08 (#4316 ) ⚒️ Publish Worker (v4) / build (supervisor) (push) Has been cancelled Details	2026-07-21 12:00:58 +01:00
nicktrn	0ff0abd776	fix(webapp): recover from stale /build assets via a bounded reload (#4282 ) ## Problem The webapp's HTML references content-hashed `/build` assets, and each running instance contains exactly one build and returns 404 for asset hashes it doesn't have. During a rolling deploy a client can hold HTML from one build while a request for one of its assets is served by an instance on a different build → missing styles or a failed chunk load. ## What this does On a `/build` stylesheet/script/chunk load failure, the client does a bounded full-document reload (at most 2 per 5 minutes, tracked in `sessionStorage`) so the page reloads onto a single consistent build. That's the whole mechanism — no polling, no `fetch` interception, no blocking overlay, no form snapshotting. - `apps/webapp/app/components/StaleAssetRecovery.tsx` — authored as a typed, lint-checked function and serialized to an inline script via `.toString()` (so the logic is real, reviewable code, not an opaque string), injected before `<Links />`, production only. - Detection: capture-phase `error` listener for `<link>`/`<script>`/modulepreload failures under `/build/`, plus an `unhandledrejection` guard for dynamic-import failures. - Guards: once-per-page re-entrancy guard, the bounded reload budget, and a `navigator.onLine` check so it never reloads into an offline error page. - Unit tests in `StaleAssetRecovery.test.ts`. ## Relationship to #4260 Replaces the recovery introduced in #4260 (reverted in #4280) with a much smaller, reload-only approach — the previous version intercepted `fetch` and could turn a data request into a navigation, and showed a full-screen overlay on any asset error; this drops both. ## `/build-version` compatibility shim `apps/webapp/server.ts` adds a tiny `GET /build-version` endpoint (build id only, `no-store`). A previously-deployed client build polls it after an asset failure and reloads once it sees a newer build, so those older tabs recover in one reload instead of getting stuck. Temporary — safe to remove once older clients have cycled out. It deliberately does not re-add an `X-Build-Id` response header. ## Also Restores the `.server-changes` writing guidance in `.claude/rules/server-apps.md` (reverted alongside #4260). ## Self-hosting note Recovery is most reliable when your load balancer keeps a client on one instance for the duration of a deploy (short session stickiness) — the reload then lands on a consistent build in one hop.	2026-07-17 16:11:25 +01:00
nicktrn	051d7080d6	Revert "fix(webapp): survive asset hash rotation across rolling deploys (#4260 )" (#4280 ) Standard `git revert` of #4260. Its client-side stale-asset recovery is net-negative during normal deploys: - The `fetch` interception treats any `?_data=` request (Remix loader and action traffic) as a navigation and, on a build-id mismatch, `location.assign`es the tab to the fetched URL — an open dashboard tab can be hard-navigated to a raw data URL during a rolling deploy, losing unsaved input. - Any transient `/build` asset error (a network blip, an extension, an unrelated failed dynamic import) blanks the page behind a full-screen overlay for ~60s before offering a manual reload. - It serialized form field values to `sessionStorage` to restore them across the reload. This returns the webapp to the pre-#4260 baseline as a fast, low-risk step. Follow-ups (separate PRs): - a minimal reload-only recovery to replace this, - restore the unrelated `.claude/rules/server-apps.md` docs tidy-up from #4260 (via cherry-pick), - a load-balancer stickiness change addressing the root cause.	2026-07-17 14:42:52 +02:00
Katia Bulatova	2aa64200f8	fix(webapp): survive asset hash rotation across rolling deploys (#4260 ) ### Problem Webapp HTML references content-hashed /build assets, and each Docker image contains exactly one build with a hard 404 for unknown hashes. During a rolling deploy, a client holding HTML from the old build may request old asset hashes from a replica running the new image, causing missing styles or failed chunk loads. The page should recover automatically once a compatible build becomes available, without reload loops or unnecessary interruptions during normal deployments. ### What changed - Build changes alone do nothing — no polling, no automatic reloads. - If a CSS or JavaScript asset fails to load, a recovery overlay is shown immediately. - While the server still reports the same build, the client polls for a newer build using exponential backoff (up to ~60s). As soon as a newer build is detected, the page reloads automatically. - If no newer build appears within the timeout, recovery falls back to a manual Reload action. - If recovery still fails after the automatic reload, the client stops retrying and displays a final recovery screen instead of entering a reload loop. - Recovery preserves form values and scroll position across the automatic reload.	2026-07-14 23:48:21 +01:00
Chris Arderne	64e5d732ad	chore(webapp,core): remove the unused ResourceMonitor server logging helper (#4244 ) The `ResourceMonitor` server-side logging helper is no longer used. It periodically logged the webapp process own memory, disk, and CPU usage behind the `RESOURCE_MONITOR_ENABLED` flag (off by default), and was also exported from `@trigger.dev/core/v3/serverOnly` with no other consumers. This removes the helper, its `@trigger.dev/core` export, the webapp wiring, and the `RESOURCE_MONITOR_ENABLED` env var. The supervisor has its own unrelated `ResourceMonitor` class, which is left untouched.	2026-07-13 20:32:29 +01:00
Eric Allam	5ba8557a51	chore(webapp,core): remove the end-of-life v3 (engine V1) execution stack (#4236 ) ## Summary v3 (the engine that ran the SDK v3 era, internally `RunEngineVersion.V1`) is end-of-life. Following the removal of the v3 execution apps ([#4194](https://github.com/triggerdotdev/trigger.dev/pull/4194)) and the legacy dev websocket ([#4198](https://github.com/triggerdotdev/trigger.dev/pull/4198)), this removes the remaining v3 execution stack from the server. Clients still on v3 (an old SDK or CLI that has not upgraded) keep getting a clear "upgrade to v4" response. Triggers, batch triggers, reschedules, and deploys that resolve to v3 are rejected with a graceful 4xx pointing at the migration guide, never a 5xx, so a stale client cannot affect server health. Self-hosted instances still running v3 should stay on the 4.5.x release line until they migrate. ## What is removed - The MarQS queue and its shared/dev queue consumers. - The v3 socket.io namespaces (coordinator, provider, shared-queue) and the v3 run lifecycle services (attempt, checkpoint, and batch-resume). - The graphile-worker background job system; all live jobs already run on `@trigger.dev/redis-worker`. - The `DEPRECATE_V3_ENABLED` flag: v3 is now rejected unconditionally, so the flag is gone. - Unused v3 exports from `@trigger.dev/core` (the `v3/zodNamespace` subpath and the legacy socket message catalogs) and the now-dead MarQS environment variables. ## What stays The v4 engine is untouched. The graceful v3 rejection boundary stays, `determineEngineVersion` still detects a v3 project so it can reject it, and the batch service plus batch-completion worker stay for current clients. Live queue concurrency limits and metrics now read from the v4 run engine instead of MarQS, and a brand-new dev environment now defaults to v4. ## Dependency cleanup Removes webapp dependencies left unused by this change: `seedrandom` and `semver` (only the removed v3 code used them) plus a set that was already dead, their orphaned `@types` packages, and two dead files. Adds a `knip:deps` script and a `knip.json` config so unused dependencies can be found the same way going forward.	2026-07-13 11:32:06 +01:00
Daniel Sutton	8465ac5ac3	feat(run-ops): webapp db topology, flags, and split-mode resolver wiring (#4117 ) ## What Wires the run-ops split into the webapp: database topology, environment flags, split-mode gating, and the control-plane resolver/cache layer that the run-store and run-engine seams from the previous PR plug into. - DB topology & env (`apps/webapp/app/db.server.ts`, `env.server.ts`, `entry.server.tsx`): adds the run-ops database clients/topology and the environment variables that configure and gate the split. - runOpsMigration module (new `apps/webapp/app/v3/runOpsMigration/`): the webapp-side machinery — `splitMode.server.ts`, `controlPlaneResolver.server.ts` + `controlPlaneCache.server.ts`, `readThrough.server.ts`, `crossSeamGuard.server.ts`, `distinctDbSentinel.server.ts`, id-minting helpers (`mintBatchFriendlyId`, `runOpsMintKind`, `resolveInheritedMintKind`), `runOpsCascadeCleanup.server.ts`, the split read gate, and route/unblock catalogs. - Store/engine wiring (`app/v3/runStore.server.ts`, `runEngine.server.ts`, `runEngineHandlers.server.ts` + new `runEngineHandlersShared.server.ts`): points the webapp's store/engine construction at the resolver, and factors shared handler logic out so both seams use one path. - Read-path touch-ups: `runtimeEnvironment.server.ts`, `eventRepository/index.server.ts`, `taskRunHeartbeatFailed.server.ts`, `engineVersion.server.ts` route their run/environment lookups read-through the resolver. - `413a94511` — interlocks split mode against the native realtime backend so the two aren't enabled in an incompatible combination (see `.server-changes/run-ops-split-realtime-interlock.md`). - `dc74c57fd` — drops the earlier "known-migrated" read layer; residency is determined by id-shape only. ## Why PR5 of the run-ops split stack. This is the webapp foundation layer: it stands up the DB topology, flags, and resolver/cache the rest of the stack depends on, and repoints webapp read paths through the resolver. Additive when the split is not enabled (existing single-DB behavior preserved behind flags); behavior-changing on the read-through paths and the realtime interlock. ## Tests New vitest coverage across `apps/webapp/test/` and colocated `.server.test.ts` files: db topology, split mode, split read gate, cross-seam guard, mint cutover / flip latency, control-plane cache, control-plane resolver, distinct-db sentinel, read-through loaders (route loaders, run-detail loaders, `findEnvironmentFromRun`), and the run-engine handlers. Testcontainers-backed; no mocks. `pnpm-lock.yaml` synced for the two new webapp deps. ## Notes Draft, stacked on #4116* (`runops/pr04-store-engine`). Review that first; this diff is against it. Server-change / changeset note to be added at stack-assembly time. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-03 18:02:22 +01:00
Chris Arderne	c7861be520	chore: activate no-unused-vars and import linters (#4096 ) Once this is merged, oxlint is at a pretty sensible baseline. Enable `no-unused-vars`, `typescript/consistent-type-imports`, and `import/no-duplicates` lint rules Turns on three previously-disabled oxlint rules across the monorepo and fixes all violations: - `no-unused-vars` – enabled as an error with standard ignore patterns: unused function arguments are ignored by default (`args: "none"`), variables/caught errors/destructured array elements prefixed with `_` are allowed, and rest siblings are permitted. - `typescript/consistent-type-imports` – enforced as an error; all type-only imports now use the `import type` syntax. - `import/no-duplicates` – enforced as an error; duplicate import statements from the same module have been merged. The remaining commits clean up the violations found across the codebase: removing unused variables/imports/type aliases, adding `_` prefixes to intentionally unused bindings, fixing duplicate imports, and converting value imports to `import type` where appropriate.	2026-07-02 11:37:05 +01:00
Katia Bulatova	b1987dc090	feat(webapp): billing limits — pause, reject, recovery, and settings UI (#3996 ) ## Summary Adds Billing Limits to the webapp. Customers can set a monthly spend cap. When usage crosses the limit, billable environments enter a grace period. If the limit is not resolved before grace expires, new triggers are rejected until the organization increases or removes the limit.	2026-06-26 17:12:53 +02:00
Chris Arderne	4fde283e76	chore: format and lint webapp also (#4056 ) #3977 added formatting and linting everywhere else. This extends it to the webapp.	2026-06-26 13:02:53 +01:00
nicktrn	7aa871f37b	feat(webapp): plan-aware compute migration (#3957 ) Adds an opt-in mechanism to route a configurable percentage of organizations onto the compute (MicroVM) backing of their region at trigger time, without changing their stored region settings. Routing is gated by three global feature flags - `computeMigrationEnabled`, `computeMigrationFreePercentage`, `computeMigrationPaidPercentage` - plus a per-org `computeMigrationEnabled` override that wins in both directions. A region's compute backing is resolved from a new `WorkerInstanceGroup.region` column: a container group and its MicroVM group share one geo `region`, so the migration swaps the resolved worker queue to the backing group's queue. Orgs are bucketed deterministically by id, so ramping a percentage down keeps a strict subset rather than reshuffling, and a region with no compute backing is never touched. Everything is off by default - behaviour is unchanged unless the flags are set. The flags and the worker-region groups are read on the trigger hot path from in-memory snapshots rather than the database: a small `createReloadingRegistry` helper loads each at startup and refreshes them on an interval, so no per-trigger query is added and a percentage or kill-switch change propagates within the reload interval. A cold replica whose snapshot hasn't loaded yet reads as not-migrated (the container path) and self-corrects on the next load - the same cold-start contract as the datastore / LLM-pricing registries, with a `reloading_registry_loaded` metric so a never-loaded registry is alertable. The same migration decision is consulted at deploy-time template creation so a migrated org gets a compute template built ahead of its first run. This runs in shadow mode (best-effort, never fails the deploy) by default, or - when the `computeMigrationRequireTemplate` flag is on - in required mode, built synchronously at deploy so the first run never builds on-demand and template errors surface at deploy time. So operators keep "which runs ran where" while customers only see geography: the run's actual worker queue is stored raw, and the geo region is stamped separately on `TaskRun.region` (and a new ClickHouse `region` column) at trigger time. Read surfaces - the dashboard, the API, and the Query/Logs page - show the geo region, falling back to the worker queue for runs written before the column existed. Minor follow-ups left out of scope: the percentage flags render as text inputs on the admin flags page (the catalog UI has no numeric control type yet), and `createReloadingRegistry` could later gain pub/sub for sub-second cross-replica propagation if the reload interval proves too slow.	2026-06-17 08:28:15 +01:00
Eric Allam	f9d57d3bd5	feat(webapp): add a new backend for the realtime runs feed (#3864 ) ## Summary Adds a second backend for the realtime runs feed (`useRealtimeRun`, `subscribeToRunsWithTag`, `subscribeToBatch`), built to stay healthy when a single busy environment has many subscribers watching many runs at once. It is gated behind a feature flag with the existing backend as the default, so nothing changes for users until it is enabled per environment. ## Design A run change is published once, as a small self-describing record, to a single per-environment channel. Every feed is then a predicate over that one stream rather than owning a channel: - A per-instance router indexes the currently-held feeds by run, tag, and batch. When a run changes it hydrates the affected rows once and serializes them once, then fans the result to every matching feed. One hot shared tag watched by many subscribers costs a single database query and serialize, not one per subscriber. - Feeds that don't match a change are never woken, wake delivery per environment is coalesced on a leading edge (250ms default) so a burst of changes costs one wake, and cold reads coalesce onto a single short-TTL-cached resolve. - An admission gate bounds how many cold ClickHouse resolves run concurrently, so a mass reconnect across many distinct filters queues instead of stampeding the database. - Changes that land while a client is between long-polls are delivered on its next poll instead of waiting for the periodic backstop: each environment buffers its recent change records, subscriptions linger briefly after the last feed closes, and a newly-armed poll replays exactly the connection's gap. - The per-connection replay cursors behind that are shared across instances via Redis (a single timestamp each), so a poll landing on a different instance behind the load balancer still reads the connection's true gap instead of falling back to a cold resolve. Cursor reads have a bounded deadline and degrade to the cold-read path on any Redis trouble. - Tag subscriptions with multiple tags match runs carrying all of the tags, mirroring the existing backend's filter semantics, and live long-polls hold for about 20 seconds to match its cadence. - The per-environment channel supports Redis Cluster sharded pub/sub, so the wake path scales horizontally across shards by environment. - The backend reports its health through OpenTelemetry metrics (delivery lag, poll resolution paths, backstop outcomes, replay and cursor-store activity), with a provisioned Grafana dashboard for local development. Everything is behind the feature flag and tunable via env vars; the existing backend remains the default.	2026-06-11 07:56:10 +01:00
Daniel Sutton	4745754a7a	feat(webapp,run-engine): mollifier drainer replay + stale sweep + cancelled-run engine API (#3754 ) ## Summary The replay side of the mollifier: - `DrainerHandler`: reads buffered snapshots and replays them through `engine.trigger` to materialise PG rows. - `RunEngine.createCancelledRun`: new public method the handler uses to write CANCELED rows directly from snapshots (bypass queue + waitpoint, emit `runCancelled`). Tolerates the cjson empty-table tags edge case found during validation. - Drainer fairness: org → env rotation so a heavy env doesn't starve light ones in the same org. - Stale-entry sweep + telemetry + alertable gauge so a stuck/offline drainer surfaces in alerts. Both the drainer and sweep default-off; nothing fires unless flagged on (`TRIGGER_MOLLIFIER_DRAINER_ENABLED`, `TRIGGER_MOLLIFIER_STALE_SWEEP_ENABLED`). Stacked on the trigger-time decisions PR. ## Test plan - [x] \`pnpm run typecheck --filter webapp\` passes - [x] \`pnpm run test --filter webapp test/mollifierDrainerHandler.test.ts\` passes - [x] \`pnpm run test --filter webapp test/mollifierStaleSweep.test.ts\` passes - [x] \`pnpm run test --filter @internal/run-engine src/engine/tests/createCancelledRun.test.ts\` passes - [x] \`pnpm run test --filter @trigger.dev/redis-worker packages/redis-worker/src/mollifier/drainer.test.ts\` passes --- ## Ship-gate follow-up fix Drainer writes SYSTEM_FAILURE on max-attempts exhaustion. Adds an `onTerminalFailure` callback on `MollifierDrainerOptions` so the customer's run lands a SYSTEM_FAILURE PG row even when the drainer exhausts `MAX_ATTEMPTS` on a retryable PG error (previously `buffer.fail()` was called with no row written → silent data loss). The callback runs before `buffer.fail()` on every terminal path (non-retryable AND max-attempts-exhausted), and re-throwing a retryable error from the callback causes the drainer to requeue rather than fail. Bumps `@trigger.dev/redis-worker` to a minor changeset (additive option + new exported types). Includes 5 unit tests covering both terminal causes plus the requeue-on-retryable-callback-failure path and no-callback back-compat. --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-06-01 13:20:05 +01:00
Matt Aitken	596a9bb90c	fix(webapp): retain sessions-replication singleton import via globalThis assignment (#3738 ) Linear: [TRI-9864](https://linear.app/triggerdotdev/issue/TRI-9864) (Urgent) Production incident: [TRI-9863](https://linear.app/triggerdotdev/issue/TRI-9863) (mitigated by image revert in cloud#910) ## Bug `apps/webapp/package.json` declares `"sideEffects": false`. PR #3333 (`71d98b4e`) replaced the previous real method-call retention idiom at the two `sessionsReplicationInstance` import sites with: ```ts import { sessionsReplicationInstance } from "..."; void sessionsReplicationInstance; ``` esbuild treats `void <identifier>;` as a pure expression statement under `sideEffects: false` and tree-shakes the entire import — including the `singleton(...)` call inside `sessionsReplicationInstance.server.ts` which is the only thing that fires `initializeSessionsReplicationInstance()`. The sessions→ClickHouse logical replication worker never starts, the slot is unconsumed, lag grows. ### How it manifested in production cloud#907's image bump rolled the `SessionReplicationService` ECS task on prod at 14:32 UTC. The new container's startup log emitted `🗃️ Runs replication service enabled` but not `🗃️ Sessions replication service enabled` or `🗃️ Sessions replication service started`. CloudWatch `OldestReplicationSlotLag` grew at ~220 MB/min and the `High replication lag` alarm fired at 14:37 UTC. Prod was reverted to the previous image (cloud#910) to stop the bleed. ### Verification `grep` of the built bundle `apps/webapp/build/index.js` (built from `c0365d36`): - 3 occurrences of `Runs replication` / `runsReplicationInstance` strings ✅ - 0 occurrences of `Sessions replication` / `sessionsReplicationInstance` / `SessionsReplicationService` ❌ The runs path survives tree-shaking because `adminWorker.server.ts` and `admin.api.v1.runs-replication.` routes have real method calls (`.start()`, `.teardown()`, `.backfill()`) — observable uses the tree-shaker must preserve. The sessions singleton has no real callers, only the `void` no-ops, hence its complete elimination from the bundle. ## Fix Replace `void sessionsReplicationInstance;` with an assignment to `globalThis`, an unambiguous observable side effect the bundler cannot eliminate: ```ts (globalThis as Record<string, unknown>).__sessionsReplicationInstance = sessionsReplicationInstance; ``` Applied at both call sites: `apps/webapp/app/entry.server.tsx` and `apps/webapp/app/v3/services/adminWorker.server.ts`. Surrounding comments updated to document the bundler interaction so the next maintainer doesn't reintroduce `void`. ## Out of scope (follow-ups) - Robustness improvement: change `apps/webapp/package.json` from `"sideEffects": false` to an allowlist that includes `Instance.server.ts` files. Prevents the same regression shape via any future `Instance` singleton. - Build-time check*: add a `grep` post-build step in `publish.yml` requiring `"Sessions replication"` to appear in `apps/webapp/build/index.js`. Catches this exact regression at CI time. ## Test plan - [x] `pnpm run typecheck --filter webapp` clean - [ ] After merge + publish: confirm new image's `SessionReplicationService` container logs `🗃️ Sessions replication service enabled` and `🗃️ Sessions replication service started` at startup - [ ] After re-deploying to prod: confirm `OldestReplicationSlotLag` stops growing and drains	2026-05-24 16:21:15 +01:00
Daniel Sutton	1015876b98	feat(webapp): user-based Sentry attribution with tenant tags (#3678 ) ## Summary Stamp every Sentry event with the signed-in user and the tenant (org / project / env) the request belongs to, so "Users Impacted" counts distinct humans and events become filterable per tenant. Design after review (current): - `user.id = real user cuid` (from `requireUser`). "Users Impacted" counts humans, not tenants. - Tenant context (org / project / env slugs, IDs, env type) moves entirely onto tags: `org_slug`, `project_slug`, `env_slug`, `org_id`, `project_id`, `project_ref`, `environment_id`, `env_type`, plus `impersonating` when set. - Backed by an `AsyncLocalStorage` scope established at the HTTP entry. Each entry point fills what it knows; loaders enrich the same scope with what they already have. Zero new database queries. The middleware does a regex match only. Dashboard loaders that already query Prisma gain a couple of extra selected columns; nothing new round-trips. ## How it's wired - Express middleware (`tenantContextResolver.server.ts`) — parses the URL with a regex and always opens an ALS scope. Populates whatever subset of slugs is present: `/orgs/:o` → just `orgSlug`; `/orgs/:o/projects/:p` adds `projectSlug`; the full triple adds `envSlug`. Non-tenant paths get an empty scope so loaders can still enrich. - `_app/route.tsx` — already calls `requireUser`. Adds `tenantContext.enrich({ userId: user.id })` for every authenticated dashboard request. No new query. - Env layout loader (`_app.orgs.$o.projects.$p.env.$e/route.tsx`) — its existing `prisma.project.findFirst` gains two columns in `select` (`externalRef`, `organization.id`). After it picks an env, calls `tenantContext.enrich({ orgId, projectId, projectRef, envId, envType })`. Same query, +2 columns. - API path (`apiBuilder.server.ts`) — wraps every handler in `tenantContext.run(tenantContextFromAuthEnvironment(authenticationResult.environment), …)`. The mapper pulls `userId` from `env.orgMember?.userId` (already selected by `authIncludeBase` — no schema change). Covers `createLoaderApiRoute`, `createActionApiRoute`, and `createMultiMethodApiRoute`. - Event processor (`sentryTenantContext.server.ts`) — registered in `entry.server.tsx` so it lives in the Remix bundle and shares the same `tenantContext` ALS instance as the middleware and loaders. Stamps whatever's present; nothing forced. ## Example events from local verification \| URL \| `user.id` \| Tags \| \|-----\|-----------\|------\| \| `/orgs/:o/projects/:p/env/:e/...` \| real user cuid \| `org_slug`, `project_slug`, `env_slug`, `org_id`, `project_id`, `project_ref`, `environment_id`, `env_type` \| \| `/orgs/:o/settings` (non-env-scoped) \| real user cuid \| `org_slug` only \| \| API request with `orgMember` \| `orgMember.userId` \| full tenant set \| \| API request without `orgMember` \| (unset) \| full tenant set \| ## Trade-offs 1. On env-scoped pages, errors that fire before the env layout loader's enrich callback runs get slugs + `user.id` but not the tenant IDs / `env_type`. Realistic errors deep in async work get the full set. (Same race as before, narrower window now that slugs/`user.id` are populated up-front by the middleware and `_app` enrich.) 2. API requests where the environment has no `orgMember` get tenant tags but no `user.id`. Those events still show in the issue but don't contribute to "Users Impacted". ## Out of scope (deferred) Background workers (`redis-worker`, `schedule-engine`) and socket handlers. Those entry points don't set `tenantContext.run` yet — their events ship without tenant attribution until each is wired in a follow-up. ## Tests 31 unit tests across 4 files. New tests notably cover: - `parseTenantPath`: org-only, org+project, and full-triple URL variants. - `tenantContext.enrich`: in-place patch, no-op outside `run()`, concurrent-scope isolation, empty-scope + enrich pattern (for non-tenant pages). - `tenantContextFromAuthEnvironment`: with and without `orgMember` — verifies the API path's `user.id` mapping. - `addTenantContextToEvent`: empty scope, userId-only, slugs-only, full enrichment, conditional tag emission, preservation of prior `event.user` fields. ## Test plan - [ ] `pnpm run typecheck --filter webapp` - [ ] `pnpm run test --filter webapp -- test/tenantContext.test.ts test/sentryTenantContext.test.ts test/tenantContextResolver.test.ts test/tenantContextFromAuthEnvironment.test.ts` - [ ] Local manual: with `SENTRY_DSN` set, hit a dashboard URL and an API route, confirm the captured events carry `user.id` + the expected tag set in Sentry. - [ ] After ship: confirm "Users Impacted" on a real Sentry issue reflects distinct users (not tenants). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-05-22 15:10:41 +01:00
Matt Aitken	71d98b4e6b	Support for org-scoped ClickHouse (#3333 ) Added `OrganizationDataStore` which allows orgs to have data stored in specific separate services. For now this is just used for ClickHouse. When using ClickHouse we get a client for the factory and pass in the org id. Particular care has to be made with two hot-insert paths: 1. RunReplicationService 2. OTLPExporter --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Claude <noreply@anthropic.com>	2026-05-22 14:43:03 +01:00
Daniel Sutton	906d5fafb6	feat(mollifier): trigger burst smoothing — Phase 1 (monitoring) (#3614 ) ## Summary - Introduce the Mollifier: a Redis-backed buffer for `trigger()` API calls during traffic spikes, with a per-env trip evaluator and a drainer ack-loop. - Phase 1 is dual-write monitoring — every mollified trigger is buffered to Redis AND continues to `engine.trigger`. No customer-facing behaviour change. - Telemetry events: `mollifier.would_mollify`, `mollifier.buffered`, `mollifier.drained`, plus the `mollifier.decisions` counter. - Gated behind a feature flag (default off). ## Test plan - [x] `pnpm run test --filter @trigger.dev/redis-worker` - [x] `pnpm run test --filter webapp -- mollifier` - [x] Manual: with flag off, no behaviour change vs main - [x] Manual: with flag on + threshold lowered, observe `mollifier.buffered` + `mollifier.drained` log pairs with matching `runId` --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-18 14:58:38 +01:00
Eric Allam	c69e939c34	feat: Sessions - bidirectional durable agent streams (#3417 ) > ⚠️ Not released yet. This PR is the server-side foundation only. The SDK changes that customers will actually use (`chat.agent` migration, `chat.createStartSessionAction`, `useTriggerChatTransport` updates) live on a separate branch and ship together in an upcoming `@trigger.dev/sdk` prerelease. Until that prerelease is published, this surface is reachable only via direct HTTP. ## What this gives Trigger.dev users A new first-class primitive, Session, for durable, task-bound, bidirectional I/O that outlives any single run. Sessions are the run manager for `chat.agent` going forward, and they unblock anything else that needs "one identifier, many runs over time" with a stable channel pair the client can write to and subscribe to. ### Use cases unblocked - Chat agents that persist across many runs. One session per chat (keyed on your own `chatId` via `externalId`), turns 1..N attach to the same Session, the UI subscribes once and keeps receiving output as new runs take over. - Approval loops and long-running tasks with user feedback. The task waits on `.in`, the client writes to `.in`, the server enforces no-writes-after-close. - Workflow progress streams that live past the run. Subscribe to `.out` after the task finishes to replay history. - Resume-next-day flows. A session is a durable row, not a transient stream. Send a message a day later and the server triggers a fresh run on the same session. ### How it works (Session-as-run-manager) A Session row is task-bound (`taskIdentifier` + `triggerConfig` are required) and owns its current run via `currentRunId` + `currentRunVersion` for optimistic claim. Three trigger paths: 1. Session create — `POST /api/v1/sessions` creates the row and triggers the first run synchronously. 2. Append-time probe — `POST /realtime/v1/sessions/:session/in/append` checks if the current run is alive; if it has terminated (idle exit, crash, etc.), the server triggers a new run before processing the append. 3. End-and-continue handoff — `POST /api/v1/sessions/:session/end-and-continue`, called by the running agent, triggers a fresh run and atomically swaps `currentRunId`. Used by `chat.requestUpgrade()` for version handoffs. Every triggered run is recorded in the `SessionRun` audit table with a reason (`initial`, `continuation`, `upgrade`, `manual`). ## Public API surface ### Control plane - `POST /api/v1/sessions` — create. Idempotent on `(env, externalId)`. Triggers the first run, returns the session and a session-scoped public access token. Returns 409 if the upserted row is already closed. - `GET /api/v1/sessions/:session` — retrieve by friendlyId (`session_abc...`) or by your own externalId (server disambiguates by prefix). - `GET /api/v1/sessions` — list with filters (`type`, `tag`, `taskIdentifier`, `externalId`, derived `status` ACTIVE/CLOSED/EXPIRED, created-at range) and cursor pagination. Backed by ClickHouse. - `PATCH /api/v1/sessions/:session` — update tags / metadata / externalId. - `POST /api/v1/sessions/:session/close` — terminate. Idempotent, hard-blocks new server-brokered writes. - `POST /api/v1/sessions/:session/end-and-continue` — agent-only handoff to a fresh run. ### Realtime - `PUT /realtime/v1/sessions/:session/:io` — initialize a channel. Returns S2 credentials in headers so high-throughput clients can write direct to S2. - `GET /realtime/v1/sessions/:session/:io` — SSE subscribe. Supports Last-Event-ID resume and an opt-in `X-Peek-Settled: 1` header that fast-closes the stream when the upstream is already settled (`trigger:turn-complete`), eliminating long-poll wait on reconnect-on-reload paths. - `POST /realtime/v1/sessions/:session/:io/append` — server-side appends. - `POST /api/v1/runs/:runFriendlyId/session-streams/wait` — runs wait on a session stream as a waitpoint, with a race-check to avoid suspending if data already landed. ### Auth scopes `sessions` is a new resource type. `read:sessions:{id}`, `write:sessions:{id}`, `admin:sessions:{id}` flow through the existing JWT validator. Session-scoped public access tokens minted by the server replace browser-held trigger-task tokens for chat-style flows — the browser never sees a run identifier or a run-scoped token in steady state. ## What's coming after this PR - SDK + chat.agent migration: separate branch, separate PR, ships in the next `@trigger.dev/sdk` prerelease alongside this server deploy. Customers using the prerelease `chat.agent` will follow the [upgrade guide](https://github.com/triggerdotdev/trigger.dev/blob/docs/tri-7532-ai-sdk-chat-transport-and-chat-task-system/docs/ai-chat/upgrade-guide.mdx). - Dashboard surfaces: dedicated agent list, agent playground, agent view on the run dashboard. Tracking separately. ## Implementation notes - Postgres `Session` table: scalar scoping columns (`projectId`, `runtimeEnvironmentId`, `environmentType`, `organizationId`) without FKs, matching the January TaskRun FK-removal decision. Point-lookup indexes only — list queries go to ClickHouse. Terminal markers (`closedAt`, `expiresAt`) are write-once. - ClickHouse `sessions_v1`: ReplacingMergeTree, partitioned by month, ordered by `(org_id, project_id, environment_id, created_at, session_id)`. Tags indexed via `tokenbf_v1` skip index. - `SessionsReplicationService`: mirrors `RunsReplicationService` exactly — leader-locked logical replication consumer, `ConcurrentFlushScheduler`, retry with exponential backoff + jitter, identical metric shape. Dedicated slot + publication so the two consume independently. - S2 keys: `sessions/{addressingKey}/{out\|in}`. The existing `runs/{runId}/{streamId}` key format for run-scoped streams is untouched. - Optimistic claim: `ensureRunForSession` triggers a run upfront (cheap to cancel if it loses the race), then attempts an `updateMany` keyed on `currentRunVersion`. Loser cancels its triggered run and reuses the winner's. No DB lock held across the trigger. ### What did NOT change Run-scoped `streams.pipe` / `streams.input` and the existing `/realtime/v1/streams/{runId}/...` routes are unchanged. Sessions are net-new — not a reshaping of the current streams API. ## Deploy notes - Set `SESSION_REPLICATION_CLICKHOUSE_URL` and `SESSION_REPLICATION_ENABLED=1` to enable the replication consumer. - The `Session` table needs `REPLICA IDENTITY FULL` set on the prod source DB before the publication is created (same one-time DDL we did for `TaskRun`). Required for delete events to carry full column values. - Cross-form authorization on the `GET /api/v1/sessions/:session` loader (a JWT minted for either form authorizes both URL forms). Action routes are URL-form-specific, matching how the SDK mints PATs. ## Verification - Webapp typecheck clean (10/10). - `apps/webapp/test/sessionsReplicationService.test.ts` — round-trip tests for insert/update/delete through Postgres logical replication into ClickHouse via testcontainers. - Live end-to-end against local dev: create + retrieve (both forms) + update + close, `.out.initialize` + `.out.append` x2 + `.in.send` + `.out.subscribe` over SSE, list with all filter combinations + pagination, `end-and-continue` swap, `X-Peek-Settled` fast-close (verified in browser via reconnect-on-reload and via curl). Replicated row lands in ClickHouse within ~1s. - Multi-round Devin + CodeRabbit review feedback addressed (read-after-write paths use `prisma` writer, info-leak on auth-routes masked as 403, peek-settled discriminator parsing fix, etc.). ## Test plan - [ ] `pnpm run typecheck --filter webapp` - [ ] `pnpm run test --filter webapp ./test/sessionsReplicationService.test.ts --run` - [ ] Start the webapp with `SESSION_REPLICATION_CLICKHOUSE_URL` and `SESSION_REPLICATION_ENABLED=1`. Confirm the slot and publication auto-create on boot. - [ ] `POST /api/v1/sessions` and verify the row replicates to `trigger_dev.sessions_v1` within a couple of seconds. - [ ] `POST /api/v1/sessions/:id/close`, then confirm `POST /realtime/v1/sessions/:id/out/append` returns 400. - [ ] Reuse a closed session's `externalId` on `POST /api/v1/sessions` and confirm 409. - [ ] `GET /realtime/v1/sessions/:id/out` with `X-Peek-Settled: 1` after a turn completes and confirm `X-Session-Settled: true` response header + immediate close.	2026-04-28 12:35:55 +01:00
Eric Allam	486f49791d	fix(webapp): eliminate SSE abort-signal memory leak (#3430 ) ## Summary Fixes a server-side memory leak in the webapp's SSE helper. Every aborted SSE connection (client tab close, navigation, timeout) was pinning its full request/response graph indefinitely on Node 20, so any long-running webapp process accumulated retained memory proportional to streaming-request churn. ## Root cause `apps/webapp/app/utils/sse.ts` combined four abort signals via `AbortSignal.any([requestAbortSignal, timeoutSignal, internalController.signal])`. The composite signal tracks its source signals in an internal `Set<WeakRef>` registered against a `FinalizationRegistry`; under sustained traffic those entries accumulate faster than they're cleaned up, pinning every source signal (and its listeners, and anything those listeners close over) until the parent signal itself is GC'd or aborts. This is a long-standing Node issue with multiple open reports: - [nodejs/node#54614](https://github.com/nodejs/node/issues/54614) — original report, still open. A [follow-up from ChainSafe](https://github.com/nodejs/node/issues/54614#issuecomment-4055656572) describes the exact same shape in a Lodestar production workload (req + timeout signals composed per request accumulating in long-running worker) and the same mitigation: drop `AbortSignal.any`, compose manually. - [nodejs/node#55351](https://github.com/nodejs/node/issues/55351) — mechanism confirmed by Node member @jasnell: "the set of dependent signals known to the AbortSignal are kept in an internal Set using WeakRefs. The AbortSignals are being properly gc'd but the Set is never cleaned out of the WeakRefs making those leak." Partially fixed by [PR #55354](https://github.com/nodejs/node/pull/55354), shipped in Node 22.12.0 — but only covers the tight-loop case, not long-lived parent signals. - [nodejs/node#57584](https://github.com/nodejs/node/issues/57584) — circular-dependency variant, still open. - [nodejs/node#62363](https://github.com/nodejs/node/issues/62363) — regression in Node 24/25 from an unrelated V8 change ("Don't pretenure WeakCells"). Different root cause, same symptom. A separate issue in `apps/webapp/app/entry.server.tsx` — `setTimeout(abort, ABORT_DELAY)` with no `clearTimeout` on success paths — kept the React render tree + `remixContext` alive for 30s per successful HTML request. Same pattern fixed upstream in React Router templates ([react-router#14200](https://github.com/remix-run/react-router/pull/14200)), never backported to Remix v2. ## What changed - `apps/webapp/app/utils/sse.ts` — single-signal abort chain. `AbortSignal.any` removed; `AbortSignal.timeout` replaced by a plain `setTimeout` cleared when the controller aborts; named sentinel constants used as stackless abort reasons; request-abort handler explicitly removed on cleanup. - `apps/webapp/app/entry.server.tsx` — clears the `setTimeout(abort, ABORT_DELAY)` timer in `onShellReady` / `onAllReady` / `onShellError`. - `apps/webapp/app/v3/tracer.server.ts` + `env.server.ts` — gates OpenTelemetry `HttpInstrumentation` and `ExpressInstrumentation` behind `DISABLE_HTTP_INSTRUMENTATION=true` as an escape hatch for future OTel-listener retention patterns. Defaults to enabled. - `apps/webapp/app/presenters/v3/RunStreamPresenter.server.ts` — uses the shared `ABORT_REASON_SEND_ERROR` sentinel. ## Verification ### Full-app reproduction (memlab) Isolated local harness, 500 abrupt SSE disconnects against a dev-presence route, GC between passes, heap snapshot diff with [memlab](https://facebook.github.io/memlab/): \| Run \| Heap delta after 500 conns + GC \| memlab retained leaks \| \| --- \| --- \| --- \| \| Before \| +16.0 MB (linear with request count) \| 158 clusters; 250 `ServerResponse`, 1000 `AbortController`, 250 `SpanImpl` retained \| \| After \| +3.3 MB (noise) \| 0 app-code leaks \| ### Standalone mechanism isolation To confirm which axis of the change is load-bearing, a separate standalone Node script (`/tmp/abort-leak-test.mjs`) ran 2000 requests × 200 KB payload per variant: \| Variant \| Heap delta after GC \| \| --- \| --- \| \| baseline (no signal machinery) \| 0 MB \| \| V1: `AbortSignal.any` + string abort reason \| +9.1 MB \| \| V2: `AbortSignal.any` only (no reason) \| +10.8 MB \| \| V3: string reason only (no `AbortSignal.any`) \| 0 MB \| \| V4: neither (the fix) \| 0 MB \| \| V5: `AbortSignal.any` with no listener on the composite \| +10.2 MB \| This proves `AbortSignal.any` is the sole mechanism. The reason type (`.abort()` vs `.abort("string")`) is irrelevant for retention — V3 is clean, V5 leaks even without a listener on the composite. ## Risk - `sse.ts` is used by the dev-presence routes. Behaviour is equivalent — timeouts and client disconnects still abort the stream. `signal.reason` is now a named string sentinel (`"timeout"`, `"request_aborted"`, etc.) instead of the previous string arg or default `AbortError`. No in-tree reader of `signal.reason` exists. - `entry.server.tsx` change is a standard cleanup of an abort timer, matches upstream React Router guidance. - `tracer.server.ts` change is env-gated and defaults to current behaviour. - Three other webapp `AbortSignal.timeout()` callsites (alert delivery, remote-build status) are fire-and-forget passed directly to `fetch` — not composed with anything long-lived, no retention risk, untouched. ## Test plan - [ ] Existing SSE integration tests pass - [ ] Dev-presence SSE behaves normally across tab open/close cycles - [ ] No heap growth under sustained aborted-connection traffic (heap snapshot diff) ## Follow-up The same `AbortSignal.any([userSignal, internalSignal])` pattern exists in several SDK/core callsites that ship to customers (`packages/core/src/v3/realtimeStreams/manager.ts`, `packages/trigger-sdk/src/v3/{ai,chat,chat-client,sessions}.ts`, `packages/core/src/v3/workers/warmStartClient.ts`). Whether those leak in practice depends on the user passing a long-lived signal. Tracked separately.	2026-04-23 15:54:05 +02:00
Eric Allam	a999d9ea3f	feat(engine): Batch trigger reloaded (#2779 ) New batch trigger system with larger payloads, streaming ingestion, larger batch sizes, and a fair processing system. This PR introduces a new `FairQueue` abstraction inspired by our own `RunQueue` that enables multi-tenant fair queueing with concurrency limits. The new `BatchQueue` is built on top of the `FairQueue`, and handles processing Batch triggers in a fair manner with per-environment concurrency limits defined per-org. Additionally, there is a global concurrency limit to prevent the BatchQueue system from creating too many runs too quickly, which can cause downstream issues. For this new BatchQueue system we have a completely new batch trigger creation and ingestion system. Previously this was a single endpoint with a single JSON body that defined details about the batch as well as all the items in the batch. We're introducing a two-phase batch trigger ingestion system. In the first phase, the BatchTaskRun record is created (and possibly rate limited). The second phase is another endpoint that accepts an NDJSON body with each line being a single item/run with payload and options. At ingestion time all items are added to a queue, in order, and then processed by the BatchQueue system. ## New batch trigger rate limits This PR implements a new batch trigger specific rate limit, configured on the `Organization.batchRateLimitConfig` column, and defaults using these environment variables: - `BATCH_RATE_LIMIT_REFILL_RATE` defaults to 10 - `BATCH_RATE_LIMIT_REFILL_INTERVAL` the duration interval, defaults to `"10s"` - `BATCH_RATE_LIMIT_MAX` defaults to 1200 This rate limiter is scoped to the environment ID and controls how many runs can be submitted via batch triggers per interval. The SDK handles the retrying side. ## Batch queue concurrency limits The new column `Organization.batchQueueConcurrencyConfig` now defines an org specific `processingConcurrency` value, with a backup of the env var `BATCH_CONCURRENCY_LIMIT_DEFAULT` which defaults to 10. This controls how many batch queue items are processed concurrently per environment. There is also a global rate limit for the batch queue set via the `BATCH_QUEUE_GLOBAL_RATE_LIMIT` which defaults to being disabled. If set, the entire batch queue system won't process more than `BATCH_QUEUE_GLOBAL_RATE_LIMIT` items per second. This allows controlling the maximum number of runs created per second via batch triggers. ## Batch trigger settings - `STREAMING_BATCH_MAX_ITEMS` controls the maximum number of items in a single batch - `STREAMING_BATCH_ITEM_MAXIMUM_SIZE` controls the maximum size of each item in a batch - `BATCH_CONCURRENCY_DEFAULT_CONCURRENCY` controls the default environment concurrency - `BATCH_QUEUE_DRR_QUANTUM` how many credits each environment gets each round for the DRR scheduler - `BATCH_QUEUE_MAX_DEFICIT` the maximum deficit for the DRR scheduler - `BATCH_QUEUE_CONSUMER_COUNT` how many queue consumers to run - `BATCH_QUEUE_CONSUMER_INTERVAL_MS` how frequently they poll for items in the queue ### Configuration Recommendations by Use Case High-throughput priority (fairness acceptable at 0.98+): ```env BATCH_QUEUE_DRR_QUANTUM=25 BATCH_QUEUE_MAX_DEFICIT=100 BATCH_QUEUE_CONSUMER_COUNT=10 BATCH_QUEUE_CONSUMER_INTERVAL_MS=50 BATCH_CONCURRENCY_DEFAULT_CONCURRENCY=25 ``` Strict fairness priority (throughput can be lower): ```env BATCH_QUEUE_DRR_QUANTUM=5 BATCH_QUEUE_MAX_DEFICIT=25 BATCH_QUEUE_CONSUMER_COUNT=3 BATCH_QUEUE_CONSUMER_INTERVAL_MS=100 BATCH_CONCURRENCY_DEFAULT_CONCURRENCY=5 ```	2025-12-16 14:32:49 +00:00
Eric Allam	128bc437f6	feat(otel): Add support for storing run spans and log data in Clickhouse (#2567 )	2025-10-01 12:41:18 -07:00
Eric Allam	71693eb9b3	stop double reporting errors to sentry (#2312 )	2025-07-24 16:02:26 +01:00
Eric Allam	09d0e80804	Add sentry error reporting (#2309 ) * Sentry WIP * Configure sentry for uploading and releasing during the publish webapp step * Delete source maps after uploading * Forward logger.error calls to sentry through Logger.onError * Couple tweaks to the dockerfile	2025-07-24 15:09:50 +01:00
nicktrn	e7795a06ad	Fix: fixes and prerequisites for v4 self-hosting (#2150 ) * remove pgadmin * remove V3_ENABLED * v3 is always enabled * enfore docker machine presets by default * rename autoremove env var * prefix more k8s-specific env vars * same prefix for all docker settings * improve profile switcher copy * supervisor can load token from file * optional webapp worker group bootstrap * fix error message * fix app origin fallback for otlp endpoint * use pnpm cache for webapp docker builds * increase default org and env concurrency limit to 100 * optional machine preset overrides * improve s3 pre-signing errors * fix DOCKER_ENFORCE_MACHINE_PRESETS bool coercion * shard unit tests * fix for s3-compatible services * optional object store region * Update apps/supervisor/src/workerToken.ts Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> * fix DEPLOY_REGISTRY_HOST example * fix platform mock * remove remaining v3Enabled refs * fix error type.. bad bot --------- Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>	2025-06-04 18:02:38 +01:00
nicktrn	f603725393	Feat: unified deploys for self-hosted and cloud users incl. multi-platform support (#2138 ) * remove registry proxy * remove --self-hosted flag * automatically set network build flag * update syncEnvVars debug log * improve switch command * always display deploy errors if they exist * fix stuck deploy command after finalize error * webapp-driven deploys, multi-platform support, lots of fixes * add worker deployment migration * rename image platform env var * only try to sync parent env vars for preview deployments * add KEEP_TMP_DIRS * supervisor: docker api version lock, auth, multi-platform * set image ref on create, validate digest * use metadata for digest, fix local multi-platform builds * print git meta branch before commit * improve push and load flag handling * make runs after local builds compatible with load and push * small improvement for platform overrides * add image platform to dequeued message * remove deprecated init request body fields * fix fail deployment id param * remove build debug logs * pass report merge with no tests * structured run debug logs * add required env var for tests * should not be an error log * add changeset	2025-06-04 15:54:10 +01:00
Matt Aitken	6cf86d5916	Deleting v2 dashboard/api (#1628 ) * Delete v2 Stripe routes * Delete v2 billing/usage pages * Delete v2 integration pages * Delete v2 project pages * Deleted a load of components and services * Deleted a load more components, presenters and services * Deleted another 100 files or so… * Removed old v2 paths * Removed named icons from form titles * Removed more string icons * Delete NamedIcon * Fixed some type errors * Delete endpointApi * Removed v2 from core/sdk * Post merge fixes * added explicit return types * using the new sdk export without v3 * Delete old v2 file * Added explicit return types because TS was complaining… * Don’t export RuntimeEnvironmentType from two core files. Was causing TS issue * Fix for removal of NamedIcon in new route * Removed strange eslintrc rule * Use the new redis client --------- Co-authored-by: James Ritchie <james@trigger.dev>	2025-03-08 14:53:06 +00:00
Eric Allam	d855d55ea0	re2: env based queue selection algo (#1775 ) * re2: fix @trigger.dev/core exports * re2: WIP env based queue selection algo * more wip * WIP * Get run engine tests to pass * Adding tests for the fair dequeueing strat in the run engine * Configure the new queue selection strategy in the webapp and get it all building and typechecks passing * webapp now uses built packages, building redis-worker, run-engine, database, using better tsconfig setups for tests, moving isomorphic code into core/v3/isomorphic * Fixed webapp typechecks * dev now depends on build, fixed supervisor typecheck * Fixed run engine tests * Fixed e2e tests	2025-03-07 14:30:19 +00:00
nicktrn	e97704d904	Run Engine 2.0 (WIP) (#1575 ) * bump worker version * Suggested glossary for the RunEngine, TBC * Removed BatchTaskRun changes from this branch, they were done in main * Set the BatchTaskRun status to completed when all runs are completed * When dequeuing respect passed in maxResources * Ported over the new run props: idempotencyKeyExpiresAt, versions, oneTimeUseToken, maxDurationInSeconds * Didn’t hit save… the new props when triggering tasks passed through * Idempotency expiration + waitpoint edge case * WIP on creating checkpoint, parking for now * fix worker routes * upgrade webapp node types to support generic event emitter * separate event bus handler singleton and run failure alerts * duration waits * fix execution snapshot debug spans * task waits * fix event bus types * temporary fix for react hook run handle type * disable run notifications for now * convert any typecasts to expect errors to more easily fix later * fix webapp types after node types upgrade * updateEnvConcurrencyLimits across marqs and the runqueue * Pass proper values into the run engine * RunQueue settings and removed unused rebalancing workers * Remove rebalancing prop * Tidied more things up * Update/remove queue limits for MARQS and RunQueue * taskQueue/concurrencyLimit changes ported back into the RunEngine * Reworked completing waitpoints to improve performance and reduce race conditions * Improved test robustness * Down to a single run lock only when a run is totally unblocked and ready to continue * warm starts, worker notifications, wait fixes * Fix for Run Engine poll interval env var * Expect the waitpoint to be completed quickly * If a run is locked then it’s too late to expire it * Added VALKEY_ env vars and plugged them into the run engine * Extracted and updated the guard queue function so it can be used when batching * Added logging and universal concurrency changes to trigger task v1 * Added notes back in * Bump @trigger.dev/worker to 3.3.7 * reportInvocationUsage for the runAttemptStarted event * improve execution snapshot span debug span start times * Unfriendly IDs * update lockfile * Created a shared determineEngineVersion function * disable unfinished commands * save new cli config to different location, misc fixes * add basic engine version check via current deploy * new run engine will default to node 22 runtime * block some actions for projects on previous run engine * fix worker group tests * fix triggerAndWait test * one typescript version to rule them all * redlock type patch * fix type issues caused by ts-reset * improve cleanup scripts * add missing socket.io dep * fix run notification handler type * fix worker group test again * generate prisma client for e2e tests * remove worker group tests for now * prevent image pull rate limits during unit tests * increase timeout for queue concurrency limit test * generate prisma client for preview release * same node types everywhere * Updated engine readme, removed legacy system notes * use default machine preset from platform package * worker instances plural in schema * disable pnpm update notifications * return worker group details from connect call * add workers admin route * fix heartbeat route return type * move deployment labels to core apps * refactor run controller env schema * Add firstAttemptStartedAt to TaskRun * RunEngine 2.0 batch trigger support (#1581) * Make it clear when BatchTriggerV2Service is used * Copy of BatchTriggerV2Service * WIP batch triggering * Allow blocking a run with multiple waitpoints at once. Made it atomic * Removed unused param * New batch service * Pass through the parentRunId and resumeParentOnCompletion * Use the new batch service, and correct trigger task version * Force V1 engine if using BatchTriggerV2Service, we’ve already done the check at this point * Removed the $transaction and early exit if nothing changed * Adedd a simple batch task to the hello world reference catalog * Fix for batch waits not working * Added parentRunId in a couple more places * Removed waitForBatch log * Added another parentRunId * Expanded the example to include all the different triggers * More changes to blocking to support continuing after idempotent completed runs * Fix for the wrong type when blocking a run * remove @map * optimise worker auth query * add engine version header to core api client requests * remove unique constraint for default group id * consolidate migrations * the first managed worker becomes the global default * Debug events off by default, added an admin toggle to show them * worker group name can't be an empty string * add exec helper to core * move machine resources to core * add pre-dequeue callback to determine max resources * optionally skip dequeue * bump worker package * move worker to core * fix ReadableStream type error * fix another type issue * update a few more tsconfigs * add metadata changes introduced in #1563 * Run Engine 2.0 trigger idempotency (#1613) * Return isCached from the trigger API endpoint * Fix for the wrong type when blocking a run * Render the idempotent run in the inspector * Event repository for idempotency * Debug events off by default, added an admin toggle to show them * triggerAndWait idempotency span * Some improvements to the reference idempotency task * Removed the cached tracing from the SDK * Server-side creating cached span * Improved idempotency test task * Create cached task spans in a better way * Idempotency span support inc batch trigger * Simplified how the spans are done, using more of the existing code * Improved the idempotency test task * Added Waitpoint Batch type, add to TaskRunWaitpoint with order * Pass batch ids through to the run engine when triggering * Added batchIndex * Better batch support in the run engine * Added settings to batch trigger service, before major overhaul * Allow the longer run/batch ids in the filters * Changed how batching works, includes breaking changes in CLI * Removed batch idempotency because it gets put on the runs instead * Added `runs` to the batch.retrieve call/API * Set firstAttemptStartedAt when creating the first attempt * Do nothing when receiving a BATCH waitpoint * Some fixes in the new batch trigger service… mostly just passing missing optional params through * Tweaked the idempotency test task for more situations * Only block with a batch if it’s a batchTriggerAndWait… 🤦‍♂️ * Added another case to the idempotency test task: multiple of the same idempotencyKey in a single batch * Support for the same run multiple times in the same batch * Small tweaks * Make sure to complete batches, even if they’re not andWait ones * Export RunDuplicateIdempotencyKeyError from the run engine * Latest lockfile * Trigger with a machine (old run engine) * RE2, allow setting machine when triggering * Fix for new glob patterns * add max run count to dequeue from version route * add worker instance name env var and header * queue consumer pre skip callback * poll for more runs after final execution errors * fix dequeue search param schema * add shortcut to debug switch * expose run engine timeouts as env vars * make warm start durations configurable * add optional status to json reply helper * fix preSkip hook, add debug logs * BLOCKED_BY_WAITPOINTS -> SUSPENDED * exit controller when run suspended * check if already replied before http reply * run controller will wait for next run after the current one is suspended * cancel run button shortcut * minimal event repository environment type * fix update metadata call * run suspension and misc fixes wip * change debug shortcut to shift + D * Started work on the Dev supervisor * Formatting * Fix for bad imports * Before rebuilding SSE * Presence updating from the CLI working via SSE * add worker notification debug logs * send run:stop when exiting run phase * skip current snapshot poll on worker notification * add more logs and route to submit run debug logs * add worker and runner ids to snapshots * improve run notification debug logs * add workload debug log route * misc run controller fixes and refactor * prevent parallel execution of critical functions * update bun to 1.2.1 * WIP with dev dequeuing * Method to convert friendlyIds to non-friendly, do nothing with actual ids * Set the engine on BackgroundWorker, lazily upgrade projects to engine V2 * Runs with ttls were getting immediately expired… oops. * Pass the Waiting for deploy reason through, so we have it on the execution snapshots * Fixed the logic for getting the right background worker for a run * Use the correct ID when dequeuing… * determineEngineVersion is now fully functional * Rate limiter ignores the dev endpoints * Retrieving a batch gives you the runIds * Set a unique version for the RE2 BatchTaskRun * add provisional changeset * The start of dev run execution is working * First dev run working * Moved the dev run controller closer to what Nick did with the managed one * export exec output type * Heartbeat fix: don’t heartbeat if _isHeartbeating == false * Dev runs get notifications, some dev bug fixes * Improved logging or dequeuing * We need to dequeue runs from the latest version too, for triggerAndWait * Ported Eric’s validateWorkerManifest with nicer errors * When flattening an idempotency key if part is undefined, return undefined * Dev logging fixes * Remove sigterm listener * Deprecating workers. Don’t specify a BackgroundWorker when dequeuing an environment * Deleted some old files. Renamed “managed” to “deploy” * When a build finishes, always copy the build dir (otherwise the first one gets trampled on by the 2nd) * Dev master queues should work differently * Deleting old workers * Added debounce function to core * Improvement to canceling * WIP on debounce canceling on socket disconnection * Added environment data to execution snapshots * Dev runs that have stalled get “Canceled” with a reason explaining why * Show CLI messaged when a connection to the platform is lost/restored * Fix TriggerTask after merge * Add trigger task v2 max attempts, replace some findUniques * Port the new queue logic to the run engine * More fixes post-merge * We weren’t setting a `retryConfig` up for the tests… it’s now required * Start the Redis worker inside the Run Engine… 🤦‍♂️ * Trying to make the testcontainers more reliable * Added keyPrefix: "engine:” * Badly placed bracket in trigger task * Better Redis namespacing * Fix for expired run not getting removed from the queue * Don’t create a redis client in the testcontainers, return the redisOptions instead * Cleanup redis client in the run lock tests * Fix for the RunQueue not supporting keyPrefix * Updated more of the RunQueue scripts rebalancing * Trying to make Redis more robust in the tests… * Improved test resiliciency more * Fix for delays (checkpoint check) * Increase the timeout slightly to fix ttl test * Added priority support when triggering * More wip trying to make test containers more reliable * batchTriggerAndWait test is still failing… some wip to try fix it * Fixed redis tests now we’re not providing a client * Separate Redis clients for the run engine worker/queue/runlock * Made the wait for duration test more resilient * Added idempotencyKeyExpiresAt to Waitpoints * Waitpoint timeouts and idempotency expiry * Use finishWaitpoint, removed extra worker job * Added waitpoint idempotency tests * Creating resume tokens is working * Some improvements to the resume tokens * Moved resumeTokens to just be wait functions 🥳 * Delete old RuntimeManagers * Wait for token is working * Better test for the wait tokens * Improved the test task some more * Hide the accessories in the span inspector * WIP on waitpoint inspector * WIP on complete waitpoint form * Span overview panel can be changed based on the entity type * Improved the waitpoint display * WIP on completing waitpoint form * Use the existing CodeBlock for the tip * Style improvements * Complete waitpoint * All waitpoint sidebar variants * Waits now use a pause icon * Durations waits use the API to create/block with a waitpoint, not the runtime * Fix for engine.blockRunWithWaitpoint required org id * Removed old wait code from the run controllers/task run process * Form action for skipping a datetime waitpoint * Move testDockerCheckpoint to a separate core package export (it can’t be bundled on the client) * Fix for glitchy hourglass animation * Completed waitpoints display better * Increase Redis maxRetriesPerRequest to 20 (default) * Completing and skipping waitpoints is working * Remove the database prisma dev command, since we need to use create only now. Updated docs * Added skip timeout, reworked the UI * Tweaked spacing * Added payload limit to waitpoint token completion from dashboard * Test idempotency works on wait.for and wait.until * Moved the worker-actions to /engine/ from /api/ * Moved dev engine endpoints to /engine/ from /api/ * Separate /engine/ rate limiter * Added parallel wait prevention, it’s working for duration waits but not well for triggerAndWait yet * WIP post-merge conflicts * Set taskEventStore column in the new engine * Remove duplicate keys * Post-merge fixes * Fix for span merge layout * Use executedAt instead of firstAttemptStartedAt --------- Co-authored-by: Matt Aitken <matt@mattaitken.com>	2025-03-05 14:40:14 +00:00
Dan	5164935f9c	Added X-Frame-Options and CSP (#1690 )	2025-02-28 11:06:27 +00:00
Eric Allam	e04d44866f	v3: better handle null unicode characters when creating task events and completing a failed attempt	2024-07-08 15:59:19 +01:00
Eric Allam	4f95c9de4e	v3: Cancel awaited subtasks and reliable rate-limit recovery (#1200 ) * v3: cancel subtasks when parent task runs are cancelled * v3: recover from server rate limiting errors in a more reliable way - Changing from sliding window to token bucket in the API rate limiter, to help smooth out traffic - Adding spans to the API Client core & SDK functions - Added waiting spans when retrying in the API Client - Retrying in the API Client now respects the x-ratelimit-reset - Retrying ApiError’s in tasks now respects the x-ratelimit-reset - Added AbortTaskRunError that when thrown will stop retries - Added idempotency keys SDK functions and automatically injecting the run ID when inside a task - Added the ability to configure ApiRequestOptions (retries only for now) globally and on specific calls - Implement the maxAttempts TaskRunOption (it wasn’t doing anything before) * Adding some docs about the request options * Fix type error * Remove context propagation through graphile jobs * Remove logger * only select a subset of task run columns * limit columns selected in batchTrigger as well * added idempotency doc * allow scoped idempotency keys, and fixed an issue with the unique index on BatchTaskRun and TaskRun * Removed old cancel task run children code	2024-07-05 10:30:52 +01:00
Eric Allam	ccbeff47e6	Improve trace view performance	2024-06-21 12:46:46 +01:00
Eric Allam	596bf78e55	Add an event loop monitor	2024-06-20 23:26:38 +01:00
nicktrn	1642fd7baf	perf: migrate to graphile worker v0.16.6 (#1097 ) * migrate to graphile worker v0.16.6 * remove stale docs link * fix jobs cleanup query	2024-05-10 18:07:16 +01:00
Eric Allam	e3cf456c69	v3: fix string and non-standard outputs (#983 ) * Fixed string outputs missing and incorrectly formatted * Handle non-serializable outputs like a function * Add changeset	2024-03-28 19:51:29 +00:00
Eric Allam	f93eae300e	v3: superjson dynamic import and deploy fixes (#982 ) * v3: Dynamically import superjson and fix some bundling issues * Added changeset * Better error handling in the registry proxy and catch uncaught exceptions and unhandled promise rejections instead of crashing the server * Await the prettyPrintPackage	2024-03-28 16:22:00 +00:00
Eric Allam	395abe1b92	v3: Various fixes to support papermark use-case (#970 ) * Adding a changeset for v3 * Add a version field to @trigger.dev/core-apps package.json * Build trigger.dev when doing a prerelease * bundle @trigger.dev/core-apps with trigger.dev cli * Fix the init command config template * Don’t use * for the @trigger.dev/core dep version specifier * strip workspace: from the package version before installing it * Added dependenciesToBundle config option to bundle ESM only packages * Added logging around resolving dependency paths * Try again * Resolve dependencies based on the project dir first * flip the bundled default * Adding some logs around dev task completion notifications * Adding some additional logs * Add more logs * Write out the log using process.stdout * Store pending completion notifications and resume them when awaited (fixes race condition) * Cleanup some of the logs * Copy over the postinstall step from the projects package.json * Add support for including additional files when deploying (e.g. prisma schema) * Don’t run scripts when resolving deps * copy all the files just in case anything is needed in postinstall * Remove duplicate option * Use the tag when outputting the dev command * Remove the postinstall script * add the trigger dir to the config if the default is not chosen * Remove the “hud” display in the dev command * trigger file names with dashes now work * Better file watching in dev * Much better duplicate ID experience now * Much better “Project not found” error * Export the handleError function types from sdk * Add support for configuring instrumentation * Upgrade and unify @opentelemetry/* packages (and remove storybook from the webapp) * Fix typescript error in react package * Upgrade react types in webapp * Allow span icons to be determined based on the span name (e.g. prisma:) * Ignore built-in env vars when checking for env vars, and allow continuing the deployment even if missing env vars were detected * Improve the retry.fetch default behavior and option structure * Fixed typescript errors with packages/email react types * Update the retry.fetch docs	2024-03-25 17:02:49 +00:00
Matt Aitken	b35eebb666	Rate limit API requests and changed SQS reading speed (#969 ) * Trying to use the @upstash/ratelimit package with ioredis… * WIP using the redis package instead * Revert the action back * Removed redis * Started refactoring * SQS setting for the poll interval. Set the default queue reading to be slower * API rate limiter as Express middleware * Organise imports * Fixed spelling mistake “limitter” * No authorization header response is problem+json	2024-03-25 16:39:20 +00:00
Eric Allam	c385db63be	v3: cli deploy command (#931 ) * WIP proxy/deploy * WIP, registry proxy in express working * A couple of notes, preparing for indexing * Move the changes to prod-worker into the new file * Deploy command working with indexing and runs (docker provider only for now) * Removed ts-expect-error directive * Fixed build command	2024-03-08 10:49:42 +00:00
nicktrn	52c9d485f8	v3: cli build command, prod runs, checkpoints (#919 ) * fix trailing slash in api url * add maybe platform down error * shorten config path * extend protobuf compiler install instructions * build command and image model * new image triggers task indexing * index support for prod builds * shorten example provider secret * multi-stage task prod build * lock prod tasks to node 18 image * extract shared build and dev command libs * pull out more shared deps * add coordinator and providers * fix core-apps build * add dev builds for new apps * fix cwd * enable corepack * some build fixes * enable buildkit for old docker versions * coordinator image fixes * update provider containerfile * build dev images in parallel * upgrade pgadmin * fix prod facade build * prod runs * fix merge * don't knock out platform on invalid attempt id * fix prod facade * rename to build.ts * fix prod builds * set to executing after fetching payload * make prod worker listen on random port if in use * remove experimental warnings in dev * prod resume * prevent execution after completion * exit prod worker after completion * always restart otel collector * docker checkpoints and prod runtime messaging * don't retry indexing without chance of success * make platform checkpoint aware * deploy with existing hash sets latest worker * log restore requests * only try to checkpoint long waits * tidying up * lockfile * fix build * prod worker merge fixes * fix prod complete and cancel * fix lua nil checks * socket namespace abstraction * cleanup * make all build args optional * add build script * don't require env vars for dev * fix schema * prod merge * small fix * bind correct logger * fix v3 ref catalog entry * resume prod batch * pass socket to error and disconnect handlers * fix non-batch resume * fix batch resume * send connection env vars when not in dev * create worker via socket * move api client back into v3 cli * fix lockfile * fix resume with failures * marqs replace message * typecheck prior to build * don't define api url in prod builds * support prod retries after resume * skip typecheck option	2024-03-04 13:10:07 +00:00
Eric Allam	baf3a84cda	v3 dev cli command + more (#894 )	2024-02-12 14:24:32 +00:00
Matt Aitken	2b48e6d04b	Feature: sendEvent can use an AWS SQS queue to increase reliability and throughput (#729 ) * Initial commit (by create-cloudflare CLI) * Changed the prettier rules * Started writing Readme and got dev working * Validate the request, parse the event and respond with the correct format * Use ulidx instead of ulid in core * Removed some boilerplate and use new function for sendEvent * Removed old ulid package * When handling a webhook, it’s not an error if the HTTP endpoint environment isn’t found * If the env vars aren’t set, log the request and pass it through * Pass requests through. Setup the env vars * Flipped the API key detection because we can enforce that it’s the shape of a private API key * Progress with sendEvent proxy * Events are being put on the SQS queue * The SQS event queue is working * Make sure there’s a timestamp before enqueuing the evnet * Log the sqs_event, not API key * Added a bit more detail to the readme * Improved the Cloudflare logs * We don’t need the global.window hack for AWS, just globalThis * Removed commented out wrangler.toml values * Reworked the proxy to make it easier to add more endpoints * Use json utility to send Responses * Start work on proxying bulk events	2023-11-13 15:08:01 +00:00
Eric Allam	c0f9f10b07	Upgrade to Remix v2.1.0	2023-10-26 15:54:21 +01:00
Liran Tal	e20fa3c2ec	fix: explicitly set a charset for rendered HTML documents (#355 )	2023-08-18 06:49:39 +01:00
Matt Aitken	9ff4c0dbd5	Project-wide Prettier setup (#237 ) * Setup project-wide prettier * Remove old workspace file * Remove old debugging directives * New top-level .prettierignore * Updated Prettier config settings * Contrubuting guide: Fix for some bad code blocks * Added more ignores * Improved the format script command * printWidth set to 100 * Formatted entire repo (pnpm run format)	2023-08-01 10:21:22 +01:00
Matt Aitken	da5d9779d4	Check if the Highlight id is defined before attempting to send the error	2023-07-06 21:03:20 +01:00

1 2

75 Commits