7707 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
nicktrn	b42e5c3771	fix(supervisor): count pods from a limit=1 list instead of an aggregate metric (#4442) ... The pod-count backpressure source read `apiserver_storage_objects{resource="pods"}` from an apiserver `/metrics` scrape. That gauge is a periodically-refreshed cached count, and it is served by whichever apiserver replica the scrape lands on — replicas disagree with each other at the same instant, by enough to swamp the engage/release hysteresis band. Engage and release timing was therefore partly a function of scrape routing. This replaces it with a single `limit=1` list of the workload namespace and computes `remainingItemCount + items.length`. One pod object transferred, no informer, no watch cache. Two request-shape constraints are load-bearing and called out in the code: passing a label or field selector makes the apiserver omit `remainingItemCount` entirely, and setting `resourceVersion` serves a cached count rather than a quorum read. Neither is passed. `remainingItemCount` is only set when the list is truncated, so `_continue` is the truncation signal — if it is absent the returned page is the whole collection and `items.length` is already exact. If the list *is* truncated and the count is missing or implausible, the fetcher throws rather than guessing. Failure semantics are unchanged: a throw lands in the monitor's existing catch, exactly as the previous parse did. The hysteresis, verdict shape, and gauge are untouched. RBAC is unchanged — the existing role already grants `pods: list`. The `/metrics` non-resource grant in the deployment role becomes unused, and the scrape-timeout env var is now a slight misnomer. Both left alone deliberately: the grant may be wanted again for other apiserver signals, and renaming the var would need a coordinated config change for no behavioural gain. Tests cover the not-truncated, truncated, missing-count, negative-count and timeout paths.	2026-07-31 19:45:37 +01:00
Eric Allam	f10bc23785	perf(run-engine,run-store): one execution snapshot per triggered run (#4419) ... A non-delayed run used to get two execution snapshots the moment it was triggered: `RUN_CREATED` nested in the run-create transaction, immediately followed by `QUEUED` from its own `BEGIN`/`INSERT`/`COMMIT`. It now gets a single `QUEUED` snapshot written inside the create, and the trigger path only publishes to the queue. One fewer row per run on `TaskRunExecutionSnapshot`, and one fewer round trip on the trigger hot path. `EnqueueSystem` gains a `publishRun` seam that enqueues without writing a snapshot. Every re-enqueue path (waitpoint resume, checkpoint restore, delayed enqueue, pending version, retry requeue) still calls `enqueueRun` and writes its own `QUEUED`, so only the first enqueue changes. The `QUEUED` snapshot still commits before the queue message, so a dequeue sees a dequeueable status exactly as before. Two things for reviewers. Nesting the write skips `createExecutionSnapshot`, which is what emits `executionSnapshotCreated` and therefore the run timeline's `[engine] QUEUED` entry, so the trigger path now emits it directly, the same way the dequeue and attempt-start paths already do for their nested creates. And `RUN_CREATED` is still written when a dequeued run has no background worker yet, so the status and both `statuses.ts` helpers stay live and existing rows keep reading correctly. Delayed runs are untouched: `DELAYED` then `QUEUED` are two genuinely different moments and stay two snapshots. Rollback is a revert. Create-and-enqueue happen in one request in one process, so no in-flight run needs both code paths to agree during a rollout. One note for whoever debugs this path later. The `QUEUED` snapshot now commits before the queue publish, so a failed publish leaves the run recorded as `QUEUED` with no queue message. That state was already reachable, since the publish was never part of the snapshot transaction, but it used to be recorded as `RUN_CREATED`, which was distinctive because it never otherwise persisted. `QUEUED` with no message is indistinguishable from a run waiting on a concurrency slot, so trigger-time publish failure is now one more cause of an apparently stuck queued run.	2026-07-31 16:12:07 +01:00
nicktrn	a91c08c731	fix(core): retry run start-attempt on transient connection errors (#4441) ... ## What `startRunAttempt` — the run controller's first call when a run starts — had no retry on transient connection errors. A brief connection blip on that call would abandon the start and send the run back through the queue, delaying its first attempt. This adds a jittered backoff retry, matching the existing `continueRunExecution` path with a shorter budget, so a transient blip is ridden out in place instead of bouncing the run. ## Why a shorter budget The continue path retries generously. Start-attempt keeps a tighter budget (6 attempts, ~25-40s jittered) so it rides out a transient blip but never keeps retrying past the point the run would already have been requeued. ## Safety Retrying is safe: start-attempt is guarded server-side by the snapshot id — a retry after a start has already committed is rejected, so it can never double-start an attempt. A pure connection error (the common case) never reached the server. ## Scope One retry-options object on `startRunAttempt`; no other behavior change. Warm starts share this path and get the same resilience.	2026-07-31 13:57:45 +00:00
Eric Allam	c72ebf9084	fix(webapp,run-engine): stop batchTriggerAndWait hanging when item streaming never completes (#4397) ... ## Summary `batchTriggerAndWait()` could leave a parent run waiting forever. The 2-phase batch API blocks the parent on the batch's waitpoint as soon as the batch is created, but the batch is only sealed at the end of item streaming. If streaming never completed, nothing sealed the batch, nothing completed the waitpoint, and the parent stayed suspended with no timeout and no way to recover. Supersedes #4016, which added the reaper alone. ## Fix Admission for item streaming was being decided twice. Batch creation passes its own rate limiter, which fixes `expectedCount` and blocks the parent, and then the item stream had to pass the general API limiter as well, competing with unrelated traffic. A second limiter could therefore veto work the first had already committed the parent to. Creation now mints a bounded grant that the item stream spends, so an admitted batch can finish streaming. The grant is capped per batch rather than exempting the path, and every failure mode (no grant, spent grant, unreachable store) falls back to the normal limiter. That makes stranding much rarer but not impossible, since a request timeout or a crash can still end streaming for good. So a seal-timeout reaper aborts any batch still unsealed after `BATCH_SEAL_TIMEOUT_MS` and completes the parent's waitpoint with an error, letting `batchTriggerAndWait()` reject instead of hang. It is race-safe against a late seal, and it is only scheduled for batches that actually block a parent, so fire-and-forget batches cost nothing. Finally, the batches page used to report "Batch completion checked." for these batches while doing nothing, because the completion path returns early on an unsealed batch. It now says the batch cannot be resumed. Rate limiting is no longer the reason a batch strands, so the reaper's default stays at 30 minutes, comfortably above the SDK's worst-case stream-retry budget. ## Verification Unit and container tests cover the grant cap, the bypass ordering (it runs after the authorization check, so it can never skip authentication), and the reaper's abort, seal race, idempotency, and no-waitpoint cases. Also verified end-to-end against a running stack. With the general limit exhausted, batch creation and other API calls returned 429 while a granted batch still streamed and sealed; an ungranted batch id was rate limited rather than bypassed; and the grant cut off exactly at its configured attempt count. Reproducing the stranded state on a real parent run, the batch was aborted at the timeout, the waitpoint completed with an error, and the parent resumed and finished instead of hanging. A parentless batch left unsealed was untouched well past the reaper window. ## Verified against deployed runs The reaper was proven end to end with a real deployed run (locally-run supervisor, containerised run) and a real network fault, rather than a simulated one: toxiproxy severs the phase 2 item stream mid-flight so every SDK stream retry genuinely fails, while phase 1 still succeeds. Only the batch calls traverse the fault, so control-plane traffic is untouched. The reproduction is the shape that actually strands a parent: the task catches the `BatchTriggerError` the SDK throws and carries on, so the phase 1 block outlives the thrown error and the parent hangs at its next suspension point. With the reaper disabled, the parent sat in `EXECUTING_WITH_WAITPOINTS` for over 24 minutes holding two blockers, and stayed stuck across a full infrastructure restart: ``` type | status | has_timeout BATCH | PENDING | f <- orphan, completedAfter NULL DATETIME | COMPLETED | t <- the wait already elapsed ``` With the reaper enabled the same task under the same fault completed in about 75 seconds with zero blockers left, the batch `ABORTED`, and its waitpoint completed carrying the error. Two conditions are required to observe this at all, which is worth knowing for any future test: the run must be deployed rather than `trigger dev` (dev runs execute in process and finish while still holding blocker rows), and the wait after the caught error must exceed the checkpoint threshold, or it is served in process and never suspends. ### Why completing the batch waitpoint is sufficient `batchTriggerAndWait` runs create, then stream, then wait. A phase 2 failure throws before the wait is ever reached, and the reaper only fires on an unsealed batch, so the parent is never suspended awaiting the batch when it runs. The parent therefore does not need a synthetic result, only to stop being blocked. Note this reasoning depends on that ordering: if the wait were ever reached with an unsealed batch, completing the batch waitpoint alone would not settle the caller. ## Follow-ups - Batches stranded before this ships still need a one-off recovery; the reaper only schedules at creation time. - That same property leaves a gap if the process dies between creating the batch and scheduling the job. A periodic sweep would close it, but wants a supporting index. - When a partially streamed batch aborts, children already enqueued keep running while the parent fails. Left as-is deliberately, since cancelling triggered work is a bigger semantic call.	2026-07-31 11:55:25 +01:00
claude[bot]	17d849b2d6	feat(cli): expose region option on the MCP trigger_task tool (#4439) ... <!-- ccr-slack-attribution --> _Requested by **Eric Allam** · [Slack thread](https://triggerdotdev.slack.com/archives/C0BEM9Z73TM/p1785491472104199)_ ## Checklist - [ ] I have followed every step in the [contributing guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md) - [x] The PR title follows the convention. - [ ] I ran and tested the code works --- ## Testing Static checks only, all clean: - `pnpm run typecheck --filter trigger.dev` - `pnpm run format` - `pnpm run lint` No live task was triggered against a running project, so the "ran and tested" box above is left unchecked. --- ## Changelog **Before:** triggering a task through the MCP server always ran it in the project's default region. There was no way to pick one. **After:** the `trigger_task` tool accepts an optional `region` option, so you can choose the region a run executes in. **How:** `region: z.string().optional()` was added to `TriggerTaskInput.options` in `packages/cli-v3/src/mcp/schemas.ts`. No call-site change was needed — `tools/tasks.ts` passes `options` through verbatim, and `TriggerTaskRequestBody.options.region` already existed. The tool description in `docs/mcp-tools.mdx` gained a matching line, and a patch changeset is included. There is no batch-trigger MCP tool, so there is no sibling tool to mirror this change on. --- ## Screenshots N/A — no UI changes. Co-authored-by: Claude <noreply@anthropic.com>	2026-07-31 11:54:42 +01:00
Wes Mason	efcb89ac26	fix(webapp): add hasAdminDisplayAccess to the env param test mock (#4430) ... `test/envParamRoute.ownership.test.ts` fails on main: 3 of its 4 tests throw ``` Error: [vitest] No "hasAdminDisplayAccess" export is defined on the "~/services/session.server" mock. Did you forget to return it from "vi.mock"? ``` #4421 added a `hasAdminDisplayAccess(user)` call to the `env.$envParam` loader, and the test's `vi.mock` of `session.server` only returns `requireUser`, so the call blows up. Both changes were green in their own PR and only conflict once merged together, which is why nobody caught it. The mock now mirrors the real implementation rather than returning a constant, so it stays correct if the test's user fixture is ever varied. No assertions were changed: the tests were right, the mock was stale. Worth flagging separately: no workflow runs on push to main, so this has been red since #4421 landed without showing up anywhere. Every PR opened since has inherited the failure.	2026-07-31 09:44:16 +01:00
claude[bot]	debfa2b733	feat(webapp): impersonation consent page and a view-as-user toggle (#4421)	2026-07-30 21:28:44 +01:00
Chris Arderne	68ed809416	test(clickhouse): keep queue metrics fixtures within TTL (#4428) ... ## Summary Keeps the queue metrics ClickHouse tests stable as wall-clock time advances. ## Root cause The fixtures used fixed timestamps. Once those timestamps crossed the tables' 30-day retention boundary, ClickHouse immediately expired the inserted aggregate rows and all six tests read empty results. The fixtures now derive a recent minute-aligned timestamp once per test file. The second 10-second bucket and ranking window are derived from the same anchor, preserving deterministic bucket relationships while keeping rows inside both the raw and aggregate table TTLs. Verified with `pnpm --filter @internal/clickhouse exec vitest run src/queueMetrics.test.ts`.	2026-07-30 21:20:44 +01:00
Oskar Otwinowski	4efe0a07c4	fix(webapp): create dev environments for SSO and Directory Sync members (#4426) ... Members added by SSO just-in-time provisioning or Directory Sync never got their per-member DEVELOPMENT environments - only invite acceptance and project creation created them. `trigger dev` returned "Environment not found" for those members and the dashboard had no dev view. ensureOrgMember now queues provisioning for every membership it settles, so both paths are covered and members missing environments are repaired on their next sync. Provisioning runs as a common-worker job to keep sign-in and directory webhooks off the per-project write loop. A failed enqueue surfaces for Directory Sync, whose worker retries the idempotent effect, and is swallowed for sign-in, where the next login enqueues again. Environment creation now tolerates a concurrent creator so the project-creation loop and the job cannot collide on the unique index. Also fixes environment resolution ignoring dev-environment ownership: a member without their own dev environment could be handed a colleague's and have it persisted as their dashboard preference.	2026-07-30 21:50:15 +02:00
Iss	d90f06ba5e	feat(webapp): migrate Plain to @team-plain/graphql + attribute support threads to org tenant (#4368) ... ## What Two changes, shipped together: 1. **SDK migration (TRI-12460).** `@team-plain/typescript-sdk` is deprecated. Move the webapp to its successors — `@team-plain/graphql` (client) and `@team-plain/ui-components` (`uiComponent` builder). Behaviour-preserving: the `PlainClient` customer upsert + thread creation move to the new `client.mutation.*({ input })` shape; the client now throws on failure, so `sendToPlain` wraps its calls and logs, staying best-effort. 2. **Org tenant attribution (TRI-12461).** When org context is available, `sendToPlain` now upserts a Plain tenant keyed by `externalId = org_id`, links the customer to it, and stamps the created thread with that tenant — so support threads become attributable to a Trigger.dev org. Wired into the four add-on quota requests and the plan-cancellation feedback (which already have org context). The tenant steps are isolated in their own try/catch and the thread's `tenantIdentifier` is gated on their success, so a tenant failure never blocks thread creation. ## Not affected - `customer.externalId` stays `User.id` — the customer cards + impersonation link are unchanged. - No ticket content leaves Plain. - Callers without a single org (e.g. the feedback widget) are unchanged — the org params are optional. ## Deploy prerequisite The webapp's Plain API key needs three **new** scopes for attribution to work (it already has `customer:create`, `customer:edit`, `thread:create`): - [x] `tenant:create` - [x] `tenant:edit` - [x] `customerTenantMembership:create` Until granted, nothing breaks — `sendToPlain` logs the forbidden error and creates the thread without attribution. ## Testing - `pnpm typecheck --filter webapp` passes; oxfmt + oxlint clean. - Ran the real `sendToPlain` end-to-end via a throwaway vitest harness against live Plain — confirmed the code path executes; the live write is gated only by the key scopes above.	2026-07-30 14:55:20 -04:00
github-actions[bot]	86b948b47a	chore: release v4.5.9 (#4408) helm-v4.5.9 v.docker.4.5.9 v4.5.9	2026-07-30 10:12:14 +01:00
Eric Allam	6e5f0f0fe7	fix(webapp,clickhouse): stop invalid customer queries alerting, and isolate Sentry scope per request (#4372) ... ## Summary A query sent to the query API with a typo in it, like a column name that does not exist, was being reported as a server error. That put customer SQL mistakes into our error alerting, where they made up almost all of the volume on one of our noisiest alerts, and it drowned out the failures that are actually ours to fix. This makes the level match who is at fault, and fixes two related problems found alongside it. ## Invalid queries are the caller's, not ours The query API route already got this right. It checks for `QueryError`, logs at warn, and returns a 400, with a comment saying the system handles it gracefully and no alert is needed. The layer underneath ignored that. `executeTSQL` logged every exception out of its catch block at error, including the compile failures the route was about to turn into a 400, and error-level logs are forwarded to error reporting. The TSQL package already draws the line we need: ```ts export class ExposedTSQLError extends BaseTSQLError { /** An exception that can be exposed to the user. */ } export class InternalTSQLError extends BaseTSQLError { /** An internal exception in the TSQL engine. */ } ``` `SyntaxError` and `QueryError` extend the first. So the catch block now branches on `ExposedTSQLError` and logs those at warn, keeping error for `InternalTSQLError` and anything unanticipated, which is a genuine compiler bug. ## SQL the caller wrote is their mistake, not ours The same asymmetry showed up one level down. A query that compiles fine can still be rejected by ClickHouse at execution, and most of those rejections mean the caller's SQL is wrong rather than that we generated something bad. This is where the volume actually is. Checking production, one error group alone, a missing `GROUP BY` on the public query API (`NOT_AN_AGGREGATE`), accounts for over a million events across hundreds of users. It is by far the largest error group in the project, and classifying only by resource limit would have left every one of those at error level. So rejections are split three ways in `ClickhouseClient`, which is the only place holding the parsed `ClickHouseError` and its symbolic type. By the time the error reaches `executeTSQL` it has been wrapped and the type is gone, and the type never appears in the message text, so it cannot be recovered by string matching. - **Resource limits** (memory ceiling, timeout, row/byte caps) log at warn. The query is valid, it just asked for more than it is allowed to spend. - **Invalid SQL** (`NOT_AN_AGGREGATE`, `UNKNOWN_IDENTIFIER`, `SYNTAX_ERROR`, the type and parse families) logs at warn **only when the caller wrote the SQL**. - **Everything else** keeps alerting. That gate matters. The client is shared, so the identical rejection on TRQL *we* generated is our bug and has to stay at error. Callers opt in with `userAuthoredQuery`: | caller | who wrote the SQL | opts in | | --- | --- | --- | | public query API | the customer | yes | | query editor | the customer | yes | | agent charts | the agent's model | yes | | built-in dashboard tiles | us, in code | no | | queue metric cards | us, in code | no | | health report | us, in code | no | The agent is the one judgement call. Its TRQL is not typed by a person, but it is also not something a code fix makes correct, so a query it gets wrong is not worth waking anyone for. The same endpoint serves built-in tiles whose TRQL we do write, so the opt-in lives with the caller rather than the route. Separately, when one of these queries did fail, the log recorded the generated ClickHouse SQL but not the query the caller actually wrote, which made the reports hard to act on. `queryWithStats` takes an optional `logFields` that `executeTSQL` uses to attach the original TSQL. ## Events were attributed to the wrong request Chasing the above turned up something broader: only a tenth of the events on that alert pointed at the query API. The rest were pinned to unrelated requests that happened to be in flight at the same time, so the alert looked like the trigger endpoint was failing. `Sentry.init` runs with `skipOpenTelemetrySetup: true`, because we register our own OTel pipeline. That skips `initOpenTelemetry`, and one of the things it does is: ```js api.context.setGlobalContextManager(new SentryContextManager()); ``` The async-context strategy is still installed, but `withIsolationScope` only marks the OTel context and delegates the actual fork to that context manager: ```js // "We depend on the otelContextManager to handle the context/hub" return api.context.with(ctx.setValue(SENTRY_FORK_ISOLATION_SCOPE_CONTEXT_KEY, true), ...) ``` `provider.register()` installed a plain `AsyncLocalStorageContextManager`, which does not know that key. The lookup found no scopes on the context and fell back to the process-global default isolation scope, so every request wrote its request data into the same object and the last writer won. The tracer now registers `SentryContextManager`, which subclasses `AsyncLocalStorageContextManager`, so OTel behaviour is unchanged. It is also registered on the path where tracing is disabled, which previously never called `register()` at all and so had no context manager of its own. Tenant tags were always correct, because those come from our own async local storage rather than the isolation scope. That is why the attribution being wrong was not obvious. This affects every error report the webapp sends, not just the query API. ## Verification `internal-packages/clickhouse`: 76 tests pass, including eight covering each level decision against a real ClickHouse container. Three pairs pin the gate open and shut at both layers: an invalid query, a compile failure, and a real limit breach driven with `max_rows_to_read` each log at warn with `userAuthoredQuery` and at error without it. The isolation fix has a test that reproduces the leak before asserting the fix. Two overlapping requests each tag their own isolation scope; with the plain context manager the slower one reads back the other's tag, and with `SentryContextManager` each reads back its own. Measured separately against a faithful reproduction of the server's wiring (own OTel pipeline, CommonJS entry) at 200 concurrent requests: per-request attribution goes from 0.5% to 100%, while span nesting, context propagation across awaits, and distinct trace IDs are identical before and after.	2026-07-30 09:04:15 +01:00
Chris Arderne	2f1734c858	fix(core,webapp): redact sensitive fields in logs by default and cap their size (#4401)	2026-07-29 17:59:47 +01:00
Chris Arderne	8ebc8a41af	fix(webapp,redis-worker): stop logging raw metadata, alert payloads, and job items (#4403)	2026-07-29 17:59:36 +01:00
Chris Arderne	a09817169f	fix(webapp): stop logging full batch item contents in batchTriggerV3 (#4404)	2026-07-29 17:59:27 +01:00
Chris Arderne	ed8f5e1297	fix(webapp): stop logging every environment on a lookup miss (#4402)	2026-07-29 17:37:58 +01:00
Chris Arderne	878c15811a	fix(cli): redact environment values from build debug logs (#4420)	2026-07-29 16:36:25 +00:00
Chris Arderne	a81ad4949c	feat(database,rbac): add multiple environment API key foundations (#4388) ... Adds the storage model and authorization contracts needed for multiple environment API keys. Credentials are represented by hashed values, revocation and expiration state, and persisted effective scopes. The built-in authorization fallback exposes full-access policy preparation, while optional authorization extensions can supply additional presets and task-aware scope generation. This change does not create, display, or authenticate additional keys.	2026-07-29 16:24:00 +00:00
Eric Allam	4eb9292cbe	feat(webapp,run-engine): queue metrics and health dashboard (#4131) ... ## Summary Three related changes, each independently gated: **Queue metrics and health.** Per-queue depth, throughput (enqueued, started, completed), concurrency, whether a queue is throttled, and scheduling delay (how long a run waits between becoming eligible and actually starting), plus a per concurrency-key breakdown for keyed queues. Collected from inside the run queue itself, stored in ClickHouse, and surfaced on the Queues list, a new per-queue detail page, the task pages, and the run inspector. The question it answers is "does this queue have enough concurrency to keep up, and if not, which key or which limit is the constraint". **Percent-based queue concurrency limits.** A queue's concurrency override can now be expressed as a percentage of the environment limit, stored as the source of truth and re-materialized whenever the environment limit changes. Absolute overrides above the environment limit are now **rejected with a 400** instead of being silently capped, which is a behavior change on `POST /api/v1/queues/:queue/concurrency/override`. **The `health` report.** A server-computed verdict on whether work is flowing, whether the runs that do start are healthy, and whether telemetry is fresh, rendered as text with sparklines. Available as `GET /api/v1/reports/:key`, `trigger report`, and the `get_report` MCP tool (plus a `report` MCP prompt, which shows up as a slash command in hosts that support prompts). With the flags off, the Queues page renders the pre-metrics component verbatim, nothing is emitted, and nothing is written to ClickHouse. ## Configuration Two independent gates, on purpose. Emission is global so data accrues for everyone before anyone can look at it; the view is per organization so it can be turned on for one org at a time without a deploy. **Runtime flags (no restart)** | Flag | Store | Gates | | --- | --- | --- | | `queue_metrics:enabled` | run-queue Redis key (`"1"`/`"0"`, off by default) | All emission, gauges and counters. Cached in-process for 10s with stale-while-revalidate, warmed eagerly at boot so the first op after a deploy is not dropped. | | `queue_metrics:gauge_sample_rate` | run-queue Redis key, `0..1` | Fraction of queue ops that emit a gauge. Counters are never sampled, so throughput stays exact at any rate. | | `queueMetricsUiEnabled` | feature-flag catalog: global `FeatureFlag` row, per-org `Organization.featureFlags` override wins | Whether an org sees the metrics view at all: the Queues list variant, the queue detail route, the built-in Queues dashboard, the concurrency-keys endpoint, and the metrics blocks on task pages and the run inspector. Off by default; a gated org gets a 404 on the detail route rather than an empty page. | Both Redis keys are readable and writable from `/admin/queue-metrics` (super-admin UI, with a live per-shard stream-health table) and `GET`/`POST /admin/api/v1/queue-metrics` (admin PAT). The admin surface uses its own Redis client, so it works on any instance regardless of whether that instance runs the emitter or the consumer. **Environment variables (boot time)** | Variable | Default | Notes | | --- | --- | --- | | `QUEUE_METRICS_EMIT_ENABLED` | `0` | Constructs the emitter and injects it into the run engine. Without it the run queue has no emitter at all. | | `QUEUE_METRICS_CONSUMER_ENABLED` | `0` | Boots the stream consumer on this instance. Independent of emission, so consumers can be sized separately from the API. | | `QUEUE_METRICS_STREAM_SHARD_COUNT` | `4` | Stream shards, hashed per queue. | | `QUEUE_METRICS_CONSUMER_BATCH_SIZE` | `1000` | Poll batch equals insert batch, so an ack can never outrun a write. | | `QUEUE_METRICS_REDIS_{HOST,PORT,USERNAME,PASSWORD,TLS_DISABLED}` | falls back to the run-queue Redis | Set `HOST` to move the metrics stream onto a dedicated instance so a metrics backlog cannot compete with the run queue for memory. Self-hosters can leave it unset and get a single-Redis deployment. | | `QUEUE_METRICS_COUNTER_STREAM_MAXLEN` | `2000000` shared, `8000000` dedicated | Bound on how much a stalled consumer can hold. The default is deliberately lower when the stream shares the queue-critical Redis. | | `QUEUE_METRICS_COUNTER_ODOMETER_TTL_SECONDS` | `604800` | TTL on the per-queue cumulative counter key, refreshed on every write, so only queues idle for the whole window are purged. | | `QUEUE_METRICS_MAX_QUEUE_NAMES_PER_ENV` | `1000` | Distinct queue names tracked per environment; overflow collapses into `__overflow__`. | | `QUEUE_METRICS_MAX_CONCURRENCY_KEYS_PER_QUEUE` | `10000` | Same idea one level down, per queue. | | `QUEUE_METRICS_GAUGE_SAMPLE_RATE` | `1` | Default for the live sample-rate key above. | | `QUEUE_METRICS_QUERY_TABLES_VISIBLE` | `0` | Lists the queue-metrics tables in the Query page, its schema docs, the schema API and the AI query context. Off keeps them unlisted while the feature is dark; a query naming them still runs either way. | | `QUEUE_METRICS_CLICKHOUSE_URL` | falls back to the shared wiring | Runs queue metrics on their own ClickHouse service: the consumer's inserts and every queue-metrics read go through it, so a metrics-heavy chart refresh never competes with runs-list or trace reads. Unset reproduces the previous split exactly (inserts on `CLICKHOUSE_URL`, reads on the query pool). | | `QUEUE_METRICS_CLICKHOUSE_READER_URL` | the write URL | Reader split, so the consumer's inserts can never land on a read endpoint. | | `QUEUE_METRICS_CLICKHOUSE_{KEEP_ALIVE_ENABLED,KEEP_ALIVE_IDLE_SOCKET_TTL_MS,MAX_OPEN_CONNECTIONS,LOG_LEVEL,COMPRESSION_REQUEST}` | `1`, unset, `10`, `info`, `1` | Pool tuning, matching the other per-workload ClickHouse clients. | Migrations to apply: ClickHouse `036_create_queue_metrics_v1.sql`, and a Postgres migration adding the nullable `TaskQueue.concurrencyLimitOverridePercent`. Both are additive. ## How collection works Queue operations produce two kinds of signal, and they have opposite failure modes, so they are handled differently. **Gauges** (queued, running, queue limit, env queued, env running, env limit, throttled, plus keys-with-backlog and worst-key wait on keyed queues) are read *inside* the same Redis script that performs the enqueue or dequeue, so the reading is atomic with the operation it describes rather than a racy follow-up read. The script returns them on its reply and the app forwards them to the stream. Gauges are sampled and drop-tolerant: they are aggregated with `max`, so a lost reading costs resolution, never correctness. **Counters** (enqueued, started, completed, plus nack and dead-lettered) are cumulative odometers. Each event increments a per-queue key on the metrics Redis and emits the absolute total, and ClickHouse takes the difference across buckets at read time. This is the important property of the design: a summed-delta counter undercounts permanently on any lost event, while a cumulative one self-heals, because the next surviving reading restates the whole total. Only bucket granularity can be lost, never the total. A queue returning after its odometer TTL expired restarts at 1 and reset detection handles it, which is safe precisely because expiry only spans a window with no activity. Both land on one sharded Redis stream. A consumer reads it with a consumer group, reclaims stale pending entries on a 15s interval rather than on every poll, maps one entry to one or two ClickHouse rows (whole-queue and, for keyed queues, per-key), and acks only after the insert lands. Each batch carries a dedup token derived from its stream-entry ids, and the target tables set `non_replicated_deduplication_window`, so a retried batch cannot double-count either the raw rows or the aggregates that hang off them. Consumer and emitter both emit OTel metrics (`queue_metrics.emitter.emitted`, `queue_metrics.consumer.{entries,rows_inserted,insert_errors,insert_duration,stream_depth,group_lag,pending,lag_unknown}`); stream depth and group lag are the two worth alerting on, and `lag_unknown` exists because Redis can report a null lag after a trim, which must not be read as zero. ## Storage and read path `queue_metrics_raw_v1` is a short landing table with a 6 hour TTL. Four aggregate tiers are materialized straight from raw, never cascaded off each other, each with a 30 day TTL: - `queue_metrics_v1`, 10 second buckets per queue, the default read path - `queue_metrics_5m_v1`, 5 minute buckets per queue, for wide ranges and cross-queue ranking - `env_metrics_v1`, 10 second buckets per environment, queue-independent so it stays cheap at any range - `queue_metrics_ck_v1`, 10 second buckets per concurrency key Every tier is an MV from raw because the counter states do not survive a cascade: their merge is order sensitive, so a `-MergeState` chain off the 10s table inflates the result, and the same property means an aggregate state may only be merged inside one queue. That constraint is now enforced by the query engine rather than by reviewer discipline: a column can declare a `mergeGroupKey`, and any query that references it without grouping by, or pinning to a single value of, every named key fails to compile with an actionable message. On the read side, TRQL gains three tables (`queue_metrics`, `env_metrics`, and a `queue_metrics_by_key` that is hidden from the editor, schema docs and schema API but still queryable, so per-key rows can never silently merge into a plain per-queue query), plus `deltaSumTimestampMerge` and `quantilesTDigestMerge`. Two schema-level optimizations ride along: a table can declare coarser rollups, so a query whose bucket interval is 5 minutes or wider is routed to the 5m table with no change to the query itself, and it can opt into the ClickHouse query cache with time bounds floored to a fixed grid, so the auto-refreshing dashboards actually share cache entries instead of missing on every tick. Both are caller-side substitutions, so the printer stays unaware of physical layout. All of this can also live on its own ClickHouse service. A table declares the pool its reads run on, the three queue-metrics tables name the dedicated one, and the ingestion consumer writes through the same client, so both directions move together with one env var and nothing else routes differently. The other engine change is opt-in gap filling: charts can request rows for empty buckets, where counters zero-fill and gauges carry forward. Grouped gauge series are densified per group and carried inside a partition, so a quiet queue's line holds its last value without bleeding another queue's value into it. ## Queue concurrency limits `concurrencyLimitOverridePercent` on `TaskQueue` is the source of truth when an override is set as a percentage; the absolute `concurrencyLimit` is materialized from it (floored, clamped to at least 1 so a percentage can never act as a pause, and never above the environment limit). Every path that changes an environment limit now recalculates the environment's percent-based overrides afterwards, outside the transaction, and pushes changed limits to the engine. The push is attempted even when the stored value did not change, so a previously failed sync self-heals rather than leaving the database and the engine diverged; paused queues are skipped so a recalculation cannot effectively unpause one. The API accepts exactly one of `concurrencyLimit` or `percent`, and the reject-instead-of-clamp change above means a request asking for more than the environment allows now fails loudly. The percent bound (greater than 0, at most 100) is defined once and shared by the zod schema, the dashboard mutation handler and the service, so the three cannot drift. The concurrency-keys table on a queue is now paginated against the ClickHouse per-key tier, ranked by peak backlog with the total on every row from a single scan, and only the keys on the current page are enriched with live counts from Redis. That replaces a hard top-50 cap with something whose cost is a function of page size rather than key cardinality. ## The health report `GET /api/v1/reports/:key?period=&format=markdown|ansi|json`. The verdict is computed on the server and is deterministic, not model-generated. Three independent analyzers run over one input snapshot: flow (is work moving, and if not, is the cause a limit, throttling, one bad queue, or dead-lettering), execution (are the runs that start succeeding, and at what latency), and liveness (how fresh is the telemetry). When telemetry is genuinely stale, the first two are forced to unknown and every actionable field is stripped, so no surface ever advises action off stale data. Authorization is per query table rather than a blanket query grant: a JWT must be scoped to every table the report reads (`runs`, `env_metrics`, `queue_metrics`), so a narrowly scoped token cannot pull a report that reads more than it was granted. `period` is validated as a shorthand with a 90 day ceiling at the edge. The report catalog is a registry of `{ load, interpret }` entries, so the next report is a new entry and no change to the route, the view model, the renderers, the CLI or the MCP tool. `trigger mcp` no longer launches the install wizard when stdout is a TTY, which fixed a real failure: hosts spawn the server over a PTY, so the wizard would open and the client would time out waiting for a server that never started. The wizard now needs `trigger mcp --install`. ## The part that is live regardless of every flag The enqueue and dequeue scripts now return a 2-tuple so a gauge reading can ride back on the reply. Every return site in the eight affected scripts is wrapped, and a `nil` original is converted to `false` on the way out, because a raw `nil` in the first slot would make Lua truncate the multi-bulk reply and silently drop the gauge on the throttled and empty-queue paths. The reply shape and the destructuring on the app side are exercised on every queue operation whether or not metrics are enabled, so that is the part of `run-engine` worth the closest review. One behavior fix in the same area: the scheduling-delay anchor is set only on a run's first entry into the queue. Anchoring it to trigger time on re-enqueues made waitpoint and checkpoint resumes report the entire wait as scheduling delay. Queue ordering is untouched, so a re-enqueued run keeps its position, and nacks deliberately keep the original anchor because a rolled-back dequeue is the same continuous wait. A pending-version promotion still anchors to trigger time, on purpose: that promotion is the run's first real entry into the queue, since the trigger deliberately held it back waiting for a worker version, and the TTL is armed at the same point for the same reason. The consequence is worth naming, because it is a judgement call: a run that waits on a deployment reports that wait as scheduling delay on its queue, which is time unrelated to queue capacity. ## Verification Unit and integration suites across the new package, the run queue, the mapping layer, the query engine and ClickHouse (including a test that applies migration 036 through the same splitter CI uses, and a regression test that inserts the same batch three times to prove the aggregates do not inflate). Beyond that, the whole path was driven end to end against a live stack with real runs: emitter to Redis stream to consumer to ClickHouse to the dashboards, for both the local dev path and the deployed path where a supervisor drives the dequeue, with assertions on exact counter reconstruction per queue and per concurrency key, throttling, environment saturation, scheduling delay, and a deliberate mid-stream reading drop to confirm the cumulative counters still reconstruct the correct total. The gated-off state was checked on every touched surface. The dedicated ClickHouse service was verified against a second, separately-schema'd instance: with it configured, the driven counters reconstruct exactly on the dedicated instance, the shared instance gains no rows for that window, a read through the query API returns the value that exists only on the dedicated instance, and a `runs` query still succeeds (it would fail outright if it were mis-routed to a service without that table). With the variable unset, the full suite passes unchanged. --------- Co-authored-by: Katia Bulatova <katia@trigger.dev> Co-authored-by: Katia Bulatova <katherine.bulatova@gmail.com> Co-authored-by: James Ritchie <james@trigger.dev>	2026-07-29 16:45:24 +01:00
claude[bot]	639eaf6e82	fix(webapp): don't apply an invite's role to an existing org member (#4409) ... <!-- ccr-slack-attribution --> _Requested via [Slack thread](https://triggerdotdev.slack.com/archives/C097ZHVKZFA/p1785249693523749)_ ## Summary Accepting an old invitation could change the role of someone who was already in the organization. A long-pending invite can carry a lower role than the member has since been promoted to, so accepting it was a silent demotion. When the accepting user was the organization's only Owner, the role layer refused that demotion, and the refusal (an expected, protective outcome) was logged as an error. An invitation now only sets a role on a membership the accept actually created, and people who are already in an organization are skipped when invitations are sent. ## How `acceptInvite` already skipped the `OrgMember` create when it found an existing membership, but the `rbac.setUserRole` call below it was gated only on `invite.rbacRoleId`. It now also tracks whether this accept created the membership. A create that loses the unique-constraint race counts as pre-existing, since whichever flow won it owns that membership's role. Skipping existing members outright would regress one case: a member with no RBAC role at all would never receive the invitation's role. `ensureOrgMember` handles that with `healMissingRoleAssignment`, which fills in a null role but never overwrites a real one, so `assignInviteRbacRole` takes the same gate. An established role is never touched; an absent one is filled in. `assignInviteRbacRole` branches on the result's machine-readable `code` instead of logging every refusal at `error`. `last_owner` goes to `logger.info`, matching the two directory-sync role paths; everything else, including a refusal that carries no code, goes to `logger.warn`. The helper is best-effort and never throws, so no outcome it produces warrants `error`. No string matching on the error text is involved. `inviteMembers` resolves the organization's members by email and skips those addresses before creating invites. The invite table's `@@unique([organizationId, email])` only dedupes *pending invites*, so it could never catch this. ## Invite surfaces Skipping addresses means a batch can now come back empty, and neither caller handled that: - The dashboard action built its redirect from `invites[0].organization`, so a batch where every address was skipped threw a `TypeError` that reached the admin as a raw error string. It also reported the submitted count rather than the created one. It now names what it skipped ("No invitations sent: 1 already a member of this organization") and counts what it actually created. - The invites API derived `alreadyInvited` as "everything not created", so an existing member was reported as though they had already been invited. `inviteMembers` now returns the two groups separately and the endpoint reports `alreadyMembers` alongside `alreadyInvited`. ## Testing `apps/webapp/test/member.server.test.ts` passes 16/16 locally, up from 12. Getting there needed a harness fix. The `~/db.server` mock did not export `Prisma`, so any code reaching `PrismaNamespace.PrismaClientKnownRequestError` threw before it could branch, leaving every duplicate-key path in `member.server.ts` unreachable from tests. The mock now re-exports the real `Prisma`, and there is a case covering the pending-invite skip. New cases: the invite role is applied when the accept creates the membership; it is not applied when the member already has a role; it is applied when an existing member has no role assigned; the organization is still joined when the assignment is refused with `last_owner`; and `inviteMembers` reports members separately from pending invites. Forcing the gate off fails exactly the "already has a role" case, so the coverage is load-bearing. `pnpm run typecheck --filter webapp` and `oxfmt --check` both pass. ## Changelog Accepting an old invitation could change the role of someone who was already in the organization. An invitation now leaves an existing member's role untouched, people who are already in an organization are no longer sent invitations to it, and the invite form says which addresses it skipped instead of failing with an unhelpful error. --- ## ✅ Checklist - [x] I have followed every step in the [contributing guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md) - [x] The PR title follows the convention. - [x] I ran and tested the code works. ## Screenshots No visual changes. The invite form's toast copy changes, as described above. --------- Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: Matt Aitken <matt@mattaitken.com>	2026-07-29 15:31:59 +01:00
DKP	8d321f8d6e	docs: give docs pages unique title tags and redirect stale pages (#4416) ... ## Summary Several docs pages rendered identical `<title>` tags, which weakens search indexing and makes results ambiguous. Each affected page now has a unique, descriptive title while keeping its existing sidebar label unchanged. Alongside the retitles: - Removed two stale build-system upgrade pages that were no longer in the navigation, with redirects to the current package upgrade guide. - Redirected the build-extensions group index to its overview page so the two URLs stop sharing a title. - Dropped a leftover orphaned API reference page (its old URL already redirects to the management overview). No links break: nothing in the docs points at the removed pages, and every redirect target exists.	2026-07-29 12:09:45 +01:00
Chris Arderne	15e160d767	chore(ci): cache typecheck work across runs (#4415)	2026-07-29 11:19:16 +01:00
James Ritchie	a11e5ffbc6	fix(webapp): fade overflowing side menu selector labels (#4412) ... Long organization, project, and environment names in the side menu were cut off mid-character. They now fade out at the right edge like the rest of the side menu items already did. ### Example of faded long names: <img width="246" height="200" alt="CleanShot 2026-07-28 at 22 59 24" src="https://github.com/user-attachments/assets/efa60b87-286f-4ab0-9d4e-490ef2de53e5" />	2026-07-29 10:16:45 +01:00
James Ritchie	1e14e29d71	fix(webapp): restyle the leave and remove team member dialogs (#4411) ... ## Summary The confirmation dialog for leaving a team or removing a teammate was still built on the old `Alert` primitive: the entire question sat in the title, there was no header divider or `Esc` affordance, and the footer used small buttons pinned to the right. It now uses the standard `Dialog` layout the rest of the dashboard uses. The title is static ("Remove team member" / "Leave team"), the question moves into the body with the person's name and the organization highlighted, and the footer is a bordered row with medium Cancel and confirm buttons. A member who has not set a name is now identified by their email instead of "them". Verified against a local dashboard on both dialogs. Confirming a removal posts the member id, deletes the membership and shows the success toast. Cancel, `Esc`, and Enter while Cancel is focused all close the dialog without issuing a request, leaving the member in place. No release note needed: this is a visual restyle of an existing dialog with no behaviour change.	2026-07-28 23:05:20 +01:00
Matt Aitken	205bdc3103	docs(wait): separate compute billing from concurrency release (#4405) ... The wait docs describe the 5 second compute-billing threshold as if it were also the suspension threshold. It isn't, and the gap is confusing when you're sizing a poll interval: - **Compute** stops being charged for any wait longer than 5 seconds. - **Concurrency** is only released once the machine has been snapshotted and shut down. For `wait.for` and `wait.until` that happens 60 seconds into the wait — a shorter wait stays `EXECUTING` and holds its concurrency slot for the whole wait, even though the compute is free. So `await wait.for({ seconds: 30 })` in a polling loop never releases its slot, which looks like a bug if the docs told you waits over 5 seconds checkpoint. ## Changes **`docs/snippets/paused-execution-free.mdx`** — rendered on `/wait`, `/wait-for` and `/wait-until`. Drops "we checkpoint and" from the billing sentence so it's purely about compute, then adds one paragraph for the concurrency half. **`docs/queue-concurrency.mdx`** — the "Waits and concurrency" section states flatly that waiting runs don't consume slots. Adds a short subsection for the time-based exception. **`docs/how-to-reduce-your-spend.mdx`** — "Waits longer than 5 seconds automatically checkpoint your task, meaning you don't pay for compute" → the compute claim only. Code comments follow, plus a pointer that waiting doesn't always free concurrency. **`docs/how-it-works.mdx`** — the Checkpoint-Resume walkthrough used `wait.for({ seconds: 30 })` as *the* example of a wait that suspends. Bumped to 5 minutes and noted the sub-60s exception. No behaviour change — docs only.	2026-07-28 15:20:00 +01:00
Chris Arderne	38bf82aebe	feat(cli,webapp): target notifications by minimum CLI version (#4407)	2026-07-28 14:23:41 +01:00
Saadi Myftija	44eca4d166	feat(webapp): org-gated internal API origin in run env vars (#4366) ... Adds an opt-in way for operators to route deployed runs' API traffic through a different origin than the public one, per organization. Set `INTERNAL_API_ORIGIN` on the webapp and enable the `internalApiOriginEnabled` feature flag (globally or per org, with the org override winning in both directions): deployed runs for enabled orgs then get `TRIGGER_API_URL` set to the internal origin instead of `API_ORIGIN`. Useful for gradually moving run traffic onto a private network path. ## Design The origin is resolved when an attempt starts, so flag changes take effect on the next attempt and roll back the same way, with no task redeploys. The org override is read fresh per attempt; the global default comes from the cached flags registry (a cold read fails safe to the public origin). When `INTERNAL_API_ORIGIN` is unset the flag is a no-op and no extra queries run, so existing deployments are unaffected. Dev runs always use the public origin, and `TRIGGER_STREAM_URL` remains unchanged.	2026-07-28 11:28:09 +02:00
claude[bot]	ec562c0e68	fix(webapp): remove unused Electric sync trace routes (#4400) ... <!-- ccr-slack-attribution --> _Requested by **Eric Allam** · [Slack thread](https://triggerdotdev.slack.com/archives/C0AU83M3136/p1785222101937829?thread_ts=1785207509.304669&cid=C0AU83M3136)_ Removes two dead Remix routes and the helpers only they used. `app/routes/sync.traces.runs.$traceId.ts` (`/sync/traces/runs/:traceId`) and `app/routes/sync.traces.$traceId.ts` (`/sync/traces/:traceId`) were added with the original ElectricSQL run page and lost their only consumers when the dashboard hooks that called them were deleted. Nothing in the repo references either route today. Also removed, because the deleted routes were their only callers: - `OtelTraceIdSchema`, `RESERVED_ELECTRIC_SHAPE_PARAMS`, `TraceScope`, `buildElectricTraceWhereClause` from `app/v3/electricShape.server.ts` (the file stays — `UNSAFE_REALTIME_TAG_CHARS` / `sanitizeRealtimeTagForSql` / `sanitizeRealtimeTagsForSql` are still used by `realtime.v1.runs.ts` and `realtimeClient.server.ts`) - the loader-specific cases in `apps/webapp/test/spanTraceRoutes.replicaLag.test.ts` and `internal-packages/run-store/src/runOpsStore.routesSpanTraceReadView.replicaLag.test.ts` `app/utils/longPollingFetch.ts` is untouched — `realtimeClient.server.ts` still uses it. `runOpsStore.ts` / `PostgresRunStore.ts` are untouched too; the unrouted-lookup mechanism there is generic and stays. As a plain code fact: the run lookup these loaders performed keyed on `TaskRun.traceId` alone, which is not an index-backed query shape. That is noted only as context for why the code is not worth keeping around unused. ### Judgement call worth a maintainer's opinion The request was specifically about `/sync/traces/runs/:traceId`, the route that looks up a run by `traceId`. This PR **also** deletes its sibling `/sync/traces/:traceId`. The reasoning: - both routes came in with the same ElectricSQL run-page work - both lost their only consumers in the same later commit - neither has any caller anywhere in the repo - they share the same helper module, so keeping one means keeping the helpers half-used If you would rather keep the sibling, reverting just that one file deletion is easy and does not affect the rest of this PR — say the word and I will restore it along with the helpers it needs. ## ✅ Checklist - [x] I have followed every step in the [contributing guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md) - [x] The PR title follows the convention. - [x] I ran and tested the code works --- ## Testing Verification run locally from the repo root: | Command | Result | | --- | --- | | `pnpm run format` | clean, no changes produced | | `pnpm run lint:fix` | clean | | `pnpm run lint` | pass (exit 0, no findings) | | `pnpm run typecheck --filter webapp` | pass | | `pnpm run typecheck --filter @internal/run-store` | pass | A ripgrep sweep for `sync.traces`, `sync/traces`, `syncTraceRunsLoader`, `buildElectricTraceWhereClause`, `OtelTraceIdSchema` and `RESERVED_ELECTRIC_SHAPE_PARAMS` (excluding `node_modules`) returns zero hits. **Not fully verified:** both edited test files are testcontainers suites and need a Docker runtime, which was not available in my environment. I confirmed each file *collects* correctly with exactly the three intended remaining tests and no import errors — notably, dropping the `session.server` / `controlPlaneResolver.server` / `longPollingFetch` / `env.server` mocks does not break module loading for the surviving loaders. The assertions themselves then failed only on `Could not find a working container runtime strategy`. CI should be the real signal here. Per `apps/webapp/CLAUDE.md`, `pnpm run build --filter webapp` was deliberately not run. --- ## Changelog Removed two unused sync routes left over from the original ElectricSQL run page, along with the helpers and tests that existed only to serve them. No behaviour change — neither route had any caller. --- ## Screenshots _n/a — no user-visible surface changes._ 💯 --------- Co-authored-by: Claude <noreply@anthropic.com>	2026-07-28 09:00:48 +01:00
Matt Aitken	3ed48516df	ci: let the claude bot trigger the PR audit workflows (#4392) ... ## Summary PRs opened by the claude GitHub app fail both the agent instructions audit and the REVIEW.md drift audit before Claude gets a chance to run. `claude-code-action` refuses any actor whose account type is not `User` unless the actor is listed in `allowed_bots`: ``` Workflow initiated by non-human actor: claude (type: Bot). Add bot to allowed_bots list or use '*' to allow all bots. ``` So those PRs land with two permanently red checks and no audit coverage at all. Both workflows already allowlist Devin; this adds the claude app alongside it. ## Why this does not open the workflows up to outside contributors `allowed_bots` is only consulted for non-`User` actors. Humans, contributor or maintainer, take the separate write-permission path and are unaffected by what is in the list. Beyond that, both jobs are guarded by `github.event.pull_request.head.repo.full_name == github.repository`, so a fork PR skips the job entirely, and they trigger on `pull_request` rather than `pull_request_target`, so a fork-triggered run would get no API key and a read-only token anyway. The bot is named explicitly instead of using `"*"`, which would let every bot trigger these audits, dependabot's PR stream included.	2026-07-27 17:22:46 +01:00
Eric Allam	fc576436e2	perf(run-ops-database): index BatchTaskRun for the batches list on the dedicated schema (#4396) ... ## Summary The batches list page orders by `(createdAt DESC, id DESC)`, which is why [#4361](https://github.com/triggerdotdev/trigger.dev/pull/4361) added a matching index on `BatchTaskRun`. That index only landed in `@trigger.dev/database`. The dedicated run-ops database has its own migration history, so it never received the index. `BatchListPresenter` reads both databases and merges, so for environments whose batches live in the dedicated database the page kept falling back to a scan and in-memory sort, which is the exact behaviour #4361 set out to fix. ## Fix Adds the index to the run-ops schema with its own migration. `CREATE INDEX CONCURRENTLY IF NOT EXISTS`, so it is a no-op where the index already exists and still records its ledger row. The second half is the interesting part. Because the two packages own separate migration histories, a run-graph schema change has to be authored twice, and nothing made the miss visible: the run-ops status check truthfully reports "up to date" against its own history, so the apply step just skips. `schemaParity.test.ts` compares the physical shape of every model the run-ops schema declares against its counterpart in `@trigger.dev/database`: scalar fields with their attributes, plus `@@index`, `@@unique`, `@@id` and `@@map`. Relation navigation fields are excluded, since the run-ops schema deliberately drops relations that would cross a database boundary while keeping the scalar FK column. A field counts as a relation when its type resolves to a model name, which keeps enum-typed columns in scope. Two models are listed as run-ops-only: `CompletedWaitpoint` and `WaitpointRunConnection`, both explicit FK-free replacements for a control-plane implicit many-to-many, since an implicit m2m carries a foreign key that cannot resolve across databases. The test also asserts that exception list is exhaustive, so a new unpaired model fails rather than being silently skipped. Confirmed the guard actually fails: reverting the index turns `BatchTaskRun` red with the missing `@@index` named in the diff.	2026-07-27 16:59:43 +01:00
github-actions[bot]	d189ce17d3	chore: release v4.5.8 (#4364) ... ## Summary 2 new features, 9 improvements, 3 bug fixes. ## Highlights - Allow additional environment API keys to create scoped public access tokens through the Trigger.dev API. Use server-issued public access tokens for batch operations so environment-scoped API keys can read batch results. ([#4387](https://github.com/triggerdotdev/trigger.dev/pull/4387)) ## Improvements - Preserve the partial assistant message when a chat turn's model stream fails mid-response. `chat.agent` now passes the recovered partial to `onTurnComplete`, and `chat.createSession`'s `turn.complete()` keeps it before rethrowing, instead of dropping the streamed-so-far output. ([#4348](https://github.com/triggerdotdev/trigger.dev/pull/4348)) ## Server changes These changes affect the self-hosted Docker image and Trigger.dev Cloud: - Favorite any dashboard page to a new Favorites section in the side menu, and customize the sidebar by renaming favorites, hiding items, and reordering items and sections. ([#4375](https://github.com/triggerdotdev/trigger.dev/pull/4375)) - List API endpoints now clamp the page size to a maximum of 100. Requests asking for a larger page size return up to 100 items and keep paginating, rather than pulling an unbounded page. ([#4360](https://github.com/triggerdotdev/trigger.dev/pull/4360)) - Organizations without billing alerts now get default spend alert thresholds, so you're notified before usage grows unexpectedly. The billing limit page no longer pre-selects an option before you've set a limit and prompts you to configure one. Alert previews now update immediately after you change your billing limit. ([#4328](https://github.com/triggerdotdev/trigger.dev/pull/4328)) - When you create a Personal Access Token, the generated token now shows its first and last few characters instead of being fully hidden, so you can confirm you copied the right value. ([#4363](https://github.com/triggerdotdev/trigger.dev/pull/4363)) - Add metrics to the realtime backend that measure how often a single changed run is served to multiple subscriptions in one batch. ([#4341](https://github.com/triggerdotdev/trigger.dev/pull/4341)) - Realtime run subscriptions can now be configured to read run data straight from the primary database, so a run's latest state is never served from a lagging replica. Off by default; replica reads are unchanged unless you turn it on. ([#4378](https://github.com/triggerdotdev/trigger.dev/pull/4378)) - SSO and Directory Sync are no longer restricted to Enterprise plans — get in touch and we can turn them on for your organization whatever plan you're on. ([#4393](https://github.com/triggerdotdev/trigger.dev/pull/4393)) - Improved supervisor observability: it now reports metrics for its outbound requests, making failed calls to upstream services easier to monitor. ([#4350](https://github.com/triggerdotdev/trigger.dev/pull/4350)) - The runs list on a task's page now updates live — run statuses change and newly triggered runs appear without a manual refresh, matching the main Runs page. ([#4377](https://github.com/triggerdotdev/trigger.dev/pull/4377)) - Speed up the Batches list page for environments with a large number of batches, which could previously time out while loading. ([#4361](https://github.com/triggerdotdev/trigger.dev/pull/4361)) - Container startup no longer prints database and ClickHouse connection strings (with credentials) to the logs. ([#4346](https://github.com/triggerdotdev/trigger.dev/pull/4346)) - The tasks page no longer runs two queries whose results were never displayed, cutting wasted work on every page load and removing a source of hidden server errors ([#4380](https://github.com/triggerdotdev/trigger.dev/pull/4380)) <details> <summary>Raw changeset output</summary> # Releases ## @trigger.dev/build@4.5.8 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.8` ## trigger.dev@4.5.8 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.8` - `@trigger.dev/build@4.5.8` - `@trigger.dev/schema-to-json@4.5.8` ## @trigger.dev/core@4.5.8 ### Patch Changes - Allow additional environment API keys to create scoped public access tokens through the Trigger.dev API. Use server-issued public access tokens for batch operations so environment-scoped API keys can read batch results. ([#4387](https://github.com/triggerdotdev/trigger.dev/pull/4387)) ## @trigger.dev/python@4.5.8 ### Patch Changes - Updated dependencies: - `@trigger.dev/sdk@4.5.8` - `@trigger.dev/core@4.5.8` - `@trigger.dev/build@4.5.8` ## @trigger.dev/react-hooks@4.5.8 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.8` ## @trigger.dev/redis-worker@4.5.8 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.8` ## @trigger.dev/rsc@4.5.8 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.8` ## @trigger.dev/schema-to-json@4.5.8 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.8` ## @trigger.dev/sdk@4.5.8 ### Patch Changes - Preserve the partial assistant message when a chat turn's model stream fails mid-response. `chat.agent` now passes the recovered partial to `onTurnComplete`, and `chat.createSession`'s `turn.complete()` keeps it before rethrowing, instead of dropping the streamed-so-far output. ([#4348](https://github.com/triggerdotdev/trigger.dev/pull/4348)) - Allow additional environment API keys to create scoped public access tokens through the Trigger.dev API. Use server-issued public access tokens for batch operations so environment-scoped API keys can read batch results. ([#4387](https://github.com/triggerdotdev/trigger.dev/pull/4387)) - Updated dependencies: - `@trigger.dev/core@4.5.8` </details> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> helm-v4.5.8 v.docker.4.5.8 v4.5.8	2026-07-27 16:40:31 +01:00
James Ritchie	73eb4c5c16	feat(webapp): Improve the Integrations page layout (#4379) ... ## Summary The project Integrations page now uses the same settings layout as the org SSO page: a centered column of titled rows with dividers, instead of headings over bordered boxes. GitHub, Vercel and build settings read as one consistent list, and the page titles itself "Integrations". Confirmations persist rather than vanishing once you move past them (`GitHub app: Installed`, `Vercel project: Connected`), plan-gated rows offer an Upgrade button instead of a dead toggle, a disabled toggle explains why in place and highlights the control that unlocks it, and warnings are rows with a hazard icon and their recovery action on the right. Copy throughout leads with the outcome instead of restating the field label. Two fixes along the way: a nested `<form>` in the Vercel panel that failed hydration and silently truncated the page, and every settings row carrying a few pixels more space above its title than below its description. ### Before <img width="1160" height="1972" alt="CleanShot 2026-07-26 at 21 56 42@2x" src="https://github.com/user-attachments/assets/ed0fd676-36d8-4eb7-a16e-827a24f007d9" /> ### After <img width="1358" height="4455" alt="CleanShot 2026-07-26 at 19 14 28@2x" src="https://github.com/user-attachments/assets/6a635e6a-c0eb-4a4c-a68f-fcde4d25e8a6" />	2026-07-27 16:34:44 +01:00
James Ritchie	d30ee6e570	feat(webapp): favorite pages and sidebar customization (#4375) ... ## Summary Favorite any dashboard page and it appears in a new "Favorites" section at the top of the side menu. The star next to the page title (or Option+F) saves the exact view, filters and tabs included, with a name derived from the URL ("Runs: Completed successfully, last 7d", "Run: 05hrqq9n") that you can rename inline from each item's hover menu. The sidebar is customizable too: "Customize sidebar" (on section header menus and in each "More" menu) opens a modal where you can reorder sections, drag items into a new order, hide items behind a per-section "More" popover, and rename or remove favorites. Changes apply on Confirm, Reset restores the default layout without touching favorites, and everything is stored per user in dashboard preferences. ## Screenshots | Favorites in the side menu | Customize sidebar modal | | --- | --- | |  |  |  ## Design notes - Favorite links carry a small marker search param so the favorite, not its identical main menu item, highlights as active. Markers from shared or stale links are cleaned on load, and changing any filter hands the highlight back to the regular menu item. - Preference writes are serialized with a row lock: several writers (debounced collapse and width saves, favorite toggles, the customize modal) can land concurrently and would otherwise clobber each other's read-modify-write of the JSON column. - Option+F is matched on `event.code` with a raw listener because macOS reports Option-modified letters as symbols, which the `event.key` based shortcut hook can't capture. Verified end-to-end in the browser: star toggle and shortcut, instant section appearance, inline rename and staged modal removal, filter-aware labels and unique active states, shared-link normalization, drag reordering, and persistence across reloads.	2026-07-27 16:29:36 +01:00
Chris Arderne	efd0ee8d74	feat(core,sdk): support additional environment API keys (#4387) ... ## Summary Additional environment API keys can use SDK APIs that require public access tokens. The SDK detects the additional-key format and asks the Trigger.dev server to mint scoped tokens instead of attempting to sign them locally. Root environment keys retain their existing local-signing behavior. Trigger and batch clients also prefer server-issued tokens returned in response headers while preserving compatibility with older servers. ## Deployment notes This package update is safe to publish before servers expose additional key creation. Existing root keys continue to use the current path, while an additional key used with an older server fails with an actionable upgrade error.	2026-07-27 15:19:27 +00:00
James Ritchie	3e53404f40	feat(webapp): live-update the runs list on task pages (#4377) ... ## Summary The runs list on a task's page now updates live, matching the main Runs page. Run rows update their status, duration, and cost in place as runs progress, and a "N new runs" button appears in the header when newer runs come in so you can pull them into the list without a manual refresh. This applies to both standard and scheduled task pages. ## Design It reuses the Runs page's polling hook. A task page scopes its runs by the task in the URL path rather than a `tasks` query filter, so the hook now takes an optional task slug and scopes new-run detection to it. The "new runs" button sits in the header, outside the deferred runs table, so the count is lifted to the page and the click action is passed through a ref. That keeps the table streaming on first load instead of blocking the header on the runs query. When newer runs come in, a `1 new run` button appears in the task page header, to the left of the time filter. Clicking it pulls the new runs into the list.	2026-07-27 15:41:47 +01:00
nicktrn	e8a2dbd605	chore: ignore local docs/superpowers planning docs (#4395) ... Adds a gitignore rule for `**/docs/superpowers/` so locally-generated planning and design scratch docs under that path aren't committed; preventive only, no-op for existing tree.	2026-07-27 12:44:14 +00:00
Matt Aitken	269470fd87	feat(webapp): gate SSO on an entitlement instead of the Enterprise plan (#4393) ... The SSO & Directory Sync settings page decided access by comparing the organization's plan code against the literal string `"enterprise"`. The webapp now reads a `hasSso` entitlement from plan limits. ## Changes - **`settings.sso` route** — `planAllowsSso` reads `limits.hasSso` rather than the plan code; the loader and the action gate on a shared `getSsoEntitlement` helper. - **`platform.v3.server`** — new `getSsoEntitlement(orgId)` returning `entitled | not_entitled | unknown`, behind a new SWR cache namespace (60s fresh / 120s stale, memory + Redis). This replaces an uncached billing round-trip that previously ran on every settings load, so the page gets cheaper than it was. - **`directorySyncEffects`** — the entitlement is now checked before applying membership effects, per organization and memoised across a batch. - **`@trigger.dev/platform` 1.2.0 → 1.3.0** — required, see below. ## Behaviour worth reviewing **Revocation now stops SCIM.** Previously the plan check existed only on the settings page, so an org that lost access kept receiving directory-sync pushes indefinitely; only the config UI froze. Provision *and* deprovision are gated, so a revoked entitlement can't remove members either. **An unreadable entitlement throws instead of skipping.** Effects are idempotent and the worker retries, so retrying is lossless where dropping would silently lose a directory change. It's raised at `warn` level so a transient billing blip doesn't page anyone. **The login path is deliberately untouched.** A hard entitlement check there turns a billing outage into a login outage. Consequence: an org that loses the entitlement keeps its existing SSO logins working until the connection is removed. Gating sign-in is a separate decision. **Self-hosted is unaffected.** With no billing service configured the helper returns `entitled`, leaving plugin presence and the kill switch as the only gates — a self-hoster who installed the plugin isn't locked out of it. ## The dependency bump is load-bearing The `Limits` schema is a plain `z.object`, so it *strips* unknown keys. On 1.2.0 the `hasSso` field was silently discarded during parsing and read as `undefined` no matter what billing sent — a structural accessor would not have helped. Verified against both builds: ``` 1.2.0 → parsed: true | hasSso survives: false 1.3.0 → parsed: true | hasSso survives: true ``` This PR therefore cannot merge before 1.3.0 is published, which it now is. ## Testing `apps/webapp/test/directorySyncEffects.server.test.ts` — 7 tests over the gate: applies when entitled, skips provision and deprovision when not, throws a warn-level retryable error when unreadable, resolves once per org across a batch, and gates per org so one unentitled org doesn't block another. `pnpm run typecheck --filter webapp` passes (18/18), oxfmt and oxlint clean.	2026-07-27 13:08:16 +01:00
claude[bot]	72c2b2c650	chore(deps): bump express-rate-limit and ip-address (#4391) ... **Before:** `ip-address` resolved twice in `pnpm-lock.yaml` — `8.1.0` under `@jsonhero/json-infer-types`, and `10.0.1` under `express-rate-limit`. **After:** a single `ip-address@10.2.0` entry, shared by both chains. **How:** `express-rate-limit@8.2.1` pinned `ip-address` to an exact version, so the parent itself had to move — `8.5.1` onwards declares a range instead, and `@modelcontextprotocol/sdk` already allows `^8.2.1`, so scoping that parent to `^8.6.0` lets `ip-address` resolve on its own. `@jsonhero/json-infer-types` caps `ip-address` at `^8.1.0` and is already at its latest published release, so that chain gets a scoped override instead of a parent bump. `jsbn` and `sprintf-js` drop out of the tree as a side effect. Both overrides are parent-scoped, so the `cli-v3` chain is deliberately untouched: it resolves `@modelcontextprotocol/sdk` 1.25.2, which declares `express-rate-limit ^7.5.0` and pulls in no `ip-address` at all. `pnpm-lock.yaml` regenerated. `package.json` and `pnpm-lock.yaml` are the only two files changed. Nothing in the repo imports `ip-address` or `express-rate-limit` directly. Both chains are transitive under `apps/webapp` — `@jsonhero/schema-infer` (used by `TestTaskPresenter.server.ts`) and `@vercel/sdk` — so no published `@trigger.dev/*` package is affected. --- ## Testing - `pnpm install --lockfile-only` regenerates cleanly, and `pnpm install --frozen-lockfile --lockfile-only` passes, so the lockfile matches the manifests. - Package churn is limited to the intended set: `express-rate-limit` 8.2.1 to 8.6.0, `ip-address` 8.1.0 and 10.0.1 collapsing to 10.2.0, and `jsbn` / `sprintf-js` removed. No other resolution moved. - `@jsonhero/json-infer-types` only calls `new Address4()` / `new Address6()` inside a try/catch to classify strings. Ran that exact logic against both `8.1.0` and `10.2.0` over 27 inputs (v4, v6, zone IDs, CIDR, IPv4-mapped, malformed, empty, non-strings): identical results in all 27. Both are still CJS named exports in `10.2.0`, with the same `engines` floor. - Drove the real `inferSchema()` path from `@jsonhero/schema-infer` with `ip-address` forced to `10.2.0`; it still detects `ipv4` and `ipv6` formats correctly. - `express-rate-limit` 8.6.0 keeps the same `express` peer range (`>= 4.11`) and the same node floor as 8.2.1. Its new `debug` dependency resolves to a version already present in the tree. - `oxfmt --check` passes on the modified `package.json`. - Both bumped versions clear the repo's `minimumReleaseAge` window; the newest `express-rate-limit` (8.6.1) and `ip-address` (10.2.1+) releases do not yet, which is why this lands on 8.6.0 and 10.2.0. - Not run here: a full monorepo install, typecheck and test suite. No TypeScript changed, and neither package leaks types into ours — `ip-address` is not referenced in `json-infer-types`' or `schema-infer`'s declaration files — so CI should be the judge of the wider suite. --- ## Changelog Routine dependency maintenance, no behaviour change. No changeset or `.server-changes/` entry: the diff touches only the root `package.json` and `pnpm-lock.yaml`, not `packages/*`, `integrations/*`, `apps/webapp/` or `apps/supervisor/`. Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>	2026-07-27 11:09:49 +00:00
claude[bot]	d91818f198	fix(webapp): remove unawaited task list metrics promises (#4380) ... <!-- ccr-slack-attribution --> _Requested via [Slack thread](https://triggerdotdev.slack.com/archives/C097ZHVKZFA/p1785082528841609)_ `TaskListPresenter` created promises that nothing ever consumed. Two of the three deferred metrics promises it returned had no reader, no `await` and no `.catch()`, so when the query behind one of them failed the rejection had nowhere to go. ## Before / After **Before** - `TaskListPresenter.call()` returned four things: `tasks`, `activity`, `runningStats` and `durations`. Its only caller reads `tasks` and `runningStats`. - Every load of the tasks page therefore fired two ClickHouse queries whose results were thrown away. - If either of those two queries failed, the resulting promise rejection was unhandled — nothing was awaiting it and nothing had attached an error handler, so it surfaced as an unhandled rejection at the process level rather than as an error anyone could attribute to a request. **After** - `TaskListPresenter.call()` returns `tasks` and `runningStats` only. - Two fewer queries run per tasks-page load. - There is no longer an unconsumed promise that can reject without a handler. `runningStats` is awaited by its caller, so its failures continue to be handled the way they always were. Nothing changes on screen: the tasks page renders `hourlyActivity` and `runningStates`, and neither of the removed values fed either of those. ## How The removed values were verified unreferenced before deleting anything: - `TaskListPresenter` has exactly one caller, `UnifiedTaskListPresenter`, which reads `taskResult.tasks` and `taskResult.runningStats` and nothing else. - No file anywhere in the repo — app code, tests, or type re-exports — reads an `activity` or `durations` field off the presenter's result. - `UnifiedTaskListPresenter` builds its own `unifiedTaskListHourlyActivity` query for the 24h chart the page actually renders, which is what made the presenter's separate 7-day daily activity data redundant. - `getDailyTaskActivity` and `getAverageDurations` on `ClickHouseEnvironmentMetricsRepository` had no callers other than the two lines being deleted, so they and their now-orphaned helpers and types were removed too. Changes: - `apps/webapp/app/presenters/v3/TaskListPresenter.server.ts` — drop the `activity` and `durations` fields (both from the main return and from the no-current-worker early return) and the two repository calls behind them. Drop the unreferenced `TaskActivity` type alias. The "don't await this" comment on the remaining `runningStats` promise now spells out that the caller has to consume it. - `apps/webapp/app/services/environmentMetricsRepository.server.ts` — remove `getDailyTaskActivity` and `getAverageDurations` from the `EnvironmentMetricsRepository` interface and its ClickHouse implementation, along with `fillInDailyTaskActivity` and the `DailyTaskActivity` / `AverageDurations` types. `getCurrentRunningStats` is the control that shows the diagnosis is right. It throws on query failure in exactly the same way as the two removed methods — `if (queryError) throw queryError` — but it never produced an unhandled rejection, because `UnifiedTaskListPresenter` passes its promise into a `Promise.all(...).then(...)` chain that the route then awaits. Same failure mode, opposite outcome, and the only difference is whether anything consumes the promise. Follow-ups, not in this PR: - `AgentListPresenter` returns three sparkline promises in the same shape and they look similarly unconsumed. Left alone here to keep this change reviewable. - With these two callers gone, the `getTaskActivity` and `getAverageDurations` query builders in `@internal/clickhouse` have no remaining callers in this repo. Whether to remove them is a separate call for someone who owns that package. ## ✅ Checklist - [x] I have followed every step in the [contributing guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md) - [x] The PR title follows the convention. - [x] I ran and tested the code works --- ## Testing - `pnpm run typecheck --filter webapp` — passes. This is the meaningful check here: it proves nothing still references the removed fields, methods or types. - `pnpm run format` and `pnpm run lint:fix` — clean, no changes produced. - No test file referenced the removed symbols, so no test needed updating. --- ## Changelog Server-only change, so this carries a `.server-changes/` note rather than a changeset: `.server-changes/task-list-remove-unused-metrics-queries.md`. > The tasks page no longer runs two queries whose results were never displayed, cutting wasted work on every page load and removing a source of hidden server errors --- ## Screenshots _No visual change — the removed data was never rendered._ Co-authored-by: Claude <noreply@anthropic.com>	2026-07-27 11:39:43 +01:00
claude[bot]	4d8b5d6f87	chore(ci): remove per-repo Dependabot alert workflows (#4384) ... ## ✅ Checklist - [x] I have followed every step in the [contributing guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md) - [x] The PR title follows the convention. - [ ] I ran and tested the code works — n/a, this PR only deletes two workflow files --- ## Summary **Before:** two scheduled workflows in this repo posted Dependabot digests to Slack — a critical-alert check every morning at 08:00 UTC, and a summary of all open alerts on Mondays at 08:00 UTC. **After:** neither runs. This reporting is handled centrally now, so the two in-repo workflows were duplicating it. **How:** deletes `.github/workflows/dependabot-critical-alerts.yml` and `.github/workflows/dependabot-weekly-summary.yml`. Both were self-contained — inline shell, no shared scripts or composite actions — so nothing else in `.github/` referenced them. Dependabot itself is unchanged: `.github/dependabot.yml`, alerts, and version updates all keep working. This removes only the two Slack notifiers. The `ENABLE_DEPENDABOT_ALERTS` repository variable existed only to switch these two workflows off. Nothing else reads it, so it can be removed from the repository settings if it's set. --- ## Testing No runtime code changes — this PR only removes two scheduled workflow files. Verified that nothing else in the repo references either filename, either workflow name, or the `ENABLE_DEPENDABOT_ALERTS` variable. --- ## Changelog Removed the two in-repo scheduled workflows that posted Dependabot digests to Slack. Co-authored-by: Claude <noreply@anthropic.com>	2026-07-27 09:51:05 +00:00
Eric Allam	d3906241a5	feat(webapp): read realtime run rows from the primary, not the replica (#4378) ... ## Summary The realtime runs feed hydrates run rows from read replicas, which means it needs a replica-lag gate to avoid serving a run's previous state right after a write. Setting `REALTIME_BACKEND_NATIVE_RUN_READS_FROM_PRIMARY=1` reads those rows from each run store's primary instead, so there is no lag to gate against: no probe, no wake delay, no stale-read retries. Off by default, so nothing changes unless you set it. ## Design The run stores already decide replica-vs-primary from the *brand* on the read client they are handed: a branded replica keeps the read on the owning store's replica, an unbranded writer escalates it to that store's own primary. So this is a one-line choice at the hydrator, and it stays correct across topologies. With the run-ops split on, each leg lands on its own writer and the caller's client is never forwarded across databases; with the split off, it is the single database's primary. ```ts const runReader = new RunHydrator({ readClient: runReadsFromPrimary ? prisma : $replica, runStore, }); ``` The same flag skips constructing the lag estimator, since probing a replica the feed no longer reads would be measuring the wrong thing. Independently, `AuroraReplicaLagSource` detected Aurora by letting `aurora_replica_status()` fail, on the assumption that the app-level catch made that free. It isn't: an unresolvable function is a query error the driver reports to the error log on every sample, so a non-Aurora replica produced a continuous stream of error events while the estimator quietly fell through to its next candidate. It now resolves the function with `to_regproc` and memoizes the answer, so the unparseable call never reaches the wire.	2026-07-26 19:43:41 +01:00
Matt Aitken	be45cf9e61	fix(sdk): preserve partial assistant message on chat stream failure (#4348) ... ## Summary When a `chat.agent` (or `chat.createSession`) turn's model stream fails mid-response (e.g. a transport timeout like `UND_ERR_BODY_TIMEOUT`), the assistant output that already streamed was dropped: `onTurnComplete` fired with `responseMessage: undefined`, and the manual loop's `turn.complete()` rethrew without keeping the partial. Apps that register `hydrateMessages` are hit hardest, since boot-time tail-replay recovery is off by design. This preserves the streamed-so-far assistant output while still reporting the turn as errored, so persistence and recovery keep it. ## Scope of behavior change Only the **error path** changes. Successful turns are unaffected: the same chunks stream to the client in the same order, and backpressure/cancel behave as before. Everything here is a correctness improvement on a turn that hit a source-stream failure. ## What it does Follow-up to #4304 (`chat.pipeAndCapture`), extending the same partial-recovery to the two loops that lacked it: - **`chat.agent`**: taps the response stream (via a `TransformStream`, so pass-through backpressure and cancel are preserved) to buffer chunks, and on a source-stream failure reconstructs the partial (preferring the `onFinish` message). It's surfaced on the error-path `onTurnComplete` (`responseMessage`, `rawResponseMessage`, `uiMessages`, `newUIMessages`, `newMessages`) and committed to the accumulator so the next turn and the reboot snapshot keep it. - **`chat.createSession` / `turn.complete()`**: the reconstructed partial is accumulated (so `turn.uiMessages` reflects it and the caller can persist after catching) before `turn.complete()` rethrows. `onBeforeTurnComplete` stays skipped on the error path (it hands out a writer for a stream that has already broken). ## Correctness properties (each covered by a regression test) Each test below was confirmed to fail without its fix: - The recovered partial reaches `onTurnComplete` and the next turn's accumulated messages. - An already-committed (possibly enriched) response is not overwritten if a post-response hook then throws. - Incomplete tool parts are cleaned from the recovered partial (text kept), so the UI and model views agree and the next turn isn't poisoned. - A prior turn's model-only compaction survives an errored turn (append only the new tail, don't reconvert the full history). - A reconstructed fragment that reuses an existing message id does not clobber the complete message. - Queued `chat.response` data parts are folded into the recovered partial, matching the success path. - `newMessages` (model delta) stays symmetric with `newUIMessages`. ## Tests New `chat-agent-source-stream-error.test.ts` covers the cases above. The full `@trigger.dev/sdk` unit suite passes and the package build is green across all supported runtimes (Node 20 to 26, Bun, Deno, Cloudflare Workers). re2-prod-supervisor-dispatch-url	2026-07-24 14:20:20 +01:00
claude[bot]	109e245d56	feat(webapp): show the first and last characters of a new PAT (#4363)	2026-07-24 12:55:31 +01:00
nicktrn	bf41c5d5fc	feat(supervisor): configurable warm-start dispatch url (#4362) ... Adds an optional `TRIGGER_WARM_START_DISPATCH_URL`. The warm-start dispatch request uses it when set, otherwise falls back to `TRIGGER_WARM_START_URL`, so the dispatch target can differ from the default warm-start URL. No behavior change when unset.	2026-07-24 12:11:55 +01:00
Eric Allam	7188eecd83	perf(webapp): clamp list-endpoint page size to 100 (#4360) ... ## Summary Several list endpoints accepted an unbounded page size (`perPage` / `per_page` / `pageSize`). An unbounded page lets one request pull an arbitrarily large result set and do a proportional amount of work, which is a poor default for a shared API. This clamps the page size to 100 on every list endpoint that was uncapped, matching the existing cap on `api.v1.runs` and `api.v1.sessions`. Clamping rather than rejecting keeps existing clients working: a request for a larger page returns up to 100 items and offset pagination continues from there. ## Endpoints capped - `api.v1.schedules` (`perPage`) - `api.v1.queues` (`perPage`) - `resources.…versions` (`per_page`) - `resources.…queues` (`per_page`) - `admin.api.v1.…engine.report` (`per_page`) - `admin.api.v1.llm-models` (`pageSize`) Already capped, left as-is: `api.v1.runs`, `api.v1.sessions`, `api.v1.deployments`.	2026-07-24 11:56:58 +01:00
Eric Allam	9c85e0ecdc	perf(database): index BatchTaskRun on (runtimeEnvironmentId, createdAt, id) for the batches list (#4361) ... ## Summary The batches list page orders by `createdAt DESC, id DESC` filtered by environment and a created-at window, but the only supporting index on `BatchTaskRun` was `(runtimeEnvironmentId, id)`. That index can't satisfy the `createdAt` ordering, so on environments with a large number of batches the query fell back to a full table scan and in-memory sort, which could run long enough to hit the statement timeout. ## Fix Adds `(runtimeEnvironmentId, createdAt DESC, id DESC)` on `BatchTaskRun`. The query now reads straight from the index in order with no sort step, returning a page with only a handful of heap fetches instead of scanning the whole environment slice. The migration uses `CREATE INDEX CONCURRENTLY IF NOT EXISTS`, so it takes no table lock and is a no-op if the index already exists.	2026-07-24 11:52:34 +01:00
nicktrn	722e240e4d	feat(supervisor): add prometheus metric for outbound http requests (#4350) ... Adds Prometheus metrics so the supervisor's outbound HTTP calls are observable - including client-side failures that previously only surfaced as a log line. - `supervisor_outbound_request_total{name, method, status, outcome}` - counts every outbound request. `outcome` separates a transport failure (`network_error`), an HTTP error response (`http_error`), a response that failed schema validation (`invalid_response`), and success (`ok`). - `supervisor_outbound_request_duration_seconds{name, outcome}` - latency histogram. Leaner labels than the counter (no `status`) to avoid bucket×label cardinality; buckets match the existing dequeue-latency histogram since these calls share the same retrying HTTP client and long-poll envelope. Coverage: - The warm-start request (a one-off `fetch`) - instrumented inline; the response status code is now also included in the failure log (it was previously dropped). - All worker API client calls (`SupervisorHttpClient`: dequeue, run attempt start/complete, heartbeats, snapshots, continue, suspend, debug-log, connect) - routed through a single instrumented `request()` helper that reports via an optional `onHttpRequestComplete` callback on the client, which the supervisor wires into the counter + histogram. Low cardinality by design: `name` is a **static per-endpoint label** (e.g. `dequeue`, `start_run_attempt`), never the interpolated URL - so no run/snapshot IDs land in labels, mirroring the templated `route` labels on the inbound HTTP server. Registered on the existing metrics registry, exposed on `/metrics` with no new wiring. Internal-only change (no package release needed), so the changelog note is a single `.server-changes` entry.	2026-07-23 18:18:58 +01:00
Eric Allam	88ca0091a9	fix(docker): stop the container entrypoint printing database connection strings in logs (#4346) ... ## Summary The container entrypoint runs under `set -x`, which echoes every command to the logs with its variables expanded. Several startup guards reference full database connection strings, so the DSN (including the password) was printed to the container logs on every boot. This turns tracing off around those lines so connection strings are never traced, while leaving migration behavior and ordinary startup logging unchanged. ## Fix The leaking lines are the `[ -n "$RUN_OPS_DATABASE_URL" ]` and `[ -n "$RUN_OPS_LEGACY_DIRECT_URL" ]` guards, and the ClickHouse block (its `[ -n "$CLICKHOUSE_URL" ]` guard plus the lines that build `GOOSE_DBSTRING` from `CLICKHOUSE_URL`). `set -x` prints each of these with the credential expanded. Tracing is now disabled around each region and restored afterward, so non-secret tracing is preserved everywhere else. The existing legacy-migration subshell already protected its own command body; this adds the missing protection for the guards and the ClickHouse block. ```sh { set +x; } 2>/dev/null if [ -n "$RUN_OPS_DATABASE_URL" ]; then set -x ... ``` ## Verification Built the webapp image and ran it with dummy sentinel connection strings whose password token is `S3NTINEL_PW_DoNotLog`, then grepped the boot logs. Before (unmodified), the token appears in the traced guards: ``` + [ -n postgresql://user:S3NTINEL_PW_DoNotLog@fake-host:6432/run-ops ] + [ -n postgresql://user:S3NTINEL_PW_DoNotLog@fake-host:5432/legacy ] + [ -n https://default:S3NTINEL_PW_DoNotLog@fake-host:8443 ] ``` After, `grep S3NTINEL_PW_DoNotLog` on the same run returns nothing, and the normal "skipping ... migrations" lines still log.	2026-07-23 11:51:34 +01:00
claude[bot]	3c82248940	chore(deps): bump tar to 7.5.19 (#4345) ... Pins `tar` to `7.5.19` via a root `pnpm.overrides` entry, replacing a stale range override (`tar@>=7 <7.5.11`) that no longer matched any installed copy. The single override collapses all resolved `tar` copies onto one version: - `packages/cli-v3` — direct dependency (was 7.5.13) - `@kubernetes/client-node` (apps/supervisor) — transitive (was 7.5.13) - `cacache` — transitive (was 6.2.1) - `giget` — transitive (was 6.2.1) No source changes; cli-v3's published `^7.5.13` spec already permits `7.5.19`, so no changeset is needed.	2026-07-23 10:59:16 +01:00
Eric Allam	e9ac98b7a1	perf(run-store): route id-set reads to the owning store, not both DBs (#4342) ... ## Summary The split run-store's id-set read path (`#findRunsByIdSet`, used by the runs-list hydrate, the realtime hydrator, and engine sweeps) queried the new store for the entire id set and then probed the legacy store for the misses. A run's residency is a total function of its id (run-ops ids live in the new store, every other id in legacy), so each id belongs to exactly one store. Route each id to its owner and query each store only for its own ids, in parallel. Same result set, and while a split is active with most runs still on legacy it removes a wasted new-store query from every id-set read. ## Change `#findRunsByIdSet` now partitions the ids by `classifyResidency` and runs one bounded query per store (skipping an empty side), in parallel, mirroring `expireRunsBatch` and the single-run `#route`. `finalizeRows` still applies orderBy/take/skip globally over the merged set. This drops the id-set path's cross-store fallback, which existed to prefer the new-store copy when the same id was present in both stores. That collision cannot arise when each id maps to exactly one store (nothing writes a legacy-shaped id into the new store), so the fallback is dead code. The two id-set tests that asserted "new copy wins on collision" now assert the routing invariant: a legacy-shaped id resolves to the legacy store and the path never consults the new store. The open-predicate path (`#findRunsOpen`) is unchanged: an open `where` has no id to route on, so it still unions both stores and dedupes. docs-release-2026-07-23 v4.5.7	2026-07-22 23:03:24 +01:00