302 Commits

Author	SHA1	Message	Date
github-actions[bot]	86b948b47a	chore: release v4.5.9 (#4408)	2026-07-30 10:12:14 +01:00
Eric Allam	6e5f0f0fe7	fix(webapp,clickhouse): stop invalid customer queries alerting, and isolate Sentry scope per request (#4372) ... ## Summary A query sent to the query API with a typo in it, like a column name that does not exist, was being reported as a server error. That put customer SQL mistakes into our error alerting, where they made up almost all of the volume on one of our noisiest alerts, and it drowned out the failures that are actually ours to fix. This makes the level match who is at fault, and fixes two related problems found alongside it. ## Invalid queries are the caller's, not ours The query API route already got this right. It checks for `QueryError`, logs at warn, and returns a 400, with a comment saying the system handles it gracefully and no alert is needed. The layer underneath ignored that. `executeTSQL` logged every exception out of its catch block at error, including the compile failures the route was about to turn into a 400, and error-level logs are forwarded to error reporting. The TSQL package already draws the line we need: ```ts export class ExposedTSQLError extends BaseTSQLError { /** An exception that can be exposed to the user. */ } export class InternalTSQLError extends BaseTSQLError { /** An internal exception in the TSQL engine. */ } ``` `SyntaxError` and `QueryError` extend the first. So the catch block now branches on `ExposedTSQLError` and logs those at warn, keeping error for `InternalTSQLError` and anything unanticipated, which is a genuine compiler bug. ## SQL the caller wrote is their mistake, not ours The same asymmetry showed up one level down. A query that compiles fine can still be rejected by ClickHouse at execution, and most of those rejections mean the caller's SQL is wrong rather than that we generated something bad. This is where the volume actually is. Checking production, one error group alone, a missing `GROUP BY` on the public query API (`NOT_AN_AGGREGATE`), accounts for over a million events across hundreds of users. It is by far the largest error group in the project, and classifying only by resource limit would have left every one of those at error level. So rejections are split three ways in `ClickhouseClient`, which is the only place holding the parsed `ClickHouseError` and its symbolic type. By the time the error reaches `executeTSQL` it has been wrapped and the type is gone, and the type never appears in the message text, so it cannot be recovered by string matching. - **Resource limits** (memory ceiling, timeout, row/byte caps) log at warn. The query is valid, it just asked for more than it is allowed to spend. - **Invalid SQL** (`NOT_AN_AGGREGATE`, `UNKNOWN_IDENTIFIER`, `SYNTAX_ERROR`, the type and parse families) logs at warn **only when the caller wrote the SQL**. - **Everything else** keeps alerting. That gate matters. The client is shared, so the identical rejection on TRQL *we* generated is our bug and has to stay at error. Callers opt in with `userAuthoredQuery`: | caller | who wrote the SQL | opts in | | --- | --- | --- | | public query API | the customer | yes | | query editor | the customer | yes | | agent charts | the agent's model | yes | | built-in dashboard tiles | us, in code | no | | queue metric cards | us, in code | no | | health report | us, in code | no | The agent is the one judgement call. Its TRQL is not typed by a person, but it is also not something a code fix makes correct, so a query it gets wrong is not worth waking anyone for. The same endpoint serves built-in tiles whose TRQL we do write, so the opt-in lives with the caller rather than the route. Separately, when one of these queries did fail, the log recorded the generated ClickHouse SQL but not the query the caller actually wrote, which made the reports hard to act on. `queryWithStats` takes an optional `logFields` that `executeTSQL` uses to attach the original TSQL. ## Events were attributed to the wrong request Chasing the above turned up something broader: only a tenth of the events on that alert pointed at the query API. The rest were pinned to unrelated requests that happened to be in flight at the same time, so the alert looked like the trigger endpoint was failing. `Sentry.init` runs with `skipOpenTelemetrySetup: true`, because we register our own OTel pipeline. That skips `initOpenTelemetry`, and one of the things it does is: ```js api.context.setGlobalContextManager(new SentryContextManager()); ``` The async-context strategy is still installed, but `withIsolationScope` only marks the OTel context and delegates the actual fork to that context manager: ```js // "We depend on the otelContextManager to handle the context/hub" return api.context.with(ctx.setValue(SENTRY_FORK_ISOLATION_SCOPE_CONTEXT_KEY, true), ...) ``` `provider.register()` installed a plain `AsyncLocalStorageContextManager`, which does not know that key. The lookup found no scopes on the context and fell back to the process-global default isolation scope, so every request wrote its request data into the same object and the last writer won. The tracer now registers `SentryContextManager`, which subclasses `AsyncLocalStorageContextManager`, so OTel behaviour is unchanged. It is also registered on the path where tracing is disabled, which previously never called `register()` at all and so had no context manager of its own. Tenant tags were always correct, because those come from our own async local storage rather than the isolation scope. That is why the attribution being wrong was not obvious. This affects every error report the webapp sends, not just the query API. ## Verification `internal-packages/clickhouse`: 76 tests pass, including eight covering each level decision against a real ClickHouse container. Three pairs pin the gate open and shut at both layers: an invalid query, a compile failure, and a real limit breach driven with `max_rows_to_read` each log at warn with `userAuthoredQuery` and at error without it. The isolation fix has a test that reproduces the leak before asserting the fix. Two overlapping requests each tag their own isolation scope; with the plain context manager the slower one reads back the other's tag, and with `SentryContextManager` each reads back its own. Measured separately against a faithful reproduction of the server's wiring (own OTel pipeline, CommonJS entry) at 200 concurrent requests: per-request attribution goes from 0.5% to 100%, while span nesting, context propagation across awaits, and distinct trace IDs are identical before and after.	2026-07-30 09:04:15 +01:00
Eric Allam	4eb9292cbe	feat(webapp,run-engine): queue metrics and health dashboard (#4131) ... ## Summary Three related changes, each independently gated: **Queue metrics and health.** Per-queue depth, throughput (enqueued, started, completed), concurrency, whether a queue is throttled, and scheduling delay (how long a run waits between becoming eligible and actually starting), plus a per concurrency-key breakdown for keyed queues. Collected from inside the run queue itself, stored in ClickHouse, and surfaced on the Queues list, a new per-queue detail page, the task pages, and the run inspector. The question it answers is "does this queue have enough concurrency to keep up, and if not, which key or which limit is the constraint". **Percent-based queue concurrency limits.** A queue's concurrency override can now be expressed as a percentage of the environment limit, stored as the source of truth and re-materialized whenever the environment limit changes. Absolute overrides above the environment limit are now **rejected with a 400** instead of being silently capped, which is a behavior change on `POST /api/v1/queues/:queue/concurrency/override`. **The `health` report.** A server-computed verdict on whether work is flowing, whether the runs that do start are healthy, and whether telemetry is fresh, rendered as text with sparklines. Available as `GET /api/v1/reports/:key`, `trigger report`, and the `get_report` MCP tool (plus a `report` MCP prompt, which shows up as a slash command in hosts that support prompts). With the flags off, the Queues page renders the pre-metrics component verbatim, nothing is emitted, and nothing is written to ClickHouse. ## Configuration Two independent gates, on purpose. Emission is global so data accrues for everyone before anyone can look at it; the view is per organization so it can be turned on for one org at a time without a deploy. **Runtime flags (no restart)** | Flag | Store | Gates | | --- | --- | --- | | `queue_metrics:enabled` | run-queue Redis key (`"1"`/`"0"`, off by default) | All emission, gauges and counters. Cached in-process for 10s with stale-while-revalidate, warmed eagerly at boot so the first op after a deploy is not dropped. | | `queue_metrics:gauge_sample_rate` | run-queue Redis key, `0..1` | Fraction of queue ops that emit a gauge. Counters are never sampled, so throughput stays exact at any rate. | | `queueMetricsUiEnabled` | feature-flag catalog: global `FeatureFlag` row, per-org `Organization.featureFlags` override wins | Whether an org sees the metrics view at all: the Queues list variant, the queue detail route, the built-in Queues dashboard, the concurrency-keys endpoint, and the metrics blocks on task pages and the run inspector. Off by default; a gated org gets a 404 on the detail route rather than an empty page. | Both Redis keys are readable and writable from `/admin/queue-metrics` (super-admin UI, with a live per-shard stream-health table) and `GET`/`POST /admin/api/v1/queue-metrics` (admin PAT). The admin surface uses its own Redis client, so it works on any instance regardless of whether that instance runs the emitter or the consumer. **Environment variables (boot time)** | Variable | Default | Notes | | --- | --- | --- | | `QUEUE_METRICS_EMIT_ENABLED` | `0` | Constructs the emitter and injects it into the run engine. Without it the run queue has no emitter at all. | | `QUEUE_METRICS_CONSUMER_ENABLED` | `0` | Boots the stream consumer on this instance. Independent of emission, so consumers can be sized separately from the API. | | `QUEUE_METRICS_STREAM_SHARD_COUNT` | `4` | Stream shards, hashed per queue. | | `QUEUE_METRICS_CONSUMER_BATCH_SIZE` | `1000` | Poll batch equals insert batch, so an ack can never outrun a write. | | `QUEUE_METRICS_REDIS_{HOST,PORT,USERNAME,PASSWORD,TLS_DISABLED}` | falls back to the run-queue Redis | Set `HOST` to move the metrics stream onto a dedicated instance so a metrics backlog cannot compete with the run queue for memory. Self-hosters can leave it unset and get a single-Redis deployment. | | `QUEUE_METRICS_COUNTER_STREAM_MAXLEN` | `2000000` shared, `8000000` dedicated | Bound on how much a stalled consumer can hold. The default is deliberately lower when the stream shares the queue-critical Redis. | | `QUEUE_METRICS_COUNTER_ODOMETER_TTL_SECONDS` | `604800` | TTL on the per-queue cumulative counter key, refreshed on every write, so only queues idle for the whole window are purged. | | `QUEUE_METRICS_MAX_QUEUE_NAMES_PER_ENV` | `1000` | Distinct queue names tracked per environment; overflow collapses into `__overflow__`. | | `QUEUE_METRICS_MAX_CONCURRENCY_KEYS_PER_QUEUE` | `10000` | Same idea one level down, per queue. | | `QUEUE_METRICS_GAUGE_SAMPLE_RATE` | `1` | Default for the live sample-rate key above. | | `QUEUE_METRICS_QUERY_TABLES_VISIBLE` | `0` | Lists the queue-metrics tables in the Query page, its schema docs, the schema API and the AI query context. Off keeps them unlisted while the feature is dark; a query naming them still runs either way. | | `QUEUE_METRICS_CLICKHOUSE_URL` | falls back to the shared wiring | Runs queue metrics on their own ClickHouse service: the consumer's inserts and every queue-metrics read go through it, so a metrics-heavy chart refresh never competes with runs-list or trace reads. Unset reproduces the previous split exactly (inserts on `CLICKHOUSE_URL`, reads on the query pool). | | `QUEUE_METRICS_CLICKHOUSE_READER_URL` | the write URL | Reader split, so the consumer's inserts can never land on a read endpoint. | | `QUEUE_METRICS_CLICKHOUSE_{KEEP_ALIVE_ENABLED,KEEP_ALIVE_IDLE_SOCKET_TTL_MS,MAX_OPEN_CONNECTIONS,LOG_LEVEL,COMPRESSION_REQUEST}` | `1`, unset, `10`, `info`, `1` | Pool tuning, matching the other per-workload ClickHouse clients. | Migrations to apply: ClickHouse `036_create_queue_metrics_v1.sql`, and a Postgres migration adding the nullable `TaskQueue.concurrencyLimitOverridePercent`. Both are additive. ## How collection works Queue operations produce two kinds of signal, and they have opposite failure modes, so they are handled differently. **Gauges** (queued, running, queue limit, env queued, env running, env limit, throttled, plus keys-with-backlog and worst-key wait on keyed queues) are read *inside* the same Redis script that performs the enqueue or dequeue, so the reading is atomic with the operation it describes rather than a racy follow-up read. The script returns them on its reply and the app forwards them to the stream. Gauges are sampled and drop-tolerant: they are aggregated with `max`, so a lost reading costs resolution, never correctness. **Counters** (enqueued, started, completed, plus nack and dead-lettered) are cumulative odometers. Each event increments a per-queue key on the metrics Redis and emits the absolute total, and ClickHouse takes the difference across buckets at read time. This is the important property of the design: a summed-delta counter undercounts permanently on any lost event, while a cumulative one self-heals, because the next surviving reading restates the whole total. Only bucket granularity can be lost, never the total. A queue returning after its odometer TTL expired restarts at 1 and reset detection handles it, which is safe precisely because expiry only spans a window with no activity. Both land on one sharded Redis stream. A consumer reads it with a consumer group, reclaims stale pending entries on a 15s interval rather than on every poll, maps one entry to one or two ClickHouse rows (whole-queue and, for keyed queues, per-key), and acks only after the insert lands. Each batch carries a dedup token derived from its stream-entry ids, and the target tables set `non_replicated_deduplication_window`, so a retried batch cannot double-count either the raw rows or the aggregates that hang off them. Consumer and emitter both emit OTel metrics (`queue_metrics.emitter.emitted`, `queue_metrics.consumer.{entries,rows_inserted,insert_errors,insert_duration,stream_depth,group_lag,pending,lag_unknown}`); stream depth and group lag are the two worth alerting on, and `lag_unknown` exists because Redis can report a null lag after a trim, which must not be read as zero. ## Storage and read path `queue_metrics_raw_v1` is a short landing table with a 6 hour TTL. Four aggregate tiers are materialized straight from raw, never cascaded off each other, each with a 30 day TTL: - `queue_metrics_v1`, 10 second buckets per queue, the default read path - `queue_metrics_5m_v1`, 5 minute buckets per queue, for wide ranges and cross-queue ranking - `env_metrics_v1`, 10 second buckets per environment, queue-independent so it stays cheap at any range - `queue_metrics_ck_v1`, 10 second buckets per concurrency key Every tier is an MV from raw because the counter states do not survive a cascade: their merge is order sensitive, so a `-MergeState` chain off the 10s table inflates the result, and the same property means an aggregate state may only be merged inside one queue. That constraint is now enforced by the query engine rather than by reviewer discipline: a column can declare a `mergeGroupKey`, and any query that references it without grouping by, or pinning to a single value of, every named key fails to compile with an actionable message. On the read side, TRQL gains three tables (`queue_metrics`, `env_metrics`, and a `queue_metrics_by_key` that is hidden from the editor, schema docs and schema API but still queryable, so per-key rows can never silently merge into a plain per-queue query), plus `deltaSumTimestampMerge` and `quantilesTDigestMerge`. Two schema-level optimizations ride along: a table can declare coarser rollups, so a query whose bucket interval is 5 minutes or wider is routed to the 5m table with no change to the query itself, and it can opt into the ClickHouse query cache with time bounds floored to a fixed grid, so the auto-refreshing dashboards actually share cache entries instead of missing on every tick. Both are caller-side substitutions, so the printer stays unaware of physical layout. All of this can also live on its own ClickHouse service. A table declares the pool its reads run on, the three queue-metrics tables name the dedicated one, and the ingestion consumer writes through the same client, so both directions move together with one env var and nothing else routes differently. The other engine change is opt-in gap filling: charts can request rows for empty buckets, where counters zero-fill and gauges carry forward. Grouped gauge series are densified per group and carried inside a partition, so a quiet queue's line holds its last value without bleeding another queue's value into it. ## Queue concurrency limits `concurrencyLimitOverridePercent` on `TaskQueue` is the source of truth when an override is set as a percentage; the absolute `concurrencyLimit` is materialized from it (floored, clamped to at least 1 so a percentage can never act as a pause, and never above the environment limit). Every path that changes an environment limit now recalculates the environment's percent-based overrides afterwards, outside the transaction, and pushes changed limits to the engine. The push is attempted even when the stored value did not change, so a previously failed sync self-heals rather than leaving the database and the engine diverged; paused queues are skipped so a recalculation cannot effectively unpause one. The API accepts exactly one of `concurrencyLimit` or `percent`, and the reject-instead-of-clamp change above means a request asking for more than the environment allows now fails loudly. The percent bound (greater than 0, at most 100) is defined once and shared by the zod schema, the dashboard mutation handler and the service, so the three cannot drift. The concurrency-keys table on a queue is now paginated against the ClickHouse per-key tier, ranked by peak backlog with the total on every row from a single scan, and only the keys on the current page are enriched with live counts from Redis. That replaces a hard top-50 cap with something whose cost is a function of page size rather than key cardinality. ## The health report `GET /api/v1/reports/:key?period=&format=markdown|ansi|json`. The verdict is computed on the server and is deterministic, not model-generated. Three independent analyzers run over one input snapshot: flow (is work moving, and if not, is the cause a limit, throttling, one bad queue, or dead-lettering), execution (are the runs that start succeeding, and at what latency), and liveness (how fresh is the telemetry). When telemetry is genuinely stale, the first two are forced to unknown and every actionable field is stripped, so no surface ever advises action off stale data. Authorization is per query table rather than a blanket query grant: a JWT must be scoped to every table the report reads (`runs`, `env_metrics`, `queue_metrics`), so a narrowly scoped token cannot pull a report that reads more than it was granted. `period` is validated as a shorthand with a 90 day ceiling at the edge. The report catalog is a registry of `{ load, interpret }` entries, so the next report is a new entry and no change to the route, the view model, the renderers, the CLI or the MCP tool. `trigger mcp` no longer launches the install wizard when stdout is a TTY, which fixed a real failure: hosts spawn the server over a PTY, so the wizard would open and the client would time out waiting for a server that never started. The wizard now needs `trigger mcp --install`. ## The part that is live regardless of every flag The enqueue and dequeue scripts now return a 2-tuple so a gauge reading can ride back on the reply. Every return site in the eight affected scripts is wrapped, and a `nil` original is converted to `false` on the way out, because a raw `nil` in the first slot would make Lua truncate the multi-bulk reply and silently drop the gauge on the throttled and empty-queue paths. The reply shape and the destructuring on the app side are exercised on every queue operation whether or not metrics are enabled, so that is the part of `run-engine` worth the closest review. One behavior fix in the same area: the scheduling-delay anchor is set only on a run's first entry into the queue. Anchoring it to trigger time on re-enqueues made waitpoint and checkpoint resumes report the entire wait as scheduling delay. Queue ordering is untouched, so a re-enqueued run keeps its position, and nacks deliberately keep the original anchor because a rolled-back dequeue is the same continuous wait. A pending-version promotion still anchors to trigger time, on purpose: that promotion is the run's first real entry into the queue, since the trigger deliberately held it back waiting for a worker version, and the TTL is armed at the same point for the same reason. The consequence is worth naming, because it is a judgement call: a run that waits on a deployment reports that wait as scheduling delay on its queue, which is time unrelated to queue capacity. ## Verification Unit and integration suites across the new package, the run queue, the mapping layer, the query engine and ClickHouse (including a test that applies migration 036 through the same splitter CI uses, and a regression test that inserts the same batch three times to prove the aggregates do not inflate). Beyond that, the whole path was driven end to end against a live stack with real runs: emitter to Redis stream to consumer to ClickHouse to the dashboards, for both the local dev path and the deployed path where a supervisor drives the dequeue, with assertions on exact counter reconstruction per queue and per concurrency key, throttling, environment saturation, scheduling delay, and a deliberate mid-stream reading drop to confirm the cumulative counters still reconstruct the correct total. The gated-off state was checked on every touched surface. The dedicated ClickHouse service was verified against a second, separately-schema'd instance: with it configured, the driven counters reconstruct exactly on the dedicated instance, the shared instance gains no rows for that window, a read through the query API returns the value that exists only on the dedicated instance, and a `runs` query still succeeds (it would fail outright if it were mis-routed to a service without that table). With the variable unset, the full suite passes unchanged. --------- Co-authored-by: Katia Bulatova <katia@trigger.dev> Co-authored-by: Katia Bulatova <katherine.bulatova@gmail.com> Co-authored-by: James Ritchie <james@trigger.dev>	2026-07-29 16:45:24 +01:00
claude[bot]	639eaf6e82	fix(webapp): don't apply an invite's role to an existing org member (#4409) ... <!-- ccr-slack-attribution --> _Requested via [Slack thread](https://triggerdotdev.slack.com/archives/C097ZHVKZFA/p1785249693523749)_ ## Summary Accepting an old invitation could change the role of someone who was already in the organization. A long-pending invite can carry a lower role than the member has since been promoted to, so accepting it was a silent demotion. When the accepting user was the organization's only Owner, the role layer refused that demotion, and the refusal (an expected, protective outcome) was logged as an error. An invitation now only sets a role on a membership the accept actually created, and people who are already in an organization are skipped when invitations are sent. ## How `acceptInvite` already skipped the `OrgMember` create when it found an existing membership, but the `rbac.setUserRole` call below it was gated only on `invite.rbacRoleId`. It now also tracks whether this accept created the membership. A create that loses the unique-constraint race counts as pre-existing, since whichever flow won it owns that membership's role. Skipping existing members outright would regress one case: a member with no RBAC role at all would never receive the invitation's role. `ensureOrgMember` handles that with `healMissingRoleAssignment`, which fills in a null role but never overwrites a real one, so `assignInviteRbacRole` takes the same gate. An established role is never touched; an absent one is filled in. `assignInviteRbacRole` branches on the result's machine-readable `code` instead of logging every refusal at `error`. `last_owner` goes to `logger.info`, matching the two directory-sync role paths; everything else, including a refusal that carries no code, goes to `logger.warn`. The helper is best-effort and never throws, so no outcome it produces warrants `error`. No string matching on the error text is involved. `inviteMembers` resolves the organization's members by email and skips those addresses before creating invites. The invite table's `@@unique([organizationId, email])` only dedupes *pending invites*, so it could never catch this. ## Invite surfaces Skipping addresses means a batch can now come back empty, and neither caller handled that: - The dashboard action built its redirect from `invites[0].organization`, so a batch where every address was skipped threw a `TypeError` that reached the admin as a raw error string. It also reported the submitted count rather than the created one. It now names what it skipped ("No invitations sent: 1 already a member of this organization") and counts what it actually created. - The invites API derived `alreadyInvited` as "everything not created", so an existing member was reported as though they had already been invited. `inviteMembers` now returns the two groups separately and the endpoint reports `alreadyMembers` alongside `alreadyInvited`. ## Testing `apps/webapp/test/member.server.test.ts` passes 16/16 locally, up from 12. Getting there needed a harness fix. The `~/db.server` mock did not export `Prisma`, so any code reaching `PrismaNamespace.PrismaClientKnownRequestError` threw before it could branch, leaving every duplicate-key path in `member.server.ts` unreachable from tests. The mock now re-exports the real `Prisma`, and there is a case covering the pending-invite skip. New cases: the invite role is applied when the accept creates the membership; it is not applied when the member already has a role; it is applied when an existing member has no role assigned; the organization is still joined when the assignment is refused with `last_owner`; and `inviteMembers` reports members separately from pending invites. Forcing the gate off fails exactly the "already has a role" case, so the coverage is load-bearing. `pnpm run typecheck --filter webapp` and `oxfmt --check` both pass. ## Changelog Accepting an old invitation could change the role of someone who was already in the organization. An invitation now leaves an existing member's role untouched, people who are already in an organization are no longer sent invitations to it, and the invite form says which addresses it skipped instead of failing with an unhelpful error. --- ## ✅ Checklist - [x] I have followed every step in the [contributing guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md) - [x] The PR title follows the convention. - [x] I ran and tested the code works. ## Screenshots No visual changes. The invite form's toast copy changes, as described above. --------- Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: Matt Aitken <matt@mattaitken.com>	2026-07-29 15:31:59 +01:00
Saadi Myftija	44eca4d166	feat(webapp): org-gated internal API origin in run env vars (#4366) ... Adds an opt-in way for operators to route deployed runs' API traffic through a different origin than the public one, per organization. Set `INTERNAL_API_ORIGIN` on the webapp and enable the `internalApiOriginEnabled` feature flag (globally or per org, with the org override winning in both directions): deployed runs for enabled orgs then get `TRIGGER_API_URL` set to the internal origin instead of `API_ORIGIN`. Useful for gradually moving run traffic onto a private network path. ## Design The origin is resolved when an attempt starts, so flag changes take effect on the next attempt and roll back the same way, with no task redeploys. The org override is read fresh per attempt; the global default comes from the cached flags registry (a cold read fails safe to the public origin). When `INTERNAL_API_ORIGIN` is unset the flag is a no-op and no extra queries run, so existing deployments are unaffected. Dev runs always use the public origin, and `TRIGGER_STREAM_URL` remains unchanged.	2026-07-28 11:28:09 +02:00
claude[bot]	ec562c0e68	fix(webapp): remove unused Electric sync trace routes (#4400) ... <!-- ccr-slack-attribution --> _Requested by **Eric Allam** · [Slack thread](https://triggerdotdev.slack.com/archives/C0AU83M3136/p1785222101937829?thread_ts=1785207509.304669&cid=C0AU83M3136)_ Removes two dead Remix routes and the helpers only they used. `app/routes/sync.traces.runs.$traceId.ts` (`/sync/traces/runs/:traceId`) and `app/routes/sync.traces.$traceId.ts` (`/sync/traces/:traceId`) were added with the original ElectricSQL run page and lost their only consumers when the dashboard hooks that called them were deleted. Nothing in the repo references either route today. Also removed, because the deleted routes were their only callers: - `OtelTraceIdSchema`, `RESERVED_ELECTRIC_SHAPE_PARAMS`, `TraceScope`, `buildElectricTraceWhereClause` from `app/v3/electricShape.server.ts` (the file stays — `UNSAFE_REALTIME_TAG_CHARS` / `sanitizeRealtimeTagForSql` / `sanitizeRealtimeTagsForSql` are still used by `realtime.v1.runs.ts` and `realtimeClient.server.ts`) - the loader-specific cases in `apps/webapp/test/spanTraceRoutes.replicaLag.test.ts` and `internal-packages/run-store/src/runOpsStore.routesSpanTraceReadView.replicaLag.test.ts` `app/utils/longPollingFetch.ts` is untouched — `realtimeClient.server.ts` still uses it. `runOpsStore.ts` / `PostgresRunStore.ts` are untouched too; the unrouted-lookup mechanism there is generic and stays. As a plain code fact: the run lookup these loaders performed keyed on `TaskRun.traceId` alone, which is not an index-backed query shape. That is noted only as context for why the code is not worth keeping around unused. ### Judgement call worth a maintainer's opinion The request was specifically about `/sync/traces/runs/:traceId`, the route that looks up a run by `traceId`. This PR **also** deletes its sibling `/sync/traces/:traceId`. The reasoning: - both routes came in with the same ElectricSQL run-page work - both lost their only consumers in the same later commit - neither has any caller anywhere in the repo - they share the same helper module, so keeping one means keeping the helpers half-used If you would rather keep the sibling, reverting just that one file deletion is easy and does not affect the rest of this PR — say the word and I will restore it along with the helpers it needs. ## ✅ Checklist - [x] I have followed every step in the [contributing guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md) - [x] The PR title follows the convention. - [x] I ran and tested the code works --- ## Testing Verification run locally from the repo root: | Command | Result | | --- | --- | | `pnpm run format` | clean, no changes produced | | `pnpm run lint:fix` | clean | | `pnpm run lint` | pass (exit 0, no findings) | | `pnpm run typecheck --filter webapp` | pass | | `pnpm run typecheck --filter @internal/run-store` | pass | A ripgrep sweep for `sync.traces`, `sync/traces`, `syncTraceRunsLoader`, `buildElectricTraceWhereClause`, `OtelTraceIdSchema` and `RESERVED_ELECTRIC_SHAPE_PARAMS` (excluding `node_modules`) returns zero hits. **Not fully verified:** both edited test files are testcontainers suites and need a Docker runtime, which was not available in my environment. I confirmed each file *collects* correctly with exactly the three intended remaining tests and no import errors — notably, dropping the `session.server` / `controlPlaneResolver.server` / `longPollingFetch` / `env.server` mocks does not break module loading for the surviving loaders. The assertions themselves then failed only on `Could not find a working container runtime strategy`. CI should be the real signal here. Per `apps/webapp/CLAUDE.md`, `pnpm run build --filter webapp` was deliberately not run. --- ## Changelog Removed two unused sync routes left over from the original ElectricSQL run page, along with the helpers and tests that existed only to serve them. No behaviour change — neither route had any caller. --- ## Screenshots _n/a — no user-visible surface changes._ 💯 --------- Co-authored-by: Claude <noreply@anthropic.com>	2026-07-28 09:00:48 +01:00
github-actions[bot]	d189ce17d3	chore: release v4.5.8 (#4364) ... ## Summary 2 new features, 9 improvements, 3 bug fixes. ## Highlights - Allow additional environment API keys to create scoped public access tokens through the Trigger.dev API. Use server-issued public access tokens for batch operations so environment-scoped API keys can read batch results. ([#4387](https://github.com/triggerdotdev/trigger.dev/pull/4387)) ## Improvements - Preserve the partial assistant message when a chat turn's model stream fails mid-response. `chat.agent` now passes the recovered partial to `onTurnComplete`, and `chat.createSession`'s `turn.complete()` keeps it before rethrowing, instead of dropping the streamed-so-far output. ([#4348](https://github.com/triggerdotdev/trigger.dev/pull/4348)) ## Server changes These changes affect the self-hosted Docker image and Trigger.dev Cloud: - Favorite any dashboard page to a new Favorites section in the side menu, and customize the sidebar by renaming favorites, hiding items, and reordering items and sections. ([#4375](https://github.com/triggerdotdev/trigger.dev/pull/4375)) - List API endpoints now clamp the page size to a maximum of 100. Requests asking for a larger page size return up to 100 items and keep paginating, rather than pulling an unbounded page. ([#4360](https://github.com/triggerdotdev/trigger.dev/pull/4360)) - Organizations without billing alerts now get default spend alert thresholds, so you're notified before usage grows unexpectedly. The billing limit page no longer pre-selects an option before you've set a limit and prompts you to configure one. Alert previews now update immediately after you change your billing limit. ([#4328](https://github.com/triggerdotdev/trigger.dev/pull/4328)) - When you create a Personal Access Token, the generated token now shows its first and last few characters instead of being fully hidden, so you can confirm you copied the right value. ([#4363](https://github.com/triggerdotdev/trigger.dev/pull/4363)) - Add metrics to the realtime backend that measure how often a single changed run is served to multiple subscriptions in one batch. ([#4341](https://github.com/triggerdotdev/trigger.dev/pull/4341)) - Realtime run subscriptions can now be configured to read run data straight from the primary database, so a run's latest state is never served from a lagging replica. Off by default; replica reads are unchanged unless you turn it on. ([#4378](https://github.com/triggerdotdev/trigger.dev/pull/4378)) - SSO and Directory Sync are no longer restricted to Enterprise plans — get in touch and we can turn them on for your organization whatever plan you're on. ([#4393](https://github.com/triggerdotdev/trigger.dev/pull/4393)) - Improved supervisor observability: it now reports metrics for its outbound requests, making failed calls to upstream services easier to monitor. ([#4350](https://github.com/triggerdotdev/trigger.dev/pull/4350)) - The runs list on a task's page now updates live — run statuses change and newly triggered runs appear without a manual refresh, matching the main Runs page. ([#4377](https://github.com/triggerdotdev/trigger.dev/pull/4377)) - Speed up the Batches list page for environments with a large number of batches, which could previously time out while loading. ([#4361](https://github.com/triggerdotdev/trigger.dev/pull/4361)) - Container startup no longer prints database and ClickHouse connection strings (with credentials) to the logs. ([#4346](https://github.com/triggerdotdev/trigger.dev/pull/4346)) - The tasks page no longer runs two queries whose results were never displayed, cutting wasted work on every page load and removing a source of hidden server errors ([#4380](https://github.com/triggerdotdev/trigger.dev/pull/4380)) <details> <summary>Raw changeset output</summary> # Releases ## @trigger.dev/build@4.5.8 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.8` ## trigger.dev@4.5.8 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.8` - `@trigger.dev/build@4.5.8` - `@trigger.dev/schema-to-json@4.5.8` ## @trigger.dev/core@4.5.8 ### Patch Changes - Allow additional environment API keys to create scoped public access tokens through the Trigger.dev API. Use server-issued public access tokens for batch operations so environment-scoped API keys can read batch results. ([#4387](https://github.com/triggerdotdev/trigger.dev/pull/4387)) ## @trigger.dev/python@4.5.8 ### Patch Changes - Updated dependencies: - `@trigger.dev/sdk@4.5.8` - `@trigger.dev/core@4.5.8` - `@trigger.dev/build@4.5.8` ## @trigger.dev/react-hooks@4.5.8 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.8` ## @trigger.dev/redis-worker@4.5.8 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.8` ## @trigger.dev/rsc@4.5.8 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.8` ## @trigger.dev/schema-to-json@4.5.8 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.8` ## @trigger.dev/sdk@4.5.8 ### Patch Changes - Preserve the partial assistant message when a chat turn's model stream fails mid-response. `chat.agent` now passes the recovered partial to `onTurnComplete`, and `chat.createSession`'s `turn.complete()` keeps it before rethrowing, instead of dropping the streamed-so-far output. ([#4348](https://github.com/triggerdotdev/trigger.dev/pull/4348)) - Allow additional environment API keys to create scoped public access tokens through the Trigger.dev API. Use server-issued public access tokens for batch operations so environment-scoped API keys can read batch results. ([#4387](https://github.com/triggerdotdev/trigger.dev/pull/4387)) - Updated dependencies: - `@trigger.dev/core@4.5.8` </details> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>	2026-07-27 16:40:31 +01:00
James Ritchie	d30ee6e570	feat(webapp): favorite pages and sidebar customization (#4375) ... ## Summary Favorite any dashboard page and it appears in a new "Favorites" section at the top of the side menu. The star next to the page title (or Option+F) saves the exact view, filters and tabs included, with a name derived from the URL ("Runs: Completed successfully, last 7d", "Run: 05hrqq9n") that you can rename inline from each item's hover menu. The sidebar is customizable too: "Customize sidebar" (on section header menus and in each "More" menu) opens a modal where you can reorder sections, drag items into a new order, hide items behind a per-section "More" popover, and rename or remove favorites. Changes apply on Confirm, Reset restores the default layout without touching favorites, and everything is stored per user in dashboard preferences. ## Screenshots | Favorites in the side menu | Customize sidebar modal | | --- | --- | |  |  |  ## Design notes - Favorite links carry a small marker search param so the favorite, not its identical main menu item, highlights as active. Markers from shared or stale links are cleaned on load, and changing any filter hands the highlight back to the regular menu item. - Preference writes are serialized with a row lock: several writers (debounced collapse and width saves, favorite toggles, the customize modal) can land concurrently and would otherwise clobber each other's read-modify-write of the JSON column. - Option+F is matched on `event.code` with a raw listener because macOS reports Option-modified letters as symbols, which the `event.key` based shortcut hook can't capture. Verified end-to-end in the browser: star toggle and shortcut, instant section appearance, inline rename and staged modal removal, filter-aware labels and unique active states, shared-link normalization, drag reordering, and persistence across reloads.	2026-07-27 16:29:36 +01:00
James Ritchie	3e53404f40	feat(webapp): live-update the runs list on task pages (#4377) ... ## Summary The runs list on a task's page now updates live, matching the main Runs page. Run rows update their status, duration, and cost in place as runs progress, and a "N new runs" button appears in the header when newer runs come in so you can pull them into the list without a manual refresh. This applies to both standard and scheduled task pages. ## Design It reuses the Runs page's polling hook. A task page scopes its runs by the task in the URL path rather than a `tasks` query filter, so the hook now takes an optional task slug and scopes new-run detection to it. The "new runs" button sits in the header, outside the deferred runs table, so the count is lifted to the page and the click action is passed through a ref. That keeps the table streaming on first load instead of blocking the header on the runs query. When newer runs come in, a `1 new run` button appears in the task page header, to the left of the time filter. Clicking it pulls the new runs into the list.	2026-07-27 15:41:47 +01:00
Matt Aitken	269470fd87	feat(webapp): gate SSO on an entitlement instead of the Enterprise plan (#4393) ... The SSO & Directory Sync settings page decided access by comparing the organization's plan code against the literal string `"enterprise"`. The webapp now reads a `hasSso` entitlement from plan limits. ## Changes - **`settings.sso` route** — `planAllowsSso` reads `limits.hasSso` rather than the plan code; the loader and the action gate on a shared `getSsoEntitlement` helper. - **`platform.v3.server`** — new `getSsoEntitlement(orgId)` returning `entitled | not_entitled | unknown`, behind a new SWR cache namespace (60s fresh / 120s stale, memory + Redis). This replaces an uncached billing round-trip that previously ran on every settings load, so the page gets cheaper than it was. - **`directorySyncEffects`** — the entitlement is now checked before applying membership effects, per organization and memoised across a batch. - **`@trigger.dev/platform` 1.2.0 → 1.3.0** — required, see below. ## Behaviour worth reviewing **Revocation now stops SCIM.** Previously the plan check existed only on the settings page, so an org that lost access kept receiving directory-sync pushes indefinitely; only the config UI froze. Provision *and* deprovision are gated, so a revoked entitlement can't remove members either. **An unreadable entitlement throws instead of skipping.** Effects are idempotent and the worker retries, so retrying is lossless where dropping would silently lose a directory change. It's raised at `warn` level so a transient billing blip doesn't page anyone. **The login path is deliberately untouched.** A hard entitlement check there turns a billing outage into a login outage. Consequence: an org that loses the entitlement keeps its existing SSO logins working until the connection is removed. Gating sign-in is a separate decision. **Self-hosted is unaffected.** With no billing service configured the helper returns `entitled`, leaving plugin presence and the kill switch as the only gates — a self-hoster who installed the plugin isn't locked out of it. ## The dependency bump is load-bearing The `Limits` schema is a plain `z.object`, so it *strips* unknown keys. On 1.2.0 the `hasSso` field was silently discarded during parsing and read as `undefined` no matter what billing sent — a structural accessor would not have helped. Verified against both builds: ``` 1.2.0 → parsed: true | hasSso survives: false 1.3.0 → parsed: true | hasSso survives: true ``` This PR therefore cannot merge before 1.3.0 is published, which it now is. ## Testing `apps/webapp/test/directorySyncEffects.server.test.ts` — 7 tests over the gate: applies when entitled, skips provision and deprovision when not, throws a warn-level retryable error when unreadable, resolves once per org across a batch, and gates per org so one unentitled org doesn't block another. `pnpm run typecheck --filter webapp` passes (18/18), oxfmt and oxlint clean.	2026-07-27 13:08:16 +01:00
claude[bot]	d91818f198	fix(webapp): remove unawaited task list metrics promises (#4380) ... <!-- ccr-slack-attribution --> _Requested via [Slack thread](https://triggerdotdev.slack.com/archives/C097ZHVKZFA/p1785082528841609)_ `TaskListPresenter` created promises that nothing ever consumed. Two of the three deferred metrics promises it returned had no reader, no `await` and no `.catch()`, so when the query behind one of them failed the rejection had nowhere to go. ## Before / After **Before** - `TaskListPresenter.call()` returned four things: `tasks`, `activity`, `runningStats` and `durations`. Its only caller reads `tasks` and `runningStats`. - Every load of the tasks page therefore fired two ClickHouse queries whose results were thrown away. - If either of those two queries failed, the resulting promise rejection was unhandled — nothing was awaiting it and nothing had attached an error handler, so it surfaced as an unhandled rejection at the process level rather than as an error anyone could attribute to a request. **After** - `TaskListPresenter.call()` returns `tasks` and `runningStats` only. - Two fewer queries run per tasks-page load. - There is no longer an unconsumed promise that can reject without a handler. `runningStats` is awaited by its caller, so its failures continue to be handled the way they always were. Nothing changes on screen: the tasks page renders `hourlyActivity` and `runningStates`, and neither of the removed values fed either of those. ## How The removed values were verified unreferenced before deleting anything: - `TaskListPresenter` has exactly one caller, `UnifiedTaskListPresenter`, which reads `taskResult.tasks` and `taskResult.runningStats` and nothing else. - No file anywhere in the repo — app code, tests, or type re-exports — reads an `activity` or `durations` field off the presenter's result. - `UnifiedTaskListPresenter` builds its own `unifiedTaskListHourlyActivity` query for the 24h chart the page actually renders, which is what made the presenter's separate 7-day daily activity data redundant. - `getDailyTaskActivity` and `getAverageDurations` on `ClickHouseEnvironmentMetricsRepository` had no callers other than the two lines being deleted, so they and their now-orphaned helpers and types were removed too. Changes: - `apps/webapp/app/presenters/v3/TaskListPresenter.server.ts` — drop the `activity` and `durations` fields (both from the main return and from the no-current-worker early return) and the two repository calls behind them. Drop the unreferenced `TaskActivity` type alias. The "don't await this" comment on the remaining `runningStats` promise now spells out that the caller has to consume it. - `apps/webapp/app/services/environmentMetricsRepository.server.ts` — remove `getDailyTaskActivity` and `getAverageDurations` from the `EnvironmentMetricsRepository` interface and its ClickHouse implementation, along with `fillInDailyTaskActivity` and the `DailyTaskActivity` / `AverageDurations` types. `getCurrentRunningStats` is the control that shows the diagnosis is right. It throws on query failure in exactly the same way as the two removed methods — `if (queryError) throw queryError` — but it never produced an unhandled rejection, because `UnifiedTaskListPresenter` passes its promise into a `Promise.all(...).then(...)` chain that the route then awaits. Same failure mode, opposite outcome, and the only difference is whether anything consumes the promise. Follow-ups, not in this PR: - `AgentListPresenter` returns three sparkline promises in the same shape and they look similarly unconsumed. Left alone here to keep this change reviewable. - With these two callers gone, the `getTaskActivity` and `getAverageDurations` query builders in `@internal/clickhouse` have no remaining callers in this repo. Whether to remove them is a separate call for someone who owns that package. ## ✅ Checklist - [x] I have followed every step in the [contributing guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md) - [x] The PR title follows the convention. - [x] I ran and tested the code works --- ## Testing - `pnpm run typecheck --filter webapp` — passes. This is the meaningful check here: it proves nothing still references the removed fields, methods or types. - `pnpm run format` and `pnpm run lint:fix` — clean, no changes produced. - No test file referenced the removed symbols, so no test needed updating. --- ## Changelog Server-only change, so this carries a `.server-changes/` note rather than a changeset: `.server-changes/task-list-remove-unused-metrics-queries.md`. > The tasks page no longer runs two queries whose results were never displayed, cutting wasted work on every page load and removing a source of hidden server errors --- ## Screenshots _No visual change — the removed data was never rendered._ Co-authored-by: Claude <noreply@anthropic.com>	2026-07-27 11:39:43 +01:00
Eric Allam	d3906241a5	feat(webapp): read realtime run rows from the primary, not the replica (#4378) ... ## Summary The realtime runs feed hydrates run rows from read replicas, which means it needs a replica-lag gate to avoid serving a run's previous state right after a write. Setting `REALTIME_BACKEND_NATIVE_RUN_READS_FROM_PRIMARY=1` reads those rows from each run store's primary instead, so there is no lag to gate against: no probe, no wake delay, no stale-read retries. Off by default, so nothing changes unless you set it. ## Design The run stores already decide replica-vs-primary from the *brand* on the read client they are handed: a branded replica keeps the read on the owning store's replica, an unbranded writer escalates it to that store's own primary. So this is a one-line choice at the hydrator, and it stays correct across topologies. With the run-ops split on, each leg lands on its own writer and the caller's client is never forwarded across databases; with the split off, it is the single database's primary. ```ts const runReader = new RunHydrator({ readClient: runReadsFromPrimary ? prisma : $replica, runStore, }); ``` The same flag skips constructing the lag estimator, since probing a replica the feed no longer reads would be measuring the wrong thing. Independently, `AuroraReplicaLagSource` detected Aurora by letting `aurora_replica_status()` fail, on the assumption that the app-level catch made that free. It isn't: an unresolvable function is a query error the driver reports to the error log on every sample, so a non-Aurora replica produced a continuous stream of error events while the estimator quietly fell through to its next candidate. It now resolves the function with `to_regproc` and memoizes the answer, so the unparseable call never reaches the wire.	2026-07-26 19:43:41 +01:00
claude[bot]	109e245d56	feat(webapp): show the first and last characters of a new PAT (#4363)	2026-07-24 12:55:31 +01:00
Eric Allam	7188eecd83	perf(webapp): clamp list-endpoint page size to 100 (#4360) ... ## Summary Several list endpoints accepted an unbounded page size (`perPage` / `per_page` / `pageSize`). An unbounded page lets one request pull an arbitrarily large result set and do a proportional amount of work, which is a poor default for a shared API. This clamps the page size to 100 on every list endpoint that was uncapped, matching the existing cap on `api.v1.runs` and `api.v1.sessions`. Clamping rather than rejecting keeps existing clients working: a request for a larger page returns up to 100 items and offset pagination continues from there. ## Endpoints capped - `api.v1.schedules` (`perPage`) - `api.v1.queues` (`perPage`) - `resources.…versions` (`per_page`) - `resources.…queues` (`per_page`) - `admin.api.v1.…engine.report` (`per_page`) - `admin.api.v1.llm-models` (`pageSize`) Already capped, left as-is: `api.v1.runs`, `api.v1.sessions`, `api.v1.deployments`.	2026-07-24 11:56:58 +01:00
Eric Allam	9c85e0ecdc	perf(database): index BatchTaskRun on (runtimeEnvironmentId, createdAt, id) for the batches list (#4361) ... ## Summary The batches list page orders by `createdAt DESC, id DESC` filtered by environment and a created-at window, but the only supporting index on `BatchTaskRun` was `(runtimeEnvironmentId, id)`. That index can't satisfy the `createdAt` ordering, so on environments with a large number of batches the query fell back to a full table scan and in-memory sort, which could run long enough to hit the statement timeout. ## Fix Adds `(runtimeEnvironmentId, createdAt DESC, id DESC)` on `BatchTaskRun`. The query now reads straight from the index in order with no sort step, returning a page with only a handful of heap fetches instead of scanning the whole environment slice. The migration uses `CREATE INDEX CONCURRENTLY IF NOT EXISTS`, so it takes no table lock and is a no-op if the index already exists.	2026-07-24 11:52:34 +01:00
nicktrn	722e240e4d	feat(supervisor): add prometheus metric for outbound http requests (#4350) ... Adds Prometheus metrics so the supervisor's outbound HTTP calls are observable - including client-side failures that previously only surfaced as a log line. - `supervisor_outbound_request_total{name, method, status, outcome}` - counts every outbound request. `outcome` separates a transport failure (`network_error`), an HTTP error response (`http_error`), a response that failed schema validation (`invalid_response`), and success (`ok`). - `supervisor_outbound_request_duration_seconds{name, outcome}` - latency histogram. Leaner labels than the counter (no `status`) to avoid bucket×label cardinality; buckets match the existing dequeue-latency histogram since these calls share the same retrying HTTP client and long-poll envelope. Coverage: - The warm-start request (a one-off `fetch`) - instrumented inline; the response status code is now also included in the failure log (it was previously dropped). - All worker API client calls (`SupervisorHttpClient`: dequeue, run attempt start/complete, heartbeats, snapshots, continue, suspend, debug-log, connect) - routed through a single instrumented `request()` helper that reports via an optional `onHttpRequestComplete` callback on the client, which the supervisor wires into the counter + histogram. Low cardinality by design: `name` is a **static per-endpoint label** (e.g. `dequeue`, `start_run_attempt`), never the interpolated URL - so no run/snapshot IDs land in labels, mirroring the templated `route` labels on the inbound HTTP server. Registered on the existing metrics registry, exposed on `/metrics` with no new wiring. Internal-only change (no package release needed), so the changelog note is a single `.server-changes` entry.	2026-07-23 18:18:58 +01:00
Eric Allam	88ca0091a9	fix(docker): stop the container entrypoint printing database connection strings in logs (#4346) ... ## Summary The container entrypoint runs under `set -x`, which echoes every command to the logs with its variables expanded. Several startup guards reference full database connection strings, so the DSN (including the password) was printed to the container logs on every boot. This turns tracing off around those lines so connection strings are never traced, while leaving migration behavior and ordinary startup logging unchanged. ## Fix The leaking lines are the `[ -n "$RUN_OPS_DATABASE_URL" ]` and `[ -n "$RUN_OPS_LEGACY_DIRECT_URL" ]` guards, and the ClickHouse block (its `[ -n "$CLICKHOUSE_URL" ]` guard plus the lines that build `GOOSE_DBSTRING` from `CLICKHOUSE_URL`). `set -x` prints each of these with the credential expanded. Tracing is now disabled around each region and restored afterward, so non-secret tracing is preserved everywhere else. The existing legacy-migration subshell already protected its own command body; this adds the missing protection for the guards and the ClickHouse block. ```sh { set +x; } 2>/dev/null if [ -n "$RUN_OPS_DATABASE_URL" ]; then set -x ... ``` ## Verification Built the webapp image and ran it with dummy sentinel connection strings whose password token is `S3NTINEL_PW_DoNotLog`, then grepped the boot logs. Before (unmodified), the token appears in the traced guards: ``` + [ -n postgresql://user:S3NTINEL_PW_DoNotLog@fake-host:6432/run-ops ] + [ -n postgresql://user:S3NTINEL_PW_DoNotLog@fake-host:5432/legacy ] + [ -n https://default:S3NTINEL_PW_DoNotLog@fake-host:8443 ] ``` After, `grep S3NTINEL_PW_DoNotLog` on the same run returns nothing, and the normal "skipping ... migrations" lines still log.	2026-07-23 11:51:34 +01:00
Katia Bulatova	23d5771d56	feat(webapp): unconfigured billing limit UX and default billing alerts (#4328) ... ## Default billing alerts + billing limit page UX - New orgs get default billing alerts: $5, $100, $500, $1000, $2500. Existing orgs are backfilled by a billing-side data migration (companion [PR](https://github.com/triggerdotdev/cloud/pull/1657)). - The billing limit form starts with nothing selected for orgs that never set a limit — the save button appears once an option is picked. - The yellow banner now also shows on the billing limits page itself, asking to configure a limit. Hidden everywhere for members who can't manage billing. - Also fixes billing limit alert preview. Tests - `apps/webapp/test/billingLimitsRoute.test.ts` — dirty logic for empty/selected mode - `apps/webapp/test/billingAlertsDefaults.test.ts` — default values - `apps/webapp/test/billingAlertsFormat.test.ts` — preview after a limit change	2026-07-22 21:23:52 +02:00
Eric Allam	a2d382b2be	feat(webapp): add emission fan-out metrics to the native realtime feed (#4341) ... ## Summary Adds two counters to the native realtime backend so we can see how much duplicate row serialization the change router does per batch. When a run changes it can match several held feeds at once (a run subscription plus one or more tag/list feeds), and today each matching feed serializes that run's wire value independently. These counters quantify that fan-out so we can decide whether a shared serialization step is worth it. ## What they measure - `realtime_native.emission_run_serializations`: total wire-value serializations performed across feeds per batch (what the current path does). - `realtime_native.emission_distinct_serializations`: distinct (columns, run) rows those serializations cover (what a serialize-once-per-batch step would do). Average feeds-per-run is `run_serializations / distinct_serializations`, and `1 - distinct / run_serializations` is the serialization work a shared step could save. Wired through a new optional `onEmissionFanout` callback on the router. No behavior change.	2026-07-22 17:48:44 +01:00
github-actions[bot]	aafc333523	chore: release v4.5.7 (#4319) ... ## Summary 5 improvements, 5 bug fixes. ## Improvements - Add `node-24` and `node-26` as supported `runtime` options in `trigger.config.ts`. The `experimental-node-24` and `experimental-node-26` names are now deprecated aliases and emit a deprecation warning; switch to `node-24` / `node-26` instead. ([#4337](https://github.com/triggerdotdev/trigger.dev/pull/4337)) ```ts import { defineConfig } from "@trigger.dev/sdk"; export default defineConfig({ runtime: "node-24", project: "<your-project-ref>", }); ``` - Avoid logging task run environment variable values at debug level ([#4336](https://github.com/triggerdotdev/trigger.dev/pull/4336)) - Custom chat agent loops get two ergonomic wins for owning the turn loop. ([#4304](https://github.com/triggerdotdev/trigger.dev/pull/4304)) `chat.writeTurnComplete()` now returns the turn boundary's resume cursors (`lastEventId` for the output stream and `sessionInEventId` for the input stream), so you can persist them straight from the task instead of round-tripping them back from the client. ```ts const { lastEventId, sessionInEventId } = await chat.writeTurnComplete(); await db.chats.update(chatId, { lastEventId, sessionInEventId }); ``` `chat.pipeAndCapture()` no longer throws when a stream is stopped or fails. It now returns a `PipeAndCaptureResult` whose `message` holds any partial output captured before the stop or failure, alongside a typed `status` (`"complete" | "aborted" | "error"`) and, on failure, the `error`. Read the message off the result: ```ts const { message, status, error } = await chat.pipeAndCapture(result, { signal, }); if (message) conversation.addResponse(message); if (status === "error") logger.error("turn failed", { error }); ``` Note: `pipeAndCapture` previously resolved to `UIMessage | undefined`. Update call sites to read `.message` from the returned result. - Suppress a build-time warning that could appear in Vite-based projects when the optional `@ai-sdk/otel` package is not installed. ([#4188](https://github.com/triggerdotdev/trigger.dev/pull/4188)) ## Bug fixes - Fixes intermittent `trigger dev` run crashes where a run could fail at boot with a cryptic `Cannot find module .../dev-run-worker.mjs` after a rebuild had cleaned up the build directory the run was launched against. Dev runs now retry cleanly instead of hard-crashing when their build directory is missing, the dev watchdog no longer removes the build tree of a still-running session, and a run assigned to a worker version that was superseded by a rebuild now fails fast with a clear message instead of silently hanging until it times out. ([#4276](https://github.com/triggerdotdev/trigger.dev/pull/4276)) ## Server changes These changes affect the self-hosted Docker image and Trigger.dev Cloud: - Refreshed the side menu: separate organization and account menus, a new project switcher, and the menu is now resizable by dragging its edge. The account Profile page has also been redesigned. ([#4066](https://github.com/triggerdotdev/trigger.dev/pull/4066)) - Allow different organization members to use the same development branch name without sharing or colliding with each other's branch environments. ([#4323](https://github.com/triggerdotdev/trigger.dev/pull/4323)) - Limit account settings email input to 254 characters. ([#4330](https://github.com/triggerdotdev/trigger.dev/pull/4330)) - Prevent duplicate Staging and Preview environments when account setup requests overlap ([#4261](https://github.com/triggerdotdev/trigger.dev/pull/4261)) - Fix the docs link on the empty Prompts page, which pointed to a page that no longer exists. ([#4247](https://github.com/triggerdotdev/trigger.dev/pull/4247)) <details> <summary>Raw changeset output</summary> # Releases ## @trigger.dev/build@4.5.7 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.7` ## trigger.dev@4.5.7 ### Patch Changes - Fixes intermittent `trigger dev` run crashes where a run could fail at boot with a cryptic `Cannot find module .../dev-run-worker.mjs` after a rebuild had cleaned up the build directory the run was launched against. Dev runs now retry cleanly instead of hard-crashing when their build directory is missing, the dev watchdog no longer removes the build tree of a still-running session, and a run assigned to a worker version that was superseded by a rebuild now fails fast with a clear message instead of silently hanging until it times out. ([#4276](https://github.com/triggerdotdev/trigger.dev/pull/4276)) - Add `node-24` and `node-26` as supported `runtime` options in `trigger.config.ts`. The `experimental-node-24` and `experimental-node-26` names are now deprecated aliases and emit a deprecation warning; switch to `node-24` / `node-26` instead. ([#4337](https://github.com/triggerdotdev/trigger.dev/pull/4337)) ```ts import { defineConfig } from "@trigger.dev/sdk"; export default defineConfig({ runtime: "node-24", project: "<your-project-ref>", }); ``` - Avoid logging task run environment variable values at debug level ([#4336](https://github.com/triggerdotdev/trigger.dev/pull/4336)) - Updated dependencies: - `@trigger.dev/core@4.5.7` - `@trigger.dev/build@4.5.7` - `@trigger.dev/schema-to-json@4.5.7` ## @trigger.dev/core@4.5.7 ### Patch Changes - Add `node-24` and `node-26` as supported `runtime` options in `trigger.config.ts`. The `experimental-node-24` and `experimental-node-26` names are now deprecated aliases and emit a deprecation warning; switch to `node-24` / `node-26` instead. ([#4337](https://github.com/triggerdotdev/trigger.dev/pull/4337)) ```ts import { defineConfig } from "@trigger.dev/sdk"; export default defineConfig({ runtime: "node-24", project: "<your-project-ref>", }); ``` ## @trigger.dev/python@4.5.7 ### Patch Changes - Updated dependencies: - `@trigger.dev/sdk@4.5.7` - `@trigger.dev/core@4.5.7` - `@trigger.dev/build@4.5.7` ## @trigger.dev/react-hooks@4.5.7 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.7` ## @trigger.dev/redis-worker@4.5.7 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.7` ## @trigger.dev/rsc@4.5.7 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.7` ## @trigger.dev/schema-to-json@4.5.7 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.7` ## @trigger.dev/sdk@4.5.7 ### Patch Changes - Custom chat agent loops get two ergonomic wins for owning the turn loop. ([#4304](https://github.com/triggerdotdev/trigger.dev/pull/4304)) `chat.writeTurnComplete()` now returns the turn boundary's resume cursors (`lastEventId` for the output stream and `sessionInEventId` for the input stream), so you can persist them straight from the task instead of round-tripping them back from the client. ```ts const { lastEventId, sessionInEventId } = await chat.writeTurnComplete(); await db.chats.update(chatId, { lastEventId, sessionInEventId }); ``` `chat.pipeAndCapture()` no longer throws when a stream is stopped or fails. It now returns a `PipeAndCaptureResult` whose `message` holds any partial output captured before the stop or failure, alongside a typed `status` (`"complete" | "aborted" | "error"`) and, on failure, the `error`. Read the message off the result: ```ts const { message, status, error } = await chat.pipeAndCapture(result, { signal, }); if (message) conversation.addResponse(message); if (status === "error") logger.error("turn failed", { error }); ``` Note: `pipeAndCapture` previously resolved to `UIMessage | undefined`. Update call sites to read `.message` from the returned result. - Suppress a build-time warning that could appear in Vite-based projects when the optional `@ai-sdk/otel` package is not installed. ([#4188](https://github.com/triggerdotdev/trigger.dev/pull/4188)) - Updated dependencies: - `@trigger.dev/core@4.5.7` </details> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>	2026-07-22 15:51:29 +01:00
Matt Aitken	81eac67069	fix(webapp): correct docs link on the blank prompts page (#4247) ... ## Summary The empty-state panel on the Prompts page linked to a docs path that no longer exists, so the "Prompts docs" button returned a 404. It now points to the current prompts documentation at /docs/ai/prompts, matching the link already used in the page header.	2026-07-22 12:51:06 +01:00
Chris Arderne	7a14188663	fix(webapp): limit account email address length (#4330) ... ## Summary Limits user account email addresses to 254 characters in profile settings and onboarding. Oversized values are rejected before the uniqueness lookup, and the form fields enforce the same limit in the browser. ## Fix Both email update flows use a shared bounded email schema. Basic validation completes before the uniqueness lookup runs.	2026-07-22 10:29:30 +01:00
Chris Arderne	bb34a2e224	fix(webapp): scope development branches to each member (#4323) ... ## Summary Allow each organization member to use the same development branch name without colliding with another member's environment. Fixes #4320. ## Fix Development branches now use the existing member-scoped project, slug, and organization-member key for upserts. Preview branches retain their project-wide shortcode behavior. New development branches receive distinct shortcodes while keeping their readable, member-scoped slugs. Existing branches continue to resolve through the member-scoped key, so this requires no migration or backfill.	2026-07-22 08:20:43 +01:00
James Ritchie	0b2919465c	feat(webapp): redesign the side menu project and organization menus (#4066) ... Redesign of the main side menu: separates Projects and Accounts from the Organization menu and makes the menu resizable. **Main changes** - **Organization & Account menus**: the top-left is now a dedicated organization menu (Settings, Usage, Billing, Team, SSO, integrations), with a separate account menu beside it (Profile, PATs, Security, Logout). - **Project switcher**: a new Project section above the Environment selector. - **Resizable side menu**: drag the right edge to set a custom width (saved per user), or click the edge to collapse/expand. - **Environment selector**: reworked to match the Project menu, including dev-branch handling. - **Account Profile page**: redesigned into the Security page's row-and-divider layout. Preview URL: https://samejr-org-menu-update.triggerlabs.dev/ https://github.com/user-attachments/assets/9b199576-6037-4ea6-9bdb-3ee15265b8c2	2026-07-21 17:25:17 +01:00
Chris Arderne	6642c8b785	fix(webapp): prevent duplicate envs from provision race (#4261) ... Fixes TRI-12078 ## Summary Prevents concurrent environment setup requests from creating duplicate Staging and Preview environments. ## Fix Adds database-enforced uniqueness for root Staging and Preview environments. If two requests race, the losing request loads the environment created by the winner and continues successfully instead of creating a duplicate or returning an error.	2026-07-21 15:40:48 +01:00
github-actions[bot]	325b906319	chore: release v4.5.6 (#4317) ... ## Summary 5 improvements, 9 bug fixes. ## Breaking changes - Self-hosted deployments no longer ship shared default credentials; fresh installs generate their own. If yours still uses a previously published default, set a unique value before upgrading, or set `ALLOW_INSECURE_DEFAULT_SECRETS=true` to keep booting while you migrate. ([#4316](https://github.com/triggerdotdev/trigger.dev/pull/4316)) ## Improvements - Require explicit browser approval for CLI and MCP login, with resilient polling while approval is pending. ([#4316](https://github.com/triggerdotdev/trigger.dev/pull/4316)) - Deployed task telemetry now reports the deployment identifier (e.g. `deployment_abc123`) in the `worker.id` attribute, instead of an opaque internal value. Upgrade to get the readable identifier in your own OpenTelemetry exporters. ([#4316](https://github.com/triggerdotdev/trigger.dev/pull/4316)) - Prevent prototype pollution when applying run metadata operations or reconstructing nested telemetry attributes, while preserving legitimate `constructor` and `prototype` fields. ([#4316](https://github.com/triggerdotdev/trigger.dev/pull/4316)) - Add helpers to mint and verify the deployment-scoped token used to authenticate run controllers to the platform. ([#4316](https://github.com/triggerdotdev/trigger.dev/pull/4316)) ## Server changes These changes affect the self-hosted Docker image and Trigger.dev Cloud: - Added optional request rate limiting for telemetry ingestion endpoints. ([#4316](https://github.com/triggerdotdev/trigger.dev/pull/4316)) - Background-worker deployment lookups are now scoped to the authenticated environment. ([#4316](https://github.com/triggerdotdev/trigger.dev/pull/4316)) - Updating a GitHub App installation from the callback flow is now scoped to your own organization, so an installation ID belonging to another organization can no longer be used to refresh that organization's installation record. The GitHub App installation session is also now single-use, so completing an installation callback invalidates its state and it can no longer be replayed. ([#4316](https://github.com/triggerdotdev/trigger.dev/pull/4316)) - Scope schedule and environment-variable writes to the caller's project and environment ([#4316](https://github.com/triggerdotdev/trigger.dev/pull/4316)) - Reject compute snapshot callbacks that do not match the snapshot request that created them. ([#4316](https://github.com/triggerdotdev/trigger.dev/pull/4316)) - Require secret-key authentication to initialize the session out (agent→client) stream, matching the append route. ([#4316](https://github.com/triggerdotdev/trigger.dev/pull/4316)) - Live run and trace subscriptions now validate their identifiers more strictly and only return data from your own organization. ([#4316](https://github.com/triggerdotdev/trigger.dev/pull/4316)) - Window-function names in the query compiler are now validated against the allowlist, matching how other function calls are handled. ([#4316](https://github.com/triggerdotdev/trigger.dev/pull/4316)) - Authenticate run controllers to the platform with a signed, deployment-scoped token. ([#4316](https://github.com/triggerdotdev/trigger.dev/pull/4316)) - Verify that worker actions (starting, completing, and continuing a run, and reading its snapshots) target a run belonging to the caller's environment. ([#4316](https://github.com/triggerdotdev/trigger.dev/pull/4316)) <details> <summary>Raw changeset output</summary> # Releases ## @trigger.dev/build@4.5.6 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.6` ## trigger.dev@4.5.6 ### Patch Changes - Require explicit browser approval for CLI and MCP login, with resilient polling while approval is pending. ([#4316](https://github.com/triggerdotdev/trigger.dev/pull/4316)) - Deployed task telemetry now reports the deployment identifier (e.g. `deployment_abc123`) in the `worker.id` attribute, instead of an opaque internal value. Upgrade to get the readable identifier in your own OpenTelemetry exporters. ([#4316](https://github.com/triggerdotdev/trigger.dev/pull/4316)) - Updated dependencies: - `@trigger.dev/core@4.5.6` - `@trigger.dev/build@4.5.6` - `@trigger.dev/schema-to-json@4.5.6` ## @trigger.dev/core@4.5.6 ### Patch Changes - Prevent prototype pollution when applying run metadata operations or reconstructing nested telemetry attributes, while preserving legitimate `constructor` and `prototype` fields. ([#4316](https://github.com/triggerdotdev/trigger.dev/pull/4316)) - Require explicit browser approval for CLI and MCP login, with resilient polling while approval is pending. ([#4316](https://github.com/triggerdotdev/trigger.dev/pull/4316)) - Add helpers to mint and verify the deployment-scoped token used to authenticate run controllers to the platform. ([#4316](https://github.com/triggerdotdev/trigger.dev/pull/4316)) ## @trigger.dev/python@4.5.6 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.6` - `@trigger.dev/build@4.5.6` - `@trigger.dev/sdk@4.5.6` ## @trigger.dev/react-hooks@4.5.6 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.6` ## @trigger.dev/redis-worker@4.5.6 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.6` ## @trigger.dev/rsc@4.5.6 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.6` ## @trigger.dev/schema-to-json@4.5.6 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.6` ## @trigger.dev/sdk@4.5.6 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.6` </details> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>	2026-07-21 12:07:52 +01:00
Chris Arderne	6997aeb05e	fix: security release 2026-07-08 (#4316)	2026-07-21 12:00:58 +01:00
github-actions[bot]	1cbe25bd1d	chore: release v4.5.5 (#4267) ... ## Summary 5 improvements, 5 bug fixes. ## Improvements - Add experimental Node.js 24 and 26 task runtimes. Set `runtime` to `experimental-node-24` or `experimental-node-26` in `trigger.config.ts`. ([#4085](https://github.com/triggerdotdev/trigger.dev/pull/4085)) - Add `defaultRegion` to the project GET and list API responses; null when unset. ([#4146](https://github.com/triggerdotdev/trigger.dev/pull/4146)) ## Server changes These changes affect the self-hosted Docker image and Trigger.dev Cloud: - Transient internal sync failures are now retried quietly instead of surfacing as errors. ([#4270](https://github.com/triggerdotdev/trigger.dev/pull/4270)) - Optionally route ClickHouse read traffic to a read replica while writes stay on the primary. Set `CLICKHOUSE_READER_URL` to move all reads, or target the busiest paths with `RUNS_LIST_CLICKHOUSE_URL` (runs list) and `EVENTS_READER_CLICKHOUSE_URL` (traces, spans, logs). All optional; unset keeps current behavior. ([#4081](https://github.com/triggerdotdev/trigger.dev/pull/4081)) - Remove the deprecated realtime stream write endpoint used by retired v3 task clients. ([#4250](https://github.com/triggerdotdev/trigger.dev/pull/4250)) - Fix batchTrigger requests that set a per-item idempotency key failing with an error instead of creating and deduplicating the runs ([#4271](https://github.com/triggerdotdev/trigger.dev/pull/4271)) - Speed up idempotency checks on `batchTrigger` calls that use idempotency keys. Large batches against a task with a big run history no longer degrade to multi-second lookups. ([#4255](https://github.com/triggerdotdev/trigger.dev/pull/4255)) - The "Preview branches" usage on the Limits page now counts only preview branches. ([#4283](https://github.com/triggerdotdev/trigger.dev/pull/4283)) - Avoid opening a redundant database connection pool when the legacy and primary databases are the same server, preventing connection usage from doubling. ([#4253](https://github.com/triggerdotdev/trigger.dev/pull/4253)) - Fix pages occasionally loading unstyled or failing to load during a deploy. The dashboard now reloads automatically to recover. ([#4282](https://github.com/triggerdotdev/trigger.dev/pull/4282)) <details> <summary>Raw changeset output</summary> # Releases ## @trigger.dev/build@4.5.5 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.5` ## trigger.dev@4.5.5 ### Patch Changes - Add experimental Node.js 24 and 26 task runtimes. Set `runtime` to `experimental-node-24` or `experimental-node-26` in `trigger.config.ts`. ([#4085](https://github.com/triggerdotdev/trigger.dev/pull/4085)) - Updated dependencies: - `@trigger.dev/core@4.5.5` - `@trigger.dev/build@4.5.5` - `@trigger.dev/schema-to-json@4.5.5` ## @trigger.dev/core@4.5.5 ### Patch Changes - Add experimental Node.js 24 and 26 task runtimes. Set `runtime` to `experimental-node-24` or `experimental-node-26` in `trigger.config.ts`. ([#4085](https://github.com/triggerdotdev/trigger.dev/pull/4085)) - Add `defaultRegion` to the project GET and list API responses; null when unset. ([#4146](https://github.com/triggerdotdev/trigger.dev/pull/4146)) ## @trigger.dev/python@4.5.5 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.5` - `@trigger.dev/build@4.5.5` - `@trigger.dev/sdk@4.5.5` ## @trigger.dev/react-hooks@4.5.5 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.5` ## @trigger.dev/redis-worker@4.5.5 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.5` ## @trigger.dev/rsc@4.5.5 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.5` ## @trigger.dev/schema-to-json@4.5.5 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.5` ## @trigger.dev/sdk@4.5.5 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.5` </details> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>	2026-07-20 14:01:37 +01:00
Chris Arderne	cecdfd94be	fix: only count preview branches toward the preview branch limit (#4283)	2026-07-17 15:54:25 +00:00
nicktrn	0ff0abd776	fix(webapp): recover from stale /build assets via a bounded reload (#4282) ... ## Problem The webapp's HTML references content-hashed `/build` assets, and each running instance contains exactly one build and returns 404 for asset hashes it doesn't have. During a rolling deploy a client can hold HTML from one build while a request for one of its assets is served by an instance on a different build → missing styles or a failed chunk load. ## What this does On a `/build` stylesheet/script/chunk load failure, the client does a **bounded full-document reload** (at most 2 per 5 minutes, tracked in `sessionStorage`) so the page reloads onto a single consistent build. That's the whole mechanism — no polling, no `fetch` interception, no blocking overlay, no form snapshotting. - `apps/webapp/app/components/StaleAssetRecovery.tsx` — authored as a typed, lint-checked function and serialized to an inline script via `.toString()` (so the logic is real, reviewable code, not an opaque string), injected before `<Links />`, production only. - Detection: capture-phase `error` listener for `<link>`/`<script>`/modulepreload failures under `/build/`, plus an `unhandledrejection` guard for dynamic-import failures. - Guards: once-per-page re-entrancy guard, the bounded reload budget, and a `navigator.onLine` check so it never reloads into an offline error page. - Unit tests in `StaleAssetRecovery.test.ts`. ## Relationship to #4260 Replaces the recovery introduced in #4260 (reverted in #4280) with a much smaller, reload-only approach — the previous version intercepted `fetch` and could turn a data request into a navigation, and showed a full-screen overlay on any asset error; this drops both. ## `/build-version` compatibility shim `apps/webapp/server.ts` adds a tiny `GET /build-version` endpoint (build id only, `no-store`). A previously-deployed client build polls it after an asset failure and reloads once it sees a newer build, so those older tabs recover in one reload instead of getting stuck. Temporary — safe to remove once older clients have cycled out. It deliberately does **not** re-add an `X-Build-Id` response header. ## Also Restores the `.server-changes` writing guidance in `.claude/rules/server-apps.md` (reverted alongside #4260). ## Self-hosting note Recovery is most reliable when your load balancer keeps a client on one instance for the duration of a deploy (short session stickiness) — the reload then lands on a consistent build in one hop.	2026-07-17 16:11:25 +01:00
Chris Arderne	73d966ad22	chore(webapp): remove deprecated realtime stream write action (#4250) ... Removes the deprecated realtime stream write action kept for retired v3 task clients. Supported clients use the targeted stream write routes, while the existing stream read loader remains unchanged.	2026-07-17 13:49:54 +01:00
nicktrn	051d7080d6	Revert "fix(webapp): survive asset hash rotation across rolling deploys (#4260)" (#4280) ... Standard `git revert` of #4260. Its client-side stale-asset recovery is net-negative during normal deploys: - The `fetch` interception treats any `?_data=` request (Remix loader **and** action traffic) as a navigation and, on a build-id mismatch, `location.assign`es the tab to the fetched URL — an open dashboard tab can be hard-navigated to a raw data URL during a rolling deploy, losing unsaved input. - Any transient `/build` asset error (a network blip, an extension, an unrelated failed dynamic import) blanks the page behind a full-screen overlay for ~60s before offering a manual reload. - It serialized form field values to `sessionStorage` to restore them across the reload. This returns the webapp to the pre-#4260 baseline as a fast, low-risk step. Follow-ups (separate PRs): - a minimal reload-only recovery to replace this, - restore the unrelated `.claude/rules/server-apps.md` docs tidy-up from #4260 (via cherry-pick), - a load-balancer stickiness change addressing the root cause.	2026-07-17 14:42:52 +02:00
Eric Allam	43250522a5	fix(run-store): fix batch idempotency lookup on the dedicated run-ops store (#4271) ... ## Summary `batchTrigger` requests that set a per-item `idempotencyKey` failed with a 500 when the run-store is split across databases: the per-item idempotency lookup errored before any run was created. Batches without per-item keys, single `trigger` idempotency, and batch-level (`idempotency-key` header) idempotency were unaffected. ## Root cause `findRunsByIdempotencyKeys` built its `UNION ALL` of per-key point-lookups with `@trigger.dev/database`'s `Prisma.sql` / `Prisma.join`, then executed it on whichever store client it was handed. On the dedicated run-ops store that client is a *separate* generated Prisma client, and a `Sql` object from a different generated client is not recognized: the bare `$queryRaw(Prisma.join(...))` form dropped the query text entirely (`Argument \`query\` is missing`). The tagged-template form is no better here: joining nested `Prisma.sql` fragments across the two clients mis-numbers the bound parameters (`syntax error at or near "$1"`). ## Fix Build the lookup as a plain parameterized string and run it via `$queryRawUnsafe` with positional placeholders and bound values, so it no longer depends on which generated client executes it. The query text contains only static SQL and integer placeholders; every value (`runtimeEnvironmentId`, `taskIdentifier`, each key) is bound, so it is not a raw-interpolation site. Same per-key point-lookup shape as before, no change on the single-client path. Verified end-to-end against a bundled build with the run-store split enabled: before the fix, `batchTrigger` with a per-item key 500s; after, it returns the runs and dedups correctly across fresh, repeat, and mixed batches.	2026-07-15 19:36:12 +01:00
Iss	80cbc46bf6	fix(webapp): log transient Attio 5xx/429 at warn instead of error (#4270) ... The signup → Attio sync (`attio.server.ts` `#assert`) logged every non-2xx response at `error` level and threw the same way regardless of status. Transient upstream failures (5xx/429) are retried by the common worker and self-heal, so treating them as errors created false alerts for something that isn't actually a bug. Now `#assert` splits the two cases: - **5xx / 429** — Logged at `warn` and thrown with `logLevel: "warn"`, so they continue to be retried but don't raise error-level alerts. This reuses the same pattern the worker already honors (`directorySyncEffects`). - **4xx** — Unchanged: logged at `error` and thrown, so genuine integration bugs (schema, permissions, auth, etc.) remain visible. There is no behavior change to retries or the signup flow. This is a server-only change. --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-07-15 10:57:39 -04:00
nicktrn	890dd66eb5	feat(webapp): route ClickHouse reads to an optional read replica (#4081) ... ## Summary Adds optional configuration to send ClickHouse read traffic to a separate instance (for example a read replica) while writes stay on the primary `CLICKHOUSE_URL`. This lets operators offload read load (runs list, traces, logs, queries) from the cluster that handles inserts. Fully backwards compatible: with nothing new set, every client resolves to `CLICKHOUSE_URL` exactly as before. ## What it adds - `CLICKHOUSE_READER_URL` (optional): a single reader endpoint that the read-only clients fall back to. Read clients resolve `<own URL> ?? CLICKHOUSE_READER_URL ?? CLICKHOUSE_URL`. The task-events client (which both inserts events and reads traces, spans, and logs) is built as a reader/writer pair so queries use the reader while inserts stay on `CLICKHOUSE_URL`. - `RUNS_LIST_CLICKHOUSE_URL` (optional): a dedicated client for the runs list (dashboard list, runs list API, live reload, child-status counts), so the highest-traffic read path can target its own instance. ## Safety Only read-only clients fall back to the reader: logs, query, admin, runs list, the pending-version lookup, and the realtime run-id resolver. The query page is constrained to read-only (the TSQL parser rejects anything that is not a `SELECT`, and a `readonly` setting is applied). The task-events client routes inserts to the writer and queries to the reader per method, so a write can never reach the reader. Pure-write clients (event inserts, replication) always use `CLICKHOUSE_URL`. Note: this PR targets a baseline branch rather than `main` so the diff stays scoped to the read-replica changes. It will be retargeted to `main` before merge. --------- Co-authored-by: Eric Allam <eallam@icloud.com>	2026-07-15 14:59:42 +01:00
Eric Allam	1ab5066ed0	perf(webapp,run-store): point-lookup batch idempotency keys (#4255) ... ## Summary Batch triggers that use per-item idempotency keys could take seconds instead of milliseconds when the target task had a large run history. This keeps the idempotency lookup fast regardless of how many runs a task has accumulated. ## Root cause The batch path checks which items already have runs by looking up their idempotency keys with a single `WHERE runtimeEnvironmentId = ? AND taskIdentifier = ? AND idempotencyKey IN (...)` query. On a very large `TaskRun` table Postgres underestimates the row count of a specific `(environment, task)` pair, so once the `IN` list grows past a handful of keys it stops doing per-key index probes and instead scans every run for that `(environment, task)` and filters the keys in memory. The cost is then flat and large regardless of how many keys are being checked, and a routine `ANALYZE` does not correct the estimate at that table size. ## Fix Look each idempotency key up on its own, batched into a `UNION ALL` of point lookups (chunked, run with bounded concurrency). Each branch is an equality on all three columns of the unique index, so the planner can only do a per-key index probe and can never fall back to the range scan. Same results, same columns, confined to the batch trigger path.	2026-07-15 08:25:41 +01:00
Katia Bulatova	2aa64200f8	fix(webapp): survive asset hash rotation across rolling deploys (#4260) ... ### Problem Webapp HTML references content-hashed /build assets, and each Docker image contains exactly one build with a hard 404 for unknown hashes. During a rolling deploy, a client holding HTML from the old build may request old asset hashes from a replica running the new image, causing missing styles or failed chunk loads. The page should recover automatically once a compatible build becomes available, without reload loops or unnecessary interruptions during normal deployments. ### What changed - Build changes alone do nothing — no polling, no automatic reloads. - If a CSS or JavaScript asset fails to load, a recovery overlay is shown immediately. - While the server still reports the same build, the client polls for a newer build using exponential backoff (up to ~60s). As soon as a newer build is detected, the page reloads automatically. - If no newer build appears within the timeout, recovery falls back to a manual Reload action. - If recovery still fails after the automatic reload, the client stops retrying and displays a final recovery screen instead of entering a reload loop. - Recovery preserves form values and scroll position across the automatic reload.	2026-07-14 23:48:21 +01:00
Daniel Sutton	a1ca64613b	fix(webapp): reuse the primary db pool for legacy run-ops when DSNs match (#4253) ... ## Summary When the run-ops split is enabled, the legacy run-ops database client was always constructed as its own connection pool, even when it points at the same database as the primary (control-plane) client. On setups where those two DSNs resolve to the same physical database, this opened a second, redundant pool and doubled the number of connections used against that database. This change makes the legacy client reuse the primary client's pool whenever their DSNs point at the same database, and only open a separate pool when they genuinely differ. ## Fix A small `sameDatabaseTarget` comparison (host, port, database name, user) decides whether the legacy DSN points at the same database as the primary. When it does, the legacy handle reuses the primary client by reference, so no second pool is opened. When the DSNs diverge, the legacy client is built independently as before, so the split still works once the databases are actually separate. Two smaller changes ride along: - An optional per-pool limit for the run-ops read replica, which connects unpooled and so draws raw backend connections; unset, it falls back to the existing default and behaviour is unchanged. - A startup warning about a missing legacy replica URL is now suppressed when the legacy client shares the primary pool, where it would be misleading. ## Verification Booted the webapp end-to-end in three modes and confirmed the pools opened as expected via the client's own startup logs and live backend connection counts: split off (single pool), split on with a shared database (legacy reuses the primary pool, no doubling), and split on with separate databases (legacy opens its own pool).	2026-07-14 11:20:21 +01:00
github-actions[bot]	165955781d	chore: release v4.5.4 (#4228) ... ## Summary 2 new features, 11 improvements, 5 bug fixes. ## Breaking changes - Trigger.dev v3 is no longer supported. For self-hosted deployments, 4.5.0 is the last version we officially support for running v3; stay on 4.5.0 or upgrade to v4. v3 triggers, batch triggers, reschedules, and deploys now return a clear upgrade message instead of running. ([#4236](https://github.com/triggerdotdev/trigger.dev/pull/4236)) ## Improvements - You can now mark environment variables synced via the `syncEnvVars` build extension as secrets. Return `{ name, value, isSecret: true }` from your callback and those variables are stored redacted in the dashboard, just like manually created secret env vars. ([#4203](https://github.com/triggerdotdev/trigger.dev/pull/4203)) - Remove the legacy `--mcp` and `--mcp-port` options from the `dev` command. Run the dedicated `trigger mcp` command to start the Trigger.dev MCP server. ([#4246](https://github.com/triggerdotdev/trigger.dev/pull/4246)) - Removed the unused `ResourceMonitor` export from `@trigger.dev/core/v3/serverOnly`. It was a server-side logging helper with no remaining consumers. ([#4244](https://github.com/triggerdotdev/trigger.dev/pull/4244)) - Removed the unused `@trigger.dev/core/v3/zodNamespace` export and the legacy v3 socket message schemas. These were only used by the now-retired v3 engine and have no v4 consumers. ([#4236](https://github.com/triggerdotdev/trigger.dev/pull/4236)) ## Bug fixes - Fix a `chat.agent` message-loss race where sending a message right after an action (such as an undo) could drop the follow-up's response from the UI until a refresh. ([#4234](https://github.com/triggerdotdev/trigger.dev/pull/4234)) ## Server changes These changes affect the self-hosted Docker image and Trigger.dev Cloud: - Added `EVENT_REPOSITORY_POSTGRES_WRITES_DISABLED` to skip all PostgreSQL task-event writes for deployments that store task events in ClickHouse. Leave it off unless `EVENT_REPOSITORY_DEFAULT_STORE` is `clickhouse_v2`, otherwise task events are lost. ([#4242](https://github.com/triggerdotdev/trigger.dev/pull/4242)) - Promo credits: a /promo signup landing page, redeeming a promo code when a new org selects a plan, and showing remaining credits on the usage page. ([#4138](https://github.com/triggerdotdev/trigger.dev/pull/4138)) - Speed up retrieving a background worker by version. The endpoint no longer runs a slow lookup that scanned the full task table for large deployments; it now reuses data it already loads, so the response is the same but returns much faster. ([#4245](https://github.com/triggerdotdev/trigger.dev/pull/4245)) - Clearer login error when an email address is blocked by the WHITELISTED_EMAILS setting: the message now explains the address isn't allowed on this instance instead of the ambiguous "This email is unauthorized". ([#4220](https://github.com/triggerdotdev/trigger.dev/pull/4220)) - Make the native build server the default in project build settings. It's now opt-out, stored as a new `disableNativeBuildServer` key. Also clarifies in the UI that build settings apply to GitHub-triggered and native build server deployments. ([#3980](https://github.com/triggerdotdev/trigger.dev/pull/3980)) - Optionally process high-volume telemetry ingestion in parallel for higher throughput under heavy load by setting `OTEL_TRANSFORM_WORKER_POOL_ENABLED=1`. Off by default. ([#4232](https://github.com/triggerdotdev/trigger.dev/pull/4232)) - Add a `REALTIME_BACKEND_DEFAULT` env var to choose the default realtime backend (`electric`, `native`, or `shadow`) for environments whose org has no per-org override. Defaults to `electric`, so existing behavior is unchanged. ([#4231](https://github.com/triggerdotdev/trigger.dev/pull/4231)) - Clarified on the Regions page that a region only affects where your runs execute, not where your data is stored. This shows as a tooltip on the Location column and in the confirmation dialog when you change your default region. ([#4226](https://github.com/triggerdotdev/trigger.dev/pull/4226)) - Improved the reliability of how run data is read and written. ([#4237](https://github.com/triggerdotdev/trigger.dev/pull/4237)) - Fixed stale login errors: an error from a previous login attempt (for example a rejected email address) no longer keeps reappearing on the login page and no longer makes later, successful attempts look like they failed. ([#4220](https://github.com/triggerdotdev/trigger.dev/pull/4220)) - The Errors page now shows better details for each error. Errors that don't carry a message — such as errors thrown without a message, or values thrown that aren't `Error` objects — get a meaningful title instead of all reading "Unknown error", and are grouped by their name (or value) rather than collapsed into a single group. The error type now shows the actual error name, and stack traces now appear where previously they were missing. ([#4225](https://github.com/triggerdotdev/trigger.dev/pull/4225)) - Return a clear client error when SSO form submissions use an unsupported content type ([#4238](https://github.com/triggerdotdev/trigger.dev/pull/4238)) - Query page: extracting fields from a run's output with JSON functions (such as JSONExtractString or JSONExtractInt) no longer fails with an "illegal type: JSON" error. ([#4221](https://github.com/triggerdotdev/trigger.dev/pull/4221)) <details> <summary>Raw changeset output</summary> # Releases ## @trigger.dev/build@4.5.4 ### Patch Changes - You can now mark environment variables synced via the `syncEnvVars` build extension as secrets. Return `{ name, value, isSecret: true }` from your callback and those variables are stored redacted in the dashboard, just like manually created secret env vars. ([#4203](https://github.com/triggerdotdev/trigger.dev/pull/4203)) - Updated dependencies: - `@trigger.dev/core@4.5.4` ## trigger.dev@4.5.4 ### Patch Changes - Remove the legacy `--mcp` and `--mcp-port` options from the `dev` command. Run the dedicated `trigger mcp` command to start the Trigger.dev MCP server. ([#4246](https://github.com/triggerdotdev/trigger.dev/pull/4246)) - Updated dependencies: - `@trigger.dev/core@4.5.4` - `@trigger.dev/build@4.5.4` - `@trigger.dev/schema-to-json@4.5.4` ## @trigger.dev/core@4.5.4 ### Patch Changes - Removed the unused `ResourceMonitor` export from `@trigger.dev/core/v3/serverOnly`. It was a server-side logging helper with no remaining consumers. ([#4244](https://github.com/triggerdotdev/trigger.dev/pull/4244)) - Removed the unused `@trigger.dev/core/v3/zodNamespace` export and the legacy v3 socket message schemas. These were only used by the now-retired v3 engine and have no v4 consumers. ([#4236](https://github.com/triggerdotdev/trigger.dev/pull/4236)) ## @trigger.dev/python@4.5.4 ### Patch Changes - Updated dependencies: - `@trigger.dev/sdk@4.5.4` - `@trigger.dev/core@4.5.4` - `@trigger.dev/build@4.5.4` ## @trigger.dev/react-hooks@4.5.4 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.4` ## @trigger.dev/redis-worker@4.5.4 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.4` ## @trigger.dev/rsc@4.5.4 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.4` ## @trigger.dev/schema-to-json@4.5.4 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.4` ## @trigger.dev/sdk@4.5.4 ### Patch Changes - Fix a `chat.agent` message-loss race where sending a message right after an action (such as an undo) could drop the follow-up's response from the UI until a refresh. ([#4234](https://github.com/triggerdotdev/trigger.dev/pull/4234)) - Updated dependencies: - `@trigger.dev/core@4.5.4` </details> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>	2026-07-14 10:07:58 +01:00
Eric Allam	29598a77b8	feat(webapp): add option to disable PostgreSQL task-event writes (#4242) ... ## Summary Adds `EVENT_REPOSITORY_POSTGRES_WRITES_DISABLED` (default off), which makes the task-event store skip all PostgreSQL `TaskEvent` writes. It's for deployments that store task events in ClickHouse (`EVENT_REPOSITORY_DEFAULT_STORE=clickhouse_v2`) and no longer want the PostgreSQL copy. ## How it works The guard sits at the single postgres write boundary, `TaskEventStore.create` / `createMany`, so it covers every write path (OTLP ingestion and run-lifecycle events) with one check. Reads are untouched (`findMany` / trace queries / streaming), so existing PostgreSQL events remain readable. Leave it off unless the default store is `clickhouse_v2`, otherwise task events for any run still routed to PostgreSQL would be dropped.	2026-07-13 17:17:02 +01:00
Daniel Sutton	e0b42a88d6	perf(webapp): avoid unindexed fileId scan in get-background-worker-by-version (#4245) ... ## What The `GET /api/v1/projects/:projectRef/background-workers/:envSlug/:version` endpoint loaded each file's tasks through the nested `files.tasks` relation. Prisma resolves that as a separate query: ```sql SELECT id, slug, "fileId" FROM "BackgroundWorkerTask" WHERE "fileId" IN (...) ``` `BackgroundWorkerTask.fileId` is not indexed — the FK constraint exists, but Postgres does not auto-create an index for foreign keys — so on a large table this can only run as a sequential scan, which gets progressively slower as the table grows and was observed taking minutes per call in production. The loader already loads every task for the worker via `tasks: true`, which uses the indexed `workerId` relation, and those rows already include `fileId`. This PR groups task slugs by `fileId` in memory from that already-loaded data and drops the `files.tasks` include entirely. ## Behavior change (latent bug fix) The response shape is unchanged, but there is a semantic correction for **source files reused across worker versions** (files are de-duplicated by `@@unique([projectId, contentHash])`, so one file row can be linked to many workers). - **Before:** `file.tasks` came from the `BackgroundWorkerFile.tasks` relation, i.e. *every* `BackgroundWorkerTask` with that `fileId` — across all workers sharing the file. So a worker's manifest could list tasks it doesn't actually have. - **After:** `file.tasks` is grouped from the queried worker's own tasks, so it reflects only that worker version's tasks. Verified on a local DB: 460 files are referenced by tasks from more than one worker; of 6819 (worker, file) pairs, 6 differ — all one file where the old union leaked a task slug (`cancellation-test`) into worker versions that never had it. The new per-worker behavior is the correct one for a worker-version manifest. (Thanks to the automated review for flagging this.) ## Analysis Captured the exact SQL before/after by instrumenting Prisma against real data (a worker with 62 files): - **Before:** 5 statements, including the `WHERE "fileId" IN (...)` scan. - **After:** 4 statements; the `fileId` query is gone and the other four are identical. EXPLAIN of the two access paths: ``` Before WHERE "fileId" IN (...) Seq Scan on "BackgroundWorkerTask" Filter: ("fileId" = ANY (...)) -- reads the whole table, scales with table size After WHERE "workerId" IN (...) Index Scan using "BackgroundWorkerTask_workerId_slug_key" Index Cond: ("workerId" = ...) -- bounded by matching rows, scale-independent ``` No new index is required: the `workerId` access path is already covered by the existing `BackgroundWorkerTask_workerId_slug_key` unique index. ## Testing - `pnpm run typecheck --filter webapp` passes. - Query capture + EXPLAIN performed against a local database seeded with real worker/file/task data.	2026-07-13 16:14:20 +01:00
Daniel Sutton	bea7e2be90	feat(webapp,run-store): route run-graph reads and writes through the run-store router (#4237) ... ## Summary Run-graph data (runs, batches, waitpoints, and their related tables) can now live in a database separate from the control plane, with every read and write routed to the correct database by each run's residency. This makes reading and writing run data more reliable once the two are split, and is a no-op for single-database installs. ## Design - Run-graph table access goes through the run-store router, which selects the legacy or the new run-ops store per run instead of assuming one shared client. - The legacy run-ops client is now independently pointable, so legacy run data can be served from its own database (and replica) rather than the control-plane connection. - Run-graph writes go straight to the run-graph database instead of being forwarded through the control plane, and replication targets are split so runs in the new database still replicate to analytics without under-counting. - Read-through slots refuse the control-plane client, so a missing residency fails loudly instead of silently reading the wrong database. - Migration `20260710120000_drop_remaining_run_graph_seam_foreign_keys` drops the foreign keys that still crossed the run-graph / control-plane seam, which is what lets the two live in separate databases. The split stays off unless explicitly enabled and the two databases are confirmed physically distinct; startup fails closed otherwise. Verified by running the full dashboard end-to-end suite against both a single-database configuration and a three-database configuration (control plane, the new database, and a physically separate legacy database), with runs on both residencies. No misrouted reads in either configuration.	2026-07-13 13:54:54 +01:00
Eric Allam	c23585710c	docs: note v3 is retired and 4.5.0 is the last version supporting v3 (#4241) ... ## Summary Refreshes the docs for the v3 sunset: v3 (SDK v3) is end of life, and 4.5.0 is the last version we officially support for running v3. - The self-hosting overview, plus the docker and kubernetes version-locking sections, now tell self-hosters on v3 to stay on 4.5.0 or migrate to v4. 4.5.1 and later reject v3 triggers and deploys with an upgrade message. - The migration guide's deprecation notice was still written in the future tense (with dates that have since passed); it now describes v3 as retired and adds the self-hosted 4.5.0 cutoff. This is the page the server's upgrade message links to. - Fixes a stale "v3 project" reference in the CLI overview. The Mintlify preview will render the callouts for a visual check.	2026-07-13 12:35:42 +01:00
Eric Allam	5ba8557a51	chore(webapp,core): remove the end-of-life v3 (engine V1) execution stack (#4236) ... ## Summary v3 (the engine that ran the SDK v3 era, internally `RunEngineVersion.V1`) is end-of-life. Following the removal of the v3 execution apps ([#4194](https://github.com/triggerdotdev/trigger.dev/pull/4194)) and the legacy dev websocket ([#4198](https://github.com/triggerdotdev/trigger.dev/pull/4198)), this removes the remaining v3 execution stack from the server. Clients still on v3 (an old SDK or CLI that has not upgraded) keep getting a clear "upgrade to v4" response. Triggers, batch triggers, reschedules, and deploys that resolve to v3 are rejected with a graceful 4xx pointing at the migration guide, never a 5xx, so a stale client cannot affect server health. Self-hosted instances still running v3 should stay on the 4.5.x release line until they migrate. ## What is removed - The MarQS queue and its shared/dev queue consumers. - The v3 socket.io namespaces (coordinator, provider, shared-queue) and the v3 run lifecycle services (attempt, checkpoint, and batch-resume). - The graphile-worker background job system; all live jobs already run on `@trigger.dev/redis-worker`. - The `DEPRECATE_V3_ENABLED` flag: v3 is now rejected unconditionally, so the flag is gone. - Unused v3 exports from `@trigger.dev/core` (the `v3/zodNamespace` subpath and the legacy socket message catalogs) and the now-dead MarQS environment variables. ## What stays The v4 engine is untouched. The graceful v3 rejection boundary stays, `determineEngineVersion` still detects a v3 project so it can reject it, and the batch service plus batch-completion worker stay for current clients. Live queue concurrency limits and metrics now read from the v4 run engine instead of MarQS, and a brand-new dev environment now defaults to v4. ## Dependency cleanup Removes webapp dependencies left unused by this change: `seedrandom` and `semver` (only the removed v3 code used them) plus a set that was already dead, their orphaned `@types` packages, and two dead files. Adds a `knip:deps` script and a `knip.json` config so unused dependencies can be found the same way going forward.	2026-07-13 11:32:06 +01:00
Chris Arderne	c0f7c803b1	fix(webapp): return 415 for invalid SSO form content types (#4238) ... ## Summary SSO form submissions with an unsupported content type now receive a 415 response instead of failing while parsing the request body.	2026-07-13 11:01:08 +01:00
Saadi Myftija	5f2541d94f	feat(webapp): make native build server the default in build settings (#3980) ... Switches the native build server from opt-in to opt-out in project build settings. - It's now enabled by default, stored as a new \`disableNativeBuildServer\` opt-out key so previously-saved \`useNativeBuildServer: false\` values aren't treated as deliberate opt-outs. - The "Use native build server" checkbox is checked by default; unchecking it persists the opt-out. - Brief wording: clarifies build settings apply to GitHub-triggered and native build server deployments, and the native build server hint no longer says "in the future".	2026-07-13 11:05:46 +02:00
Eric Allam	45527e317a	feat(webapp): opt-in worker pool for OTLP ingest transform (#4232) ... ## Summary Under high OTLP ingest volume, the whole decode, transform, and enrich pipeline runs on the request event loop, so a single CPU core becomes the ceiling while the rest sit idle. This adds an opt-in worker pool that moves decode, transform, and LLM-cost enrichment onto worker threads, keeping the main thread free for I/O. It is off by default (`OTEL_TRANSFORM_WORKER_POOL_ENABLED`), so behavior is unchanged unless enabled. ## Design Workers do decode, filter, convert, and enrich (including LLM pricing match). The main thread stays the single database reader: it loads the pricing registry and broadcasts the compiled model rows to the workers (re-broadcasting on every reload), so workers never touch the database. The pure transform is extracted into a dependency-light module (no Prisma/Redis/ClickHouse imports) so it can run inside a worker. Importantly, the main thread keeps the existing single consolidated insert path, so ClickHouse insert batching and part count are unchanged. The parallelism buys CPU headroom, not more insert streams (which would add merge pressure). The worker is bundled as a standalone file at build time and ships in the existing image with no Dockerfile change. In local load testing the pool sustained roughly 2.6x the throughput of the single-thread path and kept the main thread responsive under load.	2026-07-11 13:38:12 +01:00
Eric Allam	5d0e9d9dc5	feat(webapp): make the default realtime backend configurable (#4231) ... ## Summary The default realtime backend was hardcoded to Electric. This adds a `REALTIME_BACKEND_DEFAULT` env var (`electric` | `native` | `shadow`, default `electric`) that chooses the backend for any environment whose org has no `realtimeBackend` override. Behavior is unchanged unless you set it; per-org overrides still win. The default is applied at every point where the per-org flag falls through: the initial value, the flag lookup default, and the error fallback.	2026-07-11 09:36:09 +01:00
Matt Aitken	2cac63f13a	fix: improve error labelling, grouping, and stack traces in the Errors feature (#4225) ... ## Problem Several display/grouping issues in the **Errors** feature, all rooted in how the ClickHouse error materialized views (`errors_mv_v1`, `error_occurrences_mv_v1`) read the stored error JSON produced by `parseError`: 1. **Messageless errors show "Unknown error".** An empty message falls straight through `coalesce(nullIf(message,''), 'Unknown error')` to the literal, even though the error's class `name` is available (e.g. an Effect tagged error `ListMessagesError` with no message). 2. **Unrelated errors collapse into one group.** `calculateErrorFingerprint` keys on `type : message : stack`, where `type` is always the union tag (`BUILT_IN_ERROR`, …), `message` is empty, and the stack isn't read — so every messageless built-in error (and every string/custom error) hashes to the same constant input → one fingerprint. 3. **error_type shows the internal tag.** `coalesce(type, name, …)` always resolves to `type` (always present), so the column shows `BUILT_IN_ERROR` instead of the real class name. 4. **Stack traces never populate.** The MVs read `error.data.stack`, but the serializer stores the trace under `stackTrace` — so the column is always empty. ## Fix All display changes are `ALTER TABLE … MODIFY QUERY` on the two views (migration `035`); the fingerprint change is in the webapp. - **Fingerprint** (`errorFingerprinting.ts`): fall back **message → name → raw**. Messageless errors now group by class name (or raw value for non-Error throws); message-bearing errors are **unchanged** (short-circuits at `message`), so existing groups don't split — only currently-messageless errors get their own group going forward. - **error_message**: same `message → name → raw` fallback before `'Unknown error'`. - **error_type**: coalesce `name → code → 'Error'` (drops the reliance on the union tag). Built-in → class name, internal → `code`, string/custom → `Error`. - **stack trace**: read `error.data.stackTrace`. Bounded as before (serializer caps 50 frames / 1024 chars per line; MV clips to 2000 chars). ## Migration notes - `MODIFY QUERY` swaps the view query in place (no drop/recreate gap); Down restores the previous query. - **Existing rows are left unchanged** — changes apply only to rows inserted after the migration. No backfill. ## Tests `errorFingerprinting.test.ts` — 57 pass, incl. new cases for messageless class names, string/custom raw values, and stability of message-bearing fingerprints. Fixes the display-derivation half of TRI-11938 (error_type + stack trace); relates to TRI-9254 and TRI-9250.	2026-07-10 18:30:02 +01:00
DKP	25eb0c71a0	fix(webapp): clarify that region only affects where runs execute (#4226) ... ## Summary This adds an always-visible info tooltip on the Location column and a note in the "set default region" confirmation dialog making it explicit. It also removes the obsolete "V4" badge from the Regions page title.	2026-07-10 17:02:36 +01:00