helm-v4.5.9
4606 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Eric Allam
|
6e5f0f0fe7 |
fix(webapp,clickhouse): stop invalid customer queries alerting, and isolate Sentry scope per request (#4372)
## Summary
A query sent to the query API with a typo in it, like a column name that
does not exist, was being reported as a server error. That put customer
SQL mistakes into our error alerting, where they made up almost all of
the volume on one of our noisiest alerts, and it drowned out the
failures that are actually ours to fix. This makes the level match who
is at fault, and fixes two related problems found alongside it.
## Invalid queries are the caller's, not ours
The query API route already got this right. It checks for `QueryError`,
logs at warn, and returns a 400, with a comment saying the system
handles it gracefully and no alert is needed.
The layer underneath ignored that. `executeTSQL` logged every exception
out of its catch block at error, including the compile failures the
route was about to turn into a 400, and error-level logs are forwarded
to error reporting.
The TSQL package already draws the line we need:
```ts
export class ExposedTSQLError extends BaseTSQLError {
/** An exception that can be exposed to the user. */
}
export class InternalTSQLError extends BaseTSQLError {
/** An internal exception in the TSQL engine. */
}
```
`SyntaxError` and `QueryError` extend the first. So the catch block now
branches on `ExposedTSQLError` and logs those at warn, keeping error for
`InternalTSQLError` and anything unanticipated, which is a genuine
compiler bug.
## SQL the caller wrote is their mistake, not ours
The same asymmetry showed up one level down. A query that compiles fine
can still be rejected by ClickHouse at execution, and most of those
rejections mean the caller's SQL is wrong rather than that we generated
something bad.
This is where the volume actually is. Checking production, one error
group alone, a missing `GROUP BY` on the public query API
(`NOT_AN_AGGREGATE`), accounts for over a million events across hundreds
of users. It is by far the largest error group in the project, and
classifying only by resource limit would have left every one of those at
error level.
So rejections are split three ways in `ClickhouseClient`, which is the
only place holding the parsed `ClickHouseError` and its symbolic type.
By the time the error reaches `executeTSQL` it has been wrapped and the
type is gone, and the type never appears in the message text, so it
cannot be recovered by string matching.
- **Resource limits** (memory ceiling, timeout, row/byte caps) log at
warn. The query is valid, it just asked for more than it is allowed to
spend.
- **Invalid SQL** (`NOT_AN_AGGREGATE`, `UNKNOWN_IDENTIFIER`,
`SYNTAX_ERROR`, the type and parse families) logs at warn **only when
the caller wrote the SQL**.
- **Everything else** keeps alerting.
That gate matters. The client is shared, so the identical rejection on
TRQL *we* generated is our bug and has to stay at error. Callers opt in
with `userAuthoredQuery`:
| caller | who wrote the SQL | opts in |
| --- | --- | --- |
| public query API | the customer | yes |
| query editor | the customer | yes |
| agent charts | the agent's model | yes |
| built-in dashboard tiles | us, in code | no |
| queue metric cards | us, in code | no |
| health report | us, in code | no |
The agent is the one judgement call. Its TRQL is not typed by a person,
but it is also not something a code fix makes correct, so a query it
gets wrong is not worth waking anyone for. The same endpoint serves
built-in tiles whose TRQL we do write, so the opt-in lives with the
caller rather than the route.
Separately, when one of these queries did fail, the log recorded the
generated ClickHouse SQL but not the query the caller actually wrote,
which made the reports hard to act on. `queryWithStats` takes an
optional `logFields` that `executeTSQL` uses to attach the original
TSQL.
## Events were attributed to the wrong request
Chasing the above turned up something broader: only a tenth of the
events on that alert pointed at the query API. The rest were pinned to
unrelated requests that happened to be in flight at the same time, so
the alert looked like the trigger endpoint was failing.
`Sentry.init` runs with `skipOpenTelemetrySetup: true`, because we
register our own OTel pipeline. That skips `initOpenTelemetry`, and one
of the things it does is:
```js
api.context.setGlobalContextManager(new SentryContextManager());
```
The async-context strategy is still installed, but `withIsolationScope`
only marks the OTel context and delegates the actual fork to that
context manager:
```js
// "We depend on the otelContextManager to handle the context/hub"
return api.context.with(ctx.setValue(SENTRY_FORK_ISOLATION_SCOPE_CONTEXT_KEY, true), ...)
```
`provider.register()` installed a plain
`AsyncLocalStorageContextManager`, which does not know that key. The
lookup found no scopes on the context and fell back to the
process-global default isolation scope, so every request wrote its
request data into the same object and the last writer won.
The tracer now registers `SentryContextManager`, which subclasses
`AsyncLocalStorageContextManager`, so OTel behaviour is unchanged. It is
also registered on the path where tracing is disabled, which previously
never called `register()` at all and so had no context manager of its
own.
Tenant tags were always correct, because those come from our own async
local storage rather than the isolation scope. That is why the
attribution being wrong was not obvious.
This affects every error report the webapp sends, not just the query
API.
## Verification
`internal-packages/clickhouse`: 76 tests pass, including eight covering
each level decision against a real ClickHouse container. Three pairs pin
the gate open and shut at both layers: an invalid query, a compile
failure, and a real limit breach driven with `max_rows_to_read` each log
at warn with `userAuthoredQuery` and at error without it.
The isolation fix has a test that reproduces the leak before asserting
the fix. Two overlapping requests each tag their own isolation scope;
with the plain context manager the slower one reads back the other's
tag, and with `SentryContextManager` each reads back its own.
Measured separately against a faithful reproduction of the server's
wiring (own OTel pipeline, CommonJS entry) at 200 concurrent requests:
per-request attribution goes from 0.5% to 100%, while span nesting,
context propagation across awaits, and distinct trace IDs are identical
before and after.
|
||
|
Chris Arderne
|
2f1734c858 | fix(core,webapp): redact sensitive fields in logs by default and cap their size (#4401) | ||
|
Chris Arderne
|
8ebc8a41af | fix(webapp,redis-worker): stop logging raw metadata, alert payloads, and job items (#4403) | ||
|
Chris Arderne
|
a09817169f | fix(webapp): stop logging full batch item contents in batchTriggerV3 (#4404) | ||
|
Chris Arderne
|
ed8f5e1297 | fix(webapp): stop logging every environment on a lookup miss (#4402) | ||
|
Chris Arderne
|
a81ad4949c |
feat(database,rbac): add multiple environment API key foundations (#4388)
Adds the storage model and authorization contracts needed for multiple environment API keys. Credentials are represented by hashed values, revocation and expiration state, and persisted effective scopes. The built-in authorization fallback exposes full-access policy preparation, while optional authorization extensions can supply additional presets and task-aware scope generation. This change does not create, display, or authenticate additional keys. |
||
|
Eric Allam
|
4eb9292cbe |
feat(webapp,run-engine): queue metrics and health dashboard (#4131)
## Summary
Three related changes, each independently gated:
**Queue metrics and health.** Per-queue depth, throughput (enqueued,
started, completed), concurrency, whether a queue is throttled, and
scheduling delay (how long a run waits between becoming eligible and
actually starting), plus a per concurrency-key breakdown for keyed
queues. Collected from inside the run queue itself, stored in
ClickHouse, and surfaced on the Queues list, a new per-queue detail
page, the task pages, and the run inspector. The question it answers is
"does this queue have enough concurrency to keep up, and if not, which
key or which limit is the constraint".
**Percent-based queue concurrency limits.** A queue's concurrency
override can now be expressed as a percentage of the environment limit,
stored as the source of truth and re-materialized whenever the
environment limit changes. Absolute overrides above the environment
limit are now **rejected with a 400** instead of being silently capped,
which is a behavior change on `POST
/api/v1/queues/:queue/concurrency/override`.
**The `health` report.** A server-computed verdict on whether work is
flowing, whether the runs that do start are healthy, and whether
telemetry is fresh, rendered as text with sparklines. Available as `GET
/api/v1/reports/:key`, `trigger report`, and the `get_report` MCP tool
(plus a `report` MCP prompt, which shows up as a slash command in hosts
that support prompts).
With the flags off, the Queues page renders the pre-metrics component
verbatim, nothing is emitted, and nothing is written to ClickHouse.
## Configuration
Two independent gates, on purpose. Emission is global so data accrues
for everyone before anyone can look at it; the view is per organization
so it can be turned on for one org at a time without a deploy.
**Runtime flags (no restart)**
| Flag | Store | Gates |
| --- | --- | --- |
| `queue_metrics:enabled` | run-queue Redis key (`"1"`/`"0"`, off by
default) | All emission, gauges and counters. Cached in-process for 10s
with stale-while-revalidate, warmed eagerly at boot so the first op
after a deploy is not dropped. |
| `queue_metrics:gauge_sample_rate` | run-queue Redis key, `0..1` |
Fraction of queue ops that emit a gauge. Counters are never sampled, so
throughput stays exact at any rate. |
| `queueMetricsUiEnabled` | feature-flag catalog: global `FeatureFlag`
row, per-org `Organization.featureFlags` override wins | Whether an org
sees the metrics view at all: the Queues list variant, the queue detail
route, the built-in Queues dashboard, the concurrency-keys endpoint, and
the metrics blocks on task pages and the run inspector. Off by default;
a gated org gets a 404 on the detail route rather than an empty page. |
Both Redis keys are readable and writable from `/admin/queue-metrics`
(super-admin UI, with a live per-shard stream-health table) and
`GET`/`POST /admin/api/v1/queue-metrics` (admin PAT). The admin surface
uses its own Redis client, so it works on any instance regardless of
whether that instance runs the emitter or the consumer.
**Environment variables (boot time)**
| Variable | Default | Notes |
| --- | --- | --- |
| `QUEUE_METRICS_EMIT_ENABLED` | `0` | Constructs the emitter and
injects it into the run engine. Without it the run queue has no emitter
at all. |
| `QUEUE_METRICS_CONSUMER_ENABLED` | `0` | Boots the stream consumer on
this instance. Independent of emission, so consumers can be sized
separately from the API. |
| `QUEUE_METRICS_STREAM_SHARD_COUNT` | `4` | Stream shards, hashed per
queue. |
| `QUEUE_METRICS_CONSUMER_BATCH_SIZE` | `1000` | Poll batch equals
insert batch, so an ack can never outrun a write. |
| `QUEUE_METRICS_REDIS_{HOST,PORT,USERNAME,PASSWORD,TLS_DISABLED}` |
falls back to the run-queue Redis | Set `HOST` to move the metrics
stream onto a dedicated instance so a metrics backlog cannot compete
with the run queue for memory. Self-hosters can leave it unset and get a
single-Redis deployment. |
| `QUEUE_METRICS_COUNTER_STREAM_MAXLEN` | `2000000` shared, `8000000`
dedicated | Bound on how much a stalled consumer can hold. The default
is deliberately lower when the stream shares the queue-critical Redis. |
| `QUEUE_METRICS_COUNTER_ODOMETER_TTL_SECONDS` | `604800` | TTL on the
per-queue cumulative counter key, refreshed on every write, so only
queues idle for the whole window are purged. |
| `QUEUE_METRICS_MAX_QUEUE_NAMES_PER_ENV` | `1000` | Distinct queue
names tracked per environment; overflow collapses into `__overflow__`. |
| `QUEUE_METRICS_MAX_CONCURRENCY_KEYS_PER_QUEUE` | `10000` | Same idea
one level down, per queue. |
| `QUEUE_METRICS_GAUGE_SAMPLE_RATE` | `1` | Default for the live
sample-rate key above. |
| `QUEUE_METRICS_QUERY_TABLES_VISIBLE` | `0` | Lists the queue-metrics
tables in the Query page, its schema docs, the schema API and the AI
query context. Off keeps them unlisted while the feature is dark; a
query naming them still runs either way. |
| `QUEUE_METRICS_CLICKHOUSE_URL` | falls back to the shared wiring |
Runs queue metrics on their own ClickHouse service: the consumer's
inserts and every queue-metrics read go through it, so a metrics-heavy
chart refresh never competes with runs-list or trace reads. Unset
reproduces the previous split exactly (inserts on `CLICKHOUSE_URL`,
reads on the query pool). |
| `QUEUE_METRICS_CLICKHOUSE_READER_URL` | the write URL | Reader split,
so the consumer's inserts can never land on a read endpoint. |
|
`QUEUE_METRICS_CLICKHOUSE_{KEEP_ALIVE_ENABLED,KEEP_ALIVE_IDLE_SOCKET_TTL_MS,MAX_OPEN_CONNECTIONS,LOG_LEVEL,COMPRESSION_REQUEST}`
| `1`, unset, `10`, `info`, `1` | Pool tuning, matching the other
per-workload ClickHouse clients. |
Migrations to apply: ClickHouse `036_create_queue_metrics_v1.sql`, and a
Postgres migration adding the nullable
`TaskQueue.concurrencyLimitOverridePercent`. Both are additive.
## How collection works
Queue operations produce two kinds of signal, and they have opposite
failure modes, so they are handled differently.
**Gauges** (queued, running, queue limit, env queued, env running, env
limit, throttled, plus keys-with-backlog and worst-key wait on keyed
queues) are read *inside* the same Redis script that performs the
enqueue or dequeue, so the reading is atomic with the operation it
describes rather than a racy follow-up read. The script returns them on
its reply and the app forwards them to the stream. Gauges are sampled
and drop-tolerant: they are aggregated with `max`, so a lost reading
costs resolution, never correctness.
**Counters** (enqueued, started, completed, plus nack and dead-lettered)
are cumulative odometers. Each event increments a per-queue key on the
metrics Redis and emits the absolute total, and ClickHouse takes the
difference across buckets at read time. This is the important property
of the design: a summed-delta counter undercounts permanently on any
lost event, while a cumulative one self-heals, because the next
surviving reading restates the whole total. Only bucket granularity can
be lost, never the total. A queue returning after its odometer TTL
expired restarts at 1 and reset detection handles it, which is safe
precisely because expiry only spans a window with no activity.
Both land on one sharded Redis stream. A consumer reads it with a
consumer group, reclaims stale pending entries on a 15s interval rather
than on every poll, maps one entry to one or two ClickHouse rows
(whole-queue and, for keyed queues, per-key), and acks only after the
insert lands. Each batch carries a dedup token derived from its
stream-entry ids, and the target tables set
`non_replicated_deduplication_window`, so a retried batch cannot
double-count either the raw rows or the aggregates that hang off them.
Consumer and emitter both emit OTel metrics
(`queue_metrics.emitter.emitted`,
`queue_metrics.consumer.{entries,rows_inserted,insert_errors,insert_duration,stream_depth,group_lag,pending,lag_unknown}`);
stream depth and group lag are the two worth alerting on, and
`lag_unknown` exists because Redis can report a null lag after a trim,
which must not be read as zero.
## Storage and read path
`queue_metrics_raw_v1` is a short landing table with a 6 hour TTL. Four
aggregate tiers are materialized straight from raw, never cascaded off
each other, each with a 30 day TTL:
- `queue_metrics_v1`, 10 second buckets per queue, the default read path
- `queue_metrics_5m_v1`, 5 minute buckets per queue, for wide ranges and
cross-queue ranking
- `env_metrics_v1`, 10 second buckets per environment, queue-independent
so it stays cheap at any range
- `queue_metrics_ck_v1`, 10 second buckets per concurrency key
Every tier is an MV from raw because the counter states do not survive a
cascade: their merge is order sensitive, so a `-MergeState` chain off
the 10s table inflates the result, and the same property means an
aggregate state may only be merged inside one queue. That constraint is
now enforced by the query engine rather than by reviewer discipline: a
column can declare a `mergeGroupKey`, and any query that references it
without grouping by, or pinning to a single value of, every named key
fails to compile with an actionable message.
On the read side, TRQL gains three tables (`queue_metrics`,
`env_metrics`, and a `queue_metrics_by_key` that is hidden from the
editor, schema docs and schema API but still queryable, so per-key rows
can never silently merge into a plain per-queue query), plus
`deltaSumTimestampMerge` and `quantilesTDigestMerge`. Two schema-level
optimizations ride along: a table can declare coarser rollups, so a
query whose bucket interval is 5 minutes or wider is routed to the 5m
table with no change to the query itself, and it can opt into the
ClickHouse query cache with time bounds floored to a fixed grid, so the
auto-refreshing dashboards actually share cache entries instead of
missing on every tick. Both are caller-side substitutions, so the
printer stays unaware of physical layout.
All of this can also live on its own ClickHouse service. A table
declares the pool its reads run on, the three queue-metrics tables name
the dedicated one, and the ingestion consumer writes through the same
client, so both directions move together with one env var and nothing
else routes differently.
The other engine change is opt-in gap filling: charts can request rows
for empty buckets, where counters zero-fill and gauges carry forward.
Grouped gauge series are densified per group and carried inside a
partition, so a quiet queue's line holds its last value without bleeding
another queue's value into it.
## Queue concurrency limits
`concurrencyLimitOverridePercent` on `TaskQueue` is the source of truth
when an override is set as a percentage; the absolute `concurrencyLimit`
is materialized from it (floored, clamped to at least 1 so a percentage
can never act as a pause, and never above the environment limit). Every
path that changes an environment limit now recalculates the
environment's percent-based overrides afterwards, outside the
transaction, and pushes changed limits to the engine. The push is
attempted even when the stored value did not change, so a previously
failed sync self-heals rather than leaving the database and the engine
diverged; paused queues are skipped so a recalculation cannot
effectively unpause one.
The API accepts exactly one of `concurrencyLimit` or `percent`, and the
reject-instead-of-clamp change above means a request asking for more
than the environment allows now fails loudly. The percent bound (greater
than 0, at most 100) is defined once and shared by the zod schema, the
dashboard mutation handler and the service, so the three cannot drift.
The concurrency-keys table on a queue is now paginated against the
ClickHouse per-key tier, ranked by peak backlog with the total on every
row from a single scan, and only the keys on the current page are
enriched with live counts from Redis. That replaces a hard top-50 cap
with something whose cost is a function of page size rather than key
cardinality.
## The health report
`GET /api/v1/reports/:key?period=&format=markdown|ansi|json`. The
verdict is computed on the server and is deterministic, not
model-generated. Three independent analyzers run over one input
snapshot: flow (is work moving, and if not, is the cause a limit,
throttling, one bad queue, or dead-lettering), execution (are the runs
that start succeeding, and at what latency), and liveness (how fresh is
the telemetry). When telemetry is genuinely stale, the first two are
forced to unknown and every actionable field is stripped, so no surface
ever advises action off stale data.
Authorization is per query table rather than a blanket query grant: a
JWT must be scoped to every table the report reads (`runs`,
`env_metrics`, `queue_metrics`), so a narrowly scoped token cannot pull
a report that reads more than it was granted. `period` is validated as a
shorthand with a 90 day ceiling at the edge. The report catalog is a
registry of `{ load, interpret }` entries, so the next report is a new
entry and no change to the route, the view model, the renderers, the CLI
or the MCP tool.
`trigger mcp` no longer launches the install wizard when stdout is a
TTY, which fixed a real failure: hosts spawn the server over a PTY, so
the wizard would open and the client would time out waiting for a server
that never started. The wizard now needs `trigger mcp --install`.
## The part that is live regardless of every flag
The enqueue and dequeue scripts now return a 2-tuple so a gauge reading
can ride back on the reply. Every return site in the eight affected
scripts is wrapped, and a `nil` original is converted to `false` on the
way out, because a raw `nil` in the first slot would make Lua truncate
the multi-bulk reply and silently drop the gauge on the throttled and
empty-queue paths. The reply shape and the destructuring on the app side
are exercised on every queue operation whether or not metrics are
enabled, so that is the part of `run-engine` worth the closest review.
One behavior fix in the same area: the scheduling-delay anchor is set
only on a run's first entry into the queue. Anchoring it to trigger time
on re-enqueues made waitpoint and checkpoint resumes report the entire
wait as scheduling delay. Queue ordering is untouched, so a re-enqueued
run keeps its position, and nacks deliberately keep the original anchor
because a rolled-back dequeue is the same continuous wait.
A pending-version promotion still anchors to trigger time, on purpose:
that promotion is the run's first real entry into the queue, since the
trigger deliberately held it back waiting for a worker version, and the
TTL is armed at the same point for the same reason. The consequence is
worth naming, because it is a judgement call: a run that waits on a
deployment reports that wait as scheduling delay on its queue, which is
time unrelated to queue capacity.
## Verification
Unit and integration suites across the new package, the run queue, the
mapping layer, the query engine and ClickHouse (including a test that
applies migration 036 through the same splitter CI uses, and a
regression test that inserts the same batch three times to prove the
aggregates do not inflate). Beyond that, the whole path was driven end
to end against a live stack with real runs: emitter to Redis stream to
consumer to ClickHouse to the dashboards, for both the local dev path
and the deployed path where a supervisor drives the dequeue, with
assertions on exact counter reconstruction per queue and per concurrency
key, throttling, environment saturation, scheduling delay, and a
deliberate mid-stream reading drop to confirm the cumulative counters
still reconstruct the correct total. The gated-off state was checked on
every touched surface.
The dedicated ClickHouse service was verified against a second,
separately-schema'd instance: with it configured, the driven counters
reconstruct exactly on the dedicated instance, the shared instance gains
no rows for that window, a read through the query API returns the value
that exists only on the dedicated instance, and a `runs` query still
succeeds (it would fail outright if it were mis-routed to a service
without that table). With the variable unset, the full suite passes
unchanged.
---------
Co-authored-by: Katia Bulatova <katia@trigger.dev>
Co-authored-by: Katia Bulatova <katherine.bulatova@gmail.com>
Co-authored-by: James Ritchie <james@trigger.dev>
|
||
|
claude[bot]
|
639eaf6e82 |
fix(webapp): don't apply an invite's role to an existing org member (#4409)
<!-- ccr-slack-attribution --> _Requested via [Slack thread](https://triggerdotdev.slack.com/archives/C097ZHVKZFA/p1785249693523749)_ ## Summary Accepting an old invitation could change the role of someone who was already in the organization. A long-pending invite can carry a lower role than the member has since been promoted to, so accepting it was a silent demotion. When the accepting user was the organization's only Owner, the role layer refused that demotion, and the refusal (an expected, protective outcome) was logged as an error. An invitation now only sets a role on a membership the accept actually created, and people who are already in an organization are skipped when invitations are sent. ## How `acceptInvite` already skipped the `OrgMember` create when it found an existing membership, but the `rbac.setUserRole` call below it was gated only on `invite.rbacRoleId`. It now also tracks whether this accept created the membership. A create that loses the unique-constraint race counts as pre-existing, since whichever flow won it owns that membership's role. Skipping existing members outright would regress one case: a member with no RBAC role at all would never receive the invitation's role. `ensureOrgMember` handles that with `healMissingRoleAssignment`, which fills in a null role but never overwrites a real one, so `assignInviteRbacRole` takes the same gate. An established role is never touched; an absent one is filled in. `assignInviteRbacRole` branches on the result's machine-readable `code` instead of logging every refusal at `error`. `last_owner` goes to `logger.info`, matching the two directory-sync role paths; everything else, including a refusal that carries no code, goes to `logger.warn`. The helper is best-effort and never throws, so no outcome it produces warrants `error`. No string matching on the error text is involved. `inviteMembers` resolves the organization's members by email and skips those addresses before creating invites. The invite table's `@@unique([organizationId, email])` only dedupes *pending invites*, so it could never catch this. ## Invite surfaces Skipping addresses means a batch can now come back empty, and neither caller handled that: - The dashboard action built its redirect from `invites[0].organization`, so a batch where every address was skipped threw a `TypeError` that reached the admin as a raw error string. It also reported the submitted count rather than the created one. It now names what it skipped ("No invitations sent: 1 already a member of this organization") and counts what it actually created. - The invites API derived `alreadyInvited` as "everything not created", so an existing member was reported as though they had already been invited. `inviteMembers` now returns the two groups separately and the endpoint reports `alreadyMembers` alongside `alreadyInvited`. ## Testing `apps/webapp/test/member.server.test.ts` passes 16/16 locally, up from 12. Getting there needed a harness fix. The `~/db.server` mock did not export `Prisma`, so any code reaching `PrismaNamespace.PrismaClientKnownRequestError` threw before it could branch, leaving every duplicate-key path in `member.server.ts` unreachable from tests. The mock now re-exports the real `Prisma`, and there is a case covering the pending-invite skip. New cases: the invite role is applied when the accept creates the membership; it is not applied when the member already has a role; it is applied when an existing member has no role assigned; the organization is still joined when the assignment is refused with `last_owner`; and `inviteMembers` reports members separately from pending invites. Forcing the gate off fails exactly the "already has a role" case, so the coverage is load-bearing. `pnpm run typecheck --filter webapp` and `oxfmt --check` both pass. ## Changelog Accepting an old invitation could change the role of someone who was already in the organization. An invitation now leaves an existing member's role untouched, people who are already in an organization are no longer sent invitations to it, and the invite form says which addresses it skipped instead of failing with an unhelpful error. --- ## ✅ Checklist - [x] I have followed every step in the [contributing guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md) - [x] The PR title follows the convention. - [x] I ran and tested the code works. ## Screenshots No visual changes. The invite form's toast copy changes, as described above. --------- Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: Matt Aitken <matt@mattaitken.com> |
||
|
James Ritchie
|
a11e5ffbc6 |
fix(webapp): fade overflowing side menu selector labels (#4412)
Long organization, project, and environment names in the side menu were cut off mid-character. They now fade out at the right edge like the rest of the side menu items already did. ### Example of faded long names: <img width="246" height="200" alt="CleanShot 2026-07-28 at 22 59 24" src="https://github.com/user-attachments/assets/efa60b87-286f-4ab0-9d4e-490ef2de53e5" /> |
||
|
James Ritchie
|
1e14e29d71 |
fix(webapp): restyle the leave and remove team member dialogs (#4411)
## Summary
The confirmation dialog for leaving a team or removing a teammate was
still built on the old `Alert` primitive: the entire question sat in the
title, there was no header divider or `Esc` affordance, and the footer
used small buttons pinned to the right.
It now uses the standard `Dialog` layout the rest of the dashboard uses.
The title is static ("Remove team member" / "Leave team"), the question
moves into the body with the person's name and the organization
highlighted, and the footer is a bordered row with medium Cancel and
confirm buttons. A member who has not set a name is now identified by
their email instead of "them".
Verified against a local dashboard on both dialogs. Confirming a removal
posts the member id, deletes the membership and shows the success toast.
Cancel, `Esc`, and Enter while Cancel is focused all close the dialog
without issuing a request, leaving the member in place.
No release note needed: this is a visual restyle of an existing dialog
with no behaviour change.
|
||
|
Chris Arderne
|
38bf82aebe | feat(cli,webapp): target notifications by minimum CLI version (#4407) | ||
|
Saadi Myftija
|
44eca4d166 |
feat(webapp): org-gated internal API origin in run env vars (#4366)
Adds an opt-in way for operators to route deployed runs' API traffic through a different origin than the public one, per organization. Set `INTERNAL_API_ORIGIN` on the webapp and enable the `internalApiOriginEnabled` feature flag (globally or per org, with the org override winning in both directions): deployed runs for enabled orgs then get `TRIGGER_API_URL` set to the internal origin instead of `API_ORIGIN`. Useful for gradually moving run traffic onto a private network path. ## Design The origin is resolved when an attempt starts, so flag changes take effect on the next attempt and roll back the same way, with no task redeploys. The org override is read fresh per attempt; the global default comes from the cached flags registry (a cold read fails safe to the public origin). When `INTERNAL_API_ORIGIN` is unset the flag is a no-op and no extra queries run, so existing deployments are unaffected. Dev runs always use the public origin, and `TRIGGER_STREAM_URL` remains unchanged. |
||
|
claude[bot]
|
ec562c0e68 |
fix(webapp): remove unused Electric sync trace routes (#4400)
<!-- ccr-slack-attribution --> _Requested by **Eric Allam** · [Slack thread](https://triggerdotdev.slack.com/archives/C0AU83M3136/p1785222101937829?thread_ts=1785207509.304669&cid=C0AU83M3136)_ Removes two dead Remix routes and the helpers only they used. `app/routes/sync.traces.runs.$traceId.ts` (`/sync/traces/runs/:traceId`) and `app/routes/sync.traces.$traceId.ts` (`/sync/traces/:traceId`) were added with the original ElectricSQL run page and lost their only consumers when the dashboard hooks that called them were deleted. Nothing in the repo references either route today. Also removed, because the deleted routes were their only callers: - `OtelTraceIdSchema`, `RESERVED_ELECTRIC_SHAPE_PARAMS`, `TraceScope`, `buildElectricTraceWhereClause` from `app/v3/electricShape.server.ts` (the file stays — `UNSAFE_REALTIME_TAG_CHARS` / `sanitizeRealtimeTagForSql` / `sanitizeRealtimeTagsForSql` are still used by `realtime.v1.runs.ts` and `realtimeClient.server.ts`) - the loader-specific cases in `apps/webapp/test/spanTraceRoutes.replicaLag.test.ts` and `internal-packages/run-store/src/runOpsStore.routesSpanTraceReadView.replicaLag.test.ts` `app/utils/longPollingFetch.ts` is untouched — `realtimeClient.server.ts` still uses it. `runOpsStore.ts` / `PostgresRunStore.ts` are untouched too; the unrouted-lookup mechanism there is generic and stays. As a plain code fact: the run lookup these loaders performed keyed on `TaskRun.traceId` alone, which is not an index-backed query shape. That is noted only as context for why the code is not worth keeping around unused. ### Judgement call worth a maintainer's opinion The request was specifically about `/sync/traces/runs/:traceId`, the route that looks up a run by `traceId`. This PR **also** deletes its sibling `/sync/traces/:traceId`. The reasoning: - both routes came in with the same ElectricSQL run-page work - both lost their only consumers in the same later commit - neither has any caller anywhere in the repo - they share the same helper module, so keeping one means keeping the helpers half-used If you would rather keep the sibling, reverting just that one file deletion is easy and does not affect the rest of this PR — say the word and I will restore it along with the helpers it needs. ## ✅ Checklist - [x] I have followed every step in the [contributing guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md) - [x] The PR title follows the convention. - [x] I ran and tested the code works --- ## Testing Verification run locally from the repo root: | Command | Result | | --- | --- | | `pnpm run format` | clean, no changes produced | | `pnpm run lint:fix` | clean | | `pnpm run lint` | pass (exit 0, no findings) | | `pnpm run typecheck --filter webapp` | pass | | `pnpm run typecheck --filter @internal/run-store` | pass | A ripgrep sweep for `sync.traces`, `sync/traces`, `syncTraceRunsLoader`, `buildElectricTraceWhereClause`, `OtelTraceIdSchema` and `RESERVED_ELECTRIC_SHAPE_PARAMS` (excluding `node_modules`) returns zero hits. **Not fully verified:** both edited test files are testcontainers suites and need a Docker runtime, which was not available in my environment. I confirmed each file *collects* correctly with exactly the three intended remaining tests and no import errors — notably, dropping the `session.server` / `controlPlaneResolver.server` / `longPollingFetch` / `env.server` mocks does not break module loading for the surviving loaders. The assertions themselves then failed only on `Could not find a working container runtime strategy`. CI should be the real signal here. Per `apps/webapp/CLAUDE.md`, `pnpm run build --filter webapp` was deliberately not run. --- ## Changelog Removed two unused sync routes left over from the original ElectricSQL run page, along with the helpers and tests that existed only to serve them. No behaviour change — neither route had any caller. --- ## Screenshots _n/a — no user-visible surface changes._ 💯 --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
James Ritchie
|
73eb4c5c16 |
feat(webapp): Improve the Integrations page layout (#4379)
## Summary The project Integrations page now uses the same settings layout as the org SSO page: a centered column of titled rows with dividers, instead of headings over bordered boxes. GitHub, Vercel and build settings read as one consistent list, and the page titles itself "Integrations". Confirmations persist rather than vanishing once you move past them (`GitHub app: Installed`, `Vercel project: Connected`), plan-gated rows offer an Upgrade button instead of a dead toggle, a disabled toggle explains why in place and highlights the control that unlocks it, and warnings are rows with a hazard icon and their recovery action on the right. Copy throughout leads with the outcome instead of restating the field label. Two fixes along the way: a nested `<form>` in the Vercel panel that failed hydration and silently truncated the page, and every settings row carrying a few pixels more space above its title than below its description. ### Before <img width="1160" height="1972" alt="CleanShot 2026-07-26 at 21 56 42@2x" src="https://github.com/user-attachments/assets/ed0fd676-36d8-4eb7-a16e-827a24f007d9" /> ### After <img width="1358" height="4455" alt="CleanShot 2026-07-26 at 19 14 28@2x" src="https://github.com/user-attachments/assets/6a635e6a-c0eb-4a4c-a68f-fcde4d25e8a6" /> |
||
|
James Ritchie
|
d30ee6e570 |
feat(webapp): favorite pages and sidebar customization (#4375)
## Summary
Favorite any dashboard page and it appears in a new "Favorites" section
at the top of the side menu. The star next to the page title (or
Option+F) saves the exact view, filters and tabs included, with a name
derived from the URL ("Runs: Completed successfully, last 7d", "Run:
05hrqq9n") that you can rename inline from each item's hover menu.
The sidebar is customizable too: "Customize sidebar" (on section header
menus and in each "More" menu) opens a modal where you can reorder
sections, drag items into a new order, hide items behind a per-section
"More" popover, and rename or remove favorites. Changes apply on
Confirm, Reset restores the default layout without touching favorites,
and everything is stored per user in dashboard preferences.
## Screenshots
| Favorites in the side menu | Customize sidebar modal |
| --- | --- |
| 
| 
|
## Design notes
- Favorite links carry a small marker search param so the favorite, not
its identical main menu item, highlights as active. Markers from shared
or stale links are cleaned on load, and changing any filter hands the
highlight back to the regular menu item.
- Preference writes are serialized with a row lock: several writers
(debounced collapse and width saves, favorite toggles, the customize
modal) can land concurrently and would otherwise clobber each other's
read-modify-write of the JSON column.
- Option+F is matched on `event.code` with a raw listener because macOS
reports Option-modified letters as symbols, which the `event.key` based
shortcut hook can't capture.
Verified end-to-end in the browser: star toggle and shortcut, instant
section appearance, inline rename and staged modal removal, filter-aware
labels and unique active states, shared-link normalization, drag
reordering, and persistence across reloads.
|
||
|
James Ritchie
|
3e53404f40 |
feat(webapp): live-update the runs list on task pages (#4377)
## Summary The runs list on a task's page now updates live, matching the main Runs page. Run rows update their status, duration, and cost in place as runs progress, and a "N new runs" button appears in the header when newer runs come in so you can pull them into the list without a manual refresh. This applies to both standard and scheduled task pages. ## Design It reuses the Runs page's polling hook. A task page scopes its runs by the task in the URL path rather than a `tasks` query filter, so the hook now takes an optional task slug and scopes new-run detection to it. The "new runs" button sits in the header, outside the deferred runs table, so the count is lifted to the page and the click action is passed through a ref. That keeps the table streaming on first load instead of blocking the header on the runs query. When newer runs come in, a `1 new run` button appears in the task page header, to the left of the time filter. Clicking it pulls the new runs into the list. |
||
|
Matt Aitken
|
269470fd87 |
feat(webapp): gate SSO on an entitlement instead of the Enterprise plan (#4393)
The SSO & Directory Sync settings page decided access by comparing the organization's plan code against the literal string `"enterprise"`. The webapp now reads a `hasSso` entitlement from plan limits. ## Changes - **`settings.sso` route** — `planAllowsSso` reads `limits.hasSso` rather than the plan code; the loader and the action gate on a shared `getSsoEntitlement` helper. - **`platform.v3.server`** — new `getSsoEntitlement(orgId)` returning `entitled | not_entitled | unknown`, behind a new SWR cache namespace (60s fresh / 120s stale, memory + Redis). This replaces an uncached billing round-trip that previously ran on every settings load, so the page gets cheaper than it was. - **`directorySyncEffects`** — the entitlement is now checked before applying membership effects, per organization and memoised across a batch. - **`@trigger.dev/platform` 1.2.0 → 1.3.0** — required, see below. ## Behaviour worth reviewing **Revocation now stops SCIM.** Previously the plan check existed only on the settings page, so an org that lost access kept receiving directory-sync pushes indefinitely; only the config UI froze. Provision *and* deprovision are gated, so a revoked entitlement can't remove members either. **An unreadable entitlement throws instead of skipping.** Effects are idempotent and the worker retries, so retrying is lossless where dropping would silently lose a directory change. It's raised at `warn` level so a transient billing blip doesn't page anyone. **The login path is deliberately untouched.** A hard entitlement check there turns a billing outage into a login outage. Consequence: an org that loses the entitlement keeps its existing SSO logins working until the connection is removed. Gating sign-in is a separate decision. **Self-hosted is unaffected.** With no billing service configured the helper returns `entitled`, leaving plugin presence and the kill switch as the only gates — a self-hoster who installed the plugin isn't locked out of it. ## The dependency bump is load-bearing The `Limits` schema is a plain `z.object`, so it *strips* unknown keys. On 1.2.0 the `hasSso` field was silently discarded during parsing and read as `undefined` no matter what billing sent — a structural accessor would not have helped. Verified against both builds: ``` 1.2.0 → parsed: true | hasSso survives: false 1.3.0 → parsed: true | hasSso survives: true ``` This PR therefore cannot merge before 1.3.0 is published, which it now is. ## Testing `apps/webapp/test/directorySyncEffects.server.test.ts` — 7 tests over the gate: applies when entitled, skips provision and deprovision when not, throws a warn-level retryable error when unreadable, resolves once per org across a batch, and gates per org so one unentitled org doesn't block another. `pnpm run typecheck --filter webapp` passes (18/18), oxfmt and oxlint clean. |
||
|
claude[bot]
|
d91818f198 |
fix(webapp): remove unawaited task list metrics promises (#4380)
<!-- ccr-slack-attribution --> _Requested via [Slack thread](https://triggerdotdev.slack.com/archives/C097ZHVKZFA/p1785082528841609)_ `TaskListPresenter` created promises that nothing ever consumed. Two of the three deferred metrics promises it returned had no reader, no `await` and no `.catch()`, so when the query behind one of them failed the rejection had nowhere to go. ## Before / After **Before** - `TaskListPresenter.call()` returned four things: `tasks`, `activity`, `runningStats` and `durations`. Its only caller reads `tasks` and `runningStats`. - Every load of the tasks page therefore fired two ClickHouse queries whose results were thrown away. - If either of those two queries failed, the resulting promise rejection was unhandled — nothing was awaiting it and nothing had attached an error handler, so it surfaced as an unhandled rejection at the process level rather than as an error anyone could attribute to a request. **After** - `TaskListPresenter.call()` returns `tasks` and `runningStats` only. - Two fewer queries run per tasks-page load. - There is no longer an unconsumed promise that can reject without a handler. `runningStats` is awaited by its caller, so its failures continue to be handled the way they always were. Nothing changes on screen: the tasks page renders `hourlyActivity` and `runningStates`, and neither of the removed values fed either of those. ## How The removed values were verified unreferenced before deleting anything: - `TaskListPresenter` has exactly one caller, `UnifiedTaskListPresenter`, which reads `taskResult.tasks` and `taskResult.runningStats` and nothing else. - No file anywhere in the repo — app code, tests, or type re-exports — reads an `activity` or `durations` field off the presenter's result. - `UnifiedTaskListPresenter` builds its own `unifiedTaskListHourlyActivity` query for the 24h chart the page actually renders, which is what made the presenter's separate 7-day daily activity data redundant. - `getDailyTaskActivity` and `getAverageDurations` on `ClickHouseEnvironmentMetricsRepository` had no callers other than the two lines being deleted, so they and their now-orphaned helpers and types were removed too. Changes: - `apps/webapp/app/presenters/v3/TaskListPresenter.server.ts` — drop the `activity` and `durations` fields (both from the main return and from the no-current-worker early return) and the two repository calls behind them. Drop the unreferenced `TaskActivity` type alias. The "don't await this" comment on the remaining `runningStats` promise now spells out that the caller has to consume it. - `apps/webapp/app/services/environmentMetricsRepository.server.ts` — remove `getDailyTaskActivity` and `getAverageDurations` from the `EnvironmentMetricsRepository` interface and its ClickHouse implementation, along with `fillInDailyTaskActivity` and the `DailyTaskActivity` / `AverageDurations` types. `getCurrentRunningStats` is the control that shows the diagnosis is right. It throws on query failure in exactly the same way as the two removed methods — `if (queryError) throw queryError` — but it never produced an unhandled rejection, because `UnifiedTaskListPresenter` passes its promise into a `Promise.all(...).then(...)` chain that the route then awaits. Same failure mode, opposite outcome, and the only difference is whether anything consumes the promise. Follow-ups, not in this PR: - `AgentListPresenter` returns three sparkline promises in the same shape and they look similarly unconsumed. Left alone here to keep this change reviewable. - With these two callers gone, the `getTaskActivity` and `getAverageDurations` query builders in `@internal/clickhouse` have no remaining callers in this repo. Whether to remove them is a separate call for someone who owns that package. ## ✅ Checklist - [x] I have followed every step in the [contributing guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md) - [x] The PR title follows the convention. - [x] I ran and tested the code works --- ## Testing - `pnpm run typecheck --filter webapp` — passes. This is the meaningful check here: it proves nothing still references the removed fields, methods or types. - `pnpm run format` and `pnpm run lint:fix` — clean, no changes produced. - No test file referenced the removed symbols, so no test needed updating. --- ## Changelog Server-only change, so this carries a `.server-changes/` note rather than a changeset: `.server-changes/task-list-remove-unused-metrics-queries.md`. > The tasks page no longer runs two queries whose results were never displayed, cutting wasted work on every page load and removing a source of hidden server errors --- ## Screenshots _No visual change — the removed data was never rendered._ Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Eric Allam
|
d3906241a5 |
feat(webapp): read realtime run rows from the primary, not the replica (#4378)
🦋 Changesets PR / Create Release PR (push) Has been cancelled
🚀 Publish Trigger.dev Docker / 📣 Dispatch main image (push) Has been cancelled
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 2s
🚀 Publish Trigger.dev Docker / units (push) Failing after 2s
🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker-v4 (push) Has been skipped
🚀 Publish Trigger.dev Docker / scan-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / scan-supervisor (push) Has been skipped
## Summary
The realtime runs feed hydrates run rows from read replicas, which means
it needs a replica-lag gate to avoid serving a run's previous state
right after a write. Setting
`REALTIME_BACKEND_NATIVE_RUN_READS_FROM_PRIMARY=1` reads those rows from
each run store's primary instead, so there is no lag to gate against: no
probe, no wake delay, no stale-read retries. Off by default, so nothing
changes unless you set it.
## Design
The run stores already decide replica-vs-primary from the *brand* on the
read client they are handed: a branded replica keeps the read on the
owning store's replica, an unbranded writer escalates it to that store's
own primary. So this is a one-line choice at the hydrator, and it stays
correct across topologies. With the run-ops split on, each leg lands on
its own writer and the caller's client is never forwarded across
databases; with the split off, it is the single database's primary.
```ts
const runReader = new RunHydrator({
readClient: runReadsFromPrimary ? prisma : $replica,
runStore,
});
```
The same flag skips constructing the lag estimator, since probing a
replica the feed no longer reads would be measuring the wrong thing.
Independently, `AuroraReplicaLagSource` detected Aurora by letting
`aurora_replica_status()` fail, on the assumption that the app-level
catch made that free. It isn't: an unresolvable function is a query
error the driver reports to the error log on every sample, so a
non-Aurora replica produced a continuous stream of error events while
the estimator quietly fell through to its next candidate. It now
resolves the function with `to_regproc` and memoizes the answer, so the
unparseable call never reaches the wire.
|
||
|
claude[bot]
|
109e245d56 | feat(webapp): show the first and last characters of a new PAT (#4363) | ||
|
nicktrn
|
bf41c5d5fc |
feat(supervisor): configurable warm-start dispatch url (#4362)
Adds an optional `TRIGGER_WARM_START_DISPATCH_URL`. The warm-start dispatch request uses it when set, otherwise falls back to `TRIGGER_WARM_START_URL`, so the dispatch target can differ from the default warm-start URL. No behavior change when unset. |
||
|
Eric Allam
|
7188eecd83 |
perf(webapp): clamp list-endpoint page size to 100 (#4360)
## Summary Several list endpoints accepted an unbounded page size (`perPage` / `per_page` / `pageSize`). An unbounded page lets one request pull an arbitrarily large result set and do a proportional amount of work, which is a poor default for a shared API. This clamps the page size to 100 on every list endpoint that was uncapped, matching the existing cap on `api.v1.runs` and `api.v1.sessions`. Clamping rather than rejecting keeps existing clients working: a request for a larger page returns up to 100 items and offset pagination continues from there. ## Endpoints capped - `api.v1.schedules` (`perPage`) - `api.v1.queues` (`perPage`) - `resources.…versions` (`per_page`) - `resources.…queues` (`per_page`) - `admin.api.v1.…engine.report` (`per_page`) - `admin.api.v1.llm-models` (`pageSize`) Already capped, left as-is: `api.v1.runs`, `api.v1.sessions`, `api.v1.deployments`. |
||
|
nicktrn
|
722e240e4d |
feat(supervisor): add prometheus metric for outbound http requests (#4350)
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 5s
🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker-v4 (push) Has been skipped
🚀 Publish Trigger.dev Docker / scan-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / scan-supervisor (push) Has been skipped
🚀 Publish Trigger.dev Docker / units (push) Failing after 6s
🦋 Changesets PR / Create Release PR (push) Has been cancelled
🚀 Publish Trigger.dev Docker / 📣 Dispatch main image (push) Has been cancelled
Adds Prometheus metrics so the supervisor's outbound HTTP calls are
observable - including client-side failures that previously only
surfaced as a log line.
- `supervisor_outbound_request_total{name, method, status, outcome}` -
counts every outbound request. `outcome` separates a transport failure
(`network_error`), an HTTP error response (`http_error`), a response
that failed schema validation (`invalid_response`), and success (`ok`).
- `supervisor_outbound_request_duration_seconds{name, outcome}` -
latency histogram. Leaner labels than the counter (no `status`) to avoid
bucket×label cardinality; buckets match the existing dequeue-latency
histogram since these calls share the same retrying HTTP client and
long-poll envelope.
Coverage:
- The warm-start request (a one-off `fetch`) - instrumented inline; the
response status code is now also included in the failure log (it was
previously dropped).
- All worker API client calls (`SupervisorHttpClient`: dequeue, run
attempt start/complete, heartbeats, snapshots, continue, suspend,
debug-log, connect) - routed through a single instrumented `request()`
helper that reports via an optional `onHttpRequestComplete` callback on
the client, which the supervisor wires into the counter + histogram.
Low cardinality by design: `name` is a **static per-endpoint label**
(e.g. `dequeue`, `start_run_attempt`), never the interpolated URL - so
no run/snapshot IDs land in labels, mirroring the templated `route`
labels on the inbound HTTP server.
Registered on the existing metrics registry, exposed on `/metrics` with
no new wiring. Internal-only change (no package release needed), so the
changelog note is a single `.server-changes` entry.
|
||
|
Eric Allam
|
e9ac98b7a1 |
perf(run-store): route id-set reads to the owning store, not both DBs (#4342)
📚 Publish docs / publish (push) Has been cancelled
## Summary The split run-store's id-set read path (`#findRunsByIdSet`, used by the runs-list hydrate, the realtime hydrator, and engine sweeps) queried the new store for the entire id set and then probed the legacy store for the misses. A run's residency is a total function of its id (run-ops ids live in the new store, every other id in legacy), so each id belongs to exactly one store. Route each id to its owner and query each store only for its own ids, in parallel. Same result set, and while a split is active with most runs still on legacy it removes a wasted new-store query from every id-set read. ## Change `#findRunsByIdSet` now partitions the ids by `classifyResidency` and runs one bounded query per store (skipping an empty side), in parallel, mirroring `expireRunsBatch` and the single-run `#route`. `finalizeRows` still applies orderBy/take/skip globally over the merged set. This drops the id-set path's cross-store fallback, which existed to prefer the new-store copy when the same id was present in both stores. That collision cannot arise when each id maps to exactly one store (nothing writes a legacy-shaped id into the new store), so the fallback is dead code. The two id-set tests that asserted "new copy wins on collision" now assert the routing invariant: a legacy-shaped id resolves to the legacy store and the path never consults the new store. The open-predicate path (`#findRunsOpen`) is unchanged: an open `where` has no id to route on, so it still unions both stores and dedupes. |
||
|
Katia Bulatova
|
23d5771d56 |
feat(webapp): unconfigured billing limit UX and default billing alerts (#4328)
## Default billing alerts + billing limit page UX - New orgs get default billing alerts: $5, $100, $500, $1000, $2500. Existing orgs are backfilled by a billing-side data migration (companion [PR](https://github.com/triggerdotdev/cloud/pull/1657)). - The billing limit form starts with nothing selected for orgs that never set a limit — the save button appears once an option is picked. - The yellow banner now also shows on the billing limits page itself, asking to configure a limit. Hidden everywhere for members who can't manage billing. - Also fixes billing limit alert preview. Tests - `apps/webapp/test/billingLimitsRoute.test.ts` — dirty logic for empty/selected mode - `apps/webapp/test/billingAlertsDefaults.test.ts` — default values - `apps/webapp/test/billingAlertsFormat.test.ts` — preview after a limit change |
||
|
Eric Allam
|
a2d382b2be |
feat(webapp): add emission fan-out metrics to the native realtime feed (#4341)
## Summary Adds two counters to the native realtime backend so we can see how much duplicate row serialization the change router does per batch. When a run changes it can match several held feeds at once (a run subscription plus one or more tag/list feeds), and today each matching feed serializes that run's wire value independently. These counters quantify that fan-out so we can decide whether a shared serialization step is worth it. ## What they measure - `realtime_native.emission_run_serializations`: total wire-value serializations performed across feeds per batch (what the current path does). - `realtime_native.emission_distinct_serializations`: distinct (columns, run) rows those serializations cover (what a serialize-once-per-batch step would do). Average feeds-per-run is `run_serializations / distinct_serializations`, and `1 - distinct / run_serializations` is the serialization work a shared step could save. Wired through a new optional `onEmissionFanout` callback on the router. No behavior change. |
||
|
Katia Bulatova
|
b3b1441df9 |
fix(webapp): guard workload auth gate metric against dev HMR re-registration (#4339)
Wraps the workload_auth_gate_total Counter in the singleton helper (same pattern as reloadingRegistry.server.ts) so a dev hot reload doesn't crash with "A metric with the name workload_auth_gate_total has already been registered". No production behavior change. |
||
|
nicktrn
|
84add4ad3d |
feat(supervisor): export workload_token_enforcement_mode gauge (#4335)
Add a Prometheus gauge `workload_token_enforcement_mode` set to 1 for the active `WORKLOAD_TOKEN_ENFORCEMENT` value (`disabled`/`log`/`enforce`), emitted at startup on the shared registry. The existing mint/verify counters don't distinguish `log` from `enforce` (the verify outcome is recorded before the reject decision), so dashboards can't tell which mode a cluster is running. This gauge makes the active mode queryable at a glance. Supervisor typecheck passes. |
||
|
James Ritchie
|
11d8a05fa6 |
fix(webapp): restore admin debug tooltip on Tasks and Runs, make its IDs copyable (#4332)
Restores the debug panel on the **Tasks** and **Runs** pages, and makes the data it shows copyable. Admin/impersonation only — no change for regular users, so there's no `.server-changes` <img width="909" height="1420" alt="CleanShot 2026-07-22 at 12 05 27@2x" src="https://github.com/user-attachments/assets/ce2da167-dc23-422f-83f3-4f4aee9ed32c" /> |
||
|
Matt Aitken
|
81eac67069 |
fix(webapp): correct docs link on the blank prompts page (#4247)
## Summary The empty-state panel on the Prompts page linked to a docs path that no longer exists, so the "Prompts docs" button returned a 404. It now points to the current prompts documentation at /docs/ai/prompts, matching the link already used in the page header. |
||
|
James Ritchie
|
e7de120661 |
fix(webapp): remove Enterprise badge from SSO & Directory Sync menu item (#4333)
## What The organization side menu previously showed an "Enterprise" badge next to the SSO & Directory Sync item for any org not on the enterprise plan. That badge is now removed so the item renders without it. ## Screenshot (before) <img width="1428" height="649" alt="CleanShot 2026-07-10 at 08 24 41" src="https://github.com/user-attachments/assets/b9787363-972f-4dd5-bf61-486680f49f4c" /> |
||
|
Chris Arderne
|
7a14188663 |
fix(webapp): limit account email address length (#4330)
## Summary Limits user account email addresses to 254 characters in profile settings and onboarding. Oversized values are rejected before the uniqueness lookup, and the form fields enforce the same limit in the browser. ## Fix Both email update flows use a shared bounded email schema. Basic validation completes before the uniqueness lookup runs. |
||
|
Chris Arderne
|
bb34a2e224 |
fix(webapp): scope development branches to each member (#4323)
## Summary Allow each organization member to use the same development branch name without colliding with another member's environment. Fixes #4320. ## Fix Development branches now use the existing member-scoped project, slug, and organization-member key for upserts. Preview branches retain their project-wide shortcode behavior. New development branches receive distinct shortcodes while keeping their readable, member-scoped slugs. Existing branches continue to resolve through the member-scoped key, so this requires no migration or backfill. |
||
|
Matt Aitken
|
95307ba33c |
fix(webapp): tidy Usage page credits display (#4322)
Two small corrections to the organization **Usage** page credits display. ### 1. Label the credits panel "Credits" (was "Promo credits") The panel surfaces any credit balance, not only promo-code redemptions, so "Promo credits" is misleading when the credits come from another source. Renamed the heading to "Credits". ### 2. Don't show "Included usage" for Enterprise orgs Enterprise inherits the Pro plan's `includedUsage` value, so the Usage bar rendered an "Included usage: $50" tier marker for Enterprise organizations. Enterprise bills against prepaid credits rather than a per-month included-usage tier, so the marker was misleading. The `tierLimit` marker is now suppressed for Enterprise (`plan.type === "enterprise"`). Verified with `pnpm run typecheck --filter webapp`. |
||
|
James Ritchie
|
0b2919465c |
feat(webapp): redesign the side menu project and organization menus (#4066)
Redesign of the main side menu: separates Projects and Accounts from the Organization menu and makes the menu resizable. **Main changes** - **Organization & Account menus**: the top-left is now a dedicated organization menu (Settings, Usage, Billing, Team, SSO, integrations), with a separate account menu beside it (Profile, PATs, Security, Logout). - **Project switcher**: a new Project section above the Environment selector. - **Resizable side menu**: drag the right edge to set a custom width (saved per user), or click the edge to collapse/expand. - **Environment selector**: reworked to match the Project menu, including dev-branch handling. - **Account Profile page**: redesigned into the Security page's row-and-divider layout. Preview URL: https://samejr-org-menu-update.triggerlabs.dev/ https://github.com/user-attachments/assets/9b199576-6037-4ea6-9bdb-3ee15265b8c2 |
||
|
Chris Arderne
|
6642c8b785 |
fix(webapp): prevent duplicate envs from provision race (#4261)
Fixes TRI-12078 ## Summary Prevents concurrent environment setup requests from creating duplicate Staging and Preview environments. ## Fix Adds database-enforced uniqueness for root Staging and Preview environments. If two requests race, the losing request loads the environment created by the winner and continues successfully instead of creating a duplicate or returning an error. |
||
|
Katia Bulatova
|
d05f1a7398 |
chore(webapp): migrate from Remix compiler to Vite (#4188)
Replaces Remix compiler with the Vite plugin. The Express server (cluster, socket.io, ws) and the Docker image contract are unchanged. |
||
|
Chris Arderne
|
dc87b884e7 |
chore: upgrade to typescript 6 (#4310)
## Summary Upgrades the workspace to TypeScript 6.0.3 and applies the compiler, type, and build configuration changes required to preserve package layouts and existing runtime behavior, apart from correcting the HTTP status field used for deployment connection errors. ## Compatibility - Centralizes TypeScript 6.0.3 through the pnpm workspace catalog. - Replaces compiler options and module resolution modes that TypeScript 6 no longer accepts. - Restores explicit Node types where TypeScript 6 no longer includes them transitively. - Adds explicit declaration build roots that preserve each package's existing output layout. - Patches tsup to stop injecting the removed `baseUrl` option during declaration builds. - Uses type-only assertions for stricter typed-array and stream definitions without changing runtime behavior. - Reads the EventSource v3 HTTP status from `code`, so deployment connection errors include it correctly. - Keeps standalone CLI compatibility fixtures pinned to their existing TypeScript version and lockfiles. `turbo run typecheck` and the complete PR test suite are green. |
||
|
DKP
|
cbec61309a |
fix(webapp): fix promo page heading typography (#4311)
## What The `/promo` page heading rendered with overlapping lines — the two lines of "Promo codes are for new accounts" collided. ## Why The page used `Header2` stretched to display sizes (`sm:text-2xl md:text-3xl lg:text-4xl`), but `Header2` bakes in a fixed `leading-6` (24px). A 36px font in a 24px line box makes wrapped lines overlap. It only showed at `sm`+ widths and only on headings that wrap to 2+ lines, which is why it slipped through — the short single-line headings on the same page looked fine. ## Fix Switch both headings to `Header1` — the page-title primitive the sibling login pages (`login._index`, `login.magic`) already use for exactly this size. Add `leading-tight` (relative line-height, scales with font size, and this heading uniquely wraps to two lines) and `pb-4` to match the login pages' spacing convention. ## Testing Manually verified the signed-in view (`/promo` while logged in) renders as two clean, non-overlapping lines across breakpoints. Pure CSS/layout change — no automated test. |
||
|
Chris Arderne
|
6997aeb05e |
fix: security release 2026-07-08 (#4316)
⚒️ Publish Worker (v4) / build (supervisor) (push) Has been cancelled
|
||
|
Chris Arderne
|
cc748422d8 | test(webapp): replace slow metadata replica guard with unit test (#4312) | ||
|
Daniel Sutton
|
a7c734c223 |
test: caller-driven replica-lag + idempotency guards (stacked on #4284) (#4285)
## Stacked on #4284 — tests only This PR contains **only the tests** that guard the production fixes in #4284 (its base). Review #4284 first; this branch adds no production code. ## What Caller-driven replica-lag and idempotency guards for every fixed site: - Each guard **drives the real exported caller** (route loader/action, presenter `.call()`, service, or engine method) against a **real Postgres** with the owning replica frozen via the shared `laggingReplica` testcontainer primitive — never a store-seam reimplementation. - For a **fixed** site the guard goes **RED when the production change is reverted**; for a **tolerated read-view** site it's a caller-driven **GREEN** proof the miss self-heals (returns null/empty, no mutation, row live on primary). - The **global-scope idempotency** guard drives the real dedup + claim path through a **real `MollifierBuffer` over a Redis testcontainer** (real SETNX/poll/publish), and covers the cross-DB **andWait** waitpoint wiring and the **expired/failed clear-and-recreate** reacquire cases. Run with `vitest --no-file-parallelism` (testcontainers). Verified GREEN, and revert→RED verified per fixed site. |
||
|
Daniel Sutton
|
ae96b6c175 |
fix: read-your-writes + global-scope idempotency correctness under the run-ops split (#4284)
## What & why
Two related correctness fixes for the run-ops DB split. Under the split,
run-store reads can route to a **lagging read replica**; a just-written
run/waitpoint/batch can then be missed, causing a wrong decision.
**1. Read-your-writes → owning primary.** Surfaced first as an
intermittent `wait.until({ idempotencyKey })` re-wait on retry. Auditing
the run-store read surface found the same class at sibling sites (some
gating mutations or returning spurious 404s, others
tolerable/self-healing). Reads that must observe their own writes now
route to the owning **primary**
(`findRun`/`findWaitpoint`/`findBatchTaskRunByFriendlyId` →
`*OnPrimary`, a primary re-read on a miss, or a retryable 404 where the
SDK polls). Read-view reads stay on the replica. All additive — the
happy path is unchanged.
**2. Global-scope idempotency across the split.** A `global`-scope key
carries no per-run salt, so the same `(env, task, key)` triggered
concurrently from parents resident on **different** run-ops DBs could
dedup-miss on each DB and create a duplicate (the per-DB unique index
can't enforce cross-DB uniqueness). Such triggers (global scope, or
scope-absent, while split is active) are serialized through the existing
Redis idempotency claim, the loser resolves the winner by id across both
DBs, and the claim is reacquired on the expired/failed
clear-and-recreate path. `run`/`attempt` scope embed the run id and
never contend.
## Stacked for review
This is the **base** of a 2-PR stack, split so review is easier:
- **This PR** — production code only (34 files).
- **Stacked tests PR →
https://github.com/triggerdotdev/trigger.dev/pull/4285** — the
caller-driven guards (55 test files) on top of this branch.
## Validation
Local run-ops split, **both 2-DB and 3-DB**, fresh boot on this branch:
SDK canary 64/71 (only the known concurrency/input-streams/s3 failures),
quarantine sweep **0 unexpected** (340 pass / 16 known / 4 local) in
each topology, dashboard e2e 0 failed. No product regressions.
|
||
|
Chris Arderne
|
cecdfd94be | fix: only count preview branches toward the preview branch limit (#4283) | ||
|
Daniel Sutton
|
285666290f |
ci(webapp): wire the run-ops legacy guard into CI and add oxlint residency fences (#4279)
## What - Runs `apps/webapp/scripts/runOpsLegacyGuard.ts --check` as its own PR job (`runops-guard`), so code that reaches a run-graph table through the control-plane Prisma client instead of the RunStore fails the build. - Adds a `trigger-runops` oxlint plugin with two fast, in-editor rules scoped to `apps/webapp/app`: one for direct `prisma.taskRun`-style access, one for a control-plane client wired into a read-through slot. These are the cheap fence; the guard is the type-aware gate. - Fixes `CancelTaskRunService.callV1`: historical V1 runs are legacy-resident, so its two finalize writes now go through `runOpsLegacyPrisma` instead of the control-plane client (they'd miss the row once legacy is a separate database). - Regenerates the guard baseline, which had drifted stale (it referenced files deleted in an earlier PR). ## Why The guard existed but ran nowhere, so its baseline rotted and a real residency gap (the V1 cancel writes) sat undetected. Wiring it into CI turns it into a ratchet against new control-plane run-graph access. ## Verification Local, against a clean regen: `oxfmt --check`, `oxlint .`, `guard --check`, and `typecheck --filter webapp` all pass. Remaining baseline entries are 4 batch-results router reads through type-opaque `as PrismaReplicaClient` casts (correct at runtime, accepted) + 2 sanctioned legacy annotations. |
||
|
Daniel Sutton
|
821972176d |
fix(run-store,webapp): correct split-database read routing, write residency, and batches list ordering (#4272)
## Summary Correctness and performance fixes for deployments that split run data across more than one database. Single-database / self-hosted deployments are unaffected (they collapse to a single read/write path). - **Batches list (dashboard):** for some organizations the Batches list could hide older batches or show them out of order. It now orders and paginates by creation time (with the id as a stable tiebreak), so every batch appears exactly once, newest first. The pagination cursor format changes; older in-flight cursors simply restart from the first page. - **Reads:** waitpoint and snapshot lookups that are keyed by a single run now read only the database that holds that run instead of querying both, removing redundant queries on hot paths (unblock, snapshot reads). - **Writes:** environment-scoped writes with no owning run (standalone wait tokens, waitpoint tags, idempotency-key resets) now land in the same database as that environment's runs, rather than defaulting to the other one. An idempotency-key reset also falls back to the other database when it matches nothing, so a reset still clears the key wherever the run actually lives. ## Notes Verified end-to-end against multi-database setups: run-keyed reads and env-scoped writes land on the correct database with no cross-database writes, and the batches list surfaces every batch in creation order. New tests cover the batches ordering/reachability and the write-residency routing. |
||
|
nicktrn
|
0ff0abd776 |
fix(webapp): recover from stale /build assets via a bounded reload (#4282)
## Problem The webapp's HTML references content-hashed `/build` assets, and each running instance contains exactly one build and returns 404 for asset hashes it doesn't have. During a rolling deploy a client can hold HTML from one build while a request for one of its assets is served by an instance on a different build → missing styles or a failed chunk load. ## What this does On a `/build` stylesheet/script/chunk load failure, the client does a **bounded full-document reload** (at most 2 per 5 minutes, tracked in `sessionStorage`) so the page reloads onto a single consistent build. That's the whole mechanism — no polling, no `fetch` interception, no blocking overlay, no form snapshotting. - `apps/webapp/app/components/StaleAssetRecovery.tsx` — authored as a typed, lint-checked function and serialized to an inline script via `.toString()` (so the logic is real, reviewable code, not an opaque string), injected before `<Links />`, production only. - Detection: capture-phase `error` listener for `<link>`/`<script>`/modulepreload failures under `/build/`, plus an `unhandledrejection` guard for dynamic-import failures. - Guards: once-per-page re-entrancy guard, the bounded reload budget, and a `navigator.onLine` check so it never reloads into an offline error page. - Unit tests in `StaleAssetRecovery.test.ts`. ## Relationship to #4260 Replaces the recovery introduced in #4260 (reverted in #4280) with a much smaller, reload-only approach — the previous version intercepted `fetch` and could turn a data request into a navigation, and showed a full-screen overlay on any asset error; this drops both. ## `/build-version` compatibility shim `apps/webapp/server.ts` adds a tiny `GET /build-version` endpoint (build id only, `no-store`). A previously-deployed client build polls it after an asset failure and reloads once it sees a newer build, so those older tabs recover in one reload instead of getting stuck. Temporary — safe to remove once older clients have cycled out. It deliberately does **not** re-add an `X-Build-Id` response header. ## Also Restores the `.server-changes` writing guidance in `.claude/rules/server-apps.md` (reverted alongside #4260). ## Self-hosting note Recovery is most reliable when your load balancer keeps a client on one instance for the duration of a deploy (short session stickiness) — the reload then lands on a consistent build in one hop. |
||
|
Chris Arderne
|
73d966ad22 |
chore(webapp): remove deprecated realtime stream write action (#4250)
Removes the deprecated realtime stream write action kept for retired v3 task clients. Supported clients use the targeted stream write routes, while the existing stream read loader remains unchanged. |
||
|
nicktrn
|
051d7080d6 |
Revert "fix(webapp): survive asset hash rotation across rolling deploys (#4260)" (#4280)
Standard `git revert` of #4260. Its client-side stale-asset recovery is net-negative during normal deploys: - The `fetch` interception treats any `?_data=` request (Remix loader **and** action traffic) as a navigation and, on a build-id mismatch, `location.assign`es the tab to the fetched URL — an open dashboard tab can be hard-navigated to a raw data URL during a rolling deploy, losing unsaved input. - Any transient `/build` asset error (a network blip, an extension, an unrelated failed dynamic import) blanks the page behind a full-screen overlay for ~60s before offering a manual reload. - It serialized form field values to `sessionStorage` to restore them across the reload. This returns the webapp to the pre-#4260 baseline as a fast, low-risk step. Follow-ups (separate PRs): - a minimal reload-only recovery to replace this, - restore the unrelated `.claude/rules/server-apps.md` docs tidy-up from #4260 (via cherry-pick), - a load-balancer stickiness change addressing the root cause. |
||
|
Chris Arderne
|
939c00782d | feat(webapp): show runtime versions in deployment lists (#4273) |
