chore/react-compiler-lints
502 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Chris Arderne
|
338326c0d0 | fix(clickhouse): lowercase logs search index terms (#4705) | ||
|
Chris Arderne
|
49aff3cb39 |
fix(clickhouse): use compatible logs text index syntax (#4704)
## Summary Allow the logs search schema migration to run on ClickHouse versions that require text index options to be literals. ## Root cause The text index declared `lowerUTF8(search_text)` as a preprocessor option. Some ClickHouse versions reject that column expression while parsing index settings. The projected `search_text` is already normalized to lowercase before insertion, so removing the redundant preprocessor preserves search behavior. Verified with the task events search integration tests. |
||
|
nicktrn
|
b93904526c |
test(testcontainers): hoist container boot off the test timer (#4686)
## What The one-off worker container boot is billed to whichever test resolves the fixture first. This moves it into a `beforeAll` with its own timeout. ## Why vitest runs the fixture chain *inside* the test timer: ```js // @vitest/runner 4.1.7 setFn(task, withTimeout(...withFixtures(handler)..., timeout, ...)) ``` There is no `fixtureTimeout`. So booting Postgres (plus `CREATE DATABASE`, schema push, ClickHouse and Redis) lands on the first test and consumes a budget sized for test work. That is why losing the image pre-pull on fork PRs was fatal rather than merely slower: the extra ~10s crossed the 60s cap. Since fork time is roughly internal + 10s and forks exceed 60s, internal runs were already clearing that cap by under 10s — a latent flake regardless of forks. ## How `withWarmup` wraps each fixture family and lazily registers a `beforeAll` on first touch, with its own generous timeout. Registration is lazy so only files that actually use a family pay for it — `@internal/testcontainers` is imported by hundreds of test files, many of which only need Redis. It registers once per file, since `isolate` gives each file a fresh module registry. Eight families are wrapped. `isolatedRedisTest`, `replicationContainerTest` and `postgresAndRedisTest` are deliberately untouched: they use per-test containers by design, so there is no one-off boot to hoist. No test file or CI changes, and it applies to every package using these fixtures. ## Verification Proven by mutation. `src/warmup.test.ts` runs container tests under a deliberately tight cap: | | Result | | --- | --- | | with the warm-up | passes | | warm-up neutered | fails, `Test timed out` | It is kept as a regression test — without it, unwrapping a fixture would break nothing visibly. `triggerFailedTask.call.test.ts`, one of the five shard casualties, passes locally in 20.4s. ## Also here `@internal/testcontainers` had no `test` script, so `turbo run test --filter "@internal/*"` skipped the package and its existing `heteroDedicated.test.ts` never ran in CI. Adding the script (matching the sibling packages') runs both files; verified green through turbo exactly as CI invokes it. |
||
|
Chris Arderne
|
f4320937c5 |
chore: prefer direct iteration and function callback types (#4677)
## Summary Enable lint rules that prefer direct iteration and concise function callback types. The existing code now uses direct iteration where no index is needed, and callback contracts use function types consistently. Base: [#4675](https://github.com/triggerdotdev/trigger.dev/pull/4675) |
||
|
Chris Arderne
|
0f725cf2ba |
chore: enable lint cleanup rules (#4673)
## Summary Enable small cleanup rules for redundant boolean expressions, object ownership checks, assignments, and object construction. The existing call sites now use the simpler equivalent forms, keeping future code consistent without changing behavior. Base: [#4672](https://github.com/triggerdotdev/trigger.dev/pull/4672) |
||
|
Matt Aitken
|
444c2215ca |
fix(run-engine): stop requeued runs with a lapsed ttl being orphaned in the queue (#4669)
## Summary
A run triggered with a `ttl` could get permanently stuck showing as
queued. If the run started executing and was then requeued after a
failure (a stalled heartbeat, a worker dying mid-run) once its TTL had
already elapsed, the next dequeue pass silently dropped it from every
queue structure. The run stayed QUEUED in the database forever, and
nothing (dequeue, the TTL consumer, queue repair) could ever see it
again.
## Root cause
Enqueue registers a TTL entry for the TTL consumer, and the first
dequeue removes it ("the run is executing, not expired"). A nack rewrote
the message preserving the original `ttlExpiresAt` without
re-registering that entry. The next dequeue pass then took the
expired-TTL branch: remove the run from the queue sorted sets and leave
the message for the TTL consumer to finalize. But the consumer's entry
was gone, so nothing ever finalized the run.
The fix has two halves:
- `nackMessage` drops `ttlExpiresAt` from the rewritten message. TTL
only applies to runs that have never been dequeued (the same contract as
`includeTtl` on re-enqueues), so a requeued run stays dequeuable and is
never expired by its original deadline.
- The dequeue expired-TTL branches now (re-)register the TTL entry
instead of assuming it exists, so any message still carrying a lapsed
`ttlExpiresAt` with no TTL entry (including ones written before this
fix) finalizes as EXPIRED instead of orphaning.
## Verification
New engine test suite `ttlNackRequeue.test.ts` (testcontainers, real
Redis and Postgres). All four tests fail before the fix and pass after:
- a heartbeat-stalled EXECUTING run with a lapsed TTL is requeued and
dequeued again instead of orphaned (the full production failure chain)
- requeue-after-failure strips `ttlExpiresAt` so later dequeues do not
treat the run as expired
- a lapsed-TTL message whose TTL entry is missing is re-registered by
dequeue and finalized as EXPIRED, for both plain and concurrency-key
queues
Also ran the existing ttl, heartbeats, dequeuing and attemptFailures
engine suites plus the full run-queue suite (149 tests) against the
change.
|
||
|
Chris Arderne
|
b4313c8199 | feat: logs search v2 (#4615) | ||
|
Chris Arderne
|
53ca44dd2d | chore: cache and clean up Knip analysis (#4658) | ||
|
Katia Bulatova
|
e768d0a724 |
feat(webapp): run the dashboard agent through AWS Bedrock behind an env switch (#4609)
## What & why The dashboard agent can now run its model calls through AWS Bedrock instead of the direct Anthropic API, chosen by a single env switch. It's **off by default** (`DASHBOARD_AGENT_MODEL_PROVIDER` unset ⇒ `anthropic`), so merging changes nothing at runtime — the Bedrock path is a dormant branch until an operator sets the switch and AWS config. The default Anthropic path is byte-for-byte unchanged. This also carries a related tenant-isolation hardening for the agent's delegated token (kept together deliberately — both land the agent on Bedrock for HIPAA readiness). Refs: TRI-13251, TRI-11032. ## What's inside **Provider seam** — `internal-packages/dashboard-agent/src/model-provider.ts`: the registry now holds both `anthropic` and `bedrock`; `resolveDashboardAgentModel()` maps the canonical `"anthropic:<id>"` strings the managed prompts carry to the active provider, and the cache-breakpoint helpers emit the active provider's shape — Anthropic `cacheControl` vs Bedrock `cachePoint`. Managed prompt strings stay canonical, so stored prompts don't change meaning. Unmapped model ids throw rather than shipping a guaranteed-404 profile. All agent, watch, compaction and title callsites route through the resolver; the `dashboardAgentModelKey` locals override (test mock injection) is preserved. **Cache telemetry** — `step-cache.ts`: cache token usage is read from the active provider (Anthropic reports it on provider metadata; Bedrock reports the write on metadata and the read via standard usage), so `gen_ai.usage.cache_*` is populated on both. This also fixes a latent ordering bug where step attributes could null-overwrite the prompt-cache read count. **Webapp callsites** — `dashboardAgentHeadStart.server.ts` and the head-start route resolve the model and the cache breakpoint through the same seam, so the warm-up prefix and the following turn share one provider. The head-start firing gate is provider-aware: on Bedrock it gates on `AWS_REGION` and lets the SDK resolve credentials (IAM role / static keys / session token / bearer), so a role-based deploy still warms; on Anthropic it stays `Boolean(ANTHROPIC_API_KEY)`. `app/env.server.ts` gains the optional AWS vars and validates `DASHBOARD_AGENT_MODEL_PROVIDER`. `ANTHROPIC_API_KEY` is untouched and not required on a Bedrock deploy. **Tenant-isolation hardening** — `internal-packages/rbac/src/fallback.ts`: for a **scoped** context, the OSS `authenticateUserActor` now applies the same membership floor as the session path — a delegated user-actor token whose user is not a member of the scoped org/project is denied (403). Unscoped tokens keep their prior behavior (no tenant claim, no lookup). The user lookup falls back replica→primary so replication lag can't spuriously 401 a just-joined member. Members and admins are unaffected. Previously this invariant held only through per-route discipline; this makes it structural. ## Enabling Bedrock (later, ops) - Set `DASHBOARD_AGENT_MODEL_PROVIDER=bedrock` **identically** in both the webapp and the agent task container — the webapp warms the cache prefix and the task reads it, so a split would silently miss the cache. - Set `AWS_REGION` and provide credentials the Bedrock SDK can resolve (IAM role preferred). For v1 this runs **without** an Anthropic API key. Note: with no Anthropic key set, rollback is "turn the agent off", not "unset the switch" (unsetting falls back to the Anthropic provider, which then has no key). - Two things to confirm before rollout: the Sonnet inference-profile id is validated against the SDK's own model-id union but still warrants a live smoke test; and Bedrock prompt caching for Sonnet is a 5-minute window (not Anthropic's 1h), so input-token cost rises when flipped. ## Testing Unit tests cover both provider paths: the provider switch and per-provider cache shapes, a structural regex asserting Bedrock ids are real inference profiles (not an echo of the table), the split-metadata cache telemetry, and real-Postgres RBAC tests — member allowed, scoped non-member denied (org-only and project-only), missing user → 401, admin non-member exempt, unscoped success. `typecheck --filter webapp` and the dashboard-agent + rbac suites pass. |
||
|
Chris Arderne
|
b33197691b | chore: enforce no unused deps or code in ci (#4654) | ||
|
Wes Mason
|
a55f7cdf4d |
fix(run-engine): stop a '*' concurrency key stranding its whole base queue (#4628)
## The bug
A concurrency key is an unrestricted client string
(`ConcurrencyKeySchema` is `z.union([z.string(),
z.number()]).transform(String)`), and `concurrencyKeySection` does no
escaping, so `*` reaches the queue raw. `queueKey` then renders it as
`...:queue:<q>:ck:*`, which is byte-identical to the wildcard member the
CK scripts keep in the master queue to mean "this base queue has
concurrency-key work".
Every CK script ends with the same pair:
```lua
-- Rebalance master queue with ck:* member
redis.call('ZADD', masterQueueKey, earliestIdx[2], ckWildcardName)
-- Remove old-format entry from master queue (transition cleanup)
redis.call('ZREM', masterQueueKey, queueName)
```
`ckWildcardName` is `toCkWildcard(message.queue)`, and for a `*`-keyed
run that returns the identical string, so the cleanup on the second line
deletes what the rebalance on the first line just wrote.
The master queue then has no entry for that base queue, while `ckIndex`
and the variant queues still hold the work. **Every concurrency key on
the queue stops being dequeued**, not just the `*` one. It is silent,
and it only recovers if some later write happens to re-add the member.
Reproduced before the fix:
```
master queue AFTER normal ck enqueue: ["{org:...}:queue:task/my-task:ck:*"]
master queue AFTER ck='*' enqueue: []
ckIndex members (work still queued): [":ck:user-1", ":ck:*"]
dequeued: []
```
Blast radius is bounded to the environment that triggers it, so it is
self-inflicted rather than cross-tenant, but a single trigger stalls the
queue for everything on it.
## The fix
Guard the cleanup so it never removes the wildcard member:
```lua
if queueName ~= ckWildcardName then
redis.call('ZREM', masterQueueKey, queueName)
end
```
Applied to all 10 CK scripts (4 enqueue, 6 ack/nack/dead-letter). No
key-format change and no migration: a queue already stranded in Redis is
repaired by its next write.
I considered rejecting `*` at the API boundary instead and rejected it.
Existing Redis state and `TaskRun.concurrencyKey` rows already hold raw
`:`-bearing and `*` keys, so changing key construction would orphan
in-flight messages and split concurrency accounting mid-deploy. Boundary
validation would still be reasonable as belt-and-braces later, but the
Lua guard alone fixes it including for state already out there.
## Testing
`ckWildcardKey.test.ts` covers the enqueue, ack and nack paths. All
three pass with the guard and **all three fail without it**, verified by
reverting. Full `src/run-queue/` suite is green (166 tests).
## Note for #4367
The virtual-time branch adds three more CK scripts with the same pattern
(`enqueueMessageCkVtimeTracked`, `enqueueMessageWithTtlCkVtimeTracked`,
`nackMessageCkVtimeTracked`). They do not exist on main so they are not
in this PR; the same guard needs applying there, and I will do that on
that branch.
|
||
|
Chris Arderne
|
99f0787148 | feat(cli,webapp): default new projects to node-24 (#4649) | ||
|
nicktrn
|
f3c46f140e |
chore(deps): raise nanoid floors, drop unused declarations (#4637)
## Summary `nanoid` was pinned at exactly `3.3.8` in five manifests. Two of those five never imported it: in `internal-packages/schedule-engine` and `internal-packages/webhook-engine` the only occurrence of the string `nanoid` in the entire package was the `package.json` line itself. Both are removed rather than bumped. The three that genuinely use it move to `3.3.18`, a version already present in the tree via `postcss`, so this pulls in nothing new. | Package | Uses it | Change | | --- | --- | --- | | `internal-packages/schedule-engine` | no | removed | | `internal-packages/webhook-engine` | no | removed | | `apps/webapp` | yes | `3.3.8` to `3.3.18` | | `packages/core` | yes | `3.3.8` to `3.3.18` | | `internal-packages/run-engine` | yes | `3.3.8` to `3.3.18` | | `packages/redis-worker` | yes | `^5.0.7` to `^5.1.16` | `redis-worker` is on the 5.x line and is included because its declared range already permitted a newer release; the lockfile had simply not re-resolved, leaving it on `5.1.2`. The unused declarations were found with `pnpm run knip:deps`, which the repo already ships. `pnpm run typecheck` passes across all 57 workspaces. |
||
|
Eric Allam
|
c0b84595a3 |
feat(webapp): hosted webhook ingress, delivery pipeline, and dashboard (#4344)
## Summary The server half of hosted webhooks: the public ingress endpoint, signature verification, the delivery pipeline (Postgres partitioned storage + ClickHouse for ordering), the in-app partition manager, the HTTP API, and the dashboard (Deliveries, Endpoints, and the in-app test console). The public SDK and docs half is #4537. That PR carries the user-facing API (`webhook()`, `chat.event` / `chat.channels`, the `@trigger.dev/slack` connector) and builds on the shared `@trigger.dev/core` schemas that ship here. ## Shipping behind a flag A `WEBHOOK_ENABLED` env var (default off) gates the public ingress route and the engine worker plus partition cron, so merging and deploying this changes nothing in production until it is flipped on per environment. The dashboard is separately gated per org by the `hasWebhooksAccess` feature flag. ## Note on packages This PR includes the `@trigger.dev/core` schema additions the server compiles against, but carries no changeset. Core is not consumed independently of the SDK, so it is released together with the SDK via #4537. Keeping its changeset off `main` means no release cut from `main` publishes it early. |
||
|
Eric Allam
|
b98dd79fe4 |
feat(webapp,run-store,database): env-configurable transaction resilience (maxWait + tx-start retry) (#4623)
## What
Makes two transaction-resilience behaviors real and env-var
configurable, defaults set to the good values, so we can tune during and
after the Aug 15 database patch window without a redeploy:
- **maxWait 2s → 10s** (TRI-12982): how long Prisma waits to borrow a
connection before it can `BEGIN`. A restart freeze holds the pool full,
and the only thing that errored was transaction starts giving up at 2s.
- **Retry transaction-start P2028-at-acquisition** (TRI-12984): when
Prisma can't borrow a connection within `maxWait` it raises P2028
(`Unable to start a transaction in the given time`) and **no SQL ran**,
so retrying is safe. Scoped narrowly: only that error (never P2024
pool-exhaustion), 2 attempts, jittered backoff, and a token-bucket
budget so a mass freeze can't amplify into a retry storm.
## Env vars (`DATABASE_*` convention)
Generic defaults:
| var | default |
|---|---|
| `DATABASE_TRANSACTION_MAX_WAIT_MS` | `10000` |
| `DATABASE_TRANSACTION_START_RETRY_ENABLED` | `true` (kill switch) |
| `DATABASE_TRANSACTION_START_RETRY_MAX_ATTEMPTS` | `2` |
| `DATABASE_TRANSACTION_START_RETRY_BACKOFF_MIN_MS` | `50` |
| `DATABASE_TRANSACTION_START_RETRY_BACKOFF_MAX_MS` | `250` |
| `DATABASE_TRANSACTION_START_RETRY_BUDGET_PER_SEC` | `50` |
| `DATABASE_TRANSACTION_START_RETRY_BUDGET_BURST` | `100` |
Per-writer-pool overrides, each falling back to the generic when unset
(same pattern as the per-client pool/connect-timeout work):
`RUN_OPS_DATABASE_TRANSACTION_*` and
`RUN_OPS_LEGACY_DATABASE_TRANSACTION_*` (all 7 knobs each). Transactions
only open on writer pools, so those are the only pools with their own
knobs. Each pool gets its **own** token bucket, so a storm on one pool
can't drain another's retry budget.
## Design
- The retry primitives live in `internal-packages/database` and never
read `process.env` (IoC): a P2028-at-acquisition classifier, a
`TokenBucketRetryBudget`, and `withTransactionStartRetry`, folded into
the `$transaction` helper via a new `startRetry` option. Config is
resolved at the app boundary and threaded in.
- The `$transaction` helper is the chokepoint (wraps the whole
transaction), not the per-statement `$allOperations` extension.
- The run engine's writes go through `PostgresRunStore`'s own
`.$transaction(...)`, not the webapp helper, so both the helper and the
two `PostgresRunStore` sites apply maxWait + retry (sharing the per-pool
config). Builds on the `options?: { timeout, maxWait }` seam added in
#4514.
- Webapp `$transaction` call sites get the default `maxWait` + retry
injected at one merge point, so no call site needed editing.
## Evidence
- Unit red/green in `internal-packages/database`: reverting the helper
wiring turned the acquisition-retry test red (`Unable to start a
transaction in the given time`), re-applying it green. Full package
suite 25/25. Covers: classifier (P2028-acq yes, P2024 no, in-tx P2028
no), retry (retry-then-succeed, no-retry P2024, stop at maxAttempts,
disabled, budget-exhausted, jitter bounds), token bucket, and
`$transaction` wiring.
- Typecheck clean: webapp, run-store, run-engine.
- Full-stack run: bounded queue-ay pass (15 projects, real dev runs
through the run-engine `PostgresRunStore` transaction path). 13 pass;
the 2 failures are one documented known-failure and one
stale-worker-state flake that passes 2/2 with this change active on a
fresh app.
- Boots cleanly with per-pool overrides set.
## Configuration & rollout
Ship **inert** first (zero behavior change), then flip to the good
values **live via env** — no redeploy needed for either.
### Inert — behaves exactly as today
```
DATABASE_TRANSACTION_MAX_WAIT_MS=2000 # Prisma's built-in default (change defaults to 10000)
DATABASE_TRANSACTION_START_RETRY_ENABLED=false # disable the new retry entirely
```
`maxWait=2000` is what every path used before (Prisma's default; the
run-store sites and the helper passed no maxWait). `retry=false`
short-circuits `withTransactionStartRetry` to a single run and makes the
serialization-retry exclusion a no-op. Verified on the pooler-freeze
rig: identical fail-fast P2028 at ~2003ms with zero retries —
byte-for-byte current behavior, across all pools.
### Production ("good") — the baked defaults
Rely on defaults (nothing to set) or set explicitly:
```
DATABASE_TRANSACTION_MAX_WAIT_MS=10000
DATABASE_TRANSACTION_START_RETRY_ENABLED=true
DATABASE_TRANSACTION_START_RETRY_MAX_ATTEMPTS=3 # 3 attempts (2 retries); ~30s acquisition tolerance covers a ~20-25s freeze
DATABASE_TRANSACTION_START_RETRY_BACKOFF_MIN_MS=50
DATABASE_TRANSACTION_START_RETRY_BACKOFF_MAX_MS=250
DATABASE_TRANSACTION_START_RETRY_BUDGET_PER_SEC=50
DATABASE_TRANSACTION_START_RETRY_BUDGET_BURST=100
```
Per-pool overrides `RUN_OPS_DATABASE_TRANSACTION_*` and
`RUN_OPS_LEGACY_DATABASE_TRANSACTION_*` (all seven knobs each) are
optional and fall back to the generic set — not needed for v1; the
generic set covers the control-plane, run-ops, and run-ops-legacy writer
pools. Readers open no transactions and take nothing.
**Guardrail:** the retry only engages when a pool's `pool_timeout` >
`maxWait`. Prod is fine (`DATABASE_POOL_TIMEOUT=60` >> 10). Do not set
any writer pool's `pool_timeout` at or under `maxWait`, or saturation
failures flip from retryable P2028 to non-retryable P2024 and the retry
silently stops helping.
### Rollback
Env flip (set inert) or revert. Retry only fires where no SQL ran, and
the per-pool token bucket caps a storm. No migration.
refs TRI-13295, TRI-12982, TRI-12984
|
||
|
Eric Allam
|
dc8f90e66e |
fix(run-engine,webapp): resolve dequeue worker version fresh per task (#4622)
## Summary After a deployment promotion or rollback, newly triggered runs could keep dispatching onto the previously deployed version for up to 30 seconds. Runs now resolve the current version fresh on every dequeue, so a promotion or rollback takes effect immediately. ## Fix The dequeue path resolved the worker version through a 30s in-process cache that nothing invalidated on promotion, and it loaded the worker's entire task and queue set only to keep the single row matching the run. Both go away: the resolve now fetches just the matched task and queue by unique index and reads them fresh, so there is no cache left to serve a stale version. ``` - cache.get(env:current) # 30s TTL, never invalidated -> stale - worker + ALL tasks + ALL queues + worker + one task WHERE slug=... + one queue WHERE id/name=... # fresh ``` A kill-switch env var (`RUN_OPS_WORKER_VERSION_FRESH_READ_ENABLED`, default on) falls back to the old cached path without a code deploy. Verified end-to-end on an isolated stack: a run triggered after a mid-stream promotion now dequeues onto the new version, with the previous stale behavior reproduced first. |
||
|
Eric Allam
|
8dc8e1b58b |
perf(run-engine,webapp): narrow the control-plane worker-version read to the columns dequeue uses (#4619)
## Summary
The worker-version resolve path fetched every column of every
`BackgroundWorkerTask` for a worker (`include: { tasks: true }`), plus
full `WorkerDeployment` and `TaskQueue` rows, just to match one task at
dequeue. That pulls large JSON columns none of this path reads (task
`payloadSchema`/`config`/`queueConfig`/`description`, deployment
`externalBuildData`/`buildServerMetadata`/`errorData`/`git`, queue
`rateLimit`), so each resolve transfers and deserializes far more than
it uses.
## Fix
Replace the includes with explicit `select`s of only the columns dequeue
reads, in both the passthrough resolver and the app resolver:
- task: `id`, `slug`, `machineConfig`, `retryConfig`,
`maxDurationInSeconds`
- deployment: `id`, `friendlyId`, `imageReference`, `imagePlatform`
- queue: `id`, `name` (the queue matcher keys on both)
The shared `ResolvedWorkerVersion` element types narrow to match
(mirrored in the cache), which also shrinks each cached worker-version
entry.
## Impact
The `tasks` read fetches every task of a worker to match one, so its
cost scales with task count and payload-schema size. For a worker with
~70 registered tasks, dropping the unread columns cuts the per-query
transfer roughly:
| Task shape | Before | After | Reduction |
|---|---|---|---|
| Light (no payload schema, small config) | ~28 KB | ~14 KB | ~54% |
| Typical (mixed schemas / config) | ~62 KB | ~14 KB | ~77% |
| Schema-heavy (large `payloadSchema`) | ~200 KB | ~14 KB | ~93% |
The `after` size is roughly fixed because the kept columns are small;
the win grows with how heavy the dropped JSON is. Narrowing `deployment`
(four JSON columns off a single row) and `queues` saves further on top.
No behavior change: pure read-shape narrowing, no flag and no schema
change, so rollback is a plain revert. Verified with a red/green
run-engine test that asserts the resolved task, deployment, and queue
carry only the used columns, plus the queue feature-matrix runs (batch,
retry-policy, machine-preset, plain trigger) that exercise the kept
columns.
|
||
|
Chris Arderne
|
1240d91e43 | perf(clickhouse): add task_events_v2 inserted_at minmax index (#4620) | ||
|
Chris Arderne
|
3e7964e7fa |
feat: surface cron windows in webapp, cli, sdk (#4572)
## Summary Adds execution-window product surfaces for both declarative and imperative schedules. - Declarative schedules can set `window` through `schedules.task()`, with support for whole-minute, hour, and percentage values. - Imperative schedules can create, update, clear, and inspect windows through the API and dashboard. - Schedule API responses preserve `nextRun` as the nominal CRON time and expose `nextRunEffectiveAt` as the stable assigned time. - The dashboard displays configured windows alongside assigned upcoming-run times. - Deploy output summarizes declarative schedules and suggests adding a wider window when the default 60-second placement range is used. ## Design Window validation remains authoritative on the server and ensures each window is compatible with the schedule cadence. Omitting a window uses the default 60-second range, while explicit zero-duration windows remain supported. Deployment summaries are derived from the deployment's stored task metadata, so they reflect the declarations associated with that deployment. |
||
|
nicktrn
|
fa7eea39d8 |
fix(core): stop custom metric exporters breaking the metrics export (#4613)
## Summary Projects that configure their own `metricExporters` or `metricReaders` in `trigger.config.ts` were losing task metrics on nearly every run, and seeing an unexplained `Failed to flush tracingSDK` alongside `OTLPExporterError: Bad Request` in their run logs. Spans and logs kept working, so the runs otherwise looked healthy. ## Root cause and fix Every configured exporter gets its own `PeriodicExportingMetricReader`, and `meterProvider.forceFlush()` fans out across all readers with `Promise.all`, so two collections can land on the same millisecond. `@opentelemetry/host-metrics` divides by the elapsed interval to compute `process.cpu.utilization` ([common.ts](https://github.com/open-telemetry/opentelemetry-js-contrib/blob/main/packages/host-metrics/src/stats/common.ts)), so a zero interval yields `0/0`. `JSON.stringify(NaN)` is `null`, and a collector rejects `"asDouble": null` with a 400 that drops the **entire** request, not just the offending point. `flush()` and `shutdown()` now walk the metric readers one at a time, so collections can no longer share a timestamp. Each reader is isolated, so one failing reader cannot skip the readers behind it, and every failure is logged with the reader that produced it. The first error is still rethrown, so callers see failures exactly as before. As a second layer, non-finite data points are dropped just before our own export, so a metric that divides by zero cannot take the rest of the batch with it. Exporters and readers supplied through `trigger.config.ts` are untouched by that filter and still receive raw data. The trade-off is that configured exporters now flush after the built-in one rather than alongside it, so flush latency is the sum rather than the max. An internal test package's dependency on core was replaced with a local helper, because core now needs that package in `devDependencies` and the two together formed a workspace cycle. ## Verification Tested against a real collector in a container: a batch containing a `NaN` reading is rejected with a 400 without the fix and accepted with it, and a single flush is asserted to collect from one reader at a time. |
||
|
Katia Bulatova
|
ee854480fe |
fix(webapp): dashboard agent maintenance moves into the agent project (#4599)
## What & why The dashboard agent's upkeep — retention deletes and the investigation sweep — ran as cron jobs on the webapp's common worker, even though it only touches the agent's own datastore. This moves that upkeep into the agent's Trigger project as scheduled tasks (TRI-13182). ## What's inside **Retention** — `internal-packages/dashboard-agent/src/maintenance.ts`, a daily task (03:00 UTC). Deletes turn evals older than 30 days, hard-deletes chats soft-deleted more than 30 days ago, and purges terminal watches and submission rows older than 7 days. It used to run every 5 minutes; nothing needs a hard delete that fast, so it is daily now, draining in bounded batches and warning if it hits the cap. It retries (3 attempts) because the next run is a day away. It connects with `DASHBOARD_AGENT_DATABASE_URL`, falling back to `DATABASE_URL` like every other task in the package (the deletes are confined to the agent's own Postgres schema), and skips when neither is set. **Investigation sweep** — `src/investigation-sweep.ts`, every 5 minutes, same as before: settles investigation cards stuck `in_progress` (30-minute window, attempt cap, force-abandon note). It keeps the fast cadence because it fixes live state the UI is showing. **What stays in the webapp.** The watch finalize/deliver sweep and batch rearm: they cover a dead agent-side tick chain — a backstop can't live inside the thing it backstops — and they need the main database and the alerts worker. The org-deletion chat purge also stays: deletion must not depend on the agent project being deployed. The removed cron job keeps a cron-less tombstone entry so already-queued items drain cleanly; remove it in a follow-up. **Test plumbing** — the drizzle migration replayer that webapp tests hand-rolled is now exported once from `@internal/dashboard-agent-db/testing`; the moved tests live in the agent package as `src/*.test.ts` against real Postgres. ## Testing Agent package: retention passes (backlog drain, batch cap, no-op guard, chat-delete cascade) and the sweep, on testcontainers Postgres. Webapp: the watch/chat suites, plus a test that a settlement card stops the dashboard spinner. Full typecheck on both. |
||
|
Eric Allam
|
96b2959107 |
perf(database): index PersonalAccessToken.userId so token lookups stop seq-scanning (#4588)
## Summary The two personal-access-token lookups by `userId` (one also filtering `revokedAt is null`, the other also filtering `name`) had no index on `userId`, so each did a full sequential scan of the `PersonalAccessToken` table to return a single row. `userId` is also an unindexed foreign key. ## Fix Add a single `@@index([userId])`. A user owns only a handful of PATs, so once `userId` is indexed each lookup touches a few rows and the residual `revokedAt` / `name` filter is trivial. Both query shapes lead with `userId =`, so one index serves both and a composite would only add write cost. The migration uses `CREATE INDEX CONCURRENTLY IF NOT EXISTS`, which is online-safe under write load and reversible by dropping the index. Verified with a seeded local EXPLAIN: both queries go from a full sequential scan to an index scan on the new index. |
||
|
Eric Allam
|
4fd7cc0f55 |
perf(webapp,database): index RuntimeEnvironment.pauseSource for the billing-limit reconcile tick (#4590)
## What
The `billingLimit.reconcileTick` worker calls
`getOrgIdsWithBillingPauseSource()` on
`BILLING_LIMIT_RECONCILE_INTERVAL_MS` (~every 90s) to find which orgs
currently have billing-limit-paused environments. Two problems:
1. `RuntimeEnvironment.pauseSource` had no index, so `WHERE pauseSource
= 'BILLING_LIMIT'` was a **sequential scan of the whole table** on the
control-plane primary, every tick.
2. Prisma `distinct` dedups **after** fetching, so it read every paused
row (thousands) to produce a handful of distinct org ids.
This PR:
- Adds a **partial index** on `RuntimeEnvironment (pauseSource,
organizationId) WHERE pauseSource IS NOT NULL`. Nearly all rows have
`pauseSource = null`, so the index stays tiny. Second column lets the DB
satisfy the distinct-org lookup from the index. Defined in SQL (Prisma
can't express partial indexes), matching the existing partial-unique
indexes on this model.
- Switches the query from `findMany({ distinct })` to
`groupBy(["organizationId"])`, pushing DISTINCT into the DB so it
returns only the distinct orgs.
## Evidence
**Correctness** — colocated `postgresTest` (testcontainers, no mocks):
multiple `BILLING_LIMIT` envs in one org collapse to one org id,
`pauseSource = null` envs are excluded, each org id returned once. 5/5
tests in `billingLimitReconciliation.test.ts` pass.
**Plan change** — `EXPLAIN ANALYZE` on a synthetic table (200k rows,
5,250 `BILLING_LIMIT` across ~40 orgs, mirroring the test-side numbers
from the investigation):
| | Before (no index) | After (partial index) |
|---|---|---|
| Plan | Seq Scan (194,750 rows removed by filter) | Bitmap Index Scan
on partial index |
| Buffers | 1355 | 51 (index 6 + heap 45) |
| Exec time | 6.06 ms | 0.59 ms |
Index size 56 kB vs table 11 MB. The key win: cost now scales with the
paused-env count, not total table size, which matters most on prod where
the table is far larger.
## Rollout & rollback
- **Index**: `CREATE INDEX CONCURRENTLY IF NOT EXISTS`, in its own
migration file. Pre-apply the index manually on the control-plane
primary before deploying the migration (the migration is a no-op if the
index already exists).
- **Query change** is behavior-equivalent (same distinct org set), so no
flag needed.
- **Rollback**: revert the deploy and drop the index. No data migration
either direction.
## Notes / limitations
- The planner uses a Bitmap Heap Scan, so `organizationId` is still read
from the heap (45 blocks for the matched rows only, not the whole
table). A pure index-only scan isn't chosen for the bitmap path; the
second index column keeps that open for the index-scan path at
negligible cost.
refs TRI-13169
|
||
|
Eric Allam
|
4658cd0721 |
perf(database): index WorkerDeployment on (environmentId, status, id) for the deployments list (#4591)
## What Adds a composite index `@@index([environmentId, status, id])` to `WorkerDeployment`. The public deployments list (`GET /api/v1/deployments`) filters by `status` and paginates by `id` descending. The existing indexes cover `(environmentId, createdAt)` and the PK, but nothing covers `status`. So for a status filter Postgres walks back through the environment's deployments discarding non-matching statuses, reading roughly 350 rows for every 1 returned (p99 ~1.1s on the busiest environments). The new index makes the status filter index-satisfied and lets `id` serve both the cursor range and the `ORDER BY id DESC`, bounding the read to a single page. Full composite (not partial) because callers filter by arbitrary status values with no single dominant one. ## Query ```sql SELECT ... FROM "WorkerDeployment" WHERE "environmentId" = $1 AND "status" = $2 [AND "id" < $3] ORDER BY "id" DESC LIMIT $4; ``` Source: `apps/webapp/app/routes/api.v1.deployments.ts`. ## Evidence Reproduced on an isolated stack: one environment seeded with 7,000 deployments, the filtered status appearing 1 in 333 rows. Before (no index): ``` Seq Scan on "WorkerDeployment" (rows=21) Rows Removed by Filter: 6979 Buffers: shared hit=206 Execution Time: 2.9 ms (+ a sort for id desc) ``` After (with the index): ``` Index Scan Backward using "WorkerDeployment_environmentId_status_id_idx" Index Cond: (environmentId = $1 AND status = $2) Buffers: shared hit=23 Execution Time: 0.43 ms ``` Rows-removed-by-filter drops to 0; buffers 206 -> 23. The cursor (mid-pagination) variant uses the same index with all three predicates as the index condition. A dense/common status keeps the cheap PK backward scan (already fine); the index targets exactly the rare-status paths that were amplified. End-to-end against the running webapp API: `?status=FAILED` returns the correct newest-first page and paginates correctly across pages, and the emitted SQL matches the query above. ## Rollout - Index only, `CREATE INDEX CONCURRENTLY IF NOT EXISTS` in its own migration file. Online-safe under write load. - Pre-apply the index in production before the migration deploys, per repo convention (the migration is then a no-op). - Rollback: drop the index. No data migration. refs TRI-13171 |
||
|
Katia Bulatova
|
480bede0ad |
feat(webapp,sdk): dashboard agent plan enforcement, component gallery — and fixes (#4516)
Plan enforcement for the dashboard agent — message quota and watch limits — plus the component gallery, fixes and test hardening from the same stack (#4548, #4549, #4550, #4552, #4556 merged here). ## Plan enforcement ([TRI-12863](https://linear.app/triggerdotdev/issue/TRI-12863)) **Agent message quota.** The Free-plan allowance becomes a real server-side limit with a durable counter. New `agent_message_usage` table keyed `(organization_id, period)` — deliberately not joined to chats, so deleting a chat can't free quota within the period. Both send paths count one user message (wakes never count) and refuse at the cap with `403 message_quota_reached`, which the client renders as an upgrade panel, never a silent drop. The refusal code is a single shared constant on both sides. **Watch limits.** A watch whose window exceeds the plan's `agentWatchMaxHours`, or that would push the org past its `agentWatchers` count, is refused with `watch_limit_reached` (409 on the API, an upgrade hint on the card). Plan limits only tighten the existing code ceilings (`min(plan, 24h)`, per-chat cap of 3 still applies). A plan limit of zero means zero, not unlimited. Questions answerable instantly are answered before any plan refusal — a one-shot consumes no slot and never sees an upgrade nag. **Fails open by design.** Cloud ships the actual per-plan numbers separately (TRI-12863 P0). Until then absent limits resolve to the unlimited sentinel and the upgrade UI is gated on billing presence — self-hosted sees no cap, no upsell, with tests proving the fallback. Both quotas are nudges, not security boundaries: a failing limit read never blocks a send. ## Component gallery An admin-only gallery of every agent card state: five `storybook.agent-*` pages (chat UI, view blocks, report, investigation, watch) with their shared shell and manifest, demo fixtures, two demo-only cards, toast examples, and the screenshot script. No LLM and no data — every state renders from fixtures under `dashboard-agent/demo/`, never reachable from a production path. Designers and reviewers can look at every state, including the report states, without seeding anything. ## And fixes **SDK: watch-mode chat subscriptions survive quiet windows** (TRI-13065, TRI-13070) — watch mode keeps reconnecting across empty long-poll windows and only stops on abort or a settled session; a passive subscriber can no longer stop a turn it doesn't own (`stopOnAbort` is explicit, default off). Review findings fixed alongside: a superseded stream's async teardown no longer removes the live successor's abort controller or multi-tab claim, and stopping a generation hands the chat back to the user's other tabs. **Query boundary pinned end-to-end** ([TRI-11165](https://linear.app/triggerdotdev/issue/TRI-11165)) — a route-level test drives `api.v1.query` with a real signed environment JWT (writes refused before ClickHouse, a read passes); `readonly=1` made non-overridable; a per-turn cap stops the model burning a turn rewriting a query it can't fix (deterministic SQL errors only — busy/transport rejections don't count). **chat.agent durability regression suite** ([TRI-11166](https://linear.app/triggerdotdev/issue/TRI-11166)) — testcontainers-backed coverage of the two audit criticals (cross-tenant isolation, no duplicate mid-stream turn, both control-broken) plus crash-resume, cursor-based refresh, clean rollback of a mid-write turn failure (torn by a real constraint violation), and OOM-restart replay. **Investigation sweep backoff** — stale investigations get an attempt counter and backoff so a poison row can't pin the sweep queue head (migration `0005`: `sweep_attempts`, `last_sweep_attempt_at`). ## Screenshots <img width="1440" height="791" alt="Screenshot 2026-08-06 at 00 36 19" src="https://github.com/user-attachments/assets/6a68cd42-8580-469d-afe7-e28d1eef18e1" /> 🤖 Generated with [Claude Code](https://claude.com/claude-code) |
||
|
Chris Arderne
|
ed1bb72fb8 |
feat: implement cron window spread backend (#4566)
- New DB fields on Schedule and ScheduleInstance - Use `queueTimestamp` for the "effectiveAt" delayed start time, propagate it to Clickhouse TaskRun table - Disable fastpath for delayed jobs - Add schedule timing logic, API endpoints with windows, persistence - Calculate phase for every schedule, only persist when window is non-null - Additional o11y for phased rollout |
||
|
Matt Aitken
|
c2c6e5c705 |
fix(webapp): keep session runs off the legacy realtime streams backend (#4564)
## Summary Runs created for a Session were triggered without a realtime streams version, so they fell through to the `realtimeStreamsVersion` column default of `v1`. A Session's own `.in` / `.out` channels are always `v2`, so any run-scoped `streams.append()` or `streams.pipe()` call made inside a session run wrote to a different backend than the session it belongs to, and stayed there for the life of the run. The API trigger routes were never affected. They call `determineRealtimeStreamsVersion` with the client's `x-trigger-realtime-streams-version` header and always pass an explicit value, so a current SDK asking for v2 gets it. Only the internal callers that build trigger options by hand were leaning on the column default, which no env var can influence because that path never calls the resolver at all. ## The version resolver Fixing the call site exposed a second problem in `determineRealtimeStreamsVersion`. Its two paths disagreed: an explicit `v2` was checked against the S2 configuration first, but when the caller expressed no preference it returned `REALTIME_STREAMS_DEFAULT_VERSION` verbatim with no check. A deployment that set the default to `v2` without configuring S2 therefore stamped runs `v2`, nothing failed at trigger time, and every later read or write against those runs' streams threw `Realtime streams v2 is required for this run but S2 configuration is missing` for the life of the run. Both paths now resolve through one pure function that takes its configuration rather than reading `env`: ```ts const requested = streamVersion ?? config.defaultVersion; if (requested !== "v2") return "v1"; const hasCredentials = Boolean(config.accessToken) || config.skipAccessTokens; return hasCredentials && Boolean(config.basin) ? "v2" : "v1"; ``` ## The basin requirement `resolveStreamBasin` resolves run, session and organization basins ahead of the global setting, so a deployment that provisions a basin per organization can serve v2 with no global basin at all. Gating purely on the global setting would degrade every run there to `v1`. `determineRealtimeStreamsVersion` therefore takes an optional organization basin, and every caller that holds one passes it, including the session path: ```ts basin: organizationBasinName ?? env.REALTIME_STREAMS_S2_BASIN, ``` This is deliberately the resolved basin and not the `REALTIME_STREAMS_PER_ORG_BASINS_ENABLED` flag. The flag says the feature is on, not that a given organization has been provisioned, and provisioning happens out of band. Keying off the flag would stamp `v2` on runs for unprovisioned organizations, recreating the failure this removes. **This widens behaviour for explicit `v2` requests**, which previously required the global basin: a provisioned organization on a per-org deployment now resolves `v2` where it used to get `v1`. That is intentional, and it makes every path agree. ## Scope Only newly created runs change. A run already stamped `v1` keeps that version for its lifetime by design, since readers resolve the backend from the same column and its existing streams have to stay readable. Scheduled runs reach the same column default through `scheduleEngine.server.ts` and are deliberately left alone: that one is a policy question about `REALTIME_STREAMS_DEFAULT_VERSION` rather than an inconsistency inside a single feature. ## Verification A full-stack e2e boots the real webapp plus Postgres, Redis and s2-lite, creates a Session through the public API so the run comes from the real trigger path, appends records the way `streams.append()` does, and asserts three things at once: the version stamped on the run, that the payload is readable from S2, and that no key exists in Redis. It appends at a realistic record size so the route's body cap and S2's per-record cap are both exercised. Reverting the session-path change flips all three observations, so it fails against the old behaviour rather than passing vacuously. Unit tests cover the resolver matrix, including organization-basin-only and credential-only configurations; two of them fail against the previous resolver. Also verified by hand against a local stack: a real `chat.agent` session run writing 8 records of 250KB through `streams.append()` put 2,049,072 bytes into S2 with no Redis key, while the same agent with the session-path change removed put 2,102,360 bytes into Redis and nothing into S2. |
||
|
Chris Arderne
|
7b390e5984 | feat(cli,webapp): allow deploys with environment API keys (#4561) | ||
|
Katia Bulatova
|
0b750d00dd |
feat(webapp): dashboard agent — Watch (#4525)
Watch is the agent noticing something later: you ask it to tell you when a condition holds, and it answers when it does — or when it can't any more. A watch is a **durable one-shot promise**. The condition is checked on a schedule by deterministic code (no LLM in the checks), the answer lands in the chat once, and then the watch is over. Ten kinds: three on a run, five on a queue, error recurrence, health recovery. ## Stack Stacked on **#4529** (UI), which is stacked on **#4418** (chat, reports, investigate). Merge those first. **#4516** (storybook gallery) sits on top of this branch. ## How to review [**GUIDEBOOK.md**](https://github.com/triggerdotdev/trigger.dev/blob/feat/dashboard-agent-flows-watch/internal-packages/dashboard-agent/GUIDEBOOK.md) on this branch is the behaviour reference — it states the conditions rather than the code, so you can predict what happens without running anything. "The ten watch kinds, and what makes each fire" and "Creating a watch" describe exactly this PR, and the tables there are the spec the code is written against. ## What's inside - **Ten watch kinds**, one deterministic check each (`dashboardAgentWatch*Checks.ts`), with the spec union in `dashboard-agent-contracts/src/watch.ts`. - **Scheduling** — each watch schedules its own next check; due watches of one `(environment, cadence)` group can be checked together in one batch pass, with a sweep as the backstop for expiry, redelivery and retention. - **Delivery** — the in-chat wake and card, an optional email alert (new `DASHBOARD_AGENT_WATCH` alert channel, so it shows on the project's Alerts page with one-click unsubscribe), and an optional investigation when the outcome needs attention. - **Submission ledger** — `watch_submissions`, keyed `(chat_id, client_request_id)`, so a retried card submission replays the recorded outcome instead of creating a second watch. - **Watch token** — a delayed-execution credential accepted only by the watch endpoints, re-checked against the user's live access on every tick. - **Unread work** — the panel polls for wakes that landed while it was closed, so a chat can go unread and light the launcher dot. ## Key decisions **A check result is a 4-way, and only two of them are verdicts.** `satisfied` / `terminal_unsatisfied` are answers; `pending` and `unavailable` are not. Any exception inside any check is caught in one place and becomes `unavailable` with an unverified observation — a check that failed is never evidence. **A completed window is an answer, and whether it is good or bad news is declared per kind, never inferred.** There is a table for that in the guidebook: `run_failed` completing its window is *good* news ("hasn't failed"), `backlog_drain` completing it is not. One rule overrides the table: a window that completed on an unverified observation is neutral and says only that the watch ended without a confirmed answer. **An unreadable source is never a negative answer** — and, because investigations only open on `attention`, it never starts one either. **Identity is `(chat, project, environment)` plus the condition,** enforced by a partial unique index over active rows (`watches_chat_active_identity_key`), not by the read-then-insert check. Cadence, window, note and `ticks` are deliberately not part of it. Two different chats may watch the same thing — a watch is a promise to a chat. **The server resolves the target's name, whatever the model calls it.** The model can't tell a task queue (`task/<id>`) from a custom queue, so both spellings are tried and the stored one wins — and the rewrite happens **before** identity and before the row is written, so the identity, the checks, the link and the wording all see one spelling. **Freshness fences.** Depth falls back from the live counter to the newest 60 s ClickHouse bucket, which only counts as current within 60 s of now. A non-current reading at or below the *quiet line* is refused as `unavailable` rather than believed, so a stale empty bucket is never read as "drained". The stall streak is the one piece of carried state: it lives in the previous check's facts and *freezes* on an unreadable reading rather than breaking. **Chain reliability.** There is no shared cron — each watch (or batch group) schedules its own next tick, so the failure mode to review is the chain dying. A failed batch check is caught, the next tick is scheduled anyway and the run resolves rather than failing, so the chain survives a check that couldn't run; the sweep re-arms groups and finalizes anything still active past its deadline, even when delivery isn't configured. Wake redelivery is id-deduped rather than conditional, because the sweep can't know whether the user was already told. Access is re-authorized on **every** check against the primary — replica lag would extend access the user has already lost. **Wording lives in one place.** `watch-wording.ts` is read by the card, banner, toast, email and the agent's own narration, and the numbers come from the frozen observation rather than a fresh read, so a retry produces the same sentence. Replay reproduces the **recorded** decision instead of deciding again — the transcript is append-once, so a second decision would contradict it forever. **Cancellation is the ending without an answer** — no resolution, no wake. One exception, decided during testing: a watch the *user* cancelled leaves a single neutral transcript line ("Stopped watching …"), keyed off the watch id so a retry can't repeat it. The other four reasons stay silent. **Email is opt-in and only a fired watch emails.** An expiry is narrated in the chat and nowhere else. Both gates (agent access, a configured email transport) are checked at subscribe time *and* again at delivery, and the subscription outcome is frozen on the ledger row so a retry replays it. Neither gate is a plan check. **One watch offer per turn.** The prompt and the renderer guard this independently — if the turn already proposed a watch card, the action button is dropped, because the card is the better affordance. Two eval cases pin the prompt side: exactly one offer with the line last and the button after it, and zero offers when the rendered card already carries one — deterministic assertions, over a real-model run. ## Testing Unit tests (vitest, testcontainers, no mocks) under `apps/webapp/test/dashboardAgentWatch*.test.ts` and `internal-packages/dashboard-agent/src/watch-*.test.ts` cover the invariants above: the 4-way check results and the freshness fences, identity/dedup and the submission ledger, queue-name resolution, the batch chain surviving a failed check, sweep boundaries and alert-once, tenancy and the watch token's scope, and the wording snapshot. The load-bearing ones were verified by control-breaking the guard first and checking the test goes red. Live-tested end to end against a local stack, following the guidebook: all ten watch kinds firing and expiring, cancellation, the email pair (a fired watch mails, an expired one does not), and watch recovery from a health report. |
||
|
Katia Bulatova
|
9a3bee0288 |
feat(webapp): dashboard agent — UI (#4529)
Stacked on #4418. Merge that first. The UI slice of the dashboard agent: the side panel, the chat transport wiring, message and card rendering, suggested prompts, and chat history. #4418 works without this — the system is simply invisible. The diff is mostly components, so the notes below cover only the three decisions you can't read off the markup. Behavior and a hands-on walkthrough live in GUIDEBOOK.md, which lands with #4525. ## Decisions worth knowing - **Action rows always render at the end of a turn.** The model's emission order isn't trusted for layout, so action blocks are split out of the stream and appended last. Display only — `answered` stays keyed on the emission index. - **The last-chat memory is org-true.** It's keyed by the chat's own organization, and a foreign or deleted chat comes back as a 404 the client treats as gone, rather than an empty chat it keeps around. - **A dead stream self-heals from the settled transcript.** Terminal records are written to the chat row after the client's stream closes, so the panel re-reads it. The poll gate is any unfinished turn — a dangling tool part, not just an open investigation. ## Notes - Gated by `canAccessDashboardAgent`; no behavior change with the flag off. - Page marks: `handle.agentPageContext` on 47 routes, ~20 lines each. - Entry points: Ask Trigger button, ⌘J, Help & Feedback. The old ⌘I and `?aiHelp=` links keep working. ## Screenshots <img width="1440" height="788" alt="Screenshot 2026-08-07 at 15 14 29" src="https://github.com/user-attachments/assets/f4e89e8d-13ed-4be3-a88d-d5cca3ece0fa" /> |
||
|
Katia Bulatova
|
4569657923 |
feat(webapp): dashboard agent — chat, reports, investigate (#4418)
## What & why This is the system behind the Dashboard Agent — an assistant that answers questions about a project's runs, errors, queues, deploys and health, and can investigate failures end to end. The agent runs as a chat.agent task in its own Trigger project. It has no access to the main database or ClickHouse; all platform data is read through the public API using a delegated, read-only user token. Everything here is behind `canAccessDashboardAgent` and inert with the flag off. The UI that mounts the panel lands in #4529. ## Stack `#4418` (this, base) ← `#4529` UI ← `#4525` Watch ← `#4516` storybook gallery. The scenario/contract reference for the whole stack is `internal-packages/dashboard-agent/GUIDEBOOK.md` (it lands on the Watch branch): it states, per feature, what makes each thing happen and where that is decided. ## What's inside **Agent runtime and tools** — `internal-packages/dashboard-agent`: prompt, tool set (API reads, TRQL query, docs, navigation, evidence/investigations, repo source), conversation compaction, a prompt-prefix token budget pinned by snapshot test, and sampled LLM-judged turn evals. The package cannot import webapp server code, which is what makes the "no DB access" claim structural rather than a convention. **Contracts** — `internal-packages/dashboard-agent-contracts`: `trigger://` URIs, intents, and the block envelope every rendered card travels in. **Conversation store** — `internal-packages/dashboard-agent-db`: drizzle over postgres-js in its own `trigger_dashboard_agent` Postgres schema, plus one additive migration. **Auth boundary** — the user-actor token gains an optional environment claim; one guard (`userActorEnvironment.server.ts`) enforces it so routes don't each re-derive the rule. Token minting, cap ceiling, and the RBAC fallback path for self-hosted. **Transport** — webapp resource routes that mint the token and proxy each turn, and SDK-side mid-turn reconnect. **Public API the agent reads through** — orgs, projects, environments, runs, queue metrics, workers, a run's commit metadata, repo snapshot, reports, and `POST /api/v1/query`. **Reports** — the health report's layout is declared once and shared by the card, the markdown surface and the JSON/MCP surface, so the same report reads the same in the dashboard, the terminal and an editor. **Block renderers** — the report and investigation cards the flows above already emit (`app/components/dashboard-agent/`). The panel that hosts them, and the rest of the chat UI, is #4529. **Query safety and CSP** — see below. ## Key decisions - **The agent is a separate Trigger project, not webapp code.** It reads platform data over the public API with a delegated user-actor token whose `cap` ceilings it to read scopes. No Prisma, no ClickHouse, no webapp imports. - **The PAT-only auth helper now refuses user-actor tokens.** This is an intentional behavioral change: its callers consume only a bare userId and do not enforce delegated-token capabilities. Actor-aware routes continue through the scoped route builders instead. - **RBAC fallback builds a delegated token's ability from its own cap**, never the blanket ability a PAT gets (read-only when the token declares none). Without this, the agent's read-only cap would buy a write JWT on self-hosted. - **Org creation checks RBAC only for user-actor tokens, and only after the env gate**, so an install with `ORG_CREATION_API_ENABLED` off returns 404 rather than 403, and an ordinary PAT never consults an ability the route has no org to scope. Both orderings are pinned by test. - **The query path is read-only in depth.** TRQL rejects write statements at the grammar level (they don't parse, rather than being filtered), ClickHouse runs with `readonly=1`, and the org/project/env filters are injected server-side from the credential — the request body cannot widen scope. An unparseable query denies instead of falling through to the permissive resource. - **Document-wide img-src CSP.** Remote images are an outbound-request/exfiltration surface, so the policy permits only own-origin/data/blob, the required SSO avatar hosts, and the favicon endpoint. Operators can add exact origins through CSP_IMG_SRC_ALLOWLIST; wildcard hosts and bare schemes are intentionally not allowed. - **The chat transport reconnects on a mid-turn EOF** (`@trigger.dev/sdk`). A body that ends without a turn-complete is terminal only when the server says `X-Session-Settled: true`; otherwise the transport resubscribes from `lastEventId` with bounded backoff, and any record re-earns the budget. Previously a closed long-poll window or a proxy restart left the reply stuck as if still generating. - **Conversations live in their own datastore**, schema-scoped and foreign-key-free (it references `organizationId`/`userId` by id, because in cloud it is a different database). It is a display read-model for the History tab and transport resume; `chat.agent`'s object-store snapshot remains the model's source of truth. - **Deterministic first.** Reports and health checks contain no LLM — they are computed from the same data the dashboard shows, and the model only narrates and links them. That is what makes a number in an answer auditable. ## Testing - 63 new test files, run with `pnpm run test --filter webapp` and per-package vitest. Heaviest coverage on the auth boundary (`userActorPatOnlyBoundary`, `userActorTokenClaimsAndScopes`, `contextlessPatRoutes`, `rbacFallbackBranch`), TRQL read-only, the report layout, and the SDK reconnect. - The agent package has a separate eval lane (`pnpm run test:evals`, `vitest.eval.config.ts`) that hits the real model, so it never runs in `pnpm test`. - Live-tested against a local stack scenario by scenario; the GUIDEBOOK lists the condition each behaviour is expected under, which is what those runs were checked against. ## Changelog `.server-changes/dashboard-agent.md`, plus changesets for `@trigger.dev/core` (report schemas), `@trigger.dev/sdk` (chat reconnect) and the CLI's `mint-token` help text. |
||
|
Eric Allam
|
6449a644b9 |
feat(webapp,cli,database): track real dev onboarding progress (#4563)
## Summary
The dev environment "Get set up" panel used to be a static list of CLI
commands that only disappeared once your tasks registered, so nothing
ever changed after you ran `init` and people assumed it was stuck. It
now tracks real progress: `trigger init` records the project as
initialized, so step 1 checks off, and the panel updates live as the dev
server connects and your tasks register.
It also adds a prominent "Copy AI agent prompt" button, presented as a
clear alternative ("or") to the manual CLI steps, that copies a
ready-to-paste setup prompt pre-filled with your project reference for
Claude Code, Cursor, or any coding agent.
## Notes
- Adds a `Project.initializedAt` column (migration
`20260811065646_add_project_initialized_at`); the CLI `init` command
calls a new project-scoped `POST /api/v1/projects/:ref/init` best-effort
at the end of setup.
- The `init` scaffold now imports from `@trigger.dev/sdk` instead of the
deprecated `/v3` subpath.
## Screenshots
<img width="2400" height="1794" alt="v7-redesigned-card"
src="https://github.com/user-attachments/assets/c2fb4fa1-9484-4700-8bd3-110d66f5a44e"
/>
|
||
|
Eric Allam
|
ce368dd8e0 |
perf(database): index EnvironmentVariableValue.valueReferenceId so secret deletes stop seq-scanning (#4555)
## Why this change `EnvironmentVariableValue.valueReference` is an `onDelete: SetNull` foreign key. Deleting a `SecretReference` (the env var edit/delete path for secret values) fires the cascade `UPDATE ONLY "EnvironmentVariableValue" SET "valueReferenceId" = NULL WHERE $1 = "valueReferenceId"`. That cascade is scan-shaped: with no index on `valueReferenceId`, it reads the entire table to find the rows referencing the deleted secret. The parent `SecretReference` delete does almost no work itself; its latency is dominated by this cascade. ## Diagnosis `EnvironmentVariableValue` was indexed on `environmentId` and `(variableId, environmentId)`, but not on `valueReferenceId`. The SET NULL cascade therefore did a full sequential scan of the whole table. Two sibling SET NULL cascades on the same delete (`OrganizationIntegration.tokenReferenceId`, `User.mfaSecretReferenceId`) are index-backed and stay fast, which isolates the missing index as the cause. ## Change Add `@@index([valueReferenceId])` on `EnvironmentVariableValue`, created with `CREATE INDEX CONCURRENTLY IF NOT EXISTS` so `prisma migrate deploy` stays safe on a live table. ## Benchmark (local, seeded) Local Postgres seeded with 1,000,000 `EnvironmentVariableValue` rows, `EXPLAIN (ANALYZE, BUFFERS)` on the SET NULL cascade with zero matching rows (the worst case: reads the whole table, affects nothing): | | before | after | |---|---|---| | plan | Seq Scan (1M rows) | Bitmap Index Scan | | execution | 183 ms | 2.8 ms | In a variant where the secret matched several thousand rows, the parent `SecretReference` delete's `EnvironmentVariableValue_valueReferenceId_fkey` trigger dropped from 216 ms to 88 ms (the residual is the heap work of nulling those rows). ## Expected impact The cascade drops from a full-table sequential scan to a targeted index lookup. The win grows with the table, so the benefit is larger than the seeded numbers above. ## Risks - One extra btree to maintain on `EnvironmentVariableValue` writes; small, single-column, and it should be pre-created before the migration deploys (per the repo index rules). - No behavior change: same rows nulled, no ordering or result-set change, read paths untouched. Companion to the same fix on `ProjectAlert.channelId`. |
||
|
Eric Allam
|
4c58091973 |
perf(database): index ProjectAlert.channelId so alert-channel deletes stop seq-scanning (#4554)
## Why this change Deleting a `ProjectAlertChannel` fires the FK cascade `DELETE FROM ONLY "ProjectAlert" WHERE $1 = "channelId"`. That cascade is scan-shaped: with no index on `channelId`, it reads the entire `ProjectAlert` table to find the few child rows belonging to the deleted channel. The parent `DELETE ProjectAlertChannel` does almost no work itself; its latency is dominated by this cascade. `ProjectAlert` is append-heavy and grows over time, so the scan cost only increases. ## Diagnosis `ProjectAlert` had no index on `channelId` (only `pkey` + a `friendlyId` unique). The cascade therefore did a full sequential scan of the whole table. The sibling `ProjectAlertStorage` cascade on the same delete is index-backed and stays fast, which isolates the missing index as the cause. ## Change Add `@@index([channelId])` on `ProjectAlert`, created with `CREATE INDEX CONCURRENTLY IF NOT EXISTS` so `prisma migrate deploy` stays safe on a live table. ## Benchmark (local, seeded) Local Postgres seeded with 1,000,000 `ProjectAlert` rows across 50 channels (~20k rows per channel), `EXPLAIN (ANALYZE, BUFFERS)` on the cascade delete: | | before | after | |---|---|---| | plan | Seq Scan (1M rows) | Bitmap Index Scan | | direct child delete | 740 ms | 22 ms | | parent delete `ProjectAlert_channelId_fkey` trigger | 77.7 ms | 23.8 ms | ## Expected impact The cascade drops from a full-table sequential scan to a targeted index lookup. The win grows with the table: the more rows in `ProjectAlert`, the more a scan costs and the more the index saves, so the benefit is larger than the seeded numbers above. ## Risks - One extra btree to maintain on every `ProjectAlert` insert; acceptable for a single-column index on a high-insert table, and it should be pre-created before the migration deploys (per the repo index rules). - No behavior change: no rows orphaned, no ordering or result-set change, read paths untouched. ## Follow-up `ProjectAlert`'s other cascade FK columns (`projectId`, `environmentId`, `workerDeploymentId`) are also unindexed, but their parents are soft-deleted rather than physically removed, so those cascades do not currently fire. Lower priority unless a hard-delete path is introduced. |
||
|
Eric Allam
|
90e8bd5c12 |
feat(webapp,database): opt-in per-client Prisma driver adapters (#4539)
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 0s
🚀 Publish Trigger.dev Docker / units (push) Failing after 0s
🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker-v4 (push) Has been skipped
🚀 Publish Trigger.dev Docker / scan-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / scan-supervisor (push) Has been skipped
🦋 Changesets PR / Create Release PR (push) Has been cancelled
🚀 Publish Trigger.dev Docker / 📣 Dispatch main image (push) Has been cancelled
📚 Docs Checks / check-broken-links (push) Has been cancelled
🧭 Helm Chart Prerelease / lint-and-test (push) Has been cancelled
Workflow Checks / Actionlint (push) Has been cancelled
Workflow Checks / Zizmor (push) Has been cancelled
🧭 Helm Chart Prerelease / prerelease (push) Has been cancelled
## What
Adds an opt-in path to run each Prisma client through
**`@prisma/adapter-pg`** (the node-postgres driver) instead of the
built-in engine driver, controlled by a **per-client env var, all off by
default**:
| env var | client |
|---|---|
| `CONTROL_PLANE_DATABASE_WRITER_DRIVER_ADAPTER` | control-plane writer
|
| `CONTROL_PLANE_DATABASE_REPLICA_DRIVER_ADAPTER` | control-plane
replica |
| `RUN_OPS_DATABASE_WRITER_DRIVER_ADAPTER` | new run-ops writer |
| `RUN_OPS_DATABASE_REPLICA_DRIVER_ADAPTER` | new run-ops replica |
| `RUN_OPS_LEGACY_DATABASE_WRITER_DRIVER_ADAPTER` | legacy run-ops
writer |
| `RUN_OPS_LEGACY_DATABASE_REPLICA_DRIVER_ADAPTER` | legacy run-ops
replica |
With every flag unset the construction path is byte-identical to today
(`datasources` URL + Rust engine), so this is inert until a flag is
turned on. Per-client granularity allows enabling the adapter only where
it's wanted.
## How
- Enables the `driverAdapters` preview feature on both schemas
(`@trigger.dev/database` and `@internal/run-ops-database`). This keeps
the **Rust query engine** — it does NOT add `queryCompiler` — so query
behavior, result types, and engine tracing spans are unchanged.
- A shared `buildDriverAdapterPool` builds each client's `pg.Pool` with
an explicit `max`, a bounded `connectionTimeoutMillis` (the
node-postgres pool otherwise waits unbounded on acquire), and an
`onPoolError` handler (an unhandled idle-connection error would
otherwise crash the process). Threaded through all four client builders
via a `useDriverAdapter` flag.
- Adds `@prisma/adapter-pg` + `@types/pg` to the webapp; `pg` is already
pinned at `8.15.6` (adapter-pg 6.x requires `pg < 8.17`).
## Connect-failure handling (the important correctness/security bit)
Under the adapter an unreachable DB no longer surfaces as
`PrismaClientInitializationError` / `P1001`; it becomes a `P2010`
"Database not reachable: <host>" (or a raw
`ECONNREFUSED`/`ENOTFOUND`-class error). Two handlers are updated so a
client on the adapter behaves like today:
- **`isInfrastructureError`** now recognizes those shapes (P2010 with a
connectivity message, and raw connectivity errno codes). Without this,
the DB **hostname would leak into API-client-facing errors** and the
failure would go unlogged. Security-relevant.
- **`isPrismaRetriableError`** treats the adapter's pool-acquire timeout
("timeout exceeded when trying to connect") as retriable, preserving the
`P2024` retry behavior the adapter otherwise drops.
## Evidence
Validated on an isolated stack that mirrors the production DB topology
(chained PgBouncers in front of writer + reader):
- **Behavioral parity:** raw-query results and Prisma error codes/`meta`
are byte-identical between the engine driver and the adapter across the
queried shapes (unique-constraint `meta.target`, record-not-found,
transaction-timeout, serialization-failure, etc.).
- **Feature matrix:** a full 380-project queue-ay pass shows no
adapter-caused regressions — pass/fail parity between adapter-off and
adapter-on, with the residual failures being pre-existing
known-failures/flakes common to both.
## Rollout / rollback
All flags default off; enable per client via env var, roll back by
unsetting and redeploying (no data migration). Recommended first target
is a single writer; enable one client at a time.
## Follow-ups (not in this PR)
- `$metrics`-based pool observability is removed under the adapter (the
Prometheus route + `db.pool.connections.*` instruments); the metrics
replacement (via `pg.Pool` counters) lands in a separate PR.
- Note for operators: on the adapter path, interactive-transaction
`maxWait` does not bound pool acquisition — `connectionTimeoutMillis`
does.
## Note on connection-string parameters
The adapter pool is built from the base DSN, so Prisma-specific DSN
parameters that node-postgres does not understand are not honored when a
client is on the adapter:
- **Prisma TLS spellings** (`sslaccept`, `sslcert`, etc.) —
node-postgres uses `sslmode`/`ssl` instead. Our production DSNs do not
use these Prisma-specific TLS params, but any deployment whose DSN
relies on them must be checked before enabling a flag.
- `pgbouncer=true` and `statement_cache_size` — effectively moot under
the adapter, which uses no persistent named prepared statements.
`connection_limit`, `pool_timeout`, and `schema` are handled explicitly
(passed as `max`/`connectionTimeoutMillis` and PrismaPg's `{schema}`
option).
refs TRI-13039
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
Eric Allam
|
c526528d8f |
feat(webapp,database): bound Prisma list filter arity (#4480)
⚒️ Publish Worker (v4) / build (supervisor) (push) Has been cancelled
## Summary Prisma expands `in` / `notIn` into one bind parameter per element, so every distinct list length is a separate prepared statement. Where the length tracks data volume (a batch size, a run-graph fan-out, a prior query's id set) one call site can mint hundreds of them. Each is used about once, but inserting it evicts an entry that was being reused, so the cost lands on unrelated queries sharing the pooler's statement cache. An unbounded list also risks the 65535 bind-parameter ceiling. `boundedIn()` pads a filter list to the next power of two by repeating its last element. `IN` and `NOT IN` ignore duplicates, so results are unchanged, and a call site drops from one statement per length to at most `log2(cap)`. Applied to all existing sites. ## Enforcement Two oxlint rules require the helper: a list filter must be an inline array literal or a `boundedIn()` call. - The first covers filters reached through `where` / `having` / `cursor`, and deliberately never descends into `data`, `create`, `update`, `set` or `equals`. A key named `in` in those positions is user data, not a predicate, and rewriting it would corrupt what gets stored or compared. - The second covers bare filter objects passed to where-building helpers, which the first cannot see. It found five sites in the run-graph batch loaders that were otherwise invisible. Both rules follow filters through the shapes they are actually written in: conditional expressions, logical-and objects, spread-conditional properties, computed keys, and call arguments. An array literal only counts as fixed-arity when nothing spreads into it, since `[...new Set(ids)]` has a runtime length. Twelve sites were hidden behind those shapes until the rules handled them. Scoped to `in` and `notIn`. The scalar-list filters `hasSome` and `hasEvery` compile to `&& $1` and `@> $1`, passing the whole array as a single bind parameter, so their arity never reaches the statement text and there is nothing to bound. Both rules are `error`, so new call sites fail CI. That ratchet has already caught four sites added by other PRs while this one was in review. ## Notes `boundedIn` pads by repeating rather than with null: `x NOT IN (a, b, NULL)` is never true, so null-padding a `notIn` filter would silently return no rows. Lists above 32768 are returned unchanged so padding can never push a query past the parameter limit. Route modules reach the helper through `~/db.server` rather than importing the database barrel directly, since a value import of that barrel into a module that also exports a React component is only safe while dead-code elimination prunes it. Measured on a local rig: 300 distinct list lengths produce 300 prepared statements unpadded, 10 padded. Verified end-to-end against a local stack with the full task-suite sweep, which surfaced no regressions. |
||
|
Chris Arderne
|
0a44b88b39 | fix: security release 2026-07-21 (#4528) | ||
|
Eric Allam
|
db67a856fe |
perf(webapp,database): index the newest-task-version lookup (#4518)
📦 Preview packages (pkg.pr.new) / Build and publish previews (push) Has been cancelled
📚 Publish docs / publish (push) Has been cancelled
Implementing PlanetScale Insights improvement. ## Summary Validating a schedule (creating or updating one through the API or the dashboard, and deploying a project that declares schedules) looks up the newest version of a task by slug. That lookup reads *every* version of the task and sorts them to return one. A project gains a row per task on every deploy, so the work grows with the project's age: the oldest projects pay the most, and dev-mode redeploys make it worse. This was picked because it was the largest single consumer of database time on the schedules path, and the fix is a sort key with no index behind it. ## Fix `BackgroundWorkerTask` is indexed on `(projectId, slug)`, which serves the equality but not the `ORDER BY createdAt DESC`. Postgres seeks the index, then bitmap-scans and top-N sorts the whole group to produce a single row. Adding `createdAt` to the index lets it scan backward and stop at the first row. The same call site also selected all 21 columns, including five JSON blobs, to read one field (`triggerSource`), so it now selects that field alone. ## Benchmark Local Postgres 17, 997,000 seeded rows / 748 MB, group sizes chosen to match the distribution seen in production. | Group size | Before | After | | --- | --- | --- | | 15,000 versions of one task | 11.118 ms, 1,510 buffers, 15,000 rows scanned | 0.027 ms, 4 buffers, 1 row | | 2,000 versions of one task | 2.081 ms, 1,455 buffers, 2,000 rows scanned | 0.022 ms, 4 buffers, 1 row | ``` before: Limit -> Sort (top-N heapsort) -> Bitmap Heap Scan after: Limit -> Index Scan Backward using BackgroundWorkerTask_projectId_slug_createdAt_idx ``` An ascending index scanned backward is enough here, so no descending index is needed. ## Impact and risk Real-world gain lands between the two rows above and scales with how many deploys a project has accumulated. Projects with few deploys will see little change, since there is barely anything to sort. The new index costs noticeably more than the existing two-column one: 43 MB against 7.3 MB on the benchmark rig. Adding `createdAt` makes every key unique, which defeats btree deduplication, so this is a real disk and write cost rather than a rounding error. Writes to this table happen at deploy time, not on the run path, so the write amplification is acceptable. The existing `(projectId, slug)` index is now a redundant prefix and could be dropped, but this PR keeps it so index usage can be observed before removing it. Behavior is unchanged: same predicate, same ordering, same row returned. The narrowed select is the only code change, and the field it keeps is the only one the caller read. Deploy note: the migration is `20260806100000_add_background_worker_task_project_id_slug_created_at_index` and uses `CREATE INDEX CONCURRENTLY IF NOT EXISTS`, so it can be pre-applied by hand before the deploy. |
||
|
Matt Aitken
|
04f9c4e1a5 |
fix(webapp,run-engine,core): drop the hidden debounce ceiling, fail fast on an unusable maxDelay (#4521)
Debouncing with a `delay` longer than an hour did nothing at all. The engine applied a server-side ceiling on how long a debounced run could be pushed back, measured from the run's `createdAt` and defaulting to one hour. A run is only pushed back while its new execution time stays inside that ceiling, so a `delay` at or above it could never push anything: the waiting run was released, the trigger started its own run, and the next trigger repeated it. A `delay: "12h"` produced one run per trigger, each correctly delayed by 12h, with no error raised and nothing on the run to show the debounce key had been ignored. The ceiling is now unset by default. A debounce key with no `maxDelay` keeps collapsing triggers for as long as they keep arriving, which is what the docs have always described. Self-hosters who want a bound can still set `RUN_ENGINE_MAXIMUM_DEBOUNCE_DURATION_MS`. That has a consequence worth stating plainly, so the docs now carry a warning for it: with no `maxDelay`, a continuously triggered key never executes. Set `maxDelay` when the work has to happen eventually. **Failing fast on an unusable `maxDelay`.** A caller who sets `maxDelay` no longer than their `delay` hits exactly the dead end described above, so that pair is now rejected at trigger time instead of silently behaving as if no debounce were set: ``` debounce.maxDelay (1h) must be longer than debounce.delay (12h). A debounced run is only pushed back while it stays inside maxDelay, so with these values every trigger would create its own run. ``` An unparseable `maxDelay` is rejected too, rather than quietly falling back to no bound at all, and so is a `delay` given as a date rather than a duration, which could never work because the value is re-applied on every push. The same check runs against a configured server ceiling, so a self-hosted deployment that sets `RUN_ENGINE_MAXIMUM_DEBOUNCE_DURATION_MS` gets the error rather than the silent failure this PR is about. With no `maxDelay` and no configured ceiling, which is the default, there is nothing to conflict with and nothing is rejected. The docs, the `TriggerOptions` JSDoc and the engine option all now state that the room available to push is the gap between `delay` and `maxDelay`. The run engine suite gains the case that motivated this: four triggers on one key with a 12h delay now collapse to a single run. |
||
|
Wes Mason
|
66940c0384 |
fix(observability-map): narrow the required check and the report bot's comment lookup (#4507)
## Findings addressed - **Report bot edited the wrong comment.** The comment-lookup step matched on the marker body text with no author predicate, so it would silently PATCH a human's comment that happened to quote the marker (GitHub gates comment editing on write access, not authorship, so it never 403'd). Now constrained to `.user.login == "github-actions[bot]"`, the same identity `helm-prerelease.yml` already pins. - **A required check asserted facts about the whole webapp namespace.** `webappSymbols.test.ts` asserted that nobody anywhere in `apps/webapp` (walking locals, params, object keys) declares names like `createJWT`/`updateEnvVars`, so an unrelated PR naming a local variable failed a required check with a message pointing at nothing. Those negative self-tests move onto a package-owned fixture tree; the positive resolution assertions stay required (their absence rotted the tool before) but now name the list to edit. - **The suite ran twice on shared paths.** `obsmap` and `internal` path filters shared four generic paths (`package.json`, both lockfiles, `pr_checks.yml`), so any lockfile bump ran the observability-map suite in both jobs. Dropped from `obsmap` (where `internal` already covers them). The test that should have caught it only checked the package's own source path; it now asserts the two filters' path intersection is empty. - **PR-comment footer** reworded: it said the report gates nothing, which is true of the report but misled now that the tool's test suite does gate webapp PRs. Names both failure directions and where to read the rules. - **Nightly corpus** comment corrected (stale entry count; the failure-notification gap is documented, not silently implied). ## Review Two adversarial reviewers ran over the diff; both findings were verified and fixed: a hollow fixture assertion (a shared name satisfied either walker branch — now one name per declaration form, revert-confirmed) and a filter-intersection test that could be fooled by apostrophes in comment prose (now strips comment lines first). Full package suite green (877 passed), typecheck and format clean. |
||
|
Eric Allam
|
b20806247f |
fix(run-store): stop run-create failing on a brief write stall (#4514)
## Summary On the run-ops store, creating a run could intermittently fail with a "Transaction already closed" error, and the run would never be created. Single-write run creates no longer run inside an interactive transaction, so a brief database write stall can't blow the transaction budget and drop the run. ## Fix The dedicated run-ops `createRun` / `createFailedRun` wrapped a single nested `taskRun.create` in an interactive `$transaction`. Its default 5s budget is wall-clock from `BEGIN`, so when a write briefly stalls the transaction expires before the create completes and throws, even though the statement itself is fast at the database. A single-write create does not need an interactive transaction: Prisma's implicit nested create is already atomic and holds no app-side budget, so it now runs directly. Only the `triggerAndWait` path (run plus its associated waitpoint, two writes that must commit together) keeps an interactive transaction, now with headroom over the default. Verified with a red/green test against the real split topology (reproduces the exact expiry on the unchanged code, green after) and an end-to-end run created and completed through the dedicated store. |
||
|
Chris Arderne
|
85f5b37c68 |
chore: upgrade to TypeScript 7 (#4318)
## Summary Upgrade the monorepo to TypeScript 7.0.2 and update package build tooling for compatibility with the native compiler. ## Design Package builds now use `tshy` 4, while the packages still using `tsup` move to `tsdown`. The few scripts that depend on the legacy TypeScript compiler API use an explicit TypeScript 6 alias; declaration portability coverage invokes the TypeScript 7 CLI directly. Turbo is updated so workspace tasks can read the regenerated pnpm lockfile. --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Wes Mason
|
ca9a74e84a |
feat(observability-map): static observability scorer for webapp route entry points (#4455)
A static observability scorer for the webapp's route entry points,
Lighthouse-style. The idea comes from evlog's `map` command, but that
tool has no Remix adapter and checks for its own logging API, so the
idea is ported rather than the tool.
It scans all 427 loader/action entry points in `apps/webapp/app/routes`
with the TypeScript compiler API and scores each against five checks:
error-classification, auth-boundary, auth-scope, request-context and
audit-trail. Current output on the real tree is **19/100** over 412
measured entry points.
```
cd internal-packages/observability-map
pnpm exec tsx src/cli.ts # terminal report
pnpm exec tsx src/cli.ts --json # machine output
pnpm exec tsx src/cli.ts api/v1/token # one entry, per-check detail
```
The two findings at the top of the fix list are real: `/auth/sso` and
`/api/v1/authorization-code` mint or exchange credentials
unauthenticated, and `/_app/orgs/:organizationSlug/settings/team`
resolves its org from a URL slug and gates each mutating branch on an
RBAC check alone, which per `apps/webapp/CLAUDE.md` is not the tenant
floor on self-hosted.
Decisions worth knowing, all with the reasoning in the README:
- The score started at 83 during development and fell to 19. Every drop
was a perverse incentive being removed, not a regression: routes were
being paid for having no error handling, two checks were reading the
same fact, suppressing a failure raised the score, and a no-op `catch
(e) { throw e }` was worth 50 points a route.
- **A mutation corpus is the tool's main defence.** 44 entries apply
semantics-preserving edits to a copy of the real route tree and assert
the score cannot rise, per route as well as globally, because a mean can
hide one route going up by taking another down. One entry runs as a live
expected failure: `try { String(0); }` with a deciding catch is a known
open hole worth 19 to 44, and it is disclosed rather than quietly
excluded.
- `audit-trail` and `request-context` are reported as headline figures
rather than one finding repeated hundreds of times. Both still count in
full where they should.
- A cohort change moves the number without anything in the codebase
getting better. Widening the sensitive cohort from 26 to 67 took the
global from 15 to 19 with no webapp change at all, so the report prints
per-check applicability and what the global would be without each one.
CI: a report-only job posts a sticky comment when a PR moves the report,
and says nothing when it does not. The package's own tests gate through
`pr_checks.yml`. The diff-scoped merge gate is still deferred until the
report has been used in anger.
524 tests plus the corpus. No runtime or dependency changes to anything
that ships.
<!-- GitButler Footer Boundary Top -->
---
This is **part 1 of 4 in a stack** made with GitButler:
- <kbd> 4 </kbd> #4485
- <kbd> 3 </kbd> #4484
- <kbd> 2 </kbd> #4483
- <kbd> 1 </kbd> #4455 👈
<!-- GitButler Footer Boundary Bottom -->
|
||
|
Katia Bulatova
|
fbd6df33b4 |
feat(webapp): Themes + contrast settings update (#4206)
Adds System Preferences, Dark and Light themes, gated by the `hasThemeSwitcher` feature flag (off by default — dark stays the default theme for everyone). Old theme is now "Classic"and set as default. "System preferences" theme has both Light and Dark modes and uses your laptop settings to use a correct one. It has less color accents (specifically less colored text), and they are the same for both modes, only grayscale values change between them. And Light/Dark themes can be used separately. New Contrast setting is available for System Preferences, Dark and Light themes - it changes the contrast for the whole app. All new visual Settings live in Account. |
||
|
Eric Allam
|
9d57aff542 |
fix(webapp): make the Queues hero charts environment-wide (#4486)
## Summary The four charts above the queues table aggregated over **at most the 25 queues on the current page**. They reused the loader's already-paginated queue array as a ClickHouse `queue IN (...)` filter, so paging or re-sorting changed the values, and a name search matching nothing blanked the whole chart row. The stat tiles above them were already environment-wide, so the two rows disagreed. They now read `env_metrics`, the environment-level rollup that already exists for exactly this (the built-in Queues dashboard and the health report read it). That is both correct and queue-count-independent: no `GROUP BY queue` across an entire environment, and no client-side summing. Note this is not only a paging artifact: page 1 under-reported too. On the seeded environment below, page 1 read 82% saturation against a true 87%, because the environment's running total is not the sum of one page of per-queue gauges. Three related fixes ride along. **Scheduling delay and throttling sawed to zero.** Both are event-driven, so at the 10-second bucket a short range picks, most buckets hold no samples at all and were drawn as `0ms`. Measured over a 1-hour window: **232 of 349 buckets had no scheduling-delay samples**. A bucket where nothing started is not a bucket where nothing waited, so the line was both ugly and wrong. TRQL grows a `minBucketSeconds` floor, plumbed through the metric resource route, and the hero tiles set 60s. Buckets that still have no samples render as a gap instead of a dive to zero. **The floor must not feed a width-dependent headline.** Two of the four headlines are not peaks, so widening the plotted buckets moved them: - **Throttled** is a share of buckets that saw any throttling, so a single brief throttle came to mark a whole minute instead of ten seconds: the same seeded events read 17% at 10s and 85% at 60s. - **Scheduling delay p95** is a percentile, and merging quantile states over a wider bucket yields a p95 between the sub-buckets' own. Two 240s samples among twenty in one 10-second sub-bucket give a worst-of-six p95 of 240,000ms against a merged 60-second p95 of 5,000ms — a 48x understatement of a headline whose tooltip claims it is the worst in the window. Both charts keep the floor, since a readable line was the point of it. Their headlines now come from a second query at the range's natural bucket width, via an optional `readout` on the tile, so each means what its tooltip says regardless of how the plotted buckets are sized. Saturation and backlog are genuinely width-invariant (a max of maxes is the same at any width), so they are unchanged and issue no extra query. Both caught by Devin in review; I had wrongly lumped p95 in with the peaks. **Charts reported a hydration mismatch on every render.** Recharts resolved victory-vendor's CJS entry on the server and its ESM entry in the browser. Those bundle different d3-shape builds, and the CJS one predates d3-path's digit rounding, so every server-rendered curve carried full-precision coordinates while the client rounded to 3 decimals: ``` Server: M0,3C0.9305555555555555,3,1.8611111111111112,3,... Client: M0,3C0.931,3,1.861,3,... ``` Bundling recharts for SSR makes both sides resolve the same ESM build. Verified: 45 of 45 server-rendered chart curves now match the client, and the page loads with an empty console. ## Verification An isolated stack with 40 seeded queues (20 heavily loaded, 20 idle) and 90 minutes of 10-second buckets written into `queue_metrics_raw_v1`, so the real materialized views built `queue_metrics_v1`, `env_metrics_v1` and the 5m rollup. Ground truth for the environment: 260 running against a limit of 300 (**87% saturation**), 800 queued. | | before | after | | -- | -- | -- | | Saturation, page 1 | 82% peak | **87% peak** | | Saturation, page 2 | 5% peak | **87% peak** | | Backlog / delay, page 2 | "No activity" | **800 peak / 59.5s** | | Name search matching nothing | all four charts blank | charts stay environment-wide | | Metric refetches on a page change | 4, each painting a skeleton | **0, no skeleton** | | Buckets drawn as 0ms with no samples | 232 of 349 | **0** | | Throttled readout | 17% | **17%**, unchanged by the wider buckets | | Worst-p95 readout source | plotted buckets | **natural width**, so a sub-minute spike is not averaged away | | Crosshair reach, hovering one detail-page chart | 2 of 4 others | **4 of 4** | | SSR chart curves mismatching the client | 45 | **0** | The bucket floor was measured across ranges: it widens 10s to 60s at 30m and 1h, and is correctly a no-op at 12h (300s) and 7d (3600s). One extra request per page load, for the throttled readout. The built-in Queues dashboard, which reads `env_metrics` independently, agrees at 86.7% and 260 of 300. `internal-packages/tsql` suite green (612 tests), including 5 new ones for the floor that fail without it. Webapp typecheck, oxfmt and oxlint clean. Spot-checked the Run metrics dashboard and the per-queue detail page for SSR regressions from bundling recharts: both render, console clean. The queue detail page carries the same event-driven series, so its scheduling delay, throttling and per-key mean delay take the same treatment. ## Screenshots <img width="2540" height="580" alt="after-page1-charts" src="https://github.com/user-attachments/assets/6cd23f9c-e7fd-4918-bcfa-b1d3340b16d1" /> ## Rollout Already behind the per-organization `queueMetricsUiEnabled` flag, so only gated orgs see any of it. Blast radius is chart values on one page plus the SSR bundling of recharts; rollback is a revert with no data migration. ## Stated limitations - `wait_ms_count` and the quantile state both only count `wait_ms > 0`, so "nothing started in this bucket" and "everything started instantly" are indistinguishable in storage. Both render as a gap. Distinguishing them needs a schema change, which is not in this PR. - The queue name search deliberately no longer narrows the charts. It only did so incidentally and incorrectly before (first 25 matches, and blanked on zero matches). Search-scoped charts would need the full unpaginated matching set and a server-side aggregate; worth its own ticket if we want it. - Bundling recharts for SSR grows the server bundle slightly. That is the cost of both sides resolving one d3-shape build. - The plotted delay line is a smoothed 60-second view, so a sub-minute spike above the one-minute warning threshold can fail to colour the line even though the headline reports it and colours itself. - Every chart inside one synced group shares the floor, because the hover crosshair is a reference line on a category x-axis and only draws where the hovered bucket exists in the other chart's own data. That costs the queue detail page's gauges some resolution (1 minute instead of 10 seconds) in exchange for the crosshair working across the row. Separately, while taking the screenshots I found a pre-existing rendering bug unrelated to this change: a **perfectly flat** saturation series draws no line at all (the readout still shows the right percentage), which looks like the threshold gradient's offset degenerating when the series min equals its max. It reproduces on `main`, so it is not a regression here and I have left it alone; filed as its own issue. Refs TRI-12784 |
||
|
Chris Arderne
|
763b5dc582 |
feat(webapp): enforce scopes for environment API keys (#4389)
## Summary Environment API keys backed by the additional-key table can authenticate API requests using their stored effective scopes. Revoked and expired keys are rejected, branch environments retain their existing routing behavior, and last-used timestamps are updated on a throttled best-effort basis. ## Design API route builders receive the resolved ability and reject restricted keys on routes without an authorization declaration. Existing deployment, environment variable, queue, run, task, batch, session, and waitpoint routes declare the resources they access. Trigger and batch responses return server-signed public access tokens, so additional keys never need access to the environment signing secret. Root-key rotation also keeps public tokens valid for the existing grace window. ## Feature notes - Root environment keys remain unrestricted for backward compatibility. Additional keys enforce their persisted scopes and fail closed on routes without an authorization declaration. - Machine-key requests never exchange one credential for another. Additional keys cannot retrieve the root key, and rotated root keys are not upgraded during their grace window. - Public JWT validation remains host-owned, while installed RBAC plugins continue to supply root-key abilities. - Unfiltered session and run listings preserve existing broad task-read behavior. Filtered requests enforce the supplied task identifiers. - Related-run summaries remain embedded in run retrieval for API compatibility. Retrieving or mutating a related run independently still requires permission for that run. - Queue management authorizes at collection scope, matching the queue permissions currently issued. - Batch responses deliberately include server-signed public access tokens for all clients. Selected-task credentials continue using their original credential for per-item authorization. - Two-phase batches authorize declared task identifiers before creation and authorize every streamed item. Streaming paths that cannot declare the complete task set remain fail closed. - Authentication telemetry records successful credential resolution separately from subsequent resource-authorization failures. - API keys are high-entropy random tokens. SHA-256 is intentionally used for deterministic indexed lookup, not password hashing. ## Deployment notes The schema migration must be present before this code is deployed. Because bearer resolution runs on every authenticated request, deploy the resolver with additional-key lookup disabled, verify root-key and public-token parity, then enable lookup before any additional keys can be issued. The multi-task authorization tightening changes the result for narrowly scoped tokens that request tasks outside their grants. Observe would-deny results before enforcing that check. Request-idempotency keys are also newly isolated by environment and task, so a retry crossing the deployment boundary may execute once more before old cache entries expire. ## Follow-ups - [x] Add a system-wide kill switch for additional-key lookup, defaulted off for the initial deployment. - [x] Add authentication observability by credential kind, result, latency, and lookup path without recording credential values. - [ ] ~Add would-deny observability and an independent enforcement switch for multi-task authorization.~ - [ ] ~Add an independent switch for server-issued batch tokens while root-key parity is verified.~ - [ ] Confirm every API route reachable by a restricted key has an explicit authorization declaration or intentionally fails closed. - [x] Verify root-key rotation, revoked-key grace, and public-token validation through each bearer resolver path. |
||
|
Eric Allam
|
db6228dd1e | chore(webapp,core,sdk): upgrade @s2-dev/streamstore to 0.25 and migrate S2 hosts (#4349) | ||
|
Eric Allam
|
f9c8d518c7 | perf(webapp,run-engine,database): resolve the newest worker and deployment by createdAt (#4452) | ||
|
Eric Allam
|
0445b8ec27 |
fix(webapp,clickhouse): keep the rest of a ClickHouse batch when one run or span has un-ingestable JSON (#4358)
## Summary A single run output, trace span, or payload carrying JSON that ClickHouse can't ingest (for example nesting past its depth limit) used to fail the whole insert batch, so unrelated runs and spans silently disappeared from the runs list, traces, and logs. This keeps the rest of the batch and handles the offending row instead of dropping everything around it. ## Fix Recovery is per-table, matched to what each table needs: - **Runs** (`task_runs_v2`) keep their status. We follow ClickHouse's failing-row hint to strip just the un-ingestable JSON column(s) so the run still lands (its output reads from Postgres on the detail page), up to a configurable limit (`RUN_REPLICATION_MAX_POISON_STRIPS_PER_BATCH`, default `1`). Past the limit we stop and land the batch with `allow_errors` in a single pass, skipping the remainder. Cost stays a fixed handful of inserts no matter how large or poisoned a flush is. - **Trace events and payloads** (high volume, append-only) recover with a single `allow_errors` insert: the good rows land in one pass and only the un-ingestable rows are skipped. Before falling back, a lightweight sanitizer still repairs what it can losslessly (lone UTF-16 surrogates, out-of-range integers) so a repairable row lands in full. To read the failing-row hint we patch `@clickhouse/client-common`: its error parser truncates the server response and discards the `(at row N)` position, so the patch preserves the full text for the recovery path to read. |
||
|
Eric Allam
|
f10bc23785 |
perf(run-engine,run-store): one execution snapshot per triggered run (#4419)
A non-delayed run used to get two execution snapshots the moment it was triggered: `RUN_CREATED` nested in the run-create transaction, immediately followed by `QUEUED` from its own `BEGIN`/`INSERT`/`COMMIT`. It now gets a single `QUEUED` snapshot written inside the create, and the trigger path only publishes to the queue. One fewer row per run on `TaskRunExecutionSnapshot`, and one fewer round trip on the trigger hot path. `EnqueueSystem` gains a `publishRun` seam that enqueues without writing a snapshot. Every re-enqueue path (waitpoint resume, checkpoint restore, delayed enqueue, pending version, retry requeue) still calls `enqueueRun` and writes its own `QUEUED`, so only the first enqueue changes. The `QUEUED` snapshot still commits before the queue message, so a dequeue sees a dequeueable status exactly as before. Two things for reviewers. Nesting the write skips `createExecutionSnapshot`, which is what emits `executionSnapshotCreated` and therefore the run timeline's `[engine] QUEUED` entry, so the trigger path now emits it directly, the same way the dequeue and attempt-start paths already do for their nested creates. And `RUN_CREATED` is still written when a dequeued run has no background worker yet, so the status and both `statuses.ts` helpers stay live and existing rows keep reading correctly. Delayed runs are untouched: `DELAYED` then `QUEUED` are two genuinely different moments and stay two snapshots. Rollback is a revert. Create-and-enqueue happen in one request in one process, so no in-flight run needs both code paths to agree during a rollout. One note for whoever debugs this path later. The `QUEUED` snapshot now commits before the queue publish, so a failed publish leaves the run recorded as `QUEUED` with no queue message. That state was already reachable, since the publish was never part of the snapshot transaction, but it used to be recorded as `RUN_CREATED`, which was distinctive because it never otherwise persisted. `QUEUED` with no message is indistinguishable from a run waiting on a concurrency slot, so trigger-time publish failure is now one more cause of an apparently stuck queued run. |