main
515 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Daniel Sutton
|
092b9ef07a |
fix(run-ops): DNS-safe, sortable base32hex run id (replace base62 KSUID) (#4154)
## Problem
The run-ops split mints NEW-store run ids as **27-char base62 KSUIDs**.
The supervisor writes the run id into the Kubernetes pod name
(`runner-<id>`), and pod names must be DNS-1123 labels (lowercase
`[a-z0-9-]`) — so uppercase base62 ids make k8s reject the pod (422) and
**those runs never launch** (they loop in `PENDING_EXECUTING` until the
heartbeat-stall handler nacks them, forever). `.toLowerCase()` can't fix
it: base62 has both `A`(10) and `a`(36) as distinct symbols, so folding
collides distinct ids and destroys sort order.
## Fix: change the encoding, not the structure
Mint a **26-char lowercase base32hex** run id:
```
run_<24-char base32hex core><region char><version char>
[ 6-byte ms timestamp ][ 9 CSPRNG bytes ]
```
- **base32hex** (RFC 4648 §7, alphabet `0-9a-v`): lowercase,
order-preserving, DNS-safe; 15 bytes → exactly 24 chars, no padding.
Hand-rolled encode/decode (no new dependency).
- **48-bit ms timestamp** in the leading bytes → plain string sort ==
creation order at millisecond resolution.
- **72 bits CSPRNG** entropy; PK unique constraint is the backstop (no
retry loop).
- **region / version** are raw positional chars (read via one `charAt`
before decoding/routing), version = `"1"`.
DNS-safe from birth and hyphen-free, so **firekeeper is unchanged** —
`runner-<id>-attempt-N` → strip `runner-`, cut at first hyphen still
recovers the exact id incl. region+version.
## Residency discriminator: length → version char
`classifyKind`/`classifyResidency` (`runOpsResidency.ts`) previously
distinguished NEW vs LEGACY by **id length**. That gets ambiguous with a
third format. It now discriminates on the **version char at a fixed
position** (`isRunOpsIdBody`: 26 chars, `[25] === "1"`, base32hex
alphabet) → NEW; everything else → LEGACY. Total, never throws. The
`Residency` (NEW/LEGACY) contract the routing store consumes is
unchanged; the `"ksuid"` `ResidencyKind` label is retained only because
it's the persisted `runOpsMintKsuid` feature-flag value.
## Scope / verification
- Generator + discriminator in `@trigger.dev/core` isomorphic; mint path
+ all id-shape call sites swept (~40 webapp files); changeset added
(`@trigger.dev/core` patch).
- Core unit tests (encode/decode round-trip + property, generator shape,
ms sort-order incl. intra-second, parse partitioned-vs-legacy,
firekeeper round-trip): **24 pass**. `@trigger.dev/core` builds; webapp
typechecks; format/lint clean.
## Open decisions (flagged, not silently chosen)
1. **Backward-compat**: existing 27-char base62 KSUID runs now classify
LEGACY. On test cloud these are the broken/looping runs that never
completed, so this is acceptable — but worth a conscious call before
prod. No transitional length-recognition added (keeps the discriminator
clean).
2. **Storage collation**: the sort guarantee is byte-order — if the
run-ops id column is `TEXT` with default locale collation it's silently
not honored. Confirm whether `COLLATE "C"` / `BYTEA` is needed on the
run-ops schema.
3. **Region sourcing** wiring — see `regionCharForRegion` /
`REGION_CODES`.
---
## ⚠️ Required migration — deploy in lockstep
This PR renames a persisted feature-flag key/value and an env var. These
are **not** changed by the code alone and must be migrated when this
deploys, or affected orgs silently fall back to `cuid` minting (no crash
— `defaultValue: "cuid"`):
1. **Env var** (terraform): `RUN_OPS_MINT_KSUID_ENABLED` →
`RUN_OPS_MINT_ENABLED` (carry the value over).
2. **DB** `organization.featureFlags`: migrate both the key and value
together:
- key `runOpsMintKsuid` → `runOpsMintKind`
- value `"ksuid"` → `"runOpsId"`
Until an org's flag row is migrated, its `runOpsMintKind` lookup misses
and it mints `cuid` (legacy) — so no NEW-store ids for that org until
the data lands.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
Daniel Sutton
|
d977691219 | fix(run-store): route caller-passed read clients to the owning store's primary (#4153) | ||
|
Daniel Sutton
|
962bc48738 |
feat(run-ops): automatically migrate the dedicated run-ops database (#4150)
## What
Adds the ability to **automatically migrate the dedicated run-ops
database** (the NEW DB in the run-ops split), matching how every other
database in the system is migrated. Follow-up to the run-ops split
activation.
## Changes
- **Migrate runner** — new
`internal-packages/run-ops-database/scripts/migrate.mjs`, exposed as
`db:migrate:deploy` / `db:migrate:status`. Connects via
`RUN_OPS_DATABASE_URL` (the same var the app uses) and expands `${VAR}`
refs like Prisma's dotenv.
- **Self-host** — `docker/scripts/entrypoint.sh` runs the run-ops
migration on boot when the DB is configured, gated by
`SKIP_RUN_OPS_MIGRATIONS`. Single-DB installs never set the URL, so it's
a clean no-op.
- **Single env-var family** — the run-ops DB is now addressed by one
canonical `RUN_OPS_*` family, connect path and migrations resolving the
identical URL:
- `RUN_OPS_DATABASE_URL` (writer) — replaces `TASK_RUN_DATABASE_URL`
- `RUN_OPS_LEGACY_DATABASE_URL` — replaces
`TASK_RUN_LEGACY_DATABASE_URL`
- `RUN_OPS_DATABASE_READ_REPLICA_URL` — replaces
`TASK_RUN_DATABASE_READ_REPLICA_URL`
- the old `TASK_RUN_*` aliases, the `??` coalesce, the
`runOpsNewDatabaseUrl` indirection, and the migrate-only `directUrl` are
all removed (consumers read `env.RUN_OPS_DATABASE_URL` directly).
`directUrl` was dropped because it was only ever used by `prisma
migrate` (never the app runtime) to bypass a pooler for advisory locks —
premature here since the run-ops connection isn't wired to the app yet.
If a pooler is later introduced for the app, a direct URL can be
reintroduced then.
## Safety
- **Pure rename** — nothing deployed sets any `TASK_RUN_*` var (the
split isn't activated anywhere yet; `.env.example`, docker-compose, and
cloud already use `RUN_OPS_*`), so there is no config migration.
- **Single-DB / self-host** — no new required env var; entrypoint and
migrate are no-ops when `RUN_OPS_DATABASE_URL` is unset.
- **Cloud** — runs migrations as pre-deploy ECS tasks (companion cloud
PR), calling these same `db:migrate:deploy` / `db:migrate:status`
commands.
## Verification
- Live migration against a fresh scratch DB with only
`RUN_OPS_DATABASE_URL` set: both migrations applied, no `P1012`/`P1013`;
`${VAR}` expansion, idempotent re-run, `status`, and no-op skip all
pass.
- Schema parity 4/4; `typecheck --filter webapp` 18/18; affected
split/replication tests 34/34.
## Scope
This delivers automatic migrations only. Enabling the app to *use* the
new DB (setting `RUN_OPS_DATABASE_URL` + `RUN_OPS_SPLIT_ENABLED` on the
service) is a separate activation step.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
Daniel Sutton
|
c9f427e21f |
fix(replication): key logical replication leader lock on slot name (#4151)
## Problem
`LogicalReplicationClient` uses a Redlock leader lock to guarantee a
single active consumer per Postgres logical replication slot. The lock
resource was keyed on the client `name`:
```
logical-replication-client:${this.options.name}
```
A slot permits exactly one consumer, so the lock's job is to serialize
consumers **of a given slot**. Keying it on `name` breaks that whenever
two clients target the same slot with different names — most notably
across a rolling deploy where the client `name` changes but `slotName`
does not. Both acquire *distinct* locks, both consider themselves
leader, and the second to reach `START_REPLICATION` hits `replication
slot "<slot>" is active for PID <n>`. Because that query was
fire-and-forget and its failure was only logged (no retry), the consumer
stopped and replication stalled until the process was restarted.
## Fix
**1. Key the leader lock on `slotName`** — the actual single-consumer
resource:
```
logical-replication-client:${this.options.slotName}
```
Consumers of the same slot now contend on the same lock and hand off
cleanly across restarts/deploys; different slots stay independent.
`name` is kept for logging and the pg `application_name`.
**2. Self-healing resubscribe** (`resubscribeOnFailure`, opt-in) —
instead of logging-and-dying, a client re-subscribes with exponential
backoff after a lost election or a failed `START_REPLICATION`, so a
rolling deploy self-heals: the incoming pod retries until the draining
pod releases the slot, then takes over. Safety:
- `#cleanupAttempt()` unconditionally ends the pg client (freeing the
walsender) and releases the leader lock before rescheduling — retries
never leak connections/locks.
- `shutdown()` sets an intentional-stop latch re-checked after every
`await` in `subscribe()` (and aborts the lock-acquire spin), so a
resubscribe can never race or outlive an intentional shutdown.
- Backoff resets only on genuine stream start, so a permanently stuck
slot backs off to the ceiling and logs loudly rather than tight-looping;
an epoch guard neutralises stale `START_REPLICATION` catches.
Runs- and sessions-replication opt in and use `shutdown()` for all
intentional stops.
**3. Observability** — the admin runs-replication status route probed
the old name-keyed Redis key (would report `leader:false` for every
source after fix #1); now probes the slot-keyed key.
## Tests
`internal-packages/replication/src/client.test.ts` (real Postgres +
Redis containers):
- same-slot/different-name → second client must not double-lead or race
into "slot is active" (the regression)
- a failing `START_REPLICATION` retry loop must not leak connections or
locks
- `shutdown()` during an in-flight `subscribe()` must not leave a zombie
leader
- `subscribe()` after `shutdown()` re-arms `resubscribeOnFailure`
- self-heals once the leader releases the slot
Plus the multi-source wiring test updated to the slot-keyed lock keys.
## Rollout
With the self-healing resubscribe, this ships as a **plain rolling
deploy** — the incoming pods retry across the one-time lock-key
transition and take over once the old pods drain (a brief replication
stall that the durable slot replays on reconnect — no data loss). No
stop-before-start required.
|
||
|
Daniel Sutton
|
70bca82d84 | feat(run-ops): activation — drop cross-DB FKs, provision run-ops DB, enable split (#4124) | ||
|
Daniel Sutton
|
c266e96c87 |
feat(run-ops): run-store routing seam + run-engine read seams (#4116)
## What Introduces the run-store routing seam and the run-engine read seams that let run lifecycle operations be dispatched to either the control-plane database or a separately-generated run-ops database, depending on where a run/batch resides. - **run-store** (`internal-packages/run-store`): adds `runOpsStore.ts` and substantially expands `PostgresRunStore.ts` so the store can resolve residency and route reads/writes to the correct backing client. `types.ts` grows the routing/residency types; `NoopRunStore.ts` is removed. - **run-engine** (`internal-packages/run-engine`): adds `engine/controlPlaneResolver.ts` and routes the per-system read paths (dequeue, enqueue, waitpoint, checkpoint, run-attempt, ttl, delayed-run, execution-snapshot, pending-version, debounce, batch) through the resolver/store instead of talking to a single Prisma client directly. `engine/errors.ts`, `engine/types.ts`, and `engine/index.ts` are extended to support injecting the store/resolver. Three fixes are included on top of the seam work: - `c6cadd85f` — routes read-your-writes to the owning store's **writer**, not its lagging replica, so an operation immediately reading back what it just wrote sees a consistent result. - `05c912e05` — normalizes run-ops-generation Prisma errors to the control-plane error class at the store **write boundary**, so `instanceof` checks and the `P2002` → 422 handling continue to work across the separately-generated run-ops Prisma client. - `88d12907f` — resolves NEW-resident batches in `ApiBatchResultsPresenter` by routing the batch read through the store, so a dedicated-DB batch resolves instead of returning 404. The change is heavily test-first: the bulk of the diff is new unit/integration coverage for the store routing, residency, and each run-engine system's control-plane resolver path. ## Why PR4 of the run-ops split stack (PR1–PR3 land the ClickHouse test-container and earlier plumbing). This PR is the read-path foundation: it adds the seam and read-routing but leaves the write path to route through the same seam in a later PR. Behavior-changing where the three fixes above touch existing read-your-writes / error-normalization / batch-resolution paths; otherwise additive (new store module, new resolver, injectable dependencies with existing single-client behavior preserved when no dedicated store is configured). ## Tests Extensive new vitest coverage under `run-store/src/*.test.ts` (routing, residency, dual-schema select, cross-generation error normalization, read-after-write, idempotency dedup, mixed residency, waitpoint co-location) and `run-engine/src/engine/**/*.test.ts` (per-system `controlPlaneResolver` tests, injectability, block-edge residency, waitpoint read residency, trigger-create routing, lifecycle router). Testcontainers-backed; no mocks. ## Notes Draft, **stacked on #4114** (`runops/pr03-clickhouse-tc`). Review that first; this diff is against it. Server-change / changeset note to be added at stack-assembly time. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Daniel Sutton
|
a93b101d44 |
feat(run-ops): cross-producer ClickHouse version helper + cross-Postgres-version compat tests (#4114)
## What - Adds `composeTaskRunVersion` to `@internal/clickhouse` (exported from the package index). It packs a small `originGeneration` epoch into the top 8 bits of the ReplacingMergeTree version and keeps the producer's own LSN in the low 56 bits, so `task_runs_v2` rows replicated from more than one Postgres producer become globally comparable while preserving in-producer ordering. Single-producer setups never call it and keep using the raw LSN version. - Adds unit coverage for the helper in `taskRuns.test.ts` (bit layout, ordering, epoch precedence, range validation). - Adds two run-engine tests that exercise the cross-Postgres-version testcontainer fixture: - `heteroPostgresFixture.test.ts` — a smoke test asserting byte-identity and identical `ORDER BY` (under a pinned ICU collation) across two different Postgres major versions. - `crossVersionCompat.test.ts` — mirrors the run-engine's real raw-SQL surfaces and asserts byte-identical, ordering-identical results across the two versions. Ships with an env-gated block (skipped by default) that can be pointed at a real dedicated database in CI. ## Why Third PR in the run-ops split stack. It is purely additive: it introduces one new exported helper plus tests and changes no runtime call sites, so on its own it has no runtime behavior change. It lays down the version-composition primitive and the cross-version compatibility proof that later PRs in the stack rely on. ## Tests - New unit tests for `composeTaskRunVersion` in `internal-packages/clickhouse/src/taskRuns.test.ts` (run against a real ClickHouse testcontainer). - New cross-version tests in `internal-packages/run-engine/src/engine/tests/` running against real Postgres containers of two different major versions (no mocks). The env-gated dedicated-database block is skipped unless its URL is set. ## Notes Draft, **stacked on #4113** (`runops/pr02-db-foundation`). Review that first; this diff is against it. Server-change / changeset note to be added at stack-assembly time. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Daniel Sutton
|
84588bd236 |
feat(run-ops): dedicated run-ops database package + docker service + migration runner (#4113)
## What The **dedicated run-ops database** foundation for the split: a standalone Prisma package plus the infra to run and migrate it. - **`internal-packages/run-ops-database`** — a new Prisma package (`@internal/run-ops-database`) whose schema mirrors the run-execution tables that will live on the dedicated DB, with its own generated client, migrations, and migration runner. - **`prisma/schema.parity.test.ts`** — a parity test that guards the run-ops schema against drift from the control-plane schema for the mirrored tables. - **Docker** — a Postgres 17 service (`docker/Dockerfile.postgres17`, `docker/docker-compose.yml`) so the dedicated DB is available locally under the run-ops compose profile. - **Testcontainers** — hetero fixtures (PG14 legacy + PG17 dedicated) so later PRs can exercise cross-database behaviour with real containers rather than mocks. ## Why This is the **second PR in the run-ops split stack**, stacked on the core primitives. It stands up the dedicated database and its tooling. There is **no runtime wiring** into the webapp here — the app does not read or write this DB yet; that arrives in later PRs. On its own this PR only adds a package, a docker service, and test fixtures. ## Tests Schema-parity test for the run-ops schema; hetero testcontainer fixture smoke test. ## Notes - Draft, **stacked on #4112** (`runops/pr01-core-residency`). Review that one first; this diff is against it. - Server-change / changeset note to be added at stack-assembly time. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Chris Arderne
|
448443a024 |
chore: bump internal node to 22 and standardise (#4084)
Bumps the internal/toolchain Node version to the latest 22.x LTS (`22.23.1`) and standardises it across the repo. Scope is the **platform toolchain + the repo's own runtime images** (all `20 → 22` *upgrades*, off the now-EOL node 20). ### Main changes - Node `20.20.2 → 22.23.1` across all CI workflows, `.nvmrc`, `CONTRIBUTING.md`, and the OSS `docker/Dockerfile` (digest-pinned). - `@types/node → 22.20.0` (root dep + pnpm `overrides`, so the whole workspace resolves to it); lockfile regenerated. - `sdk-compat` matrix: adds Node 24 + 26 (keeps 20, still in `engines`). - **App runtime images → node 22** (were on EOL node 20): `apps/coordinator` → `node:22.23.1-bookworm-slim`; `apps/docker-provider` + `apps/kubernetes-provider` → `node:22-alpine` (reusing the exact digest `apps/supervisor` already runs, so all four worker images are now identical). Stage aliases renamed off `node-20`. ### Possible issues / test notes - `@types/node` 22.x can surface new TS errors — typecheck (now on 22) is the gate. - **Smoke-test the v3 worker path** — `coordinator` (`crictl`/CRI calls) and the docker/kubernetes providers (talking to their daemons) now run on node 22 (alpine/musl for the providers). Upgrade off EOL so low-risk, but it's deployed runtime code with its own `publish-worker.yml` pipeline. |
||
|
Matt Aitken
|
fd4f02b2f8 |
fix(webapp): onboard new cloud orgs via plan selection; allow Free plan without GitHub verification (#4109)
## What & why Two related fixes to how new cloud organizations get onboarded onto the Free plan. ### 1. Route new cloud orgs through plan selection New cloud organizations were created already activated, so they skipped the plan-selection step and went straight to creating projects — which meant their plan and usage limits were never set up. They're now created deactivated and routed through plan selection, which activates them once a plan is chosen. Self-hosted installs have no plan-selection step, so they're activated immediately on creation and are unaffected. The `Organization.v3Enabled` field is renamed to `isActivated` to better describe what it now gates. It's mapped to the existing `v3Enabled` column, so there's no data migration — only a schema/code rename. ### 2. Allow selecting the Free plan without GitHub verification Choosing the Free plan no longer requires connecting and verifying a GitHub account. The plan is applied immediately when selected. This removes: - the "Connect to GitHub" dialog and the GitHub-verified badge from the plan picker - the account-rejected state - the now-unreachable GitHub-connect return routes ## Notes - These changes pair with the corresponding change in the billing service that applies the Free plan directly; they should be released together. ## Testing Verified locally end to end: a new cloud org is routed to plan selection, the Free plan applies in one click with no GitHub step, the org is activated, its usage allowance is provisioned, and it lands on the new-project page. 🤖 Generated with [Claude Code](https://claude.com/claude-code) |
||
|
Chris Arderne
|
c7861be520 |
chore: activate no-unused-vars and import linters (#4096)
Once this is merged, oxlint is at a pretty sensible baseline. **Enable `no-unused-vars`, `typescript/consistent-type-imports`, and `import/no-duplicates` lint rules** Turns on three previously-disabled oxlint rules across the monorepo and fixes all violations: - **`no-unused-vars`** – enabled as an error with standard ignore patterns: unused function arguments are ignored by default (`args: "none"`), variables/caught errors/destructured array elements prefixed with `_` are allowed, and rest siblings are permitted. - **`typescript/consistent-type-imports`** – enforced as an error; all type-only imports now use the `import type` syntax. - **`import/no-duplicates`** – enforced as an error; duplicate import statements from the same module have been merged. The remaining commits clean up the violations found across the codebase: removing unused variables/imports/type aliases, adding `_` prefixes to intentionally unused bindings, fixing duplicate imports, and converting value imports to `import type` where appropriate. |
||
|
Matt Aitken
|
536731a5d6 |
feat(clickhouse): infer mixed-type JSON arrays as Array(Dynamic) on insert (#4095)
## ✅ Checklist - [ ] I have followed every step in the [contributing guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md) - [x] The PR title follows the convention. - [ ] I ran and tested the code works --- ## Testing `pnpm run typecheck --filter @internal/clickhouse` passes. This only adds a ClickHouse input-format setting to existing insert calls; the setting affects type inference for newly-inserted/merged data and is non-destructive to existing rows. --- ## Changelog Sets `input_format_json_infer_array_of_dynamic_from_array_of_different_types = 1` on every native-JSON insert path: - `task_runs_v2` (`output`, `error`) — `insertTaskRuns`, `insertTaskRunsCompactArrays`, and the async-insert variants - `task_events_v1` / `task_events_v2` (`attributes`) - `metrics_v1` - `sessions_v1` ### Why Our JSON columns contain arrays with mixed element types (e.g. `[{"key":"value"}, "string", "string"]`). With this setting off — which is the effective default under `24.12` compatibility — ClickHouse infers those as deeply nested unnamed `Tuple(JSON, Nullable(String), …)` types. ClickHouse 26.2 introduced `input_format_binary_max_type_complexity` (default 1000), and those tuple type trees exceeded the limit, causing background merges to fail with **Code 117**. With the setting on (the default since 25.8), mixed-type arrays are inferred as a single `Array(Dynamic)` — a simpler, flatter type representation that never approaches the complexity limit, even once the upstream default limit is restored. Setting this explicitly at insert time keeps behavior deterministic and version-controlled, so it does not depend on the server profile or a future compatibility bump. This is a forward-only change: it only affects newly inserted/merged data and does not rewrite existing parts. Our read path re-serializes these columns to strings (`toJSONString` via the materialized `*_text` columns), so the internal Tuple → Array(Dynamic) representation change is transparent to the application. ### Companion server-side setting To also apply this on the ClickHouse side (covers merges and any writes not going through these code paths), set it on the default user: ```sql ALTER USER default SETTINGS input_format_json_infer_array_of_dynamic_from_array_of_different_types = 1; ``` --- ## Screenshots _N/A_ 💯 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01AaChyhestFMBYBWh6bgcCF --- _Generated by [Claude Code](https://claude.ai/code/session_01AaChyhestFMBYBWh6bgcCF)_ --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Chris Arderne
|
bfa902bd18 |
chore: enable more linters (#4080)
Re-enables ~15 oxlint rules that were blanket-disabled before. |
||
|
Eric Allam
|
aae1c6a512 |
fix(schedule-engine,webapp): log out-of-entitlements scheduled triggers as warnings (#4067)
## Summary When a scheduled task fires for an organization that is out of entitlements, the trigger can't proceed. That's an expected outcome, but it was being logged at error level and surfaced as a failure. ## Fix The trigger callback now classifies an out-of-entitlements result as its own error type (`OUT_OF_ENTITLEMENTS`), and the schedule engine logs both that and the existing queue-limit result as warnings rather than errors. The run still doesn't fire and the `schedule_execution_failure` metric still records the outcome (now tagged `out_of_entitlements`), so nothing about observability or behavior changes beyond the log level. |
||
|
Chris Arderne
|
0119cf8f9f | fix(dashboard-agent-db): load .env for local migrations (#4069) | ||
|
Katia Bulatova
|
b1987dc090 |
feat(webapp): billing limits — pause, reject, recovery, and settings UI (#3996)
## Summary Adds Billing Limits to the webapp. Customers can set a monthly spend cap. When usage crosses the limit, billable environments enter a grace period. If the limit is not resolved before grace expires, new triggers are rejected until the organization increases or removes the limit. |
||
|
Chris Arderne
|
b54201f986 | chore: switch to oxfmt, oxlint - add ci checks (#3977) | ||
|
Eric Allam
|
01b8dcf03b |
feat(dashboard-agent-db): run migrations over a direct (non-pooler) connection (#4054)
## Summary The in-dashboard agent's datastore now runs migrations over a direct (non-pooler) connection. A transaction-mode pooler can't run the migrator (no advisory locks, no multi-statement DDL), so when the agent's database sits behind a pooler the migration step needs a separate direct connection. The application keeps connecting over the pooled `DASHBOARD_AGENT_DATABASE_URL`. Only the migration entry points changed (`drizzle.config.ts`, `migrate.mjs`, `migrate-status.mjs`); the runtime client is untouched. ## Connection resolution (migrations) ``` DASHBOARD_AGENT_DIRECT_URL direct agent connection (used for migrations) DASHBOARD_AGENT_DATABASE_URL pooled agent connection (preserves current behavior) DIRECT_URL main direct connection (single-database fallback) DATABASE_URL last resort ``` Mirrors the existing `DATABASE_URL` / `DIRECT_URL` split. Fully backward-compatible: with nothing new set, resolution is identical to before. The agent-specific vars take precedence over the main `DIRECT_URL`, so a separate agent database is never migrated against the wrong one. When the agent falls back to the main single database, migrations now prefer its direct connection. |
||
|
Chris Arderne
|
df78ef96d9 |
feat: multi dev branches (#4023)
Closes this feature request: [https://triggerdev.featurebase.app/p/isolated-dev-sessions-for-multiple-local-trigger-dev-instances](https://triggerdev.featurebase.app/p/isolated-dev-sessions-for-multiple-local-trigger-dev-instances) ### Feature notes: - CLI `trigger dev` works as before - `trigger dev --branch my-branch` to create a new branch and run against it. - `trigger dev archive --branch my-branch` to archive (or in webapp). - New webapp page to manage and archive dev branches, currently feature flagged. ### Implementation details: - No changes to data model, no backfill. `isBranchableEnvironment` column is ignored for dev branches, we use `parentEnvironmentId IS NULL` instead. - `x-trigger-branch` overloaded for preview and dev branches - New `TRIGGER_DEV_BRANCH` env var available locally. `TRIGGER_PREVIEW_BRANCH` overloaded for child runs. - Lots of new glue code to sanitise the branch checks. ### Rollout - Deploy webapp/API changes (all backwards compatible) - Manual tests on some orgs - Deploy docs, release CLI, flip feature flag for webapp feature ### NB - `api.v1.projects.$projectRef.environments.ts` will return `isBranchableEnvironment: true` for all dev environments. ### Prerequisites - [x] Typecheck will not pass until we make a new release of `@trigger.dev/platform` and bump it here |
||
|
Eric Allam
|
5379b744ff | feat(dashboard-agent-db): add a pending-migration status check (#4037) | ||
|
Eric Allam
|
82bdd1fe0e |
fix(dashboard-agent-db): isolate the migration journal table (#4032)
## Summary The dashboard agent's database migrations could be silently skipped when its database is shared with another Drizzle application, leaving the `trigger_dashboard_agent` schema uncreated and a later migration failing with `schema "trigger_dashboard_agent" does not exist`. ## Root cause Drizzle's migrator decides what to run by reading the most recent row from its journal table by `created_at`, and skipping any migration dated at or before it. The dashboard-agent runner used Drizzle's default journal table, `drizzle.__drizzle_migrations`, which every Drizzle app shares by default. When the database is shared, another app's journal row dated between two of our migrations makes the migrator treat the earlier one (the `CREATE SCHEMA`) as already applied and run a later one against a schema that was never created. ## Fix - Track the dashboard-agent migrations in a dedicated journal table (`drizzle.__dashboard_agent_migrations`), in both the deploy runner (`migrate.mjs`) and the drizzle-kit config, so its history is independent of any other Drizzle app sharing the database. The table stays in the `drizzle` schema so the first migration's `CREATE SCHEMA "trigger_dashboard_agent"` does not collide with it. - Make the first two migrations idempotent (`CREATE SCHEMA/TABLE/INDEX IF NOT EXISTS`) so databases that already tracked them under the old journal table re-run cleanly after the rename instead of erroring on the bare `CREATE SCHEMA`. Verified against a Postgres seeded to reproduce the skip: the old default-table path fails as above, the dedicated-table path creates the schema and all tables, and re-running on an already-migrated database is a clean no-op. |
||
|
Eric Allam
|
8890d7a258 |
feat(run-engine,webapp): always report worker queue length metrics (#4029)
## Summary The `runqueue.workerQueue.length` gauge only reported a worker queue's depth while runs were being dequeued from it. When dequeues stop, the metric goes stale or missing, so a queue that has backed up because nothing is draining it can't be alerted on. This adds a small observer that refreshes the observed set of worker queues from the `WorkerInstanceGroup` records on an interval, so every active worker queue (and its scheduled split variant) keeps reporting its length regardless of dequeue activity. The observer is off by default and enabled per service via `RUN_ENGINE_WORKER_QUEUE_OBSERVER_ENABLED`, reads from the read replica, and skips a configurable set of cloud providers (`RUN_ENGINE_WORKER_QUEUE_OBSERVER_EXCLUDED_CLOUD_PROVIDERS`, default `digitalocean`). When enabled it is the source of truth for the observed set, so the per-dequeue registration is skipped on that instance, and it groups by worker queue so the per-instance duplicates collapse to the true depth. Also removes the unused `GET`/`POST /api/v1/workers` endpoints. Their only consumer was a CLI command group that is no longer registered. ## Verification Verified end to end against a local stack: the gauge reports each worker queue's length with no dequeues happening, excludes the configured providers, includes hidden groups, and the removed endpoints return as if they never existed. Added a run-engine test (`workerQueueObservation.test.ts`). |
||
|
Eric Allam
|
c06005b353 |
feat(webapp,sdk): in-dashboard AI agent (#4018)
## Summary Adds an in-dashboard AI agent: a chat panel, reachable from any environment page, that answers questions about your runs, errors, tasks, and analytics, diagnoses why a run failed, charts your data, reads your connected repo's source, and answers product and how-to questions. It is gated behind the `hasDashboardAgentAccess` feature flag (global or per-org, default off), so this PR ships disabled: the launcher is hidden unless the flag is enabled. ## Design The agent runs as a standalone `chat.agent` Trigger task in its own internal package, with no access to the webapp database, Prisma, or ClickHouse. It reads the user's data over the public API, acting as the user via a short-lived delegated user-actor token minted server-side each turn (never in the browser), building on [#3997](https://github.com/triggerdotdev/trigger.dev/pull/3997). The error and analytics tools use [#4005](https://github.com/triggerdotdev/trigger.dev/pull/4005) and the TRQL query API. The first turn of a new chat streams from a warm webapp route (Head Start) while the durable agent boots in parallel. Structured answers (a run-failure diagnosis card, a live chart) render through a small typed view catalog rather than arbitrary markup. A knowledge lane forwards product and how-to questions to the support assistant. Conversation history lives in a separate Drizzle-backed store on its own Postgres schema, kept as a display read-model so it can never corrupt the agent's model context. The SDK changes add an `apiClient` option to `chat.createStartSessionAction` and `chat.headStart`, and keep the Head Start tool-approval tail intact across a custom `prepareMessages` hook so prompt caching and Head Start compose. |
||
|
Matt Aitken
|
5667461895 |
fix(run-engine): decrement totalWeight in fair-queue weighted env shuffle (#4019)
## Summary Fixes the fair-queue weighted environment shuffle, which biased environment ordering whenever fair-queue biases are enabled (the default configuration). ## Root cause `#weightedShuffle` in `fairQueueSelectionStrategy.ts` computed the total weight once and drew its random pivot against that full-set total on every iteration, but never decremented the total as items were removed from the working set. After the first pick, the pivot frequently overshot the sum of the remaining items, so the inner selection loop ran off the end and clamped to the last remaining element. The result systematically over-selected whichever environment sat at the tail of the set. The first slot stayed fair (the full total is correct on the first draw), but later positions were ordered by environment iteration order rather than by the intended concurrency-limit and available-capacity weighting. For four equal-weight environments, the final position landed on one env ~9% of the time and another ~42%, instead of ~25% each. The two sibling selection paths (`#weightedRandomQueueOrder` and `#selectTopEnvs`) already decrement the total before splicing; this brings the env shuffle in line with them. ## Fix ```ts result.push(items[index].envId); totalWeight -= items[index].weight; items.splice(index, 1); ``` Adds a regression test that runs the weighted shuffle over equal-weight envs with biases enabled and asserts each env lands in every position roughly uniformly. It fails on the old code (tail position ~37%) and passes with the fix. Reported in #4001. |
||
|
James Ritchie
|
a90a495542 |
feat(webapp,database): show a Test column for agent sessions (#4011)
## Summary Sessions started from the agent Test playground were tagged with a `"playground"` tag that rendered in the Sessions table's Tags column. They are now flagged with a real `Session.isTest` boolean (mirroring `TaskRun.isTest`) and surfaced as a dedicated **Test** column with a check icon, to the left of Tags, on both the Sessions page and the Agent landing page, plus a matching **Test** property on the session detail page. This mirrors how Standard and Scheduled task runs already indicate test runs. ## Design `isTest` is a new `Session` column (Postgres) replicated into ClickHouse `sessions_v1` alongside the existing fields. The Sessions list reads `isTest` from Postgres for display (ClickHouse only supplies the ordered session IDs), so the column renders correctly without a ClickHouse backfill. The playground action now sets `isTest: true` on session create instead of writing the `"playground"` tag. The triggered run still carries `playground:true` in its own tags (unchanged). A migration backfills existing sessions, setting `isTest = true` and stripping the now-redundant `"playground"` tag where it is present, so the list and detail views render consistently without read-time tag filtering. |
||
|
Eric Allam
|
c6f0769299 |
fix(webapp): bound logs search memory and fix pagination at scale (#4012)
## Summary The logs search page (behind a feature flag) ran ClickHouse out of memory when browsing back over long time ranges. This keeps it within bounded memory and fixes a pagination bug that could skip or duplicate rows at a page boundary. ## Fix Memory: the list query reads in sort-key order, which opens one read stream per part in the window, and on object storage those per-part read buffers dominate peak memory, so it scaled with the number of parts scanned. Two changes bound it: - The logs ClickHouse client caps the per-part read buffers via new env-tunable settings. The object-storage-only setting is opt-in, so it is never sent to a ClickHouse version that lacks it. - Recent-first window narrowing: rows come back newest first, so the presenter probes the most recent window and only widens toward the full requested range when a page is short. A busy environment fills a page from a few recent parts instead of scanning the whole range; a quiet one still returns every row in a couple of cheap reads. Correctness: the keyset cursor ordered on (triggered_timestamp, trace_id), which is not unique because the spans of a trace share both, so rows at a tie could be skipped or duplicated across pages. The cursor and ORDER BY now include span_id, and the cursor is versioned so stale cursors reset to the first page. Guards: the effective page size is capped, and the existing per-query memory limit lets a pathological wide browse fail with an error instead of taking the node down. ## ClickHouse 26.2 The memory fix relies on lazy materialization deferring the wide attributes column to the output rows, which only holds on 26.x. Cloud already runs 26.2, so this moves the dev stack, testcontainers, and CI to match. The ClickHouse test suite passes on 26.2. 🤖 Generated with [Claude Code](https://claude.com/claude-code) |
||
|
Daniel Sutton
|
65c545da4e |
refactor(run-store,webapp,run-engine): route Postgres TaskRun reads through the run store (#3990)
## Summary Adds read methods to `RunStore` (`findRun`, `findRunOrThrow`, `findRuns`) and routes every Postgres read of `TaskRun` through them, mirroring how writes already go through the store. Behavior-preserving: each relocated read keeps its exact query, field selection, and database client (writer, replica, or transaction). This lets `TaskRun` reads be retargeted to a different backing store later without touching call sites. Stacked on #3981 (the write adapter); that PR is the base of this one. ## Scope In scope: the run engine, webapp services, presenters, and route loaders. Three reads that pulled `TaskRun` in through a parent model's relation `include` (alert delivery, batch results, attempt-dependency cancellation) are decomposed to fetch the run(s) through the store and stitch them back, since a relation include would not follow `TaskRun` to a new table. Left reading the existing table (out of scope): the legacy MarQS paths, the legacy trigger idempotency read, and one raw-SQL recovery script (commented for revisiting at cutover). ## Notes Reads default to the read replica; callers pass the writer or a transaction client wherever the original read did, so writer-vs-replica behavior is unchanged. |
||
|
Eric Allam
|
5052d895b3 |
feat(webapp,core): add a public HTTP API for errors (#4005)
## Summary
Adds an environment-scoped HTTP API over the Errors feature, mirroring
the runs API. Task-run failures are grouped by a fingerprint into "error
groups," and this exposes everything you can do with them in the
dashboard:
- `GET /api/v1/errors` lists error groups, with
`filter[taskIdentifier]`, `filter[version]`, `filter[status]`
(`unresolved`/`resolved`/`ignored`), `filter[search]`, a time range, and
cursor pagination.
- `GET /api/v1/errors/{errorId}` retrieves a single group (summary,
lifecycle state, affected versions).
- `POST /api/v1/errors/{errorId}/{resolve,ignore,unresolve}` changes its
state.
- `GET /api/v1/runs?filter[error]={errorId}` lists the runs behind a
group.
Request and response schemas are exported from `@trigger.dev/core/v3` so
the SDK can reuse them, and all endpoints are documented in the API
reference (OpenAPI). `errorId` is the `error_<fingerprint>` friendly id.
## Attribution
State changes record who made them. A plain environment API key has no
user, so `resolvedBy`/`ignoredByUserId` stay null. When the caller uses
an environment JWT obtained by exchanging a personal access token or a
delegated user token at `POST /api/v1/projects/:ref/:env/jwt`, that
exchange now stamps an `act` delegation claim, and the write endpoints
read `act.sub` to attribute the change to the acting user. This is the
first endpoint to consume the `act` claim, so two small pieces of
plumbing ride along: the exchange stamps `act` for personal-access-token
subjects too (it was delegated-token-only), and the public-JWT
bearer-auth path surfaces `act.sub` to the handler.
Built on the delegated-token work in #3997.
|
||
|
Eric Allam
|
06969b254a |
feat(cli,webapp): mint short-lived delegated tokens that act as a user (#3997)
## Summary Adds a short-lived, delegated token (`tr_uat_...`) that authenticates against the API as a user without handing out a long-lived personal access token. You mint one from a PAT, optionally narrow it to a set of scopes, and give it a lifetime; the API then treats requests as that user, subject to their role. `trigger.dev mint-token` is the entry point (it uses your stored PAT): ```bash UAT=$(trigger.dev mint-token --ttl 3600 --cap read:runs) ``` The token works anywhere a PAT does for user-level endpoints, and can be exchanged for an environment JWT at `POST /api/v1/projects/:ref/:env/jwt` to reach environment-scoped data (the same exchange a PAT supports). ## How it works A user-actor token is a short-lived JWT verified by a new first-class `authenticateUserActor` method on the RBAC plugin. Self-hosters get a built-in fallback; role-aware enforcement comes from the plugin. Effective permissions are the intersection of the user's role and the token's optional scope cap, so a token is only ever narrower than the user, never broader. Minting is restricted to personal access tokens (a token can't mint another one, and an environment key can't mint one). Tokens default to a 1 hour lifetime (max 365 days). When exchanged for an environment JWT, the user is stamped on it for attribution and the scope cap is carried through. |
||
|
Daniel Sutton
|
315baf2e54 | refactor(run-engine,webapp): route TaskRun writes through a new RunStore adapter (#3981) | ||
|
Oskar Otwinowski
|
e98a547e6c | feat(sso): SAML/OIDC single sign-on (#3911) | ||
|
nicktrn
|
6bdf800a11 |
feat(clickhouse): replicate run plan type to task_runs_v2 (#3978)
Replicates `TaskRun.planType` into the `task_runs_v2` ClickHouse table so run analytics can group by plan type. Adds a `plan_type` column (goose migration `033`, `LowCardinality(String)`), the replication insert mapping, and the matching schema/column/type entries - same shape as the recent `region` addition. Write-once at trigger, so it just rides along on existing replicated rows. Internal analytics only; not exposed in the Query API. |
||
|
nicktrn
|
7aa871f37b |
feat(webapp): plan-aware compute migration (#3957)
Adds an opt-in mechanism to route a configurable percentage of organizations onto the compute (MicroVM) backing of their region at trigger time, without changing their stored region settings. Routing is gated by three global feature flags - `computeMigrationEnabled`, `computeMigrationFreePercentage`, `computeMigrationPaidPercentage` - plus a per-org `computeMigrationEnabled` override that wins in both directions. A region's compute backing is resolved from a new `WorkerInstanceGroup.region` column: a container group and its MicroVM group share one geo `region`, so the migration swaps the resolved worker queue to the backing group's queue. Orgs are bucketed deterministically by id, so ramping a percentage down keeps a strict subset rather than reshuffling, and a region with no compute backing is never touched. Everything is off by default - behaviour is unchanged unless the flags are set. The flags and the worker-region groups are read on the trigger hot path from in-memory snapshots rather than the database: a small `createReloadingRegistry` helper loads each at startup and refreshes them on an interval, so no per-trigger query is added and a percentage or kill-switch change propagates within the reload interval. A cold replica whose snapshot hasn't loaded yet reads as not-migrated (the container path) and self-corrects on the next load - the same cold-start contract as the datastore / LLM-pricing registries, with a `reloading_registry_loaded` metric so a never-loaded registry is alertable. The same migration decision is consulted at deploy-time template creation so a migrated org gets a compute template built ahead of its first run. This runs in shadow mode (best-effort, never fails the deploy) by default, or - when the `computeMigrationRequireTemplate` flag is on - in required mode, built synchronously at deploy so the first run never builds on-demand and template errors surface at deploy time. So operators keep "which runs ran where" while customers only see geography: the run's actual worker queue is stored raw, and the geo region is stamped separately on `TaskRun.region` (and a new ClickHouse `region` column) at trigger time. Read surfaces - the dashboard, the API, and the Query/Logs page - show the geo region, falling back to the worker queue for runs written before the column existed. Minor follow-ups left out of scope: the percentage flags render as text inputs on the admin flags page (the catalog UI has no numeric control type yet), and `createReloadingRegistry` could later gain pub/sub for sub-second cross-replica propagation if the reload interval proves too slow. |
||
|
Eric Allam
|
07a0e4ade9 |
feat(webapp): split Models into Your models and Model library tabs (#3958)
## Summary The Models page is now split into two tabs. **Your models** shows the models your project has actually used in the selected time range, with usage charts (cost over time, tokens over time, calls by model), a per-model table of calls / cost / avg TTFC / avg tokens-per-sec, and calls/tokens trend sparklines. **Model library** is the full catalog, reordered from alphabetical to a relevance-based provider order (Anthropic, OpenAI, Google, then the rest), newest models first within each provider, with a "New" badge on models released in the last 7 days. One time-range selector drives the whole Your models tab, so the charts, the table, and the sparklines all share the same window. Opening a model shows its own metrics with an independent range picker and a "View in AI metrics" link that opens the AI metrics dashboard filtered to that model. The active tab is kept in the URL so it survives a refresh and is shareable. ## Prompt caching & cost accuracy Both the Your models tab and the AI metrics dashboard now surface prompt-cache usage: a cache-savings column plus per-model cached-tokens and cache-hit-rate views, and a caching section on the dashboard (hit rate, cached tokens, estimated savings, and hit rate by model). Building this surfaced a cost bug. `input_tokens` is the total prompt count and already includes cache-read and cache-creation tokens, but the cost pipeline charged the full input at the input price and then added a separate cache line, so cached tokens were billed twice (and on Anthropic, cache reads were never discounted because their price is keyed differently). The input price now applies only to the non-cached remainder, with cache prices resolved across the provider-specific keys, so LLM cost and the cache hit-rate metric are accurate. Hit rate is computed as cached reads over total input. ## Notes Also fixes React "invalid DOM property" console warnings from the provider icons (the Llama and DeepSeek SVGs used raw `fill-rule` / `clip-rule` / `clip-path` attributes), which this page surfaces by rendering more provider icons. ## Screenshots **Your models tab:** usage charts and a per-model table with calls/tokens trend sparklines. <img width="2560" height="1267" alt="1-your-models-tab" src="https://github.com/user-attachments/assets/859bd24f-9047-4828-8bbb-83e5882846d6" /> **Model library:** provider-relevance ordering with a "New" badge on models released in the last 7 days. <img width="2560" height="1267" alt="2-model-library-tab" src="https://github.com/user-attachments/assets/46dd54b9-80f9-4922-ade9-5935b08dfebc" /> **Model detail, Metrics tab:** per-model range picker and a "View in AI metrics" link. <img width="2560" height="1267" alt="3-model-detail-metrics" src="https://github.com/user-attachments/assets/0f65d9d0-6142-4918-93f0-110bb277101a" /> **View in AI metrics:** the dashboard deep-linked and filtered to the selected model. <img width="2560" height="1267" alt="4-ai-metrics-filtered" src="https://github.com/user-attachments/assets/821f256c-e305-493c-98c7-eafaf2f57f83" /> |
||
|
Matt Aitken
|
5d6ea33166 |
refactor: share the public-token JWT scope decoder; make @trigger.dev/plugins internal (#3919)
## What `buildJwtAbility` — the decoder for public-token scope strings (`read:tags:…`, `read:runs:run_abc`, `admin`, …) — now lives in `@trigger.dev/plugins` as the single source of truth. `@trigger.dev/rbac` re-exports it, so the built-in fallback and any auth plugin interpret a token identically. Scope strings are split on only the first **two** colons (`action:type:id`), so a resource id that itself contains colons — e.g. a tag like `user:123` — is matched in full rather than truncated to its first segment. (The fallback already did this; this makes it the one shared implementation.) `@trigger.dev/plugins` is now **private (unpublished)** and gains a `@triggerdotdev/source` export condition, so consumers bundle it from source per-commit like `@trigger.dev/core` instead of resolving a published version — no cross-version coordination. ## Why Two hand-maintained copies of the scope grammar drift, and the difference silently changes what a token grants. One shared decoder removes that class of bug. ## Notes - No changeset: `@trigger.dev/plugins` is now private and `@trigger.dev/rbac` is internal — neither is published. - Unit coverage for the colon-id path lives in `internal-packages/rbac/src/ability.test.ts` (now exercising the shared function). |
||
|
Daniel Sutton
|
7b4443a437 |
test(webapp): stop streamBatchItems container tests timing out on cold start (#3900)
Fixes an intermittent `Test timed out in 30000ms` in the `streamBatchItems` suite. Not a logic hang — the 30s budget covers container setup, and each case boots its own per-test Redis container + a full `RunEngine`, so under CI Docker contention a cold boot can cross 30s (which is why the failure moved between tests). - New `containerTestWithIsolatedRedisNoClickhouse` fixture (Postgres clone + per-test Redis, no ClickHouse) — this suite never uses ClickHouse, but the old fixture's auto `resetClickhouse` forced a ClickHouse boot + migration onto the cold-start test. - Raised `testTimeout` 30s → 120s, matching the run-engine convention for this footprint. |
||
|
Eric Allam
|
f9d57d3bd5 |
feat(webapp): add a new backend for the realtime runs feed (#3864)
## Summary Adds a second backend for the realtime runs feed (`useRealtimeRun`, `subscribeToRunsWithTag`, `subscribeToBatch`), built to stay healthy when a single busy environment has many subscribers watching many runs at once. It is gated behind a feature flag with the existing backend as the default, so nothing changes for users until it is enabled per environment. ## Design A run change is published once, as a small self-describing record, to a single per-environment channel. Every feed is then a predicate over that one stream rather than owning a channel: - A per-instance router indexes the currently-held feeds by run, tag, and batch. When a run changes it hydrates the affected rows once and serializes them once, then fans the result to every matching feed. One hot shared tag watched by many subscribers costs a single database query and serialize, not one per subscriber. - Feeds that don't match a change are never woken, wake delivery per environment is coalesced on a leading edge (250ms default) so a burst of changes costs one wake, and cold reads coalesce onto a single short-TTL-cached resolve. - An admission gate bounds how many cold ClickHouse resolves run concurrently, so a mass reconnect across many distinct filters queues instead of stampeding the database. - Changes that land while a client is between long-polls are delivered on its next poll instead of waiting for the periodic backstop: each environment buffers its recent change records, subscriptions linger briefly after the last feed closes, and a newly-armed poll replays exactly the connection's gap. - The per-connection replay cursors behind that are shared across instances via Redis (a single timestamp each), so a poll landing on a different instance behind the load balancer still reads the connection's true gap instead of falling back to a cold resolve. Cursor reads have a bounded deadline and degrade to the cold-read path on any Redis trouble. - Tag subscriptions with multiple tags match runs carrying all of the tags, mirroring the existing backend's filter semantics, and live long-polls hold for about 20 seconds to match its cadence. - The per-environment channel supports Redis Cluster sharded pub/sub, so the wake path scales horizontally across shards by environment. - The backend reports its health through OpenTelemetry metrics (delivery lag, poll resolution paths, backstop outcomes, replay and cursor-store activity), with a provisioned Grafana dashboard for local development. Everything is behind the feature flag and tunable via env vars; the existing backend remains the default. |
||
|
Daniel Sutton
|
6afc9bfa4c |
fix(run-engine): retry getSnapshotsSince on the replica then primary when the read replica lags (#3889)
## Summary
When `RUN_ENGINE_READ_REPLICA_SNAPSHOTS_SINCE_ENABLED` is on,
`RunEngine.getSnapshotsSince` reads from the read replica. During write
spikes the replica can briefly lag, so the snapshot id a runner just
learned from the writer isn't visible there yet: the lookup threw, the
worker route returned a 500, and the runner waited for its next poll —
turning sub-second snapshot notifications into poll-interval latency
exactly when things are busiest. This PR makes the flag safe to enable:
a replica miss of the since snapshot gets one jittered retry on the
replica (most lag windows are shorter than the ~50–200ms wait, so the
writer is never touched), then falls back to the primary, observed via a
new `run_engine.snapshots_since.replica_miss` counter with an `outcome`
attribute (`replica_retry` vs `primary`). Only genuine misses — absent
on the primary too — remain errors.
## Design
- `getExecutionSnapshotsSince` now throws a typed
`ExecutionSnapshotNotFoundError` so the engine can distinguish the
expected lag miss from real failures. The message string is unchanged
and the error never leaves the engine.
- The recovery path only engages when the flag is on, a distinct replica
client is configured, and no transaction client was passed. With the
flag off, the path is behaviorally identical to before.
- Retry delay bounds are configurable
(`RUN_ENGINE_SNAPSHOTS_SINCE_REPLICA_RETRY_MIN_MS`/`MAX_MS`, default
50/200; `MAX_MS=0` skips the replica retry and goes straight to the
primary).
- The warn log fires only when the primary serves the read (the writer
spill is the operationally interesting event); replica-retry recoveries
are counted but quiet. A permanently-missing snapshot id stays an
error-level failure with a `failedDuring` field, so lag metrics aren't
polluted by bogus ids.
- Stale-tail lag (replica has the since snapshot but not newer rows)
deliberately still returns the replica's view; the next poll catches up.
- The since-snapshot anchor lookup is now scoped to the polled run
(`where: { id, runId }`), so a snapshot id from a different run raises
not-found instead of silently anchoring a too-wide window of the run's
snapshots.
## Test plan
All vitest + testcontainers, no mocks. A new `schemaOnlyPrisma` fixture
(migrated-but-empty clone database) simulates a replica that hasn't
caught up, and a real in-memory OTel meter pins the counter semantics
per outcome.
- [x] Replica catches up during the jittered retry window → served by
the replica, `outcome=replica_retry` = 1, primary never consulted
- [x] Replica permanently missing the since snapshot → served by the
primary, `outcome=primary` = 1
- [x] Snapshot missing on both replica and primary → null, counter = 0
- [x] Replica has the since snapshot but lags by one → the replica's
view is served, no fallback (verified discriminating power: the test
fails if reads secretly hit the primary)
- [x] Flag off with a replica configured → primary serves the read
- [x] Transaction client provided → bypasses the replica entirely
- [x] Since snapshot belonging to a different run → null
- [x] Existing getSnapshotsSince + waitpoints suites green; run-engine,
testcontainers, and webapp typechecks pass
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
|
||
|
Matt Aitken
|
6bcd369ea1 |
feat(webapp,rbac): REQUIRE_PLUGINS=1 fail-fast for required plugin loads [TRI-9852] (#3734)
## Summary - `internal-packages/rbac/src/index.ts` — in `LazyController.load()`'s catch block, throw an Error when `process.env.REQUIRE_PLUGINS === "1"` instead of silently falling back. The throw is captured into the lazy controller's init promise, so it surfaces on the first method call. - `apps/webapp/app/routes/healthcheck.tsx` — `await rbac.isUsingPlugin()` after the DB ping. With `REQUIRE_PLUGINS=1` and a failed plugin load, the throw surfaces here and the healthcheck returns 500 → readiness probe fails → rollout is rolled back. Noop for self-hosters. - `.server-changes/require-plugins-fail-fast.md` — server-changes entry. - `internal-packages/rbac/src/require-plugins.test.ts` — 4 unit tests covering loader branching: unset → fallback, `=1` → throw, `forceFallback: true` wins, only exactly `"1"` enforces. - `internal-packages/testcontainers/src/webapp.ts` — adds `requirePlugins?: boolean` to `StartWebappOptions`. Implies `forceRbacFallback: false`. - `apps/webapp/test/healthcheck-require-plugins.e2e.test.ts` — e2e closes the loop: spawns a real webapp, hits `/healthcheck` via HTTP, asserts 500 with `REQUIRE_PLUGINS=1` and 200 without. ## Motivation Today the RBAC plugin loader catches any plugin-load failure (missing module, broken transitive dep, init throw) and silently returns the default fallback implementation. This is the correct behaviour for self-hosters who don't ship the plugin — but it's dangerous in deployments where the plugin is expected to load: an accidentally-missing or broken plugin would silently disable enforcement. `REQUIRE_PLUGINS=1` makes the loader fail loudly in those deployments. The variable name is intentionally plural and generic — future plugin contracts (audit logs, SSO) can read the same flag without renaming. Closes [TRI-9852](https://linear.app/triggerdotdev/issue/TRI-9852/require-plugins1-fail-fast-for-required-plugin-loads). ## Test plan - [x] `pnpm run test --filter @trigger.dev/rbac` — 38/38 tests pass, including the 4 new loader tests - [x] `pnpm run typecheck --filter webapp` — passes - [x] `pnpm run typecheck --filter @trigger.dev/rbac --filter @internal/testcontainers` — passes - [x] e2e test added (`healthcheck-require-plugins.e2e.test.ts`) — CI runs it via `e2e-webapp.yml`. Couldn't run locally (no Docker daemon up); CI has Docker provisioned. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Oskar Otwinowski
|
93532cdb99 |
feat(supervisor): forward per-run labels to the compute provider (#3821)
Add an optional network_labels field to the internal compute client's create and restore request schemas and forward per-VM endpoint labels on both paths, so a restored VM keeps the same labels as a freshly-booted one. Mirrors the label the Kubernetes workload manager already sets on the run pod. --------- Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com> |
||
|
Matt Aitken
|
ef04cc39ef |
fix(webapp): use composite keyset cursor for run pagination (#3852)
## Problem
`ClickHouseRunsRepository.listRunIds` / `listRuns` order results by the
composite key `(created_at, run_id)`, but the cursor predicate cut on
`run_id` **alone**:
```ts
.where("run_id < {runId: String}", { runId: cursor })
.orderBy("created_at DESC, run_id DESC")
```
This is only sound when `run_id` lexicographic order matches
`created_at` order. `run_id`s are cuids — only coarsely time-sortable —
so when a burst of runs is created within a sub-second window, the two
orders can diverge. When they do, the next-page predicate (`run_id <
cursor`, where `cursor` is the *last* page element = the smallest
`created_at`, not necessarily the smallest `run_id`):
- **re-includes** rows already returned on a previous page (duplicates),
and
- **skips** rows it should have returned (silent data loss).
For bulk **replay** this caused runs to be replayed more than once
(replay has no idempotency guard). For the dashboard and the `runs.list`
API it could silently repeat or skip runs at page boundaries.
## Fix
Make the cursor predicate match the composite ordering:
- Cursors now encode the full `(created_at, run_id)` key as an **opaque
URL-safe base64 token**
(`base64url({"c":<createdAtMs>,"r":"<runId>"})`), and the query cuts on
the matching tuple — `(created_at, run_id) < (…)` forward / `> (…)`
backward.
- The `ORDER BY` is unchanged, so the query stays aligned with the
table's primary key — no performance regression (the tuple range
predicate is actually more index-friendly than `run_id <` alone).
- Cursors are **server-issued opaque tokens** (the SDK only echoes
`pagination.next` / `pagination.previous` back), so this needs **no
client/SDK update**. Legacy cursors were the bare internal `run_id`;
they're detected by decode failure (a cuid isn't a valid base64-wrapped
JSON payload) and fall back to the old `run_id`-only predicate, so
in-flight cursors keep working and drain naturally. New cursors also no
longer expose a bare internal run id.
- `listRunIds` is now the single cursor-aware list primitive: it returns
`{ runIds, pagination: { nextCursor, previousCursor } }`, and `listRuns`
builds on it (one place constructs cursors). Bulk actions consume the
same method and advance by `pagination.nextCursor`, finishing when it's
`null`.
- `getTaskRunsQueryBuilder` now also selects
`toUnixTimestamp64Milli(created_at) AS created_at_ms`, using a dedicated
`TaskRunListQueryResult` schema. The shared `TaskRunV2QueryResult` stays
`run_id`-only so the run-engine pending-version lookup
(`getPendingVersionIdsQueryBuilder`, which selects only `run_id`)
doesn't fail validation on a column it doesn't query.
## Tests
New `runsRepositoryCursor.test.ts` (testcontainer-backed, real
Postgres→ClickHouse replication):
- **forward** pagination returns every run exactly once when `run_id`
order is the reverse of `created_at` order (reproduces the
duplicate/skip bug — fails on `main`; this
walk-until-`nextCursor`-null-and-assert-complete is exactly the bulk
action's iteration),
- **backward** pagination round-trips to the previous page across a
boundary,
- **legacy** bare-`run_id` cursor still uses the old predicate
(backwards compatibility).
The existing `runsRepository` suites (part1–4) still pass; `part4`'s
`count new runs with listRunIds` test was updated for the new `{ runIds,
pagination }` return shape, and the `clickhouse` `taskRuns`
query-builder snapshots were regenerated for the added `created_at_ms`
column.
## Notes
- Separate, pre-existing issue (out of scope, not introduced here):
`listRuns`' backward display-slicing (`rows.slice(1, size+1)` when
`hasMore`) has an off-by-one that can return a straddled page. Tracked
separately.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
nicktrn
|
f261ff2b85 |
chore(docker): tidy dev postgres + clickhouse images (#3859)
Two small hygiene tweaks to **dev-only** images: - `docker/Dockerfile.postgres`: add `--no-install-recommends` to the partman install (leaner image, skips unneeded recommended packages). - `internal-packages/clickhouse/Dockerfile`: run the migration helper as a non-root user. Both are local-dev images (the `pnpm run docker` stack) - no impact on the published webapp image, prod, or self-hosting. |
||
|
nicktrn
|
fa15438e42 |
perf(ci): speed up unit tests with LPT sharding + container scoping (#3855)
Speeds up and de-flakes the unit-test suite: testcontainers booted once per vitest worker (per-test isolation kept only where a test runs background redis work that outlives it), a duration-weighted shard sequencer so each shard does roughly equal work, the slowest suites split, two genuine flakes fixed (`streamBatchItems` shared-redis leak; run-engine waits that relied on fixed sleeps), and transient DockerHub pulls retried. **Timings (CI, per-shard wall):** worst unit-test shard ~771s → ~294s; packages/webapp shards ~250-270s, most internal ~190-240s. All 25 shards green. A shard breaks down as ~70s fixed setup (install / image-pull / generate) + ~70s cold `^build` + the actual container tests. So the remaining cost is mostly the tests themselves plus that fixed setup. **Next (separate, timings):** - **typecheck (~6m24s)** — the slowest check overall; bound by full-graph `tsc`, not the TS version (a TS6 branch is still ~6m17s). The real lever is **tsgo** (the Go compiler). - Possible later: turbo CI caching could trim the ~70s cold build on *warm* runs, but it's conditional (cold runs rebuild anyway) and doesn't touch setup or test time — secondary. `cli-v3` e2e and `sdk-compat` are path-gated (don't run on test-infra changes) and already comfortably fast. |
||
|
Eric Allam
|
97036fb741 |
feat(webapp,clickhouse): export run traces as log, markdown, or jsonl (#3851)
## Summary Adds a trace export to the run page. From the new **Export trace** menu you can copy a run's full trace to the clipboard as Markdown (for pasting into an AI assistant) or download it as a flat Log, a Markdown table, or JSON Lines. Internal engine-debug events are filtered out by default, and errors are surfaced inline with their message. ## Design The export streams events from the store to the gzipped response one at a time and never materialises the span tree, so a trace of any size exports with bounded memory and without stalling the server. Output is flat and chronological: each line carries its own `spanId ← parentSpanId`, so the hierarchy is reconstructable without nesting. Formats share a single streaming pipeline and are pluggable via `?format=log|jsonl|markdown`, so adding a format is an isolated change. ## Screenshots **Export menu** <img width="370" height="252" alt="trace-export-menu" src="https://github.com/user-attachments/assets/3d10304a-8c49-4606-b15d-2859b137419f" /> **In context** <img width="2400" height="1802" alt="trace-export-run-page" src="https://github.com/user-attachments/assets/46c80b30-303b-47c6-9ace-a2fb06f6cb61" /> |
||
|
nicktrn
|
707bf1adb4 |
ci: reduce unit test flakiness and shard re-run cost (#3844)
A unit-test shard recently failed on a timing race rather than a real regression - a run-engine waitpoint test sleeps 1250ms waiting on a 1000ms timeout that's processed by a ~1000ms worker poll, so on a CPU-starved shard the margin evaporates and the whole matrix goes red. Because `fail-fast` defaults on, that one flake cancels the sibling shards, and the only recovery is re-running the entire matrix "just to be sure" - which is itself slow. This is the low-risk first pass at that pain: - `fail-fast: false` on the webapp and internal shard matrices, so one flaky shard no longer cancels its siblings. "Re-run failed jobs" now re-runs just the failed shard instead of the whole matrix. - CI-scoped `retry: process.env.CI ? 2 : 0` on the timing-sensitive packages (`run-engine`, `redis-worker`, `schedule-engine`). Flakes self-heal in CI; local runs stay at `retry: 0` so they still surface in dev. A stopgap until the timing tests are made deterministic. - `fetch-depth: 1` on the unit-test checkouts - they don't use git history, so the full clone was wasted setup time across ~20 jobs. - Reconcile the pre-pull image tags with what testcontainers actually pulls (`redis:7-alpine` -> `redis:7.2`, `ryuk:0.11.0` -> `ryuk:0.14.0`) and add `minio/minio:latest` to the webapp pre-pull. Otherwise those images pull unauthenticated at test time and risk Docker Hub rate-limit flakes (worst on fork PRs, where the authenticated pre-pull is skipped entirely). Deeper follow-ups - bigger runners, turbo remote cache, runtime-weighted sharding, and the real root-cause fix (container reuse / template-DB isolation + deterministic timing tests) - are tracked under TRI-10484. |
||
|
nicktrn
|
16d59aa9e7 |
chore: harden webapp docker image (#3845)
Hardens the webapp Docker image and adds a CVE scan of each published image. - Base image `bullseye-slim` → `bookworm-slim` (Debian 12), pinned by digest. Adds `apt-get upgrade` + `--no-install-recommends` + apt-cache cleanup across the build stages so OS packages are patched at build time. - Moves the `react-email` CLI to `devDependencies` in `internal-packages/emails` — only the `email dev` preview script uses it; the runtime render path is `@react-email/render` + `@react-email/components`. This also drops the bundled `esbuild` binary from the production image. - Bumps `goose` v3.26.0 → v3.27.1 and its Go builder image 1.23 → 1.26. - Adds a reusable Trivy image-scan workflow wired into `publish.yml`, so every published image (main builds and releases) is scanned for OS-package CVEs right after it's pushed to GHCR. Report-only (writes to the run summary), runs alongside the worker publishes so it never blocks a deploy. Verified locally: the image builds clean on the new base, and `@react-email/render` carries no `esbuild` dependency so email rendering is unaffected. |
||
|
Eric Allam
|
aa9f1112ea |
fix(database): include the Prisma CLI in production builds (#3843)
## Summary The Prisma CLI was missing from production builds of the webapp image, so anything that shells out to `prisma` at startup failed. The container entrypoint and the standalone migration step both run `prisma migrate deploy` / `prisma migrate status`, and those broke with `Command "prisma" not found`. ## Fix `prisma` was a `devDependency` of `@trigger.dev/database`. It had only been landing in the pruned `--prod` install as a side effect of pnpm auto-installing it as a peer of `@prisma/client`. A recent dependency change shifted peer resolution so prisma stopped being materialized into the production tree, and the CLI disappeared from the image. Moving `prisma` into `dependencies` of `@trigger.dev/database` makes the CLI an explicit part of production installs. It lands in the webapp image only: the separately deployed supervisor, coordinator, and provider images don't reach the database package in their production trees (`core` only `devDepends` on it, so it isn't transitive), so they're unaffected. Verified against a locally built production image: `pnpm --filter @trigger.dev/database exec prisma --version` now resolves the CLI and the schema engine instead of failing. |
||
|
Eric Allam
|
359e2503c9 |
feat(database,webapp): add LlmModel pricing_unit column and admin selector (#3820)
## Summary
Adds a nullable `pricing_unit` column to the LLM model registry's
`llm_models` table, recording how each model is billed ("tokens",
"characters", "images", "minutes", "requests", "free", "not_findable").
It lets pricing-coverage reporting exclude models that aren't priced
per-token (image/video/audio models currently drag the "% priced" number
down even though they can never carry a per-token price), and lays the
groundwork for non-token pricing.
The default model catalog is entirely per-token, so `seed` and
`syncLlmCatalog` set `pricing_unit="tokens"` on those rows. The admin
LLM model form (create + edit) and the admin API get a pricing-unit
selector so admin-curated models can set it; existing rows can stay
unset.
Auto-discovered models get their unit from the model-registry pipeline,
which lands separately.
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
|
||
|
nicktrn
|
55d85d0b23 |
chore(emails): upgrade react-email to latest (#3819)
## ✅ Checklist - [x] I have followed every step in the [contributing guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md) - [x] The PR title follows the convention. - [x] I ran and tested the code works --- ## Testing Upgraded the email packages in `internal-packages/emails`: | Package | Before | After | | --- | --- | --- | | `@react-email/components` | `0.0.16` | `1.0.12` | | `@react-email/render` | `^0.0.12` | `^2.0.8` | | `react-email` (CLI) | `^2.1.1` | `^6.5.0` | | `react-dom` | _(missing)_ | `^18.2.0` (now a required peer of render) | **Breaking change handled:** `render()` is now async (`Promise<string>`) in `@react-email/render` v1+. Added `await` in the `aws-ses`, `smtp` and `null` transports. `EmailClient` and the webapp callers were already async and needed no changes. Verification: - `pnpm run typecheck --filter emails` ✅ - `pnpm run typecheck --filter webapp` (consumer of the `emails` package) ✅ - **Before/after render comparison**: rendered all 11 templates (magic-link, invite, welcome, alert-attempt/run/error-group, deployment-failure/success, mfa-enabled/disabled, bulk-action-complete) to HTML with both the old and new packages and compared them visually + via HTML diff. Output is visually identical. The only HTML changes come from upstream improvements: `<Body>` now wraps content in a `<table>`/`<td>` for better email-client compatibility, an `x-apple-disable-message-reformatting` meta tag was added, and CSS shorthand (e.g. `margin`) is now also emitted as longhand. No visual regressions; the `CodeBlock`/dracula theme, buttons, and row/column layouts all render correctly. No changeset or `.server-changes/` entry is added: `emails` is a private internal package (not under `packages/`), and there is no user-facing behavior change. --- ## Changelog Upgrade `react-email` and `@react-email/{components,render}` in `internal-packages/emails` to their latest versions and adapt the mail transports to the now-async `render()` API. --- ## Screenshots Rendered email templates before vs after the upgrade (visually identical): **Before** (`@react-email/components@0.0.16`, `render@0.0.12`) |
||
|
nicktrn
|
a4d8c9f65f |
chore(deps): update OpenTelemetry suite to 0.218.0 / 2.7.1 (#3810)
Brings the OpenTelemetry packages up to the latest coherent release
across the webapp and the published packages (`@trigger.dev/core`, the
CLI, `@trigger.dev/sdk`) plus
`internal-packages/{tracing,testcontainers}`:
- `@opentelemetry/sdk-node` 0.218.0
- `@opentelemetry/core` 2.7.1
- `@opentelemetry/host-metrics` 0.38.3
We were already on the otel 2.x line, so this is a same-major minor move
- the versions are pinned to `@opentelemetry/sdk-node@0.218.0`'s own
declared dependency set so the experimental (0.2xx) and stable (2.x)
packages stay coherent (mixing them is the usual otel breakage).
**One code change:** otel 0.215 made `forceFlush()` a required method on
`LogRecordExporter`, so `ExternalLogRecordExporterWrapper` (core's
tracing SDK) gains a `forceFlush()` that delegates to the underlying
exporter.
**Notable upgrades along the way:** OTLP exporters can take a custom
HTTP agent (connection pooling/keepAlive on the export path), HTTP
request headers are captured at span creation, and core hot-path perf
improvements in 2.6.1/2.7. `host-metrics` 0.37→0.38 is a clean upgrade.
Patch changeset added for the three published packages. References
projects are intentionally untouched.
Verified: `@trigger.dev/core` / CLI / `@trigger.dev/sdk` build, webapp +
`@internal/tracing` typecheck - all green.
|