v.docker.4.4.6
349 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Eric Allam
|
2301ed608c |
refactor(run-engine): make taskIdentifier optional on run-queue messages (#3559)
## Summary Make `taskIdentifier` optional on the run-queue message schema. No behavior change in this PR; readers continue to accept payloads that include the field. A separate change will stop writing it on the wire to shrink the per-run payload that lives in Redis while runs wait to be dequeued. ## Design The field is written into every payload at enqueue time but no consumer reads it back on the dequeue path. Both the run-engine and supervisor derive `taskIdentifier` from the loaded `TaskRun` row instead. Relaxing the schema first means readers tolerate payloads that omit it, so the writer-side change can ship without producing schema-parse errors during a rolling deploy. `projectId` is left required: `WorkerQueueResolver.#getOverride` reads it for project-scoped runtime worker-queue overrides. ## Test plan - [x] `pnpm run typecheck --filter @internal/run-engine` - [x] `pnpm run typecheck --filter webapp` - [x] `pnpm run test ./src/run-queue/tests/enqueueMessage.test.ts ./src/run-queue/tests/workerQueueResolver.test.ts --run` (28/28 passing) |
||
|
Eric Allam
|
5f1a3e1653 |
refactor(run-engine): route TTL expiration through the batch path only (#3554)
## Summary
TTL expiration on queued runs was being scheduled twice: once via a
per-run `expireRun` worker job (the original implementation) and once
via the batch TTL system (added more recently). Both paths attempt to
flip the same run to `EXPIRED`. The per-run job almost always won the
race, leaving the batch consumer to observe runs already expired by the
older path.
This collapses TTL expiration onto the batch path so every queued TTLed
run goes through a single Redis-backed sorted set + batch consumer
instead of also getting its own scheduled redis-worker job.
## Design
`engine.trigger` and `delayedRunSystem.enqueueDelayedRun` no longer call
`ttlSystem.scheduleExpireRun`. The remaining `enqueueSystem.enqueueRun({
includeTtl: true })` already adds the run to the TTL sorted set;
`TtlSystem.expireRunsBatch` flips it to `EXPIRED` when the TTL fires.
Delayed runs get the same coverage by passing `includeTtl: true` on
their post-delay enqueue, so the TTL is armed from the moment the run
enters the queue (matching how the old job behaved —
`parseNaturalLanguageDuration` is evaluated at enqueue time).
The new path explicitly does not re-expire runs once they have been
allocated a concurrency slot. That is intentional: TTL is for runs that
are queued and have never started. Once a run has a slot it is on its
way to executing.
## Test plan
- [x] `pnpm run test --filter @internal/run-engine
./src/engine/tests/ttl.test.ts` — 15 tests, including a new "Re-enqueued
runs are not expired by TTL once they have started" that locks in the
queued-and-never-started contract.
- [x] `pnpm run test --filter @internal/run-engine
./src/engine/tests/delays.test.ts` — 5 tests, including "Delayed run
with a ttl" which now also asserts the TTL is armed from queue-enter
time, not `createdAt`.
- [x] `pnpm run test --filter @internal/run-engine
./src/engine/tests/lazyWaitpoint.test.ts` — 12 tests.
- [x] `pnpm run typecheck --filter @internal/run-engine`.
|
||
|
Eric Allam
|
a5ba406530 |
feat(webapp,redis): handle UNBLOCKED during ElastiCache role change (#3549)
## Summary When ElastiCache demotes a primary to replica — during a Multi-AZ failover or a vertical node-type change — the demoting primary issues an `UNBLOCKED` reply to any in-flight blocking commands (`BLPOP`, `BRPOP`, `BLMOVE`, `XREADGROUP ... BLOCK`, etc.) to clear them before the role flips. ioredis surfaces these as `ReplyError` to caller code. The shared `defaultReconnectOnError` added in #3548 only matches `READONLY` and `LOADING`. This extends it to `UNBLOCKED` so the disconnect-reconnect-retry cycle handles BLPOP-shaped errors the same way the existing two cases handle non-blocking-command errors. ## Fix ```ts export function defaultReconnectOnError(err: Error): boolean | 1 | 2 { const msg = err.message ?? ""; if ( msg.startsWith("READONLY") || msg.startsWith("LOADING") || msg.startsWith("UNBLOCKED") ) { return 2; } return false; } ``` Returning `2` tells ioredis to disconnect, reconnect, and re-issue the command. For a BLPOP that means a fresh BLPOP against the new primary instead of the `UNBLOCKED` error escaping to the caller. ## Test plan - [ ] CI green - [ ] Trigger a Multi-AZ failover or a vertical scale event on an ElastiCache replication group whose clients are running blocking commands and confirm no `UNBLOCKED` errors surface to caller code during the cutover. |
||
|
Eric Allam
|
567e2a2c32 |
feat(webapp,redis): handle READONLY / LOADING during ElastiCache failover (#3548)
## Summary
During an ElastiCache role swap (failover) or node-type change (vertical
scale), the ioredis TCP/TLS connection stays open but the server starts
answering with `READONLY` (the client is talking to a node that became a
replica) or `LOADING` (node still loading data from disk). Without an
explicit hook, those errors surface to caller code as `ReplyError`
instances — every write op on the affected connection fails until the
cluster fully cuts over.
This PR adds `reconnectOnError` to every prod ioredis client so the
disconnect + reconnect + retry cycle absorbs these errors and caller
code never sees them.
## Fix
```ts
export function defaultReconnectOnError(err: Error): boolean | 1 | 2 {
const msg = err.message ?? "";
if (msg.startsWith("READONLY") || msg.startsWith("LOADING")) return 2;
return false;
}
```
Returning `2` tells ioredis to disconnect, reconnect, and re-issue the
failed command. After reconnect, DNS / SG state routes the new socket to
a writable node.
The helper lives in `@internal/redis` and is wired into both the shared
`createRedisClient` (which covers RunQueue, schedule-engine,
redis-worker, and every other internal-package consumer) and the direct
`new Redis(...)` call sites in the webapp.
V1-only marqs files are intentionally not migrated.
## Test plan
- [x] `pnpm run typecheck --filter webapp`
- [x] `pnpm run typecheck --filter @internal/run-engine`
- [x] Verified end-to-end against a live ElastiCache vertical-scale
event — caller-surfaced errors went from tens of thousands during the
cutover window down to a handful per ioredis client
- [ ] Confirm steady-state behavior unchanged after deploy
|
||
|
Matt Aitken
|
62e006617e |
fix(cli): fail attempt on uncaught exception instead of hanging to maxDuration (TRI-9117) (#3529)
⚒️ Publish Worker (v4) / build (supervisor) (push) Has been cancelled
When a Node EventEmitter (e.g. node-redis) emits an "error" event with
no
listener attached, Node escalates it to process.on("uncaughtException")
in
the task worker. The worker reported the error via the
UNCAUGHT_EXCEPTION
IPC event but did not exit, and the supervisor-side handler in
taskRunProcess only logged the message at debug level — leaving the
run()
promise orphaned until maxDuration fired and producing empty attempts
(durationMs=0, costInCents=0).
The supervisor now rejects the in-flight attempt with an
UncaughtExceptionError and gracefully terminates the worker (preserving
the OTEL flush window) on UNCAUGHT_EXCEPTION. The attempt fails fast
with
TASK_EXECUTION_FAILED, surfacing the original error name, message, and
stack trace, and falls under the normal retry policy. This mirrors the
existing indexing-side behavior in indexWorkerManifest. Apply the same
handling to unhandled promise rejections, which Node already routes
through uncaughtException by default.
|
||
|
Eric Allam
|
386b4f65ff |
feat(webapp): per-org S2 basin migration (#3516)
## Summary Move from a single shared S2 basin to **per-org basins** with retention tied to the org's billing plan. Stops S2 from deleting streams out from under live chat sessions when basin retention fires before the chat ends, and unlocks per-org cost attribution. OSS / s2-lite installs are unaffected: provisioning is gated by `REALTIME_STREAMS_PER_ORG_BASINS_ENABLED` (default `false`), and the read precedence falls back to the global basin env var when an entity has no stamped basin. ``` basin = run.streamBasinName ?? session.streamBasinName ?? env.REALTIME_STREAMS_S2_BASIN ``` ## Design Three nullable `streamBasinName` columns (`Organization`, `TaskRun`, `Session`) plus a provisioner that idempotently creates the basin and reconfigures retention on plan changes. The trigger and session-create paths stamp the org's basin onto new rows; the realtime read path picks the basin from the entity context. Admin routes back-fill existing orgs and force-reconfigure a single org. ## Test plan - [x] `pnpm run typecheck --filter webapp --filter @internal/run-engine` - [x] Backfill admin route end-to-end (provision + DB stamp + S2 basin config). - [x] Reconfigure on plan change (all retention tiers). - [x] chat.agent multi-turn drives streams into the per-org basin. - [x] Legacy fallback when entity has no stamped basin. - [x] Provisioner is a no-op when the flag is off. |
||
|
James Ritchie
|
45ec23cc73 |
feat(webapp): app auto session logout (#3473)
<img width="2284" height="2028" alt="CleanShot 2026-05-01 at 18 53 50@2x" src="https://github.com/user-attachments/assets/4f58cbb1-0168-40fb-a523-017f2ba625a1" /> ## Performance - **Per-request DB hit**: `getUserId` runs `getEffectiveSessionDuration` (User lookup + Org `aggregate`) on *every* authenticated request, including each fetcher poll. Consider caching the effective duration in the session cookie with a short TTL (e.g. 60s) and revalidating in the background. - **Double session commit in `root.tsx`**: `getUser` already runs the expiry check; then `commitAuthenticatedSessionLazy` commits the cookie again. Fine, but doubles `Set-Cookie` headers on every page load — worth a quick perf check. ## Correctness / Edge cases - **Lazy backfill assumes a root.tsx hit first**: users whose first post-deploy request is a fetcher/API route (`/resources/*`) skip the backfill until they navigate to a page. Not a security hole, but `getUserId` could backfill itself for completeness. - **No upper bound on `Organization.maxSessionDuration`**: admin API accepts `1` second, which would instant-logout every member on next request. Add a `min(60)` (or `min(300)` to match the lowest user option) to the Zod schema. - **No clock-skew tolerance**: `isSessionExpired` is exact-millisecond. Multi-instance deploys with skewed clocks could log users out a few seconds early/late. Probably fine for the 5-min minimum, but worth noting. ## Security - **Auto-logout audit log lacks IP/orgId**: HIPAA forensics typically wants source IP and which org context. Currently logs only `userId` + path. IP isn't PII for audit purposes; orgIds help correlate. Add both. - **Cookie `Max-Age` is 1 year regardless of user's setting**: intentional (server-side `issuedAt` is the source of truth), but reviewers will ask. Add a one-line comment on the cookie config explaining why. ## API surface - **`maxSessionDuration` is admin-PAT only**: no in-app UI for org owners to set/change their own cap. If this is "Trigger staff sets it during HIPAA onboarding", say so in the PR description; otherwise add an org-settings UI. - **Auto-submit dropdown has no confirmation**: misclicking "5 minutes" immediately shortens the user's session window with no undo. Consider a save button or 3-sec undo toast. ## Schema / migration - **`User.sessionDuration NOT NULL DEFAULT 31556952`**: instant on PG 11+ (metadata-only), but call out in the PR description so reviewers don't worry about a table rewrite on the User table. - **No DB-level constraint matching `SESSION_DURATION_OPTIONS`**: if the option list changes, existing users keep orphaned values. The dropdown's tag-along behaviour hides this — fine for now, but if you ever drop an option you'll need a backfill. ## UX - **Session expiry only fires on next request**: an idle authenticated tab keeps showing UI past the cap (until SSE/polling catches it, ~60s). Add a client-side timer based on the user's effective duration that triggers a fetcher to `/account` or `/logout` at expiry. - **No "you were signed out" message on logout**: users hitting their cap are bounced to `/` with no explanation. Was intentionally reverted in this PR — call that out so reviewers don't request it. ## Tests - Unit coverage on `sessionDuration.server.ts` is solid (215 lines). Missing: integration test for `getUserId` → expired session → redirect to `/logout`, and one for the loader's clamping fix (the most recent bug). Add at least the second one to lock in the regression. --------- Co-authored-by: Matt Aitken <matt@mattaitken.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
nicktrn
|
ee3887a321 |
feat(webapp): configurable deploy template machine presets (#3492)
The webapp's compute template creation hardcoded a single machine preset (`small-1x`) at deploy time, regardless of which presets a project actually uses. Tasks running on any other preset paid full cold-snapshot creation cost on first run. Two new env vars: - `COMPUTE_TEMPLATE_MACHINE_PRESETS` - CSV of preset names to build boot snapshots for during deploy. Defaults to `small-1x` so existing deploys don't change behavior. - `COMPUTE_TEMPLATE_MACHINE_PRESETS_REQUIRED` - CSV of presets whose failure fails a required-mode deploy. Defaults to the full `PRESETS` list. Optional preset failures are logged but don't block the deploy. The compute client now sends the multi-config request shape; the service evaluates per-preset outcomes against the required set and surfaces a combined failure message when a required preset fails. Both env vars are validated at boot via the env schema - unknown preset names or `_REQUIRED` entries that aren't a subset of `_PRESETS` fail loudly at startup rather than silently per-deploy. |
||
|
Eric Allam
|
ac7177d61f |
feat(schedule-engine): stop persisting per-tick schedule state (#3476)
## Summary
Each scheduled-task tick previously issued **3 Prisma `UPDATE`s**
against
`TaskSchedule.lastRunTriggeredAt`,
`TaskScheduleInstance.lastScheduledTimestamp`,
and `TaskScheduleInstance.nextScheduledTimestamp`. All three were pure
denormalization — every value can be derived without persisting.
After this PR `TaskSchedule` and `TaskScheduleInstance` become **near
read-only**:
writes happen only on schedule create / update / delete (rare admin
actions),
so the per-tick autovacuum churn on these hot tables disappears.
## Design
The previous fire time travels forward through the **schedule worker
payload**,
not through the database. Concretely:
- The `schedule.triggerScheduledTask` worker payload gains an optional
`lastScheduleTime: z.coerce.date().optional()` field.
- When the engine fires a schedule, it re-enqueues the next tick with
`lastScheduleTime = scheduleTimestamp` (the just-fired time).
- When the next tick dequeues, `payload.lastTimestamp` is sourced from
`params.lastScheduleTime` directly. No DB round-trip, no cron-derivation
drift across DST boundaries, no caveats around recently-edited cron
expressions.
`payload.lastTimestamp` keeps its `Date | undefined` SDK shape.
First-ever
fires still report `undefined`, so customer `if
(!payload.lastTimestamp)`
first-run patterns keep working.
For Redis jobs that were enqueued **before** this change (which lack
`lastScheduleTime` in their payload), the engine falls back to
`instance.lastScheduledTimestamp` once. Once those drain, the column is
never read again. Revert is code-only; the columns stay in place and can
be dropped in a follow-up once the rollout is stable.
## Files
- `internal-packages/schedule-engine/*` — engine refactor,
`workerCatalog`
schema field, `TriggerScheduleParams` extension, tests updated to assert
on the worker-payload flow rather than DB readbacks.
- `internal-packages/database/prisma/schema.prisma` — `/// @deprecated`
triple-slash docstrings on the three columns. No migration.
- `apps/webapp/app/presenters/v3/ScheduleListPresenter.server.ts` —
drops
the `lastRunTriggeredAt` Prisma select; "Last run" cell is approximated
from the cron expression's previous slot, gated on `schedule.createdAt`
so brand-new schedules show "–". UI is best-effort; the runs page is the
source of truth.
- `apps/webapp/app/v3/utils/calculateNextSchedule.server.ts` — adds a
`previousScheduledTimestamp` helper for the UI cell above. Public API
responses (`api.v1.schedules.*`) already compute `nextRun` from cron and
don't expose `lastTimestamp` — no public API change.
- `references/scheduled-tasks/` — new reference project with declarative
schedules at multiple cadences and three throw-on-fail validators
(`first-fire-detector`, `interval-validator`, `upcoming-validator`) for
E2E-verifying the worker-payload flow.
Refs TRI-8891
## Test plan
- [x] `pnpm run typecheck --filter @internal/schedule-engine --filter
webapp`
- [x] `pnpm run build --filter @trigger.dev/core`
- [x] `pnpm run test --filter @internal/schedule-engine` — integration
test
asserts first-fire `lastTimestamp === undefined`, second fire carries
the previous fire's timestamp exactly.
- [x] E2E against local webapp via `references/scheduled-tasks`:
- Fresh schedules attached → all three deprecated columns stay `NULL`
after
multiple fires.
- Redis payload at second fire contains
`"lastScheduleTime":"<previous fire timestamp>"`.
- `TaskRun.payload` and the every-minute task's returned output both
confirm
`lastTimestamp = null` on first fire and `lastTimestamp = <prev fire>`
on
second fire, exactly 60s apart.
- All three throw-on-FAIL validators completed successfully on every
non-first fire.
- [x] Schedules REST API end-to-end (`POST` / `GET` / `PUT` / `activate`
/
`deactivate` / `DELETE`) — `nextRun` recomputed live from cron + tz on
every response, no reads of deprecated columns.
|
||
|
Eric Allam
|
dac9c83bdc |
chore(webapp,run-engine): downgrade boundary log noise to warn (#3462)
## Summary
Several boundary catches and customer-input validation paths were
logging at `error` level for failures the system already handles
gracefully — disconnect on auth failure, return undefined, skip retries,
etc. This batch routes them to `warn` (which stays in stdout) or counts
them as OTel metrics, so visibility is preserved without surfacing them
as alerts.
## Changes
**New helper / pattern:**
- `apiBuilder.server.ts` — `logBoundaryError(message, error, url)`
inspects the inner error type at loader/action boundary catches;
downgrades to `warn` for `AbortError`, `ServiceValidationError`, and
`EngineServiceValidationError`.
- `platform.v3.server.ts` — `platform_client.failures_total` OTel
counter with `{function, kind}` labels; helper
`recordPlatformFailure(fn, kind)` replaces the previous error-level
logging across all `BillingClient` wrappers.
**Log-level downgrades:**
- `handleSocketIo.server.ts` — `Worker authentication failed` → warn
(system disconnects on failure; refs TRI-8863)
- `waitpointSystem.ts` — when `runStatus === "CANCELED"` in the
suspended-without-checkpoint branch, skip the throw and warn instead
(benign cancel-vs-resume race, nothing to resume)
- `runAttemptSystem.ts` — `flushedMetadata` parse/validate failures →
warn (customer-side data shape, system returns gracefully)
- `batch-queue/index.ts` — final-attempt failures with
`result.skipRetries` → warn (callbacks already opted out of retry, e.g.
queue size limit hit)
- `queryPerformanceMonitor.server.ts` — slow queries → warn
(observability signal, not an application error)
- `timeoutDeployment.server.ts` — deployment-state mismatch in the
timeout job → warn (timeout-vs-completion race)
**Inner error preservation:**
- `waitpointCompletionPacket.server.ts` — `logger.error(uploadError)`
before throwing the `ServiceValidationError` wrapper, so the underlying
upload error stays visible.
## Why
The pattern across all of these is the same: a boundary log treated any
thrown/returned error as `error` regardless of cause, even when the
cause was an expected, system-handled condition (client disconnect,
customer quota, race condition, schema validation of customer data).
That made the logs noisy and made it harder to spot real bugs.
Where the underlying signal is still useful operationally (slow queries,
billing call failures), we route it to OTel metrics with low-cardinality
labels so dashboards and alerts can be tuned independently of error
logs.
## Test plan
- [ ] `pnpm run typecheck --filter webapp`
- [ ] `pnpm run build --filter @internal/run-engine`
- [ ] Trigger a run on hello-world and verify task lifecycle is
unaffected
- [ ] Cancel a suspended run and verify the cancel-while-suspended
branch in `waitpointSystem.ts` returns `{status: "skipped"}` instead of
throwing
- [ ] Confirm `platform_client.failures_total` counter shows up in
metrics with `{function, kind}` labels when the billing client errors
|
||
|
Eric Allam
|
c69e939c34 |
feat: Sessions - bidirectional durable agent streams (#3417)
> ⚠️ **Not released yet.** This PR is the server-side foundation only. The SDK changes that customers will actually use (`chat.agent` migration, `chat.createStartSessionAction`, `useTriggerChatTransport` updates) live on a separate branch and ship together in an upcoming `@trigger.dev/sdk` prerelease. Until that prerelease is published, this surface is reachable only via direct HTTP. ## What this gives Trigger.dev users A new first-class primitive, **Session**, for durable, task-bound, bidirectional I/O that outlives any single run. Sessions are the run manager for `chat.agent` going forward, and they unblock anything else that needs "one identifier, many runs over time" with a stable channel pair the client can write to and subscribe to. ### Use cases unblocked - **Chat agents that persist across many runs.** One session per chat (keyed on your own `chatId` via `externalId`), turns 1..N attach to the same Session, the UI subscribes once and keeps receiving output as new runs take over. - **Approval loops and long-running tasks with user feedback.** The task waits on `.in`, the client writes to `.in`, the server enforces no-writes-after-close. - **Workflow progress streams that live past the run.** Subscribe to `.out` after the task finishes to replay history. - **Resume-next-day flows.** A session is a durable row, not a transient stream. Send a message a day later and the server triggers a fresh run on the same session. ### How it works (Session-as-run-manager) A Session row is task-bound (`taskIdentifier` + `triggerConfig` are required) and owns its current run via `currentRunId` + `currentRunVersion` for optimistic claim. Three trigger paths: 1. **Session create** — `POST /api/v1/sessions` creates the row and triggers the first run synchronously. 2. **Append-time probe** — `POST /realtime/v1/sessions/:session/in/append` checks if the current run is alive; if it has terminated (idle exit, crash, etc.), the server triggers a new run before processing the append. 3. **End-and-continue handoff** — `POST /api/v1/sessions/:session/end-and-continue`, called by the running agent, triggers a fresh run and atomically swaps `currentRunId`. Used by `chat.requestUpgrade()` for version handoffs. Every triggered run is recorded in the `SessionRun` audit table with a reason (`initial`, `continuation`, `upgrade`, `manual`). ## Public API surface ### Control plane - `POST /api/v1/sessions` — create. Idempotent on `(env, externalId)`. Triggers the first run, returns the session and a session-scoped public access token. Returns 409 if the upserted row is already closed. - `GET /api/v1/sessions/:session` — retrieve by friendlyId (`session_abc...`) or by your own externalId (server disambiguates by prefix). - `GET /api/v1/sessions` — list with filters (`type`, `tag`, `taskIdentifier`, `externalId`, derived `status` ACTIVE/CLOSED/EXPIRED, created-at range) and cursor pagination. Backed by ClickHouse. - `PATCH /api/v1/sessions/:session` — update tags / metadata / externalId. - `POST /api/v1/sessions/:session/close` — terminate. Idempotent, hard-blocks new server-brokered writes. - `POST /api/v1/sessions/:session/end-and-continue` — agent-only handoff to a fresh run. ### Realtime - `PUT /realtime/v1/sessions/:session/:io` — initialize a channel. Returns S2 credentials in headers so high-throughput clients can write direct to S2. - `GET /realtime/v1/sessions/:session/:io` — SSE subscribe. Supports Last-Event-ID resume and an opt-in `X-Peek-Settled: 1` header that fast-closes the stream when the upstream is already settled (`trigger:turn-complete`), eliminating long-poll wait on reconnect-on-reload paths. - `POST /realtime/v1/sessions/:session/:io/append` — server-side appends. - `POST /api/v1/runs/:runFriendlyId/session-streams/wait` — runs wait on a session stream as a waitpoint, with a race-check to avoid suspending if data already landed. ### Auth scopes `sessions` is a new resource type. `read:sessions:{id}`, `write:sessions:{id}`, `admin:sessions:{id}` flow through the existing JWT validator. Session-scoped public access tokens minted by the server replace browser-held trigger-task tokens for chat-style flows — the browser never sees a run identifier or a run-scoped token in steady state. ## What's coming after this PR - **SDK + chat.agent migration**: separate branch, separate PR, ships in the next `@trigger.dev/sdk` prerelease alongside this server deploy. Customers using the prerelease `chat.agent` will follow the [upgrade guide](https://github.com/triggerdotdev/trigger.dev/blob/docs/tri-7532-ai-sdk-chat-transport-and-chat-task-system/docs/ai-chat/upgrade-guide.mdx). - **Dashboard surfaces**: dedicated agent list, agent playground, agent view on the run dashboard. Tracking separately. ## Implementation notes - **Postgres `Session` table**: scalar scoping columns (`projectId`, `runtimeEnvironmentId`, `environmentType`, `organizationId`) without FKs, matching the January TaskRun FK-removal decision. Point-lookup indexes only — list queries go to ClickHouse. Terminal markers (`closedAt`, `expiresAt`) are write-once. - **ClickHouse `sessions_v1`**: ReplacingMergeTree, partitioned by month, ordered by `(org_id, project_id, environment_id, created_at, session_id)`. Tags indexed via `tokenbf_v1` skip index. - **`SessionsReplicationService`**: mirrors `RunsReplicationService` exactly — leader-locked logical replication consumer, `ConcurrentFlushScheduler`, retry with exponential backoff + jitter, identical metric shape. Dedicated slot + publication so the two consume independently. - **S2 keys**: `sessions/{addressingKey}/{out|in}`. The existing `runs/{runId}/{streamId}` key format for run-scoped streams is untouched. - **Optimistic claim**: `ensureRunForSession` triggers a run upfront (cheap to cancel if it loses the race), then attempts an `updateMany` keyed on `currentRunVersion`. Loser cancels its triggered run and reuses the winner's. No DB lock held across the trigger. ### What did NOT change Run-scoped `streams.pipe` / `streams.input` and the existing `/realtime/v1/streams/{runId}/...` routes are unchanged. Sessions are net-new — not a reshaping of the current streams API. ## Deploy notes - Set `SESSION_REPLICATION_CLICKHOUSE_URL` and `SESSION_REPLICATION_ENABLED=1` to enable the replication consumer. - The `Session` table needs `REPLICA IDENTITY FULL` set on the prod source DB before the publication is created (same one-time DDL we did for `TaskRun`). Required for delete events to carry full column values. - Cross-form authorization on the `GET /api/v1/sessions/:session` loader (a JWT minted for either form authorizes both URL forms). Action routes are URL-form-specific, matching how the SDK mints PATs. ## Verification - Webapp typecheck clean (10/10). - `apps/webapp/test/sessionsReplicationService.test.ts` — round-trip tests for insert/update/delete through Postgres logical replication into ClickHouse via testcontainers. - Live end-to-end against local dev: create + retrieve (both forms) + update + close, `.out.initialize` + `.out.append` x2 + `.in.send` + `.out.subscribe` over SSE, list with all filter combinations + pagination, `end-and-continue` swap, `X-Peek-Settled` fast-close (verified in browser via reconnect-on-reload and via curl). Replicated row lands in ClickHouse within ~1s. - Multi-round Devin + CodeRabbit review feedback addressed (read-after-write paths use `prisma` writer, info-leak on auth-routes masked as 403, peek-settled discriminator parsing fix, etc.). ## Test plan - [ ] `pnpm run typecheck --filter webapp` - [ ] `pnpm run test --filter webapp ./test/sessionsReplicationService.test.ts --run` - [ ] Start the webapp with `SESSION_REPLICATION_CLICKHOUSE_URL` and `SESSION_REPLICATION_ENABLED=1`. Confirm the slot and publication auto-create on boot. - [ ] `POST /api/v1/sessions` and verify the row replicates to `trigger_dev.sessions_v1` within a couple of seconds. - [ ] `POST /api/v1/sessions/:id/close`, then confirm `POST /realtime/v1/sessions/:id/out/append` returns 400. - [ ] Reuse a closed session's `externalId` on `POST /api/v1/sessions` and confirm 409. - [ ] `GET /realtime/v1/sessions/:id/out` with `X-Peek-Settled: 1` after a turn completes and confirm `X-Session-Settled: true` response header + immediate close. |
||
|
Eric Allam
|
e134da7306 |
fix(run-engine): debounce hot-key lock contention and 5xx feedback loop (#3453)
## Changes
Three changes in
`internal-packages/run-engine/src/engine/systems/debounceSystem.ts`, in
order of impact:
1. **Fast-path skip before the lock.** In `handleExistingRun`, do an
unlocked read of `delayUntil` (and `createdAt` for the max-duration
check) from the run row before entering `runLock.lock("handleDebounce",
...)`. If `newDelayUntil <= currentDelayUntil` and the run is still
within its max-duration window, return the existing run immediately
without taking the lock. Safe because debounce is monotonic-forward only
— a stale read either matches reality or undershoots, both of which
decay correctly (re-checked properly inside the lock by whichever caller
is actually pushing forward). Trailing-mode triggers carrying
`updateData` still take the lock so the data update is applied.
2. **Quantize `newDelayUntil`.** Round the computed `newDelayUntil` to
1-second buckets (configurable via `quantizeNewDelayUntilMs`, set to 0
to disable). Without quantization, every call has a slightly larger
`newDelayUntil` than the last and they all pass the fast-path check.
With it, concurrent callers on the same key share a target time and ~95%
short-circuit. User-visible effect: a debounced run might fire up to 1s
earlier than the strict spec — non-issue for typical debounce use cases
(chat summarization, batched notifications, etc.).
3. **Graceful lock-contention fallback.** Wrap the `runLock.lock(...)`
call so `LockAcquisitionTimeoutError` and Redlock `ExecutionError` /
`ResourceLockedError` return the existing run id with success instead of
propagating a 5xx. Debounce is best-effort: if we can't take the lock,
the herd is already updating it for us; fall in line. This kills the 5xx
→ SDK-retry feedback loop. With (1)+(2) this rarely fires; without them
it's the difference between 5xx and 200.
Defaults preserve current behaviour aside from quantization (1s) and
fast-path (on). Both are configurable via `RunEngineOptions.debounce`.
## ✅ Checklist
- [x] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [x] The PR title follows the convention.
- [x] I ran and tested the code works
---
## Changelog
Reduce 5xx feedback loops on hot debounce keys by quantizing
`delayUntil`, adding an unlocked fast-path skip before the redlock, and
gracefully handling redlock contention in `handleDebounce` so the SDK no
longer retries into a herd.
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
|
||
|
devin-ai-integration[bot]
|
4b28080ed4 |
feat: add isReplay to run context (#3454)
## Summary Adds `isReplay` boolean to the run context (`ctx.run.isReplay`), following the same pattern as the existing `isTest`. The value is derived from the existing `replayedFromTaskRunFriendlyId` database field, so no schema migration is needed. ## ✅ Checklist - [x] I have followed every step in the [contributing guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md) - [x] The PR title follows the convention. - [x] I ran and tested the code works --- ## Testing - Verified `@trigger.dev/core` builds successfully - Verified `webapp` typechecks successfully - All new fields use `default(false)` for backwards compatibility --- ## Changelog - Added `isReplay` to `TaskRun` and `V3TaskRun` schemas in `common.ts` - Added `RUN_IS_REPLAY` semantic attribute and wired it in `taskContext` - Propagated `isReplay` through the dequeue system, run attempt system, and all execution context construction paths (V1 + V2) - Added `isReplay` to `DequeuedMessage` and `TaskRunExecutionLazyAttemptPayload` schemas - Added patch changeset for `@trigger.dev/core` - Updated docs: added `isReplay` to context reference, added "Detecting replays" section to replaying page --- 💯 Link to Devin session: https://app.devin.ai/sessions/1d6f1b3cc39a4623b72d05bf00f2d70c --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: nick <55853254+nicktrn@users.noreply.github.com> |
||
|
nicktrn
|
91fd8a8a03 |
chore(security): close dependabot alerts q2 (#3456)
Closes ~80 dependabot alerts (3 critical, ~25 high, ~31 medium) by
bumping direct deps where possible and narrowly overriding the rest.
Cloud uses `resend` email transport and Node 20 - all bumps are safe for
both cloud and self-hosters.
## Direct upgrades
| Package | Where | From | To | Why |
|---|---|---|---|---|
| `vite` | root devDeps | ^5.4.21 | *(removed)* | dead pin; vitest pulls
vite transitively |
| `dompurify` | apps/webapp | ^3.2.6 | ^3.4.1 | XSS CVEs |
| `effect` | apps/webapp | ^3.11.7 | ^3.21.2 | AsyncLocalStorage CVE in
Effect fibers |
| `nodemailer` | internal-packages/emails | ^7.0.11 | ^8.0.6 | SMTP CRLF
injection (only affects self-hosters w/ smtp/aws-ses transport) |
| `uuid` | apps/webapp | ^9.0.0 | ^14.0.0 | buffer bounds check;
ESM-only but bundled by Remix |
| `uuid` + `@types/uuid` | packages/trigger-sdk | ^9.0.0 | *(removed)* |
dead deps, no usage |
| `@types/uuid` | apps/webapp | ^9.0.0 | *(removed)* | uuid 14 ships its
own types |
| `tar` | packages/cli-v3 | ^7.5.4 | ^7.5.13 | path traversal CVEs |
| `testcontainers` + `@testcontainers/postgresql` +
`@testcontainers/redis` | internal-packages/testcontainers | ^10.28.0 |
^11.14.0 | dev/test cleanup; one-line API fix for
`RedisContainer(image)` |
| `rimraf` | webapp + 6 packages | ^3.0.2 / ^5.0.7 | ^6.0.1 | dev/build
tool consolidation |
## Scoped overrides
All bound by both `>=` and `<` to avoid major-version yanks.
| Override | Closes |
|---|---|
| `tar@>=7 <7.5.11` → `^7.5.11` | supervisor's `@kubernetes/client-node
1.0.0` chain |
| `axios@>=1.0.0 <1.15.0` → `^1.15.0` | replaces older 1.9.0 pin |
| `systeminformation@>=5.0.0 <5.31.0` → `^5.31.0` | bumps existing
5.27.14 pin |
| `lodash@>=4.0.0 <4.18.0` → `^4.18.0` | bumps existing 4.17.23 pin |
| `lodash-es@>=4.0.0 <4.18.0` → `^4.18.0` | new (mirrors lodash) |
| `dompurify@>=3 <3.4.0` → `^3.4.1` | catches transitive dompurify via
mermaid |
| `vite@>=5.0.0 <6.4.2` → `^6.4.2` | path traversal; vite 5 has no patch
|
| `rollup@>=4 <4.59.0` → `^4.59.0` | path traversal in vite/vitest chain
|
| `flatted@>=3 <3.4.2` → `^3.4.2` | prototype pollution in eslint
flat-cache |
| `picomatch@>=2 <2.3.2` → `^2.3.2` | ReDoS in 2.x branch (transitive) |
| `picomatch@>=4 <4.0.4` → `^4.0.4` | ReDoS in 4.x branch
(vitest/tinyglobby) |
| `minimatch@>=3 <3.1.3` → `^3.1.3` | ReDoS in eslint 8 chain |
| `protobufjs@>=7 <7.5.5` → `^7.5.5` | **critical** RCE via
@opentelemetry/otlp-transformer |
| `fast-xml-parser@>=4 <4.5.5` → `^4.5.5` | DOCTYPE bypass + others (4.x
branch via aws-sdk in supervisor) |
| `fast-xml-parser@>=5 <5.7.0` → `^5.7.0` | **critical** + others (5.x
branch via aws-sdk in webapp) |
| `path-to-regexp@>=0.1 <0.1.13` → `^0.1.13` | ReDoS in express 4 /
@remix-run/express |
| `ajv@>=8 <8.18.0` → `^8.18.0` | DoS |
| `socket.io-parser@>=4 <4.2.6` → `^4.2.6` | DoS in @trigger.dev/core's
socket.io |
| `postcss@>=8 <8.5.10` → `^8.5.10` | XSS via stringify |
| `yaml@>=2 <2.8.3` → `^2.8.3` | DoS |
| `semver@>=5 <5.7.2` → `^5.7.2` | ReDoS in 5.x |
| `defu@>=6 <6.1.5` → `^6.1.5` | prototype pollution via __proto__ in
@prisma/config c12 chain |
## Dismissed (~47)
| Reason | Cluster | Count |
|---|---|---|
| `not_used` | langsmith + next 15.x in references/* | 10 |
| `not_used` | minimatch 8.x via prisma-generator-ts-enums
(references/prisma-6) | 3 |
| `not_used` | basic-ftp via puppeteer in references/hello-world +
references/seed | 2 |
| `not_used` | hono / @hono/node-server / express-rate-limit /
path-to-regexp 8.x / @modelcontextprotocol/sdk - all via mcp-sdk chain
(dormant in webapp; dev-only localhost in cli-v3) | 22 |
| `not_used` | fastify / @fastify/static / file-type via evalite devDep
| 5 |
| `tolerable_risk` | rollup 3 + minimatch 5/8/9/10 dev/build tooling |
13 |
## Notes
- **mcp-sdk chain**: `@vercel/sdk` in webapp imports `Vercel` API client
only; `mcp-server/*` subpath isn't loaded at runtime. cli-v3's MCP
server runs only via `trigger mcp` on developer machines. Bumping
`@modelcontextprotocol/sdk` to latest (1.29.0) wouldn't close these
alerts anyway - it ships hono ^4.11.4 which is still vulnerable - so
dismissal is the cleaner call.
- **References ignore list**: confirmed with current dependabot ignore
config; added `references/seed/package.json` (only gap).
- **undici** alerts (CVE-2026-1527, 4 alerts) will auto-close: lockfile
already at 6.25.0 > patched 6.24.0; just needs Dependabot rescan.
- **Effect 3.20 fix** is a runtime-only scheduler fix, no public API
changes - verified with research agent against our four `effect/*`
imports.
- **uuid 14** is ESM-only; we only call `validate`/`version` (no crypto
needed) so Node 20 requirement isn't load-bearing for us.
## Public packages (`packages/*`)
Minimal surface, deliberately. None of these change published runtime
behaviour - all changesets-worthy public package changes are deferred to
a regular release pass.
| Package | Change | Runtime impact |
|---|---|---|
| `packages/trigger-sdk` | Removed dead `uuid` dep (no source imports) |
None - dep was unused |
| `packages/cli-v3` | `tar` ^7.5.4 → ^7.5.13 | Patch bump within
already-allowed 7.x range; nothing CLI consumers see |
| `packages/core` / `packages/build` / `packages/python` /
`packages/rsc` / `packages/react-hooks` / `packages/schema-to-json` |
`rimraf` ^3.0.2 → ^6.0.1 in devDeps | Build-time only, no runtime change
|
No changeset added because nothing in these packages affects what
published consumers run.
## Validation
- Webapp typecheck (forced, no cache) passes after every commit
- Smoke-tested testcontainers v11 changes via real `postgresTest` +
`redisTest` (sync.test.ts, releaseConcurrency.test.ts) - both pass
- Webapp built + verified `require("uuid")` no longer in CJS server
output (now bundled inline)
- Test env webapp deployed at `dependabot-q2.rc0` (cloud#740) - no
issues observed
- Test suite run with package prerelease passed
|
||
|
Matt Aitken
|
8aa1e55588 |
test: e2e auth baseline tests + webapp testcontainer infrastructure (#3438)
Adds a minimal end-to-end test harness that spawns the compiled webapp as a child process against a throwaway Postgres container, plus a baseline of 8 auth-behaviour tests. These tests will be used as a regression check before and after the upcoming apiBuilder RBAC migration to confirm auth behaviour is unchanged. ## What's included **`internal-packages/testcontainers/src/webapp.ts`** (new) Spawns `build/server.js` with a dynamically allocated port, polls `/healthcheck`, and exposes `WebappInstance` and `startTestServer()` (postgres container + webapp + PrismaClient in one call). Key details: - Uses `process.execPath` so the correct Node binary is found in forked test processes - Sets `NODE_PATH` to `node_modules/.pnpm/node_modules` so pnpm-hoisted transitive deps (e.g. `eventsource-parser`) resolve correctly inside the subprocess - Overrides both `PORT` and `REMIX_APP_PORT` so Vite's automatic `.env` loading doesn't override the dynamically allocated port **`internal-packages/testcontainers/package.json`** Adds `./webapp` sub-path export so tests can `import from "@internal/testcontainers/webapp"`. **`internal-packages/testcontainers/src/index.ts`** Exports `createPostgresContainer` (used internally by `webapp.ts`). **`apps/webapp/test/helpers/seedTestEnvironment.ts`** (new) Creates a minimal org → project → environment row set with random suffixes. **`apps/webapp/test/api-auth.e2e.test.ts`** (new) 8 tests across two suites: - API-key bearer: valid key (auth passes, 404), missing header (401), invalid key (401), error body shape - JWT bearer: valid JWT on JWT-enabled route (passes), valid JWT on non-JWT route (401), empty-scope JWT (403), wrong signing key (401) ## How to run ```bash # Build required first (one-time) pnpm run build --filter webapp cd apps/webapp && pnpm exec vitest run test/api-auth.e2e.test.ts ``` ## Test plan - [x] All 8 tests pass against the current webapp build - [x] Webapp healthcheck returns 200 on startup - [ ] CI passes --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Matt Aitken
|
f7aefb705a |
fix: disable RunQueue Worker in priority tests to prevent partial-batch race (#3440)
## Summary - The `processMasterQueueForEnvironment` call in the priority test was racing against background `processQueueForWorkerQueue` jobs scheduled 50ms after each trigger - With a 50ms debounce (`processWorkerQueueDebounceMs: 50`) and runs triggered sequentially, the RunQueue Worker could process those jobs mid-sequence, pushing partial batches to the worker queue in the wrong overall priority order - `masterQueueConsumersDisabled: true` only blocks the shard-level polling loops — it does not prevent the RunQueue's own Worker from processing these debounced jobs - Fix: add `worker.disabled: true` to the test 1 engine config, which propagates to `workerOptions.disabled` in the RunQueue constructor and prevents the Worker from starting ## Test plan - [x] Both priority tests pass: `pnpm run test ./src/engine/tests/priority.test.ts --run` - [x] Test 1 log confirms no `✅ Starting run engine worker` or worker loop messages — workers fully disabled - [x] Test 2 unaffected (uses master queue consumers for automatic promotion, no `disabled` flag added) 🤖 Generated with [Claude Code](https://claude.com/claude-code) |
||
|
Eric Allam
|
2d3b2e82e6 |
feat(run-engine): flag to route getSnapshotsSince through read replica (#3423)
## Summary Adds `RUN_ENGINE_READ_REPLICA_SNAPSHOTS_SINCE_ENABLED` (default `"0"`). When enabled, the Prisma reads inside `RunEngine.getSnapshotsSince` run against the read-only replica client instead of the primary. Offloads the snapshot-polling queries fired by every running task runner off the writer. ## Why `getSnapshotsSince` is called from the managed runner's fetch-and-process loop (once per poll interval, plus on every snapshot-change notification). It runs four sequential reads per call — one `findFirst` by snapshot id, one `findMany` on snapshots with `createdAt > X`, one raw SQL against `_completedWaitpoints`, and chunked `findMany` on `waitpoint`. Per concurrent run, every few seconds. It's read-only, tolerates a small amount of staleness, and is an obvious candidate for the replica. ## Replica-lag considerations - **Step 1 "since snapshot not found"**: if the runner just received a snapshot id from the primary and asks the replica before it replicates, the function throws and the caller treats the response as an error (runner falls back to a metadata refresh). Self-correcting, not silent. - **Step 2 missing newly-created snapshots**: the next poll's `createdAt > sinceSnapshot.createdAt` filter still picks them up once the replica catches up. - **Waitpoint junction race**: the riskiest path — if a latest snapshot is replicated but its `_completedWaitpoints` join rows aren't yet, the runner could advance past that snapshot with `completedWaitpoints: []`. WAL/storage-level replication replays commits in order, so in practice both should appear atomically on the reader, but the race window is why the flag ships disabled. Aurora reader shrinks all three windows to single-digit ms in typical conditions, and its storage-level replication gives atomic visibility of committed transactions on the reader. ## Test plan - [ ] Flip the flag on in a non-prod environment, confirm snapshot polling behaves normally and `getSnapshotsSince` errors in Sentry stay flat. - [ ] Verify writer query volume drops and reader query volume rises on the snapshot-polling queries. - [ ] Keep an eye on `AuroraReplicaLag` (or equivalent) during rollout. |
||
|
nicktrn
|
b570586899 |
fix(webapp): allow cancelling runs in DEQUEUED status from the runs list (#3421)
The cancel button was missing from the runs list for runs in `DEQUEUED` status. The runs list gates the button on `run.isCancellable`, which goes through `isCancellableRunStatus` -> `CANCELLABLE_RUN_STATUSES` = `NON_FINAL_RUN_STATUSES`. `DEQUEUED` was never added to that list when it was introduced in the run engine. The single run page uses a separate check (`!run.isFinished`, i.e. the inverse of `FINAL_RUN_STATUSES`), so cancellation already worked there - only the list was affected. Adding `DEQUEUED` to `NON_FINAL_RUN_STATUSES` also flips `isCrashableRunStatus` and `isFailableRunStatus`, but: - The crash path is the right behaviour - a `DEQUEUED` run (worker has claimed but not yet executing) can legitimately crash before `EXECUTING`, same as `PENDING`/`DELAYED` already do. - The fail path (`failedTaskRun.server.ts`) is only reached from V1 code paths (marqs consumers, v1 heartbeat handler). `DEQUEUED` is a V2-engine-only status, so V1 consumers never see it. When cancelling a `DEQUEUED` run the execution snapshot goes to `PENDING_CANCEL` (worker must ack) but `TaskRun.status` flips to `CANCELED` immediately - the UI reflects cancellation without waiting for the worker. Added an integration test in `run-engine/src/engine/tests/cancelling.test.ts` covering the full trigger -> dequeue -> cancel -> worker-ack flow. ## Stall safety The stall recovery path (PENDING_EXECUTING heartbeat miss -> nack-and-requeue -> back to QUEUED) lives entirely inside `@internal/run-engine` and never touches the webapp's `taskStatus.ts` helpers - the engine has zero imports from `~/v3/taskStatus` and doesn't know `CrashTaskRunService` / `FailedTaskRunService` exist. A stalled DEQUEUED run still goes back to the queue for retry; this change cannot cause stalls to crash or fail. The only realistic impact is the intended UI fix - the theoretical V1 crash/fail branches for DEQUEUED are unreachable in practice because V1 runs never have DEQUEUED status. |
||
|
Eric Allam
|
03e4d5fe31 |
feat(webapp,database): API key rotation grace period (#3420)
## Summary
Regenerating a RuntimeEnvironment API key no longer immediately
invalidates the previous one. Rotation is now overlap-based: the old key
keeps working for 24 hours so customers can roll it out in their env
vars without downtime, then stops working.
## Design
- **New `RevokedApiKey` table** (one row per revocation). Holds the
archived `apiKey`, a FK to the env, an `expiresAt`, and a `createdAt`.
Indexed on `apiKey` (high-cardinality equality — single-row hits) and on
`runtimeEnvironmentId`.
- **`regenerateApiKey` wraps both writes in a single `$transaction`:**
insert a `RevokedApiKey` with `expiresAt = now + 24h`, update the env
with the new `apiKey`/`pkApiKey`.
- **`findEnvironmentByApiKey` does a two-step lookup:** primary
unique-index hit on `RuntimeEnvironment.apiKey` first; on miss,
`RevokedApiKey.findFirst({ apiKey, expiresAt: { gt: now } })` with an
`include: { runtimeEnvironment }`. Two-step (not `OR`-join) keeps the
hot path identical to today and puts the fallback cost only on invalid
keys. Both lookups use `$replica`.
- **Admin endpoint** `POST /admin/api/v1/revoked-api-keys/:id` accepts
`{ expiresAt }` and updates the row. Setting to `now` ends the grace
window immediately; setting to the future extends it.
- **Modal copy** on the regenerate dialog updated — previously warned of
downtime, now explains the 24h overlap.
## Why a separate table instead of columns on `RuntimeEnvironment`
- Keeps the hot auth path's primary lookup unchanged — no
OR/nullable-apiKey semantics to reason about.
- Naturally supports multiple in-flight grace windows (regenerate twice
in a day → two old keys valid until their independent expiries).
- FK + cascade cleans up correctly when an env is deleted; nothing to
backfill.
## Test plan
Verified locally against hello-world with dev and prod env keys:
- [x] baseline — current key authenticates (`GET /api/v1/runs`) → `200`
- [x] regenerate via UI — DB shows old key in `RevokedApiKey` with
`expiresAt ≈ now+24h`, env has new key
- [x] grace window — both old and new keys → `200`; bogus key → `401`
- [x] admin endpoint: `expiresAt = now` → old key `401`
- [x] admin endpoint: `expiresAt = +1h` (after early-expire) → old key
`200` again
- [x] admin endpoint: `expiresAt = past` → old key `401`
- [x] admin 400 (invalid body), 404 (unknown id), 401 (missing/non-admin
PAT)
- [x] same flow exercised end-to-end on a PROD-typed env — behavior
identical
- [x] `pnpm run typecheck --filter webapp` passes
|
||
|
Iss
|
7d7ebdde52 | feat: Increase default project limit per org from 10 to 25 (#3409) | ||
|
devin-ai-integration[bot]
|
ff290dfe2f |
perf(run-engine): merge dequeue snapshot creation into taskRun.update transaction [TRI-8450] (#3395)
## Summary
Nests the `TaskRunExecutionSnapshot` creation inside the
`taskRun.update()` Prisma call in the dequeue flow, reducing **2 DB
commits → 1** per dequeue operation. This is the highest-volume of the
five unmerged flows identified in TRI-8450 (~9,200 commits/sec on the
engine service).
**Pattern**: Follows the same nested-write approach already used in the
completion path (`runAttemptSystem.ts:735`) and trigger path
(`engine/index.ts:674`).
**Changes**:
- `dequeueSystem.ts`: Moved snapshot creation into `executionSnapshots:
{ create: {...} }` within the existing `taskRun.update()`. Pre-generates
the snapshot ID via `generateInternalId()` (plain cuid, matching what
Prisma's `@default(cuid())` produces) so the event emission, heartbeat
enqueue, and return value can all be constructed from data already in
scope — **no extra DB read needed** after the merged write.
`SnapshotId.toFriendlyId()` is used only for the return value's
`friendlyId` field, matching the original `createExecutionSnapshot`
behavior.
- `executionSnapshotSystem.ts`: Added public
`enqueueHeartbeatIfNeeded()` method that exposes the heartbeat
scheduling logic (previously only available internally via
`createExecutionSnapshot`). This is needed because `PENDING_EXECUTING`
requires a heartbeat, unlike the `FINISHED` status in the completion
reference pattern. This method is reusable by future merge targets
(retry-immediate, checkpoint, cancel, requeue).
**Net DB change per dequeue**: eliminates 1 write transaction (the
separate `TaskRunExecutionSnapshot.create`). No extra reads added — the
snapshot ID is pre-generated and the `executionSnapshotCreated` event
payload is constructed inline from values already available in the
closure.
## Review & Testing Checklist for Human
- [ ] **Verify manually-constructed event payload matches DB state**:
The `executionSnapshotCreated` event is now built inline (not read back
from DB). Confirm the field values (`runStatus: "PENDING"`,
`attemptNumber`, `checkpointId`, `workerId`, `runnerId`,
`completedWaitpointIds`) match what Prisma actually writes. A mismatch
here would be silent — event consumers would get stale/wrong data.
- [ ] **Verify `attemptNumber` source is equivalent**: Old code used
`lockedTaskRun.attemptNumber` (post-update result). New code uses
`result.run.attemptNumber` (pre-update). The `taskRun.update()` data
payload does NOT include `attemptNumber`, so they should be identical —
but confirm this assumption holds for all dequeue scenarios (e.g.
retried runs).
- [ ] **Verify `isValid` defaults to `true` in schema**: The old
`createExecutionSnapshot` explicitly set `isValid: error ? false :
true`. The nested create omits `isValid` (no error in the dequeue happy
path). Confirm the Prisma schema default for
`TaskRunExecutionSnapshot.isValid` is `true`.
- [ ] **Verify `runStatus: "PENDING"` hardcoding matches the mapping**:
The old code passed `lockedTaskRun.status` ("DEQUEUED") to
`createExecutionSnapshot`, which mapped it to "PENDING" via `run.status
=== "DEQUEUED" ? "PENDING" : run.status`. The new code hardcodes
`"PENDING"` directly. This is correct but brittle if `status` ever
changes from "DEQUEUED" to something else upstream.
- [ ] **Spot-check `completedWaitpoints` connect + order logic**: The
nested create replicates the connect/order logic from
`createExecutionSnapshot` (lines 387-393). Verify the
`snapshot.completedWaitpoints` type provides `id` and `index` fields
compatible with this usage.
- [ ] **Verify `checkpoint` in return value**: The return now uses
`snapshot.checkpoint` (from the *previous* snapshot) instead of reading
the newly-created snapshot's checkpoint relation. Since `checkpointId`
is passed through unchanged, they should be identical — but worth a
sanity check.
**Recommended test plan**: deploy to staging, run the
`sample_pg_activity.py` sampler for a 5-minute window, and verify the
COMMIT count drop on the engine service + proportional `IO:XactSync`
reduction.
### Notes
- This only covers the **dequeue** flow (flow #1 from TRI-8450). The
remaining four flows (retry-immediate, checkpoint, requeue, cancel) are
separate follow-ups.
- The new `enqueueHeartbeatIfNeeded` method is deliberately designed for
reuse by those follow-up PRs.
- CI note: the `priority.test.ts` failure in shard 7 is a flaky ordering
assertion unrelated to this change (it compares `friendlyId` values in
dequeue order). The `audit` check is also pre-existing/unrelated.
Link to Devin session:
https://app.devin.ai/sessions/034fe0e7224f49278a2de260203e1377
Requested by: @ericallam
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Eric Allam <eallam@icloud.com>
|
||
|
Eric Allam
|
79b6053e13 |
feat(server): add TaskIdentifier registry to replace expensive distinct query (#3368)
Replace the expensive DISTINCT query for task filter dropdowns with a dedicated TaskIdentifier registry table backed by Redis. Environments migrate automatically on their next deploy, with a transparent fallback to the legacy query for unmigrated environments. Also fixes duplicate dropdown entries when a task changes trigger source, and adds active/archived grouping for removed tasks. Moves BackgroundWorkerTask reads in the trigger hot path to the read replica. |
||
|
Eric Allam
|
7c95207486 | fix(run-engine): Stop querying for associated run tags during dequeue (#3379) | ||
|
Eric Allam
|
73ea5865d8 |
fix(run-engine): distinguish oneTimeUseToken P2002 from idempotency key collision (#3374)
Prevent retrying when retrying won’t actually do any good |
||
|
Eric Allam
|
f739a5c545 | fix(db): add index to ProjectAlertStorage to prevent sequence scans (#3349) | ||
|
Eric Allam
|
ed0c3e4f38 |
fix: stop creating TaskRunTag records and join table entries during triggering (#3369)
The TaskRun.runTags string array already stores tag names, making the TaskRunTag M2M relation redundant write overhead. Remove createTags calls, connect: tags, and join table writes from both V1 and V2 trigger paths. Simplify the add-tags API to just push to runTags directly. |
||
|
Eric Allam
|
417ab876e3 | fix(batch-queue): Batch items that hit the environment queue size limit now fast-fail (#3352) | ||
|
nicktrn
|
e59614a31c |
feat(webapp): gate microvm regions behind compute access feature flag (#3366)
Adds region-level gating so MICROVM regions are only visible and usable by orgs with the `hasComputeAccess` feature flag. Admins and explicit allowlist behavior unchanged. - New shared helper (`regionAccess.server.ts`) with `resolveComputeAccess`, `defaultVisibilityFilter`, and `isComputeRegionAccessible` - `RegionsPresenter` filters out MICROVM regions for non-compute orgs - `SetDefaultRegionService` blocks setting a MICROVM region as default without compute access - `WorkerGroupService` blocks triggering runs in MICROVM regions without compute access - `computeTemplateCreation` refactored to use shared `resolveComputeAccess` - Updated snapshot callback schema |
||
|
Matt Aitken
|
0e14b6d750 |
TaskRun optimizations: dropping FKs and some indexes (#3309)
## Summary - Drop all 8 foreign key constraints on TaskRun. The run listing path is now fully ClickHouse-backed so we no longer need Postgres to enforce referential integrity on this table. The FK constraints add write overhead on every insert/update with no remaining benefit. Prisma queries are unaffected. - Remove PostgresRunsRepository and its associated feature flag (runsListRepository), which was the last remaining code path querying TaskRun directly for list/count operations. - Drop three indexes that were only useful for the Postgres run list path and have no remaining query consumers: - TaskRun_runtimeEnvironmentId_id_idx — was the cursor pagination index for PostgresRunsRepository; superseded by the (runtimeEnvironmentId, createdAt DESC) composite index - TaskRun_scheduleId_idx — redundant with the (scheduleId, createdAt DESC) composite index; no direct Postgres queries filter by scheduleId alone - TaskRun_rootTaskRunId_idx — no queries filter TaskRun by rootTaskRunId as a WHERE clause anywhere in the codebase All index drops use CONCURRENTLY IF EXISTS to avoid table locks in production. ## Test plan - pnpm run db:migrate:deploy applies all migrations cleanly - pnpm run typecheck --filter webapp passes - Run list pages load correctly in the dashboard (ClickHouse path) - Scheduled task runs still trigger and appear correctly |
||
|
Matt Aitken
|
68e88d0d71 |
Object Storage seamless migration (#3275)
This allows seamless migration to different object storage. Existing runs that have offloaded payloads/outputs will continue to use the default object store (configured using `OBJECT_STORE_*` env vars). You can add additional stores by setting new env vars: - `OBJECT_STORE_DEFAULT_PROTOCOL` this determines where new run large payloads will get stored. - If you set that you need to set new env vars for that protocol. Example: ``` OBJECT_STORE_DEFAULT_PROTOCOL=“s3" OBJECT_STORE_S3_BASE_URL=https://s3.us-east-1.amazonaws.com OBJECT_STORE_S3_ACCESS_KEY_ID=<val> OBJECT_STORE_S3_SECRET_ACCESS_KEY=<val> OBJECT_STORE_S3_REGION=us-east-1 OBJECT_STORE_S3_SERVICE=s3 ``` --------- Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com> |
||
|
Matt Aitken
|
0977c56efe |
Errors (versions) (#3187)
- Added versions filtering on the Errors list and page - Added errors stacked bars to the graph on the individual error page --------- Co-authored-by: James Ritchie <james@trigger.dev> |
||
|
nicktrn
|
2ba77d89dd |
fix: add build step to @internal/compute package (#3303)
The @internal/compute package had its main/types pointing to ./src/index.ts with no build step. This works in dev (tsc resolves .ts at compile time) but fails at runtime in Docker because Node.js can't load .ts files directly. Added tsconfig.build.json and build/clean/dev scripts matching the pattern used by schedule-engine and other internal packages. Exports now point to dist/. |
||
|
Eric Allam
|
a3407287c9 |
feat(engine): enqueue fast path; skip the queue under certain conditions (#3299)
## Summary Currently, every triggered run follows a two-step path through Redis: 1. **Enqueue** — A Lua script atomically adds the message to a queue sorted set (ordered by priority-adjusted timestamp) 2. **Dequeue** — A debounced `processQueueForWorkerQueue` job fires ~500ms later, checks concurrency limits, removes the message from the sorted set, and pushes it to a worker queue (Redis list) where workers pick it up via `BLPOP` This means every run pays at least ~500ms of latency between being triggered and being available for a worker to execute, even when the queue is empty and concurrency is wide open. ### What changed The enqueue Lua scripts now atomically decide whether to **skip the queue sorted set entirely** and push directly to the worker queue. This happens inside the same Lua script that handles normal enqueue, so the decision is atomic with respect to concurrency bookkeeping. A run takes the **fast path** when all of these are true: - **Fast path is enabled** for this worker queue (gated per `WorkerInstanceGroup`) - **No available messages** in the queue (`ZRANGEBYSCORE` finds nothing with score ≤ now) — this respects priority ordering and allows fast path even when the queue has future-scored messages (e.g. nacked retries with delay) - **Environment concurrency** has capacity - **Queue concurrency** has capacity (including per-concurrency-key limits for CK queues) When the fast path is taken: - The message is stored and pushed directly to the worker queue (`RPUSH`) - Concurrency slots are claimed (`SADD` to the same sets used by the normal dequeue path) - The `processQueueForWorkerQueue` job is **not scheduled** (no work to do) - TTL sorted set is skipped (the `expireRun` worker job handles TTL independently) When any condition fails, the existing slow path runs unchanged. ### Rollout gating - **Development environments**: Fast path is always enabled - **Production environments**: Gated by a new `enableFastPath` boolean on `WorkerInstanceGroup` (defaults to `false`), allowing region-by-region rollout ### Rolling deploy safety Each process registers its own Lua scripts via `defineCommand` (identified by SHA hash). Old and new processes never share scripts. The Redis data structures are fully compatible in both directions — ack, nack, and release operations work identically regardless of which path a message took. ## Test plan - [x] Fast path taken when queue is empty and concurrency available - [x] Slow path when `enableFastPath` is false - [x] Slow path when queue has available messages (respects priority ordering) - [x] Fast path when queue only has future-scored messages - [x] Slow path when env concurrency is full - [x] Fast-path message can be acknowledged correctly - [x] Fast-path message can be nacked and re-enqueued to the queue sorted set - [x] Run all existing run-queue tests (ack, nack, CK, concurrency sweeper, dequeue) to verify no regressions - [x] Typecheck passes for run-engine and webapp |
||
|
nicktrn
|
0e63f8317e |
feat: add ttl support at task and config levels (#3196)
Add TTL (time-to-live) defaults at task-level and config-level, with precedence: per-trigger > task > config > dev default (10m). Docs PR: #3200 (merge after packages are released) |
||
|
Eric Allam
|
b07567888d |
fix(llm-catalog): refresh default model pricing on sync (#3281)
- Rebuild llm_pricing_tiers and llm_prices in syncLlmCatalog for source=default - Add vitest config, sync regression tests, and pin vitest 3.1.4 - Update pnpm-lock.yaml for the new devDependency |
||
|
nicktrn
|
9cb3dcb07c |
feat(supervisor): compute workload manager (#3114)
Adds the `ComputeWorkloadManager` for routing task execution through the compute gateway, including full checkpoint/restore support, OTel trace integration, and template pre-warming. ## Changes **Compute workload manager** (`apps/supervisor/src/workloadManager/compute.ts`) - Routes instance create, snapshot, delete, and restore through the compute gateway API - Wide event logging on create with full timing and context - Configurable gateway timeout, auth token, image digest stripping **Compute snapshot service** (`apps/supervisor/src/services/computeSnapshotService.ts`) - Timer wheel for delayed snapshot dispatch (avoids wasted work on short-lived waitpoints) - Configurable dispatch concurrency limit (`COMPUTE_SNAPSHOT_DISPATCH_LIMIT`) - Snapshot-complete callback handler with suspend completion reporting - Trace context management and OTel span emission for snapshot operations **OTel trace service** (`apps/supervisor/src/services/otlpTraceService.ts`) - Fire-and-forget OTLP span emission for compute operations (provision, restore, snapshot) - BigInt nanosecond conversion preserving sub-ms precision for span ordering **Template creation** (`apps/webapp/app/v3/services/computeTemplateCreation.server.ts`) - Three-mode rollout: required (MICROVM projects), shadow (feature flag / percentage), skip - Integrated into deploy finalize flow **Shared compute package** (`internal-packages/compute/`) - Gateway client with namespace-based API (instances, templates, snapshots) - Zod schemas for all gateway request/response types **Database** - `COMPUTE` variant added to `TaskRunCheckpointType` enum - `WorkloadType` enum and column on `WorkerInstanceGroup` - `hasComputeAccess` feature flag **Env / config** - Compute gateway URL, auth token, timeout - Snapshot enable flag, delay, dispatch limit - Dedicated OTLP endpoint for compute spans (`COMPUTE_TRACE_OTLP_ENDPOINT`) |
||
|
Oskar Otwinowski
|
efe24f9c2a |
feat(private-link): Add private links UI (#3264)
⚒️ Publish Worker (v4) / build (supervisor) (push) Has been cancelled
|
||
|
Oskar Otwinowski
|
8244ac6f84 |
feat(Notifications): Platform notifications for CLI and Dashboard (#3254)
For human reviewer: - Check if Redis connection + code makes sense - Check CLI methods (it's on a hotpath) - Check DB Migrations and new tables ## ✅ Checklist - [x] I have followed every step in the [contributing guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md) - [x] The PR title follows the convention. - [x] I ran and tested the code works --- ## Testing Spawning new CLI / Dashboard notifications, check MVP, check if failures not produce any problems with CLI/Dashboard --- ## Changelog Added notifications mechanism for Dashboard and CLI --- ## Screenshots 💯 |
||
|
Saadi Myftija
|
38559480c9 |
feat: replicate trigger_source, root_trigger_source, and is_warm_start to ClickHouse (#3274)
Adds three new top-level columns to the ClickHouse task_runs_v2 table primarily for analytics: - `trigger_source` / `root_trigger_source` - extracted from the existing TaskRun.annotations JSON during WAL replication - `is_warm_start` - new nullable boolean on TaskRun in Postgres, set in the existing taskRun.update() at attempt start (no additional write). null until the first attempt starts. Run region is already available via the existing `worker_queue` column in ClickHouse. |
||
|
Saadi Myftija
|
97d2f72063 |
feat(supervisor): schedule-tree node affinity (#3271)
Scheduled runs create predictable hourly spikes that compete with on-demand runs for node capacity. Runs triggered "on-demand" via the SDK, API, or dashboard, are more sensitive to cold start latency since users are typically waiting on the result. When a burst of scheduled runs lands at the top of the hour, it can saturate the shared pool resources causing contention, affecting cold starts across the board. The idea in this change is to absorb these periodic spikes in a dedicated pool without affecting the cold starts of on-demand runs. Scheduled runs are inherently less sensitive to cold starts. ### Changes in this PR Follows up on run annotations (#3241), which made trigger origin available on every run in the tree. This PR exposes annotations at dequeue time to the supervisor. This enables scheduling decisions based on trigger source. The affinities are soft preferences at schedule time, so runs fall back gracefully if the target pool is out out of capacity. |
||
|
Eric Allam
|
1a6481a579 |
feat: add Model Registry feature with catalog pipeline, dashboard pages, and TSQL schema (#3270)
- Add llm-model-catalog package (renamed from llm-pricing) with Claude CLI research pipeline - Add Prisma schema: catalog columns + baseModelName on LlmModel - Add ClickHouse: llm_model_aggregates MV + base_response_model column - Add TSQL llm_models schema for query page integration - Add ModelRegistryPresenter with catalog, metrics, and comparison queries - Add 3 dashboard pages: catalog (cards+table+filters), detail (overview+metrics+cost estimator), compare - Add sidebar navigation under AI section with hasAiAccess feature flag - Add admin dashboard sync/seed for catalog metadata - Add model variant grouping (dated snapshots under base models) - Add shared formatters and design system component usage refs TRI-7941 |
||
|
Eric Allam
|
947f33d55b |
fix: downgrade queue size limit errors to warnings (#3243)
Queue limit ServiceValidationErrors were being logged at error level. These are expected validation rejections, not bugs. - Add logLevel property to ServiceValidationError (webapp + run-engine) - Set logLevel: warn on all queue limit throws - Schedule engine: detect queue limit failures and log as warn - Redis-worker: respect logLevel on thrown errors |
||
|
Saadi Myftija
|
d4772b5f60 |
feat: run annotations (#3241)
Adds an `annotations` JSONB column to task runs that captures where and how each run was triggered. This enables filtering and analyzing trigger origins without querying up the run tree. Also enables making scheduling decisions based on the trigger source, e.g., use separate affinities for scheduled runs. Each run records: - **triggerSource**: who initiated it (sdk, api, dashboard, cli, mcp, schedule) - **triggerAction**: what kind of action (trigger, replay, test) - **rootTriggerSource**: the trigger source of the root ancestor, propagated through the entire run tree - **rootScheduleId**: schedule id, in case the run tree was triggered from a schedule Currently the main motivation for annotations it to determine whether a run is part of a schedule-originated tree without traversing ancestors. ### A couple of design considerations - **Decoupled source from method**: triggerSource and triggerAction are separate fields to avoid combinatorial explosion (every new source × every new action) - **Server-side first**: all annotation values are primarily determined on the server, only a minor SDK change needed - **Forward-compatible**: annotation fields use `z.enum([...]).or(anyString)` so new values can be added without breaking validation; we currently don't need an explicit version field for annotations. Note: `metadata` would have been a more fitting name for the db column, as it is consistent with other tables where we store this type of information. It is already in use to store user metadata though, so we go with `annotations` instead. |
||
|
Eric Allam
|
54d95ee4b9 |
feat: AI prompt management dashboard and enhanced span inspectors (#3244)
- Full prompt management UI: list, detail, override, and version
management for AI prompts defined with `prompts.define()`
- Rich AI span inspectors for all AI SDK operations with token usage,
messages, and prompt context
- Real-time generation tracking with live polling and filtering
## Prompt management
Define prompts in your code with `prompts.define()`, then manage
versions and overrides from the dashboard without redeploying:
```typescript
import { task, prompts } from "@trigger.dev/sdk";
import { generateText } from "ai";
import { openai } from "@ai-sdk/openai";
import { z } from "zod";
const supportPrompt = prompts.define({
id: "customer-support",
model: "gpt-4o",
variables: z.object({
customerName: z.string(),
plan: z.string(),
issue: z.string(),
}),
content: `You are a support agent for Acme SaaS.
Customer: {{customerName}} ({{plan}} plan)
Issue: {{issue}}
Respond with empathy and precision.`,
});
export const supportTask = task({
id: "handle-support",
run: async (payload) => {
const resolved = await supportPrompt.resolve({
customerName: payload.name,
plan: payload.plan,
issue: payload.issue,
});
const result = await generateText({
model: openai(resolved.model ?? "gpt-4o"),
system: resolved.text,
prompt: payload.issue,
...resolved.toAISDKTelemetry(),
});
return { response: result.text };
},
});
```
The prompts list page shows each prompt with its current version, model,
override status, and a usage sparkline over the last 24 hours.
From the prompt detail page you can:
- **Create overrides** to change the prompt template or model without
redeploying. Overrides take priority over the deployed version when
`prompt.resolve()` is called.
- **Promote** any code-deployed version to be the current version
- **Browse generations** across all versions with infinite scroll and
live polling for new results
- **Filter** by version, model, operation type, and provider
- **View metrics** (total generations, avg tokens, avg cost, latency)
broken down by version
## AI span inspectors
Every AI SDK operation now gets a custom inspector in the run trace
view:
- **`ai.generateText` / `ai.streamText`** — Shows model, token usage,
cost, the full message thread (system prompt, user message, assistant
response), and linked prompt details
- **`ai.generateObject` / `ai.streamObject`** — Same as above plus the
JSON schema and structured output
- **`ai.toolCall`** — Shows tool name, call ID, and input arguments
- **`ai.embed`** — Shows model and the text being embedded
For generation spans linked to a prompt, a "Prompt" tab shows the prompt
metadata, the input variables passed to `resolve()`, and the template
content from the prompt version.
All AI span inspectors include a compact timestamp and duration header.
## Other improvements
- Resizable panel sizes now persist across page refreshes (patched
`@window-splitter/state` to fix snapshot restoration)
- Run page panels also persist their sizes
- Fixed `<div>` inside `<p>` DOM nesting warnings in span titles and
chat messages
- Added Operations and Providers filters to the AI metrics dashboard
## Screenshots
<img width="3680" height="2392" alt="CleanShot 2026-03-21 at 10 14
17@2x"
src="https://github.com/user-attachments/assets/f3e59989-a2fa-4990-a9d0-3cacda431868"
/>
<img width="3680" height="2392" alt="CleanShot 2026-03-21 at 10 15
37@2x"
src="https://github.com/user-attachments/assets/2f2d02df-2d2b-44fb-ac6f-9153f6a6c387"
/>
<img width="3680" height="2392" alt="CleanShot 2026-03-21 at 10 15
54@2x"
src="https://github.com/user-attachments/assets/baa161e0-ef91-4fa4-a55f-986b71cccdf0"
/>
|
||
|
Eric Allam
|
1cfc296c6b |
feat(ai): LLM metrics tracking and AI span inspector (#3213)
- Automatic LLM cost enrichment for AI SDK spans (streamText, generateText, generateObject) or any other spans that use semantic gen_ai attributes with support for 145+ models - New AI span inspector sidebar showing model, tokens, cost, messages, tool calls, and response text - LLM metrics dual-write to ClickHouse `llm_metrics_v1` table for analytics - LLM metrics built-in dashboard (unlinked at the moment) - Provider cost fallback — uses gateway/OpenRouter reported costs from `providerMetadata` when registry pricing is unavailable - Prefix-stripping for gateway/OpenRouter model names (e.g. `mistral/mistral-large-3` matches `mistral-large-3` pricing) - Admin dashboard for managing LLM model pricing (list, create, edit, delete, search, test pattern matching) - Missing models detection page — queries ClickHouse for unpriced models with sample spans and Claude Code-ready prompts for adding pricing - AI span seed script (`pnpm run db:seed:ai-spans`) with 51 spans across 12 provider systems for local dev testing - UI fixes: `completionTokens`/`promptTokens` aliases, `ai.response.object` display for generateObject, cache read/write token breakdown ## Screenshots: <img width="1030" height="104" alt="CleanShot 2026-03-17 at 16 48 54@2x" src="https://github.com/user-attachments/assets/bc8fccda-e48b-4d0c-bfb1-e620064e5979" /> <img width="1094" height="1512" alt="CleanShot 2026-03-17 at 16 49 23@2x" src="https://github.com/user-attachments/assets/c2424569-d07e-4d67-a436-e8250043a1ee" /> <img width="1074" height="1412" alt="CleanShot 2026-03-17 at 16 49 18@2x" src="https://github.com/user-attachments/assets/22342ac4-4769-45d1-a328-a24fb9a82a50" /> <img width="1012" height="2292" alt="CleanShot 2026-03-17 at 16 39 01@2x" src="https://github.com/user-attachments/assets/59e327d1-6652-4293-8be0-bb8326e5fbc5" /> <img width="3680" height="2392" alt="CleanShot 2026-03-15 at 08 29 38@2x" src="https://github.com/user-attachments/assets/1f77beb8-de67-495b-b890-bcdb8d7f1fe8" /> --------- Co-authored-by: James Ritchie <james@trigger.dev> |
||
|
Eric Allam
|
411803e49e |
fix(engine): lockless waitpoint insert for batch items to eliminate lock contention (#3232)
When processing batchTriggerAndWait items, each batch item was acquiring a Redis lock on the parent run to insert a TaskRunWaitpoint row. With high concurrency (processingConcurrency=50), this caused LockAcquisitionTimeoutError (880 errors/24h in prod), orphaned runs, and stuck parent runs. Since blockRunWithCreatedBatch already transitions the parent to EXECUTING_WITH_WAITPOINTS before items are processed, the per-item lock is unnecessary. The new blockRunWithWaitpointLockless method performs only the idempotent CTE insert and timeout scheduling without acquiring the lock. |
||
|
Matt Aitken
|
2406e850eb | RunEngine readme updates (#3223) | ||
|
Eric Allam
|
7672e8d998 |
fix(run-queue): prevent concurrency keys from bloating master queue shards (#3219)
Queues with concurrency keys now appear as a single entry in the master queue instead of one entry per key. This prevents high-CK-count tenants from consuming the entire `parentQueueLimit` window and starving other tenants on the same shard. A new per-queue **CK index** (sorted set) tracks active concurrency key sub-queues. The master queue gets one `:ck:*` wildcard entry per base queue. Dequeuing from that entry round-robins across sub-queues, maintaining per-CK concurrency tracking and fairness. All existing operations (enqueue, dequeue, ack, nack, DLQ, TTL expiry) are CK-index-aware and keep the index consistent. Old-format entries drain naturally during rollout — no migration step needed, single deploy. |
||
|
Eric Allam
|
d4d8d9fabc | fix(engine): add additional error logging around triggering runs (#3211) | ||
|
Eric Allam
|
436f20efc6 |
feat(cli): auto-cancel dev runs on CLI exit via detached watchdog (#3191)
When the dev CLI exits (e.g. ctrl+c via pnpm), runs that were mid-execution previously stayed stuck in EXECUTING status for up to 5 minutes until the heartbeat timeout fired. Now they are cancelled within seconds. The dev CLI spawns a lightweight detached watchdog process at startup. The watchdog monitors the CLI process ID and, when it detects the CLI has exited, calls a new POST /engine/v1/dev/disconnect endpoint to cancel all in-flight runs immediately (skipping PENDING_CANCEL since the worker is known to be dead). Watchdog design: - Fully detached (detached: true, stdio: ignore, unref()) so it survives even when pnpm sends SIGKILL to the process tree - Active run IDs maintained via atomic file write (.trigger/active-runs.json) - Single-instance guarantee via PID file (.trigger/watchdog.pid) - Safety timeout: exits after 24 hours to prevent zombie processes - On clean shutdown, the watchdog is killed (no disconnect needed) Disconnect endpoint: - Rate-limited: 5 calls/min per environment - Capped at 500 runs per call - Small counts (<= 25): cancelled inline with pMap concurrency 10 - Large counts: delegated to the bulk action system - Uses finalizeRun: true to skip PENDING_CANCEL and go straight to FINISHED Run engine change: - cancelRun() now respects finalizeRun when the run is in EXECUTING status, skipping the PENDING_CANCEL waiting state and going directly to FINISHED |