v4.4.6
7185 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 devin-ai-integration[bot]
|
b0131352f6 |
fix(webapp): constrain usage chart height to 320px (#3469)
## ✅ Checklist - [x] I have followed every step in the [contributing guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md) - [x] The PR title follows the convention. - [x] I ran and tested the code works --- ## Testing Ran the webapp locally with `CLOUD_ENV=development` and verified the usage page chart height at different viewport sizes. The chart now renders at a fixed 320px height instead of expanding to fill the viewport. --- ## Changelog Fix the "Usage by day" chart on the usage settings page taking up 100% of the viewport height. The regression was introduced in PR #2905 when the `UsageChart` was migrated from using `ChartContainer` directly (with `max-h-96 min-h-40 w-full`) to the new `Chart.Root` compound component. The `ChartContainer` base class includes `aspect-video` (16:9 ratio), and the `max-h-96` constraint was lost during migration, causing the chart to scale its height based on viewport width. Fix: wrap `Chart.Root` in a fixed-height container (`h-80` = 320px) and use the `fillContainer` prop, which applies `!aspect-auto` to override the `aspect-video` ratio. --- ## Screenshots Before (chart fills entire viewport):  After (chart constrained to 320px):  💯 Link to Devin session: https://app.devin.ai/sessions/6e5ed40516d3448db85950feb1115ab3 Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: nick <55853254+nicktrn@users.noreply.github.com> |
||
|
Eric Allam
|
99dfee3a57 |
fix(webapp): honor RevokedApiKey grace window for public access tokens (#3464)
## Summary Follow-up to #3420. PATs (public access tokens) minted before an API key rotation 401'd immediately on the realtime stream endpoints, even though the rotation flow advertises a 24h overlap. This fixes the gap. ## Root cause PATs are JWTs signed with the env's `apiKey` at mint time. When that secret is rotated, `validatePublicJwtKey` (`apps/webapp/app/services/realtime/jwtAuth.server.ts`) only verifies the signature against `environment.parentEnvironment?.apiKey ?? environment.apiKey` — i.e. the env's *current* canonical key. Any PAT in the wild signed with the previous key fails signature verification → 401, even within the grace window. #3420 wired up the grace-window fallback in two places — `findEnvironmentByApiKey` (raw secret-key auth) and `api.v1.auth.jwt.ts` (signs new JWTs with the canonical key when minting from an old one) — but the *verify* path for already-issued PATs was never updated. In a typical app, `POST /api/v1/tasks/.../trigger` (Bearer secret) keeps working through rotation because that path has the fallback, but `GET /realtime/v1/streams/run_*/...` and `POST /realtime/v1/streams/run_*/input/...` 401 for runs that were already in flight when the rotation happened. ## Fix After the primary `validateJWT` against the env's current `apiKey`, fall back to non-expired `RevokedApiKey` rows for the signing env (parent env when the request is against a child) — but **only on the failure path**, so the hot success path is unchanged. Uses `$replica` to match the rest of the auth path. Symmetrical to the `findEnvironmentByApiKey` two-step from #3420. ## Changes - `apps/webapp/app/services/realtime/jwtAuth.server.ts` — `validateAgainstRevokedApiKeys` helper invoked only on `!result.ok` - `apps/webapp/app/models/runtimeEnvironment.server.ts` — `findEnvironmentById` also selects `parentEnvironment.id` so we can scope the revoked-keys lookup to the correct env ## Test plan E2E verified locally via curl against `GET /realtime/v1/runs/{runId}` (PAT-authenticated): - [x] Pre-rotation, PAT signed with K1 → **200** with run body - [x] Simulate rotation (insert `RevokedApiKey` row + flip env `apiKey` to K2 in a single transaction, mirroring `regenerateApiKey`) - [x] Same PAT (K1) within grace window → **200** with run body — fallback hits - [x] Fresh PAT signed with K2 → **200** — current key still works - [x] Set `RevokedApiKey.expiresAt` to past → **401** — fallback finds no live row - [x] Bogus signature (no rotation) → **401** - [x] Cleanup verified: env `apiKey` restored, `RevokedApiKey` row deleted - [x] `pnpm run typecheck --filter webapp` passes |
||
|
Eric Allam
|
dac9c83bdc |
chore(webapp,run-engine): downgrade boundary log noise to warn (#3462)
## Summary
Several boundary catches and customer-input validation paths were
logging at `error` level for failures the system already handles
gracefully — disconnect on auth failure, return undefined, skip retries,
etc. This batch routes them to `warn` (which stays in stdout) or counts
them as OTel metrics, so visibility is preserved without surfacing them
as alerts.
## Changes
**New helper / pattern:**
- `apiBuilder.server.ts` — `logBoundaryError(message, error, url)`
inspects the inner error type at loader/action boundary catches;
downgrades to `warn` for `AbortError`, `ServiceValidationError`, and
`EngineServiceValidationError`.
- `platform.v3.server.ts` — `platform_client.failures_total` OTel
counter with `{function, kind}` labels; helper
`recordPlatformFailure(fn, kind)` replaces the previous error-level
logging across all `BillingClient` wrappers.
**Log-level downgrades:**
- `handleSocketIo.server.ts` — `Worker authentication failed` → warn
(system disconnects on failure; refs TRI-8863)
- `waitpointSystem.ts` — when `runStatus === "CANCELED"` in the
suspended-without-checkpoint branch, skip the throw and warn instead
(benign cancel-vs-resume race, nothing to resume)
- `runAttemptSystem.ts` — `flushedMetadata` parse/validate failures →
warn (customer-side data shape, system returns gracefully)
- `batch-queue/index.ts` — final-attempt failures with
`result.skipRetries` → warn (callbacks already opted out of retry, e.g.
queue size limit hit)
- `queryPerformanceMonitor.server.ts` — slow queries → warn
(observability signal, not an application error)
- `timeoutDeployment.server.ts` — deployment-state mismatch in the
timeout job → warn (timeout-vs-completion race)
**Inner error preservation:**
- `waitpointCompletionPacket.server.ts` — `logger.error(uploadError)`
before throwing the `ServiceValidationError` wrapper, so the underlying
upload error stays visible.
## Why
The pattern across all of these is the same: a boundary log treated any
thrown/returned error as `error` regardless of cause, even when the
cause was an expected, system-handled condition (client disconnect,
customer quota, race condition, schema validation of customer data).
That made the logs noisy and made it harder to spot real bugs.
Where the underlying signal is still useful operationally (slow queries,
billing call failures), we route it to OTel metrics with low-cardinality
labels so dashboards and alerts can be tuned independently of error
logs.
## Test plan
- [ ] `pnpm run typecheck --filter webapp`
- [ ] `pnpm run build --filter @internal/run-engine`
- [ ] Trigger a run on hello-world and verify task lifecycle is
unaffected
- [ ] Cancel a suspended run and verify the cancel-while-suspended
branch in `waitpointSystem.ts` returns `{status: "skipped"}` instead of
throwing
- [ ] Confirm `platform_client.failures_total` counter shows up in
metrics with `{function, kind}` labels when the billing client errors
|
||
|
Oskar Otwinowski
|
1a7943ce1b |
feat(docs): Private Links official documentation (#3466)
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Oskar Otwinowski
|
5fe72eefa5 | feat(webapp): Private Links setup wizard UI tweaks (#3465) | ||
|
nicktrn
|
fefe61f006 |
ci(helm): roll prereleases on main pushes + manual trigger (#3461)
Today the helm prerelease workflow only fires on PRs that touch `hosting/k8s/helm/**`. Two consequences we ran into: 1. The `changeset-release/main` PR's prerelease comment goes stale once the release branch gets force-pushed without a helm-touching commit (the bot's `Chart.yaml` bump alone doesn't seem to refire the trigger reliably). 2. The release PR's chart references an `appVersion` (e.g. `v4.4.5`) whose Docker images don't exist until *after* merge + tag. So that prerelease chart can't actually be installed end-to-end. Renames the workflow to `helm-prerelease.yml` and adds two new triggers: - **`push: main`** with `paths: hosting/k8s/helm/**` -> rolling prereleases versioned `<base>-main.<sha>`. `appVersion` stays at whatever `Chart.yaml` has (i.e. last released), so installs pull real images. Tests that chart structure is deployable, even if the app code is one release behind. - **`workflow_dispatch`** with optional `app_version` input -> manually trigger a prerelease and optionally override `appVersion` (e.g. pin to `main` or a specific tag). Useful for testing chart + app-version combinations on demand. PR behavior unchanged: same `<base>-pr<N>.<sha>` versioning, same posted/updated comment. Why not also bypass paths for `changeset-release/main`? The release PR's chart references not-yet-built `v4.4.5` images, so those prereleases aren't actually installable. The rolling main prerelease covers the testable case better. Why not SHA-pin `appVersion` to a built image like `main-<sha>`? Bigger change - the docker publish workflows currently only push `:main` (no SHA-suffixed tag). Worth doing later if we want first-class "install one chart, get exactly that commit's app code" testing, but out of scope here. Diff is mostly a rename. Substantive changes: - new `push` and `workflow_dispatch` triggers - `prerelease` job `if:` extended for the new event types - version logic branches per event - new "Override appVersion" step (workflow_dispatch only) - new "Write run summary" step so non-PR runs surface the install instructions - PR comment steps gated on `github.event_name == 'pull_request'` - concurrency group falls back to `github.ref` for non-PR runs |
||
|
Eric Allam
|
c69e939c34 |
feat: Sessions - bidirectional durable agent streams (#3417)
> ⚠️ **Not released yet.** This PR is the server-side foundation only. The SDK changes that customers will actually use (`chat.agent` migration, `chat.createStartSessionAction`, `useTriggerChatTransport` updates) live on a separate branch and ship together in an upcoming `@trigger.dev/sdk` prerelease. Until that prerelease is published, this surface is reachable only via direct HTTP. ## What this gives Trigger.dev users A new first-class primitive, **Session**, for durable, task-bound, bidirectional I/O that outlives any single run. Sessions are the run manager for `chat.agent` going forward, and they unblock anything else that needs "one identifier, many runs over time" with a stable channel pair the client can write to and subscribe to. ### Use cases unblocked - **Chat agents that persist across many runs.** One session per chat (keyed on your own `chatId` via `externalId`), turns 1..N attach to the same Session, the UI subscribes once and keeps receiving output as new runs take over. - **Approval loops and long-running tasks with user feedback.** The task waits on `.in`, the client writes to `.in`, the server enforces no-writes-after-close. - **Workflow progress streams that live past the run.** Subscribe to `.out` after the task finishes to replay history. - **Resume-next-day flows.** A session is a durable row, not a transient stream. Send a message a day later and the server triggers a fresh run on the same session. ### How it works (Session-as-run-manager) A Session row is task-bound (`taskIdentifier` + `triggerConfig` are required) and owns its current run via `currentRunId` + `currentRunVersion` for optimistic claim. Three trigger paths: 1. **Session create** — `POST /api/v1/sessions` creates the row and triggers the first run synchronously. 2. **Append-time probe** — `POST /realtime/v1/sessions/:session/in/append` checks if the current run is alive; if it has terminated (idle exit, crash, etc.), the server triggers a new run before processing the append. 3. **End-and-continue handoff** — `POST /api/v1/sessions/:session/end-and-continue`, called by the running agent, triggers a fresh run and atomically swaps `currentRunId`. Used by `chat.requestUpgrade()` for version handoffs. Every triggered run is recorded in the `SessionRun` audit table with a reason (`initial`, `continuation`, `upgrade`, `manual`). ## Public API surface ### Control plane - `POST /api/v1/sessions` — create. Idempotent on `(env, externalId)`. Triggers the first run, returns the session and a session-scoped public access token. Returns 409 if the upserted row is already closed. - `GET /api/v1/sessions/:session` — retrieve by friendlyId (`session_abc...`) or by your own externalId (server disambiguates by prefix). - `GET /api/v1/sessions` — list with filters (`type`, `tag`, `taskIdentifier`, `externalId`, derived `status` ACTIVE/CLOSED/EXPIRED, created-at range) and cursor pagination. Backed by ClickHouse. - `PATCH /api/v1/sessions/:session` — update tags / metadata / externalId. - `POST /api/v1/sessions/:session/close` — terminate. Idempotent, hard-blocks new server-brokered writes. - `POST /api/v1/sessions/:session/end-and-continue` — agent-only handoff to a fresh run. ### Realtime - `PUT /realtime/v1/sessions/:session/:io` — initialize a channel. Returns S2 credentials in headers so high-throughput clients can write direct to S2. - `GET /realtime/v1/sessions/:session/:io` — SSE subscribe. Supports Last-Event-ID resume and an opt-in `X-Peek-Settled: 1` header that fast-closes the stream when the upstream is already settled (`trigger:turn-complete`), eliminating long-poll wait on reconnect-on-reload paths. - `POST /realtime/v1/sessions/:session/:io/append` — server-side appends. - `POST /api/v1/runs/:runFriendlyId/session-streams/wait` — runs wait on a session stream as a waitpoint, with a race-check to avoid suspending if data already landed. ### Auth scopes `sessions` is a new resource type. `read:sessions:{id}`, `write:sessions:{id}`, `admin:sessions:{id}` flow through the existing JWT validator. Session-scoped public access tokens minted by the server replace browser-held trigger-task tokens for chat-style flows — the browser never sees a run identifier or a run-scoped token in steady state. ## What's coming after this PR - **SDK + chat.agent migration**: separate branch, separate PR, ships in the next `@trigger.dev/sdk` prerelease alongside this server deploy. Customers using the prerelease `chat.agent` will follow the [upgrade guide](https://github.com/triggerdotdev/trigger.dev/blob/docs/tri-7532-ai-sdk-chat-transport-and-chat-task-system/docs/ai-chat/upgrade-guide.mdx). - **Dashboard surfaces**: dedicated agent list, agent playground, agent view on the run dashboard. Tracking separately. ## Implementation notes - **Postgres `Session` table**: scalar scoping columns (`projectId`, `runtimeEnvironmentId`, `environmentType`, `organizationId`) without FKs, matching the January TaskRun FK-removal decision. Point-lookup indexes only — list queries go to ClickHouse. Terminal markers (`closedAt`, `expiresAt`) are write-once. - **ClickHouse `sessions_v1`**: ReplacingMergeTree, partitioned by month, ordered by `(org_id, project_id, environment_id, created_at, session_id)`. Tags indexed via `tokenbf_v1` skip index. - **`SessionsReplicationService`**: mirrors `RunsReplicationService` exactly — leader-locked logical replication consumer, `ConcurrentFlushScheduler`, retry with exponential backoff + jitter, identical metric shape. Dedicated slot + publication so the two consume independently. - **S2 keys**: `sessions/{addressingKey}/{out|in}`. The existing `runs/{runId}/{streamId}` key format for run-scoped streams is untouched. - **Optimistic claim**: `ensureRunForSession` triggers a run upfront (cheap to cancel if it loses the race), then attempts an `updateMany` keyed on `currentRunVersion`. Loser cancels its triggered run and reuses the winner's. No DB lock held across the trigger. ### What did NOT change Run-scoped `streams.pipe` / `streams.input` and the existing `/realtime/v1/streams/{runId}/...` routes are unchanged. Sessions are net-new — not a reshaping of the current streams API. ## Deploy notes - Set `SESSION_REPLICATION_CLICKHOUSE_URL` and `SESSION_REPLICATION_ENABLED=1` to enable the replication consumer. - The `Session` table needs `REPLICA IDENTITY FULL` set on the prod source DB before the publication is created (same one-time DDL we did for `TaskRun`). Required for delete events to carry full column values. - Cross-form authorization on the `GET /api/v1/sessions/:session` loader (a JWT minted for either form authorizes both URL forms). Action routes are URL-form-specific, matching how the SDK mints PATs. ## Verification - Webapp typecheck clean (10/10). - `apps/webapp/test/sessionsReplicationService.test.ts` — round-trip tests for insert/update/delete through Postgres logical replication into ClickHouse via testcontainers. - Live end-to-end against local dev: create + retrieve (both forms) + update + close, `.out.initialize` + `.out.append` x2 + `.in.send` + `.out.subscribe` over SSE, list with all filter combinations + pagination, `end-and-continue` swap, `X-Peek-Settled` fast-close (verified in browser via reconnect-on-reload and via curl). Replicated row lands in ClickHouse within ~1s. - Multi-round Devin + CodeRabbit review feedback addressed (read-after-write paths use `prisma` writer, info-leak on auth-routes masked as 403, peek-settled discriminator parsing fix, etc.). ## Test plan - [ ] `pnpm run typecheck --filter webapp` - [ ] `pnpm run test --filter webapp ./test/sessionsReplicationService.test.ts --run` - [ ] Start the webapp with `SESSION_REPLICATION_CLICKHOUSE_URL` and `SESSION_REPLICATION_ENABLED=1`. Confirm the slot and publication auto-create on boot. - [ ] `POST /api/v1/sessions` and verify the row replicates to `trigger_dev.sessions_v1` within a couple of seconds. - [ ] `POST /api/v1/sessions/:id/close`, then confirm `POST /realtime/v1/sessions/:id/out/append` returns 400. - [ ] Reuse a closed session's `externalId` on `POST /api/v1/sessions` and confirm 409. - [ ] `GET /realtime/v1/sessions/:id/out` with `X-Peek-Settled: 1` after a turn completes and confirm `X-Session-Settled: true` response header + immediate close. |
||
|
Eric Allam
|
e134da7306 |
fix(run-engine): debounce hot-key lock contention and 5xx feedback loop (#3453)
## Changes
Three changes in
`internal-packages/run-engine/src/engine/systems/debounceSystem.ts`, in
order of impact:
1. **Fast-path skip before the lock.** In `handleExistingRun`, do an
unlocked read of `delayUntil` (and `createdAt` for the max-duration
check) from the run row before entering `runLock.lock("handleDebounce",
...)`. If `newDelayUntil <= currentDelayUntil` and the run is still
within its max-duration window, return the existing run immediately
without taking the lock. Safe because debounce is monotonic-forward only
— a stale read either matches reality or undershoots, both of which
decay correctly (re-checked properly inside the lock by whichever caller
is actually pushing forward). Trailing-mode triggers carrying
`updateData` still take the lock so the data update is applied.
2. **Quantize `newDelayUntil`.** Round the computed `newDelayUntil` to
1-second buckets (configurable via `quantizeNewDelayUntilMs`, set to 0
to disable). Without quantization, every call has a slightly larger
`newDelayUntil` than the last and they all pass the fast-path check.
With it, concurrent callers on the same key share a target time and ~95%
short-circuit. User-visible effect: a debounced run might fire up to 1s
earlier than the strict spec — non-issue for typical debounce use cases
(chat summarization, batched notifications, etc.).
3. **Graceful lock-contention fallback.** Wrap the `runLock.lock(...)`
call so `LockAcquisitionTimeoutError` and Redlock `ExecutionError` /
`ResourceLockedError` return the existing run id with success instead of
propagating a 5xx. Debounce is best-effort: if we can't take the lock,
the herd is already updating it for us; fall in line. This kills the 5xx
→ SDK-retry feedback loop. With (1)+(2) this rarely fires; without them
it's the difference between 5xx and 200.
Defaults preserve current behaviour aside from quantization (1s) and
fast-path (on). Both are configurable via `RunEngineOptions.debounce`.
## ✅ Checklist
- [x] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [x] The PR title follows the convention.
- [x] I ran and tested the code works
---
## Changelog
Reduce 5xx feedback loops on hot debounce keys by quantizing
`delayUntil`, adding an unlocked fast-path skip before the redlock, and
gracefully handling redlock contention in `handleDebounce` so the SDK no
longer retries into a herd.
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
|
||
|
devin-ai-integration[bot]
|
4b28080ed4 |
feat: add isReplay to run context (#3454)
## Summary Adds `isReplay` boolean to the run context (`ctx.run.isReplay`), following the same pattern as the existing `isTest`. The value is derived from the existing `replayedFromTaskRunFriendlyId` database field, so no schema migration is needed. ## ✅ Checklist - [x] I have followed every step in the [contributing guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md) - [x] The PR title follows the convention. - [x] I ran and tested the code works --- ## Testing - Verified `@trigger.dev/core` builds successfully - Verified `webapp` typechecks successfully - All new fields use `default(false)` for backwards compatibility --- ## Changelog - Added `isReplay` to `TaskRun` and `V3TaskRun` schemas in `common.ts` - Added `RUN_IS_REPLAY` semantic attribute and wired it in `taskContext` - Propagated `isReplay` through the dequeue system, run attempt system, and all execution context construction paths (V1 + V2) - Added `isReplay` to `DequeuedMessage` and `TaskRunExecutionLazyAttemptPayload` schemas - Added patch changeset for `@trigger.dev/core` - Updated docs: added `isReplay` to context reference, added "Detecting replays" section to replaying page --- 💯 Link to Devin session: https://app.devin.ai/sessions/1d6f1b3cc39a4623b72d05bf00f2d70c --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: nick <55853254+nicktrn@users.noreply.github.com> |
||
|
nicktrn
|
91fd8a8a03 |
chore(security): close dependabot alerts q2 (#3456)
Closes ~80 dependabot alerts (3 critical, ~25 high, ~31 medium) by
bumping direct deps where possible and narrowly overriding the rest.
Cloud uses `resend` email transport and Node 20 - all bumps are safe for
both cloud and self-hosters.
## Direct upgrades
| Package | Where | From | To | Why |
|---|---|---|---|---|
| `vite` | root devDeps | ^5.4.21 | *(removed)* | dead pin; vitest pulls
vite transitively |
| `dompurify` | apps/webapp | ^3.2.6 | ^3.4.1 | XSS CVEs |
| `effect` | apps/webapp | ^3.11.7 | ^3.21.2 | AsyncLocalStorage CVE in
Effect fibers |
| `nodemailer` | internal-packages/emails | ^7.0.11 | ^8.0.6 | SMTP CRLF
injection (only affects self-hosters w/ smtp/aws-ses transport) |
| `uuid` | apps/webapp | ^9.0.0 | ^14.0.0 | buffer bounds check;
ESM-only but bundled by Remix |
| `uuid` + `@types/uuid` | packages/trigger-sdk | ^9.0.0 | *(removed)* |
dead deps, no usage |
| `@types/uuid` | apps/webapp | ^9.0.0 | *(removed)* | uuid 14 ships its
own types |
| `tar` | packages/cli-v3 | ^7.5.4 | ^7.5.13 | path traversal CVEs |
| `testcontainers` + `@testcontainers/postgresql` +
`@testcontainers/redis` | internal-packages/testcontainers | ^10.28.0 |
^11.14.0 | dev/test cleanup; one-line API fix for
`RedisContainer(image)` |
| `rimraf` | webapp + 6 packages | ^3.0.2 / ^5.0.7 | ^6.0.1 | dev/build
tool consolidation |
## Scoped overrides
All bound by both `>=` and `<` to avoid major-version yanks.
| Override | Closes |
|---|---|
| `tar@>=7 <7.5.11` → `^7.5.11` | supervisor's `@kubernetes/client-node
1.0.0` chain |
| `axios@>=1.0.0 <1.15.0` → `^1.15.0` | replaces older 1.9.0 pin |
| `systeminformation@>=5.0.0 <5.31.0` → `^5.31.0` | bumps existing
5.27.14 pin |
| `lodash@>=4.0.0 <4.18.0` → `^4.18.0` | bumps existing 4.17.23 pin |
| `lodash-es@>=4.0.0 <4.18.0` → `^4.18.0` | new (mirrors lodash) |
| `dompurify@>=3 <3.4.0` → `^3.4.1` | catches transitive dompurify via
mermaid |
| `vite@>=5.0.0 <6.4.2` → `^6.4.2` | path traversal; vite 5 has no patch
|
| `rollup@>=4 <4.59.0` → `^4.59.0` | path traversal in vite/vitest chain
|
| `flatted@>=3 <3.4.2` → `^3.4.2` | prototype pollution in eslint
flat-cache |
| `picomatch@>=2 <2.3.2` → `^2.3.2` | ReDoS in 2.x branch (transitive) |
| `picomatch@>=4 <4.0.4` → `^4.0.4` | ReDoS in 4.x branch
(vitest/tinyglobby) |
| `minimatch@>=3 <3.1.3` → `^3.1.3` | ReDoS in eslint 8 chain |
| `protobufjs@>=7 <7.5.5` → `^7.5.5` | **critical** RCE via
@opentelemetry/otlp-transformer |
| `fast-xml-parser@>=4 <4.5.5` → `^4.5.5` | DOCTYPE bypass + others (4.x
branch via aws-sdk in supervisor) |
| `fast-xml-parser@>=5 <5.7.0` → `^5.7.0` | **critical** + others (5.x
branch via aws-sdk in webapp) |
| `path-to-regexp@>=0.1 <0.1.13` → `^0.1.13` | ReDoS in express 4 /
@remix-run/express |
| `ajv@>=8 <8.18.0` → `^8.18.0` | DoS |
| `socket.io-parser@>=4 <4.2.6` → `^4.2.6` | DoS in @trigger.dev/core's
socket.io |
| `postcss@>=8 <8.5.10` → `^8.5.10` | XSS via stringify |
| `yaml@>=2 <2.8.3` → `^2.8.3` | DoS |
| `semver@>=5 <5.7.2` → `^5.7.2` | ReDoS in 5.x |
| `defu@>=6 <6.1.5` → `^6.1.5` | prototype pollution via __proto__ in
@prisma/config c12 chain |
## Dismissed (~47)
| Reason | Cluster | Count |
|---|---|---|
| `not_used` | langsmith + next 15.x in references/* | 10 |
| `not_used` | minimatch 8.x via prisma-generator-ts-enums
(references/prisma-6) | 3 |
| `not_used` | basic-ftp via puppeteer in references/hello-world +
references/seed | 2 |
| `not_used` | hono / @hono/node-server / express-rate-limit /
path-to-regexp 8.x / @modelcontextprotocol/sdk - all via mcp-sdk chain
(dormant in webapp; dev-only localhost in cli-v3) | 22 |
| `not_used` | fastify / @fastify/static / file-type via evalite devDep
| 5 |
| `tolerable_risk` | rollup 3 + minimatch 5/8/9/10 dev/build tooling |
13 |
## Notes
- **mcp-sdk chain**: `@vercel/sdk` in webapp imports `Vercel` API client
only; `mcp-server/*` subpath isn't loaded at runtime. cli-v3's MCP
server runs only via `trigger mcp` on developer machines. Bumping
`@modelcontextprotocol/sdk` to latest (1.29.0) wouldn't close these
alerts anyway - it ships hono ^4.11.4 which is still vulnerable - so
dismissal is the cleaner call.
- **References ignore list**: confirmed with current dependabot ignore
config; added `references/seed/package.json` (only gap).
- **undici** alerts (CVE-2026-1527, 4 alerts) will auto-close: lockfile
already at 6.25.0 > patched 6.24.0; just needs Dependabot rescan.
- **Effect 3.20 fix** is a runtime-only scheduler fix, no public API
changes - verified with research agent against our four `effect/*`
imports.
- **uuid 14** is ESM-only; we only call `validate`/`version` (no crypto
needed) so Node 20 requirement isn't load-bearing for us.
## Public packages (`packages/*`)
Minimal surface, deliberately. None of these change published runtime
behaviour - all changesets-worthy public package changes are deferred to
a regular release pass.
| Package | Change | Runtime impact |
|---|---|---|
| `packages/trigger-sdk` | Removed dead `uuid` dep (no source imports) |
None - dep was unused |
| `packages/cli-v3` | `tar` ^7.5.4 → ^7.5.13 | Patch bump within
already-allowed 7.x range; nothing CLI consumers see |
| `packages/core` / `packages/build` / `packages/python` /
`packages/rsc` / `packages/react-hooks` / `packages/schema-to-json` |
`rimraf` ^3.0.2 → ^6.0.1 in devDeps | Build-time only, no runtime change
|
No changeset added because nothing in these packages affects what
published consumers run.
## Validation
- Webapp typecheck (forced, no cache) passes after every commit
- Smoke-tested testcontainers v11 changes via real `postgresTest` +
`redisTest` (sync.test.ts, releaseConcurrency.test.ts) - both pass
- Webapp built + verified `require("uuid")` no longer in CJS server
output (now bundled inline)
- Test env webapp deployed at `dependabot-q2.rc0` (cloud#740) - no
issues observed
- Test suite run with package prerelease passed
|
||
|
nicktrn
|
9e99c81f64 |
ci: skip privileged PR jobs on fork PRs (#3458)
Fork PRs can't access org secrets or push to GHCR, so these two `pull_request` jobs hard-fail with no path to passing: - `claude-md-audit` - needs `CLAUDE_CODE_OAUTH_TOKEN` - `helm-pr-prerelease` `prerelease` job - needs `packages: write` to push the chart Hit this on #3449. Approving the run didn't help; the jobs ran and failed at the privileged step. The chart-validation `lint-and-test` job is fork-safe and stays untouched - that remains the merge gate for Helm changes. Gate both jobs on same-repo head: ```yaml if: github.event.pull_request.head.repo.full_name == github.repository ``` Other PR workflows already handle forks fine: `pr_checks` (typecheck/units/e2e/sdk-compat) falls back to anonymous DockerHub pulls when secrets are missing. |
||
|
ThullyoCunha
|
e8f1a7a0a1 |
fix(helm): expand CLICKHOUSE_PASSWORD in webapp CLICKHOUSE_URL via kubelet (#3449)
## Summary When the official Helm chart is deployed with an external ClickHouse and `clickhouse.external.existingSecret` set — the documented path for not committing secrets to `values.yaml` — the webapp pod crash-loops on startup: ``` goose run: parse "http://default:${CLICKHOUSE_PASSWORD}@<host>:8123?secure=false": net/url: invalid userinfo ``` Context in vouch request #3443. Re-opening in draft status per bot policy (previous attempt was #3445, closed by automation because it wasn't draft; no changes to the patch). ## Root cause Two pieces interact: 1. `hosting/k8s/helm/templates/_helpers.tpl` renders `CLICKHOUSE_URL` (and `RUN_REPLICATION_CLICKHOUSE_URL`) with a shell-style literal `${CLICKHOUSE_PASSWORD}` expecting bash expansion at container start. 2. `docker/scripts/entrypoint.sh` does `export GOOSE_DBSTRING="$CLICKHOUSE_URL"` — single-pass POSIX sh substitution, so the inner `${...}` survives as literal text and goose rejects it. Reproduces against the latest published chart (`oci://ghcr.io/triggerdotdev/charts/trigger:4.0.5`) and `main`. ## Fix Switch the two helpers (external + `existingSecret` branch) from shell-style `${CLICKHOUSE_PASSWORD}` to Kubernetes' `$(CLICKHOUSE_PASSWORD)`. Kubelet substitutes `$(VAR)` at pod-creation time from earlier env entries, and the chart already declares `CLICKHOUSE_PASSWORD` from the Secret immediately before `CLICKHOUSE_URL`, so the URL reaches the entrypoint with the real password already inlined. No entrypoint change, no image change. The plain-password branch (no `existingSecret`) is unchanged. Operator caveat added as template comments: `CLICKHOUSE_PASSWORD` must be URL-userinfo-safe since kubelet substitutes verbatim without percent-encoding. Hex-encoded passwords (e.g. `openssl rand -hex 32`) are safe by construction. ## Verification - `helm template` against `external.existingSecret` now renders `value: "http://default:$(CLICKHOUSE_PASSWORD)@<host>:8123?secure=false"` (was `${CLICKHOUSE_PASSWORD}`). - `helm template` against the plain-password branch is byte-identical to before. - Deployed end-to-end on a staging EKS cluster (Meistrari platform): webapp container reaches `goose: successfully migrated database to version: 6`, Node.js ClickHouse client connects at runtime. ## Alternatives considered - **Change `entrypoint.sh`** to `eval` / `envsubst` the URL — larger surface, touches every deployment mode (Docker Compose + k8s) and every container image. - **Mirror the Postgres pattern** (chart reads the full URL via `valueFrom.secretKeyRef`, as in `trigger-v4.postgres.useSecretUrl`) — cleaner long-term but requires a new `values.yaml` field and a migration path for existing users. Happy to follow up with that as a separate PR if the minimal fix here isn't the preferred direction. ## Changeset None added — the Helm chart isn't versioned through `@changesets/cli` (docs/chart-only PRs historically merge without a changeset, e.g. #2671). Happy to add one if the policy changed. Closes #3443. |
||
|
Eric Allam
|
4dced14ad5 |
chore: fix CONTRIBUTING.md setup steps and scope db:seed to webapp (#3450)
## Summary
Two fixes that together get a fresh-machine setup working from
`CONTRIBUTING.md` end-to-end with no manual workarounds:
### `CONTRIBUTING.md`
- Fix wrong path in the migration walkthrough: `cd packages/database` →
`cd internal-packages/database`. The current path doesn't exist; this
breaks step 2 for every contributor adding a migration.
- Renumber duplicate `4.` steps in **Adding migrations** and the skipped
`5.` in the hello-world **Running** section.
- Combine three sequential `pnpm run build --filter ...` calls into one
(Turbo parallelizes filters): `pnpm run build --filter webapp --filter
trigger.dev --filter @trigger.dev/sdk`.
- Add a `pnpm run db:seed` step after migrate. The seed creates the
local user, `References` org, and reference projects (including
`hello-world` with the stable `proj_rrkpdguyagvsoktglnod`). Removes the
manual instruction to edit the `externalRef` column in Postgres.
- Mention ClickHouse and the ClickHouse migrator alongside
Postgres/Redis in the Docker step (they're already part of `pnpm run
docker`, just invisible in the docs).
- Remove the V1-era **Add sample jobs** section.
`references/job-catalog` no longer exists; the hello-world flow above
replaces it.
### `turbo.json`
Scope `db:seed` to `webapp#db:seed → webapp#build`. The previous
root-level entry queued `build` for every workspace package — including
`references-*`, `docs`, `kubernetes-provider`, `coordinator`, etc. Only
`webapp` actually has a `db:seed` script, so the rest of those builds
were dead weight. Worse: a single broken reference (today,
`references-realtime-hooks-test` failing under Turbopack with
`node:fs/promises`) kills the whole seed pipeline.
After the change, `turbo run db:seed --dry-run` plan drops from 27 tasks
to 20 — only `webapp` and its real transitive workspace deps. Reference
projects no longer block seeding.
## Test plan
- [x] Fresh-machine setup followed end-to-end on a wiped Postgres +
ClickHouse: migrate → seed → build → webapp → CLI login → `trigger dev`
→ triggered `hello-world`, run completed with `{"message":"Hello,
world!"}`.
- [x] `turbo run db:seed --dry-run=json` confirms 20 tasks, all webapp
deps, no reference packages.
- [ ] CI green on the renamed turbo task name.
|
||
|
Eric Allam
|
5693b62cfb |
fix(webapp): propagate abort signal through realtime proxy fetch (#3442)
## Summary
Fixes an RSS-only memory leak in the three realtime proxy routes
(`/realtime/v1/runs`, `/realtime/v1/runs/:id`,
`/realtime/v1/batches/:id`). Client disconnects during an in-flight
long-poll would leave the upstream fetch to Electric running with no way
to abort it, so undici kept the socket open and buffered response chunks
that would never be consumed.
## Root cause
All three routes flow through
`RealtimeClient.streamRun/streamRuns/streamBatch` → `#streamRunsWhere` →
`#performElectricRequest` → `longPollingFetch(url, { signal })`. The
chain was already signal-aware, but `#streamRunsWhere` hardcoded
`signal=undefined` when calling `#performElectricRequest`, so no signal
ever reached `longPollingFetch`.
When a downstream client aborts a long-poll mid-flight:
1. Express tears down the downstream response socket.
2. The `longPollingFetch` promise has already resolved (it returns as
soon as upstream headers arrive) and handed back `new
Response(upstream.body, {...})`.
3. `undici` keeps the upstream socket open and continues buffering
chunks into the `ReadableStream` that nothing will ever read from.
4. The upstream connection is eventually closed by Electric's own poll
timeout (~20s). During that window the per-request buffers stay in
native memory.
These buffers live below V8's accounting — no `heapUsed` or `external`
growth, no sign in heap snapshots, only RSS. An isolated standalone
reproducer (`fetch` against a slow-streaming upstream, discard the
`Response` before consuming its body) measures **~44 KB retained per
leaked request** after GC. That's consistent with the undici socket +
receive buffer + HTTP parser state for a long-lived chunked response.
The pattern is the shape documented in
[nodejs/undici#1108](https://github.com/nodejs/undici/issues/1108) and
[#2143](https://github.com/nodejs/undici/issues/2143).
## What changed
- **`realtimeClient.server.ts`** — add optional `signal` parameter to
`streamRun`, `streamRuns`, `streamBatch`, and the shared
`#streamRunsWhere`; thread it through to `#performElectricRequest`
instead of hardcoding `undefined`.
- **`realtime.v1.runs.$runId.ts`, `realtime.v1.runs.ts`,
`realtime.v1.batches.$batchId.ts`** — pass `getRequestAbortSignal()`
(from `httpAsyncStorage.server.ts`) at the call site. This is the signal
wired to `res.on('close')` and fires reliably on downstream disconnect.
- **`longPollingFetch.ts`** — belt-and-suspenders: cancel the upstream
body explicitly in the error path, and treat `AbortError` as a clean
`499` instead of a `500`. This both releases undici's buffers
deterministically on error and avoids spurious 500s in request logs when
a client legitimately walks away.
## Verification
Standalone reproducer: slow upstream server streams 32 KB chunks every
100 ms for 5 seconds per request. The proxy does `fetch(url)` with
varying signal/cancel strategies, creates `new Response(upstream.body,
...)`, and discards it without consuming the body (simulating the leak
path).
Results from 1 000 parallel fetches per variant, measured post-GC:
| variant | Δ heap | Δ external | Δ RSS |
| --- | --- | --- | --- |
| A. no signal, body never consumed (the bug) | +0.3 MB | 0 MB | **+59.4
MB** |
| B. signal propagated, aborted after headers (this fix) | −0.1 MB | 0
MB | +15.4 MB |
| C. no signal, explicit `res.body.cancel()` | 0 MB | 0 MB | −25.4 MB |
10-round sustained test of variant B to distinguish accumulating
retention from one-time allocator overhead:
```
round 1/10 Δ=+3.2 MB round 6/10 Δ=-12.5 MB
round 2/10 Δ=-7.6 MB round 7/10 Δ=-11.9 MB
round 3/10 Δ=-11.7 MB round 8/10 Δ=-2.6 MB
round 4/10 Δ=+3.2 MB round 9/10 Δ=-8.0 MB
round 5/10 Δ=-1.2 MB round 10/10 Δ=-12.6 MB
```
RSS oscillates in a 49-65 MB band with no upward trend — signal
propagation fully releases the buffers.
## Risk
- Behavior change only on aborted long-polls: the upstream fetch now
cancels promptly instead of running to its natural timeout. This saves
both memory and outbound traffic to Electric.
- `AbortError` now surfaces as `499` rather than `500`. Any dashboard or
alert that counts 500s in request logs will see slightly fewer of them;
this is the intended behavior.
- Signal-aware parameter is optional on
`RealtimeClient.streamRun/streamRuns/streamBatch`, so callers that don't
opt in get the previous behavior.
## Test plan
- [ ] Existing realtime integration tests pass
- [ ] Dashboard realtime views (runs list, batch details) continue
working normally across tab open/close cycles
- [ ] Under a burst of aborted long-polls, server RSS returns to
baseline rather than climbing
|
||
|
Eric Allam
|
2ce981d454 |
chore: add GautamBytes to vouch list (#3447)
Adds @GautamBytes to the vouch list so they can contribute to the repository. Closes #3307 Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com> Co-authored-by: Eric Allam <ericallam@users.noreply.github.com> |
||
|
nicktrn
|
d2059556c3 |
chore: vouch for ThullyoCunha (#3444)
closes #3443 |
||
|
Iss
|
8dd1fc12c6 | docs: document auth.withAuth scoped authentication helper (#3436) | ||
|
Matt Aitken
|
8aa1e55588 |
test: e2e auth baseline tests + webapp testcontainer infrastructure (#3438)
Adds a minimal end-to-end test harness that spawns the compiled webapp as a child process against a throwaway Postgres container, plus a baseline of 8 auth-behaviour tests. These tests will be used as a regression check before and after the upcoming apiBuilder RBAC migration to confirm auth behaviour is unchanged. ## What's included **`internal-packages/testcontainers/src/webapp.ts`** (new) Spawns `build/server.js` with a dynamically allocated port, polls `/healthcheck`, and exposes `WebappInstance` and `startTestServer()` (postgres container + webapp + PrismaClient in one call). Key details: - Uses `process.execPath` so the correct Node binary is found in forked test processes - Sets `NODE_PATH` to `node_modules/.pnpm/node_modules` so pnpm-hoisted transitive deps (e.g. `eventsource-parser`) resolve correctly inside the subprocess - Overrides both `PORT` and `REMIX_APP_PORT` so Vite's automatic `.env` loading doesn't override the dynamically allocated port **`internal-packages/testcontainers/package.json`** Adds `./webapp` sub-path export so tests can `import from "@internal/testcontainers/webapp"`. **`internal-packages/testcontainers/src/index.ts`** Exports `createPostgresContainer` (used internally by `webapp.ts`). **`apps/webapp/test/helpers/seedTestEnvironment.ts`** (new) Creates a minimal org → project → environment row set with random suffixes. **`apps/webapp/test/api-auth.e2e.test.ts`** (new) 8 tests across two suites: - API-key bearer: valid key (auth passes, 404), missing header (401), invalid key (401), error body shape - JWT bearer: valid JWT on JWT-enabled route (passes), valid JWT on non-JWT route (401), empty-scope JWT (403), wrong signing key (401) ## How to run ```bash # Build required first (one-time) pnpm run build --filter webapp cd apps/webapp && pnpm exec vitest run test/api-auth.e2e.test.ts ``` ## Test plan - [x] All 8 tests pass against the current webapp build - [x] Webapp healthcheck returns 200 on startup - [ ] CI passes --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Saadi Myftija
|
496ac78484 |
feat(supervisor): optional ndots override for runner pods (#3441)
⚒️ Publish Worker (v4) / build (supervisor) (push) Has been cancelled
Adds `KUBERNETES_POD_DNS_NDOTS_OVERRIDE_ENABLED` flag (off by default) that overrides the cluster default and sets `dnsConfig.options.ndots` on runner pods (defaulting to 2, configurable via `KUBERNETES_POD_DNS_NDOTS`). Kubernetes defaults pods to `ndots: 5`, so any name with fewer than 5 dots, including typical external domains like `api.example.com`, is first walked through every entry in the cluster search list (`<ns>.svc.cluster.local`, `svc.cluster.local`, `cluster.local`) before being tried as-is, turning one resolution into 4+ CoreDNS queries (×2 with A+AAAA). Using a lower `ndots` value reduces DNS query amplification in the `cluster.local` zone.re2-prod-supervisor-ndots-override re2-test-supervisor-ndots-override |
||
|
Matt Aitken
|
f7aefb705a |
fix: disable RunQueue Worker in priority tests to prevent partial-batch race (#3440)
## Summary - The `processMasterQueueForEnvironment` call in the priority test was racing against background `processQueueForWorkerQueue` jobs scheduled 50ms after each trigger - With a 50ms debounce (`processWorkerQueueDebounceMs: 50`) and runs triggered sequentially, the RunQueue Worker could process those jobs mid-sequence, pushing partial batches to the worker queue in the wrong overall priority order - `masterQueueConsumersDisabled: true` only blocks the shard-level polling loops — it does not prevent the RunQueue's own Worker from processing these debounced jobs - Fix: add `worker.disabled: true` to the test 1 engine config, which propagates to `workerOptions.disabled` in the RunQueue constructor and prevents the Worker from starting ## Test plan - [x] Both priority tests pass: `pnpm run test ./src/engine/tests/priority.test.ts --run` - [x] Test 1 log confirms no `✅ Starting run engine worker` or worker loop messages — workers fully disabled - [x] Test 2 unaffected (uses master queue consumers for automatic promotion, no `disabled` flag added) 🤖 Generated with [Claude Code](https://claude.com/claude-code) |
||
|
nicktrn
|
3446be7b32 |
ci: automate helm chart release alongside package release (#3439)
Wires up automatic Helm chart releases to ride along with the existing
changeset-driven package release flow.
Today `Chart.yaml` is bumped by hand and `release-helm.yml` fires only
when a human pushes a `helm-v*` tag. With this, the changeset release PR
also carries a `Chart.yaml` bump so main always matches the published
version, and `release.yml` invokes `release-helm.yml` via
`workflow_call` after Docker images are published.
`helm-v${VERSION}` tag is pushed as a marker (same GITHUB_TOKEN trick as
`v.docker.*`). Manual `helm-v*` tag flow still works. Chart.yaml
consistency check in `release-helm.yml` is the safety net if the bump
job ever drifts.
First rollout: the open `changeset-release/main` PR has stale
Chart.yaml. Bump it manually on that branch before merging, otherwise
the first automated helm release fails at the consistency check.
|
||
|
Eric Allam
|
ca399565ba |
feat(webapp): add per-worker Node.js heap metrics (#3437)
## Summary Adds direct V8 heap and process-memory gauges to the webapp's OpenTelemetry meter. The webapp already exports per-cluster-worker Node.js runtime metrics (event-loop lag / utilization, active handles, active requests, libuv threadpool size) via a custom meter under the `trigger.dev` scope. Heap and memory were missing; this PR adds them alongside, in the same observable-batch pattern. ## New gauges | Metric | Source | Unit | | --- | --- | --- | | `nodejs.memory.heap.used` | `process.memoryUsage().heapUsed` | bytes | | `nodejs.memory.heap.total` | `process.memoryUsage().heapTotal` | bytes | | `nodejs.memory.heap.limit` | `v8.getHeapStatistics().heap_size_limit` | bytes | | `nodejs.memory.external` | `process.memoryUsage().external` | bytes | | `nodejs.memory.array_buffers` | `process.memoryUsage().arrayBuffers` | bytes | | `nodejs.memory.rss` | `process.memoryUsage().rss` | bytes | Gated by the existing `INTERNAL_OTEL_NODEJS_METRICS_ENABLED` flag, same as the adjacent event-loop / handle gauges. Zero overhead when disabled. ## Why `@opentelemetry/host-metrics` publishes `process.memory.usage`, which is RSS only. RSS is the sum of V8 heap, external memory (Buffers, etc.), native code, and thread stacks. Without a direct heap metric it is not possible to size the V8 heap cap (`--max-old-space-size`) from metrics alone, because RSS overstates heap by the external + native footprint. A worker can have a 4 GB RSS with a 2.5 GB heap and 1.5 GB of buffers; the former constrains `--max-old-space-size`, the latter does not. `nodejs.memory.heap.limit` also surfaces the configured `--max-old-space-size` (read from `v8.getHeapStatistics().heap_size_limit`), so operators can see the current limit in the same dashboard as actual usage rather than cross-referencing container environment variables. ## Risk Minimal. Observable gauges are sampled at the configured metric-export interval. `v8.getHeapStatistics()` and `process.memoryUsage()` are each microsecond-level calls, and six gauges are added to the same batch callback that already reads ~20 other Node.js runtime values per sample. Same registration pattern as the existing event-loop metrics in the file. ## Test plan - [ ] Deploy and confirm the six new gauges appear at the configured exporter - [ ] In cluster mode, confirm per-worker granularity (one series per cluster worker, tagged by `process.executable.name` / `service.instance.id`) - [ ] Confirm `nodejs.memory.heap.limit` reports the configured `--max-old-space-size` value in bytes |
||
|
Iss
|
41434b536b |
feat(webapp): admin Back Office tab with org API rate limit editor (#3434)
## Summary - New **Back office** tab at `/admin`, per-org detail page at `/admin/back-office/orgs/:orgId` designed to host future per-org admin actions (project count, delete account, YC deals). - First action: edit an organization's API rate limit — tokenBucket override (refill rate, interval, max tokens), with a live plain-English preview (e.g. *"1,500 requests per minute · 750 request burst allowance"*). Writes are audit-logged via the server logger. - Cleanup: removed unused `v2?` / `v3?` columns from the admin orgs list (display only — Prisma select untouched). ## Test plan - [ ] Back office tab visible in admin nav and highlighted when on a sub-route - [ ] `/admin/orgs` shows a Back office "Open" link per row; no v2/v3 columns - [ ] Empty state at `/admin/back-office` links back to `/admin/orgs` - [ ] Detail page renders the effective rate limit in view mode; Edit reveals the form - [ ] Save writes `Organization.apiRateLimiterConfig`, returns to view mode, shows "Rate limit saved." banner - [ ] Invalid values surface inline field errors and keep edit mode - [ ] Non-admins hitting any new route are redirected to `/` - [ ] Server logs show `admin.backOffice.rateLimit` info line per mutation |
||
|
devin-ai-integration[bot]
|
cbb1f35ef0 |
chore(helm): bump appVersion to v4.4.4 (#3432)
|
||
|
Eric Allam
|
486f49791d |
fix(webapp): eliminate SSE abort-signal memory leak (#3430)
## Summary Fixes a server-side memory leak in the webapp's SSE helper. Every aborted SSE connection (client tab close, navigation, timeout) was pinning its full request/response graph indefinitely on Node 20, so any long-running webapp process accumulated retained memory proportional to streaming-request churn. ## Root cause `apps/webapp/app/utils/sse.ts` combined four abort signals via `AbortSignal.any([requestAbortSignal, timeoutSignal, internalController.signal])`. The composite signal tracks its source signals in an internal `Set<WeakRef>` registered against a `FinalizationRegistry`; under sustained traffic those entries accumulate faster than they're cleaned up, pinning every source signal (and its listeners, and anything those listeners close over) until the parent signal itself is GC'd or aborts. This is a long-standing Node issue with multiple open reports: - [nodejs/node#54614](https://github.com/nodejs/node/issues/54614) — original report, still open. A [follow-up from ChainSafe](https://github.com/nodejs/node/issues/54614#issuecomment-4055656572) describes the exact same shape in a Lodestar production workload (req + timeout signals composed per request accumulating in long-running worker) and the same mitigation: drop `AbortSignal.any`, compose manually. - [nodejs/node#55351](https://github.com/nodejs/node/issues/55351) — mechanism confirmed by Node member @jasnell: *"the set of dependent signals known to the AbortSignal are kept in an internal Set using WeakRefs. The AbortSignals are being properly gc'd but the Set is never cleaned out of the WeakRefs making those leak."* Partially fixed by [PR #55354](https://github.com/nodejs/node/pull/55354), shipped in Node 22.12.0 — but only covers the tight-loop case, not long-lived parent signals. - [nodejs/node#57584](https://github.com/nodejs/node/issues/57584) — circular-dependency variant, still open. - [nodejs/node#62363](https://github.com/nodejs/node/issues/62363) — regression in Node 24/25 from an unrelated V8 change ("Don't pretenure WeakCells"). Different root cause, same symptom. A separate issue in `apps/webapp/app/entry.server.tsx` — `setTimeout(abort, ABORT_DELAY)` with no `clearTimeout` on success paths — kept the React render tree + `remixContext` alive for 30s per successful HTML request. Same pattern fixed upstream in React Router templates ([react-router#14200](https://github.com/remix-run/react-router/pull/14200)), never backported to Remix v2. ## What changed - **`apps/webapp/app/utils/sse.ts`** — single-signal abort chain. `AbortSignal.any` removed; `AbortSignal.timeout` replaced by a plain `setTimeout` cleared when the controller aborts; named sentinel constants used as stackless abort reasons; request-abort handler explicitly removed on cleanup. - **`apps/webapp/app/entry.server.tsx`** — clears the `setTimeout(abort, ABORT_DELAY)` timer in `onShellReady` / `onAllReady` / `onShellError`. - **`apps/webapp/app/v3/tracer.server.ts` + `env.server.ts`** — gates OpenTelemetry `HttpInstrumentation` and `ExpressInstrumentation` behind `DISABLE_HTTP_INSTRUMENTATION=true` as an escape hatch for future OTel-listener retention patterns. Defaults to enabled. - **`apps/webapp/app/presenters/v3/RunStreamPresenter.server.ts`** — uses the shared `ABORT_REASON_SEND_ERROR` sentinel. ## Verification ### Full-app reproduction (memlab) Isolated local harness, 500 abrupt SSE disconnects against a dev-presence route, GC between passes, heap snapshot diff with [memlab](https://facebook.github.io/memlab/): | Run | Heap delta after 500 conns + GC | memlab retained leaks | | --- | --- | --- | | Before | +16.0 MB (linear with request count) | 158 clusters; 250 `ServerResponse`, 1000 `AbortController`, 250 `SpanImpl` retained | | After | **+3.3 MB (noise)** | **0 app-code leaks** | ### Standalone mechanism isolation To confirm *which* axis of the change is load-bearing, a separate standalone Node script (`/tmp/abort-leak-test.mjs`) ran 2000 requests × 200 KB payload per variant: | Variant | Heap delta after GC | | --- | --- | | baseline (no signal machinery) | 0 MB | | V1: `AbortSignal.any` + string abort reason | **+9.1 MB** | | V2: `AbortSignal.any` only (no reason) | **+10.8 MB** | | V3: string reason only (no `AbortSignal.any`) | 0 MB | | V4: neither (the fix) | 0 MB | | V5: `AbortSignal.any` with no listener on the composite | **+10.2 MB** | This proves `AbortSignal.any` is the sole mechanism. The reason type (`.abort()` vs `.abort("string")`) is irrelevant for retention — V3 is clean, V5 leaks even without a listener on the composite. ## Risk - `sse.ts` is used by the dev-presence routes. Behaviour is equivalent — timeouts and client disconnects still abort the stream. `signal.reason` is now a named string sentinel (`"timeout"`, `"request_aborted"`, etc.) instead of the previous string arg or default `AbortError`. No in-tree reader of `signal.reason` exists. - `entry.server.tsx` change is a standard cleanup of an abort timer, matches upstream React Router guidance. - `tracer.server.ts` change is env-gated and defaults to current behaviour. - Three other webapp `AbortSignal.timeout()` callsites (alert delivery, remote-build status) are fire-and-forget passed directly to `fetch` — not composed with anything long-lived, no retention risk, untouched. ## Test plan - [ ] Existing SSE integration tests pass - [ ] Dev-presence SSE behaves normally across tab open/close cycles - [ ] No heap growth under sustained aborted-connection traffic (heap snapshot diff) ## Follow-up The same `AbortSignal.any([userSignal, internalSignal])` pattern exists in several SDK/core callsites that ship to customers (`packages/core/src/v3/realtimeStreams/manager.ts`, `packages/trigger-sdk/src/v3/{ai,chat,chat-client,sessions}.ts`, `packages/core/src/v3/workers/warmStartClient.ts`). Whether those leak in practice depends on the user passing a long-lived signal. Tracked separately. |
||
|
nicktrn
|
87b6716535 |
fix(helm): support webapp serviceAccount annotations for IRSA (#3429)
Mirrors the existing `supervisor.serviceAccount` pattern onto webapp so
operators can annotate the SA (IRSA `eks.amazonaws.com/role-arn`,
Workload Identity, etc.) or bring their own SA. Without this,
`webapp.serviceAccount.annotations` isn't exposed and operators have to
patch the SA out-of-band.
```yaml
webapp:
serviceAccount:
create: true
name: ""
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/trigger-webapp
```
Three pieces, same as supervisor:
- `webapp.serviceAccount.create` toggle on the SA block
- `webapp.serviceAccount.annotations` + `name` values
- `trigger-v4.webappServiceAccountName` helper, used by the SA, the
token-syncer RoleBinding subject, and the Deployment's
`serviceAccountName`
Role + RoleBinding are left unguarded (matching supervisor's shape where
`rbac.create` is a separate toggle from `serviceAccount.create`) -
BYO-SA users take on the responsibility of ensuring the SA they supply
has the permissions the RoleBinding grants.
Verified with `helm template` against default values, an IRSA annotation
override, and `create: false` with a custom name.
|
||
|
Matt Aitken
|
fc71e7dd75 |
fix: handle fast-completion race in batch streaming seal check (#3427)
## Problem When `batchTrigger()` is called with large payloads, each item's payload is uploaded to R2 server-side during the streaming loop before being enqueued. This makes the loop slow — around 3 seconds per item. Workers pick up and execute each item as it's enqueued, running concurrently with the ongoing stream. For the last item in the batch, a race exists between the streaming loop finishing and the batch completion cleanup: 1. The loop enqueues the last item and returns from `enqueueBatchItem()` 2. A waiting worker picks up the item almost instantly and executes it 3. `recordSuccess()` fires, `processedCount` hits the expected total, `finalizeBatch()` runs 4. `cleanup()` deletes all Redis keys for the batch, including `enqueuedItemsKey` 5. The streaming loop exits and calls `getBatchEnqueuedCount()` — reads the now-deleted key — returns 0 The count check finds `enqueuedCount (0) !== batch.runCount`, falls through to a Postgres fallback, but the fallback only checked `sealed`. The BatchQueue completion path sets `status = COMPLETED` in Postgres without setting `sealed = true` (that's the streaming endpoint's job), so the fallback misses it too. This causes the endpoint to return `sealed: false`. The SDK treats this as retryable and retries up to 5 times with exponential backoff. Each retry calls `enqueueBatchItem()`, which reads the batch meta key from Redis — also deleted by `cleanup()` — and throws "Batch not found or not initialized" (500). The final retry gets a 422 because the batch is already COMPLETED, which the SDK does not retry, causing an `ApiError` to be thrown from `await batchTrigger()` in the parent run — even though all child runs completed successfully. ## Fix In the Postgres fallback inside `StreamBatchItemsService`, also check `status === "COMPLETED"` alongside `sealed`. This covers the fast-completion path where the BatchQueue finishes all runs before the streaming endpoint gets to seal the batch normally. Also switches `findUnique` to `findFirst` per webapp convention. --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Oskar Otwinowski
|
8eb596f3fe | fix(vercel): Fix vercel settings page (#3424) | ||
|
Eric Allam
|
2d3b2e82e6 |
feat(run-engine): flag to route getSnapshotsSince through read replica (#3423)
## Summary Adds `RUN_ENGINE_READ_REPLICA_SNAPSHOTS_SINCE_ENABLED` (default `"0"`). When enabled, the Prisma reads inside `RunEngine.getSnapshotsSince` run against the read-only replica client instead of the primary. Offloads the snapshot-polling queries fired by every running task runner off the writer. ## Why `getSnapshotsSince` is called from the managed runner's fetch-and-process loop (once per poll interval, plus on every snapshot-change notification). It runs four sequential reads per call — one `findFirst` by snapshot id, one `findMany` on snapshots with `createdAt > X`, one raw SQL against `_completedWaitpoints`, and chunked `findMany` on `waitpoint`. Per concurrent run, every few seconds. It's read-only, tolerates a small amount of staleness, and is an obvious candidate for the replica. ## Replica-lag considerations - **Step 1 "since snapshot not found"**: if the runner just received a snapshot id from the primary and asks the replica before it replicates, the function throws and the caller treats the response as an error (runner falls back to a metadata refresh). Self-correcting, not silent. - **Step 2 missing newly-created snapshots**: the next poll's `createdAt > sinceSnapshot.createdAt` filter still picks them up once the replica catches up. - **Waitpoint junction race**: the riskiest path — if a latest snapshot is replicated but its `_completedWaitpoints` join rows aren't yet, the runner could advance past that snapshot with `completedWaitpoints: []`. WAL/storage-level replication replays commits in order, so in practice both should appear atomically on the reader, but the race window is why the flag ships disabled. Aurora reader shrinks all three windows to single-digit ms in typical conditions, and its storage-level replication gives atomic visibility of committed transactions on the reader. ## Test plan - [ ] Flip the flag on in a non-prod environment, confirm snapshot polling behaves normally and `getSnapshotsSince` errors in Sentry stay flat. - [ ] Verify writer query volume drops and reader query volume rises on the snapshot-polling queries. - [ ] Keep an eye on `AuroraReplicaLag` (or equivalent) during rollout. |
||
|
Eric Allam
|
7c95ee498e |
feat(webapp): tag Prisma spans with db.datasource attribute (#3422)
## Summary
Stamp every Prisma span with `db.datasource: "writer" | "replica"` so
traces can distinguish which client the query went through.
Both `PrismaClient` instances share the same global
`@prisma/instrumentation`, so their spans come out with identical names
and attributes today. This makes them trivially filterable.
## How
Two pieces in `apps/webapp/app/`:
1. **`v3/tracer.server.ts`** — a `DatasourceAttributeSpanProcessor`
reads an OTel context key in `onStart` and calls
`span.setAttribute("db.datasource", value)`. Registered as the first
span processor.
2. **`db.server.ts`** — `tagDatasource(datasource, client)` wraps each
`PrismaClient` with `$extends({ query: { $allOperations } })`. The
middleware sets the context key around the query and directly tags the
active span (to catch `prisma:client:operation`, which Prisma creates
before the middleware fires).
### Context-propagation gotcha
`PrismaPromise` is lazy — `query(args)` returns a thenable that only
starts when someone `.then()`s it. The naive `context.with(ctx, () =>
query(args))` restores ALS synchronously, so when Prisma's internal code
awaits the thenable later, the engine spans fire with the original ALS.
Wrapping as `async () => await query(args)` forces the `.then()` inside
the `context.with` callback, so ALS stays on our context for the engine
spans.
### Coverage
- **Tagged**: all `prisma:engine:*` (`connection`, `db_query`,
`serialize`, `query`, etc.), `prisma:client:operation`,
`prisma:client:serialize`, `prisma:client:connect`
- **Not tagged**: `prisma:client:load_engine` — one-time startup, fires
before any query
Concurrent `Promise.all([writer.x, replica.y])` correctly tags each pool
separately (ALS isolates per-Promise chain).
### Performance
One `context.with` (~200ns) and one `setAttribute` per span (effectively
free per OTel JS benchmarks) per Prisma op. Negligible against a query
path measured in milliseconds.
## Test plan
- [ ] Verify `db.datasource` appears on `prisma:engine:connection` spans
after the webapp is restarted
- [ ] Spot-check a handful of real traces carry the attribute
|
||
|
nicktrn
|
b570586899 |
fix(webapp): allow cancelling runs in DEQUEUED status from the runs list (#3421)
The cancel button was missing from the runs list for runs in `DEQUEUED` status. The runs list gates the button on `run.isCancellable`, which goes through `isCancellableRunStatus` -> `CANCELLABLE_RUN_STATUSES` = `NON_FINAL_RUN_STATUSES`. `DEQUEUED` was never added to that list when it was introduced in the run engine. The single run page uses a separate check (`!run.isFinished`, i.e. the inverse of `FINAL_RUN_STATUSES`), so cancellation already worked there - only the list was affected. Adding `DEQUEUED` to `NON_FINAL_RUN_STATUSES` also flips `isCrashableRunStatus` and `isFailableRunStatus`, but: - The crash path is the right behaviour - a `DEQUEUED` run (worker has claimed but not yet executing) can legitimately crash before `EXECUTING`, same as `PENDING`/`DELAYED` already do. - The fail path (`failedTaskRun.server.ts`) is only reached from V1 code paths (marqs consumers, v1 heartbeat handler). `DEQUEUED` is a V2-engine-only status, so V1 consumers never see it. When cancelling a `DEQUEUED` run the execution snapshot goes to `PENDING_CANCEL` (worker must ack) but `TaskRun.status` flips to `CANCELED` immediately - the UI reflects cancellation without waiting for the worker. Added an integration test in `run-engine/src/engine/tests/cancelling.test.ts` covering the full trigger -> dequeue -> cancel -> worker-ack flow. ## Stall safety The stall recovery path (PENDING_EXECUTING heartbeat miss -> nack-and-requeue -> back to QUEUED) lives entirely inside `@internal/run-engine` and never touches the webapp's `taskStatus.ts` helpers - the engine has zero imports from `~/v3/taskStatus` and doesn't know `CrashTaskRunService` / `FailedTaskRunService` exist. A stalled DEQUEUED run still goes back to the queue for retry; this change cannot cause stalls to crash or fail. The only realistic impact is the intended UI fix - the theoretical V1 crash/fail branches for DEQUEUED are unreachable in practice because V1 runs never have DEQUEUED status. |
||
|
Eric Allam
|
03e4d5fe31 |
feat(webapp,database): API key rotation grace period (#3420)
## Summary
Regenerating a RuntimeEnvironment API key no longer immediately
invalidates the previous one. Rotation is now overlap-based: the old key
keeps working for 24 hours so customers can roll it out in their env
vars without downtime, then stops working.
## Design
- **New `RevokedApiKey` table** (one row per revocation). Holds the
archived `apiKey`, a FK to the env, an `expiresAt`, and a `createdAt`.
Indexed on `apiKey` (high-cardinality equality — single-row hits) and on
`runtimeEnvironmentId`.
- **`regenerateApiKey` wraps both writes in a single `$transaction`:**
insert a `RevokedApiKey` with `expiresAt = now + 24h`, update the env
with the new `apiKey`/`pkApiKey`.
- **`findEnvironmentByApiKey` does a two-step lookup:** primary
unique-index hit on `RuntimeEnvironment.apiKey` first; on miss,
`RevokedApiKey.findFirst({ apiKey, expiresAt: { gt: now } })` with an
`include: { runtimeEnvironment }`. Two-step (not `OR`-join) keeps the
hot path identical to today and puts the fallback cost only on invalid
keys. Both lookups use `$replica`.
- **Admin endpoint** `POST /admin/api/v1/revoked-api-keys/:id` accepts
`{ expiresAt }` and updates the row. Setting to `now` ends the grace
window immediately; setting to the future extends it.
- **Modal copy** on the regenerate dialog updated — previously warned of
downtime, now explains the 24h overlap.
## Why a separate table instead of columns on `RuntimeEnvironment`
- Keeps the hot auth path's primary lookup unchanged — no
OR/nullable-apiKey semantics to reason about.
- Naturally supports multiple in-flight grace windows (regenerate twice
in a day → two old keys valid until their independent expiries).
- FK + cascade cleans up correctly when an env is deleted; nothing to
backfill.
## Test plan
Verified locally against hello-world with dev and prod env keys:
- [x] baseline — current key authenticates (`GET /api/v1/runs`) → `200`
- [x] regenerate via UI — DB shows old key in `RevokedApiKey` with
`expiresAt ≈ now+24h`, env has new key
- [x] grace window — both old and new keys → `200`; bogus key → `401`
- [x] admin endpoint: `expiresAt = now` → old key `401`
- [x] admin endpoint: `expiresAt = +1h` (after early-expire) → old key
`200` again
- [x] admin endpoint: `expiresAt = past` → old key `401`
- [x] admin 400 (invalid body), 404 (unknown id), 401 (missing/non-admin
PAT)
- [x] same flow exercised end-to-end on a PROD-typed env — behavior
identical
- [x] `pnpm run typecheck --filter webapp` passes
|
||
|
Iss
|
de3b9a158b | docs: document secret env vars and Vercel sync behavior (#3419) | ||
|
DKP
|
be6b490790 | docs: skills page update (#3418) | ||
|
Eric Allam
|
881288c615 |
feat(webapp): deprecate v3 CLI deploys server-side (#3415)
## ✅ Checklist - [x] I have followed every step in the [contributing guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md) - [x] The PR title follows the convention. - [x] I ran and tested the code works --- ## Summary Adds a server-side gate that detects deploy attempts from v3 CLI versions (i.e. `trigger.dev@3.x`) at the `POST /api/v1/deployments` entry point and, when enabled, rejects them with a clear upgrade message. v4 CLI deploys are completely unaffected. The last 3.x CLI release was `3.3.7`, which we can't update. This approach short-circuits the deploy before any DB writes, image-ref generation, S2 stream creation, or queue enqueue — no side effects in either mode. ## How v3 vs v4 are distinguished I pulled the published CLI tarballs for `trigger.dev@3.3.7`, `4.0.0`, `4.0.1`, `4.0.5`, `4.1.0`, `4.2.0`, and the current `4.4.4` in the repo. The cleanest, most reliable signal is the request body to `POST /api/v1/deployments`: | Field on initialize | v3.3.7 CLI | v4.x CLI | |---|---|---| | `type` | **never sent** | always sent — `"MANAGED"` (run_engine_v2) or `"V1"` | | `isNativeBuild` / `gitMeta` / `triggeredVia` / `runtime` | not sent | sent | | `registryHost` / `namespace` | sent (v3-only; stripped by current Zod schema) | not sent | Every v4 call site I inspected sets `type: features.run_engine_v2 ? "MANAGED" : "V1"` unconditionally. `payload.type` is `undefined` if and only if the client is a 3.x CLI. ## Behavior - Detection always runs and emits `logger.warn("Detected deploy from deprecated v3 CLI", { environmentId, projectId, organizationId, enforced })`, which lets us watch how many v3 deploys are still happening before enforcement is flipped. - Enforcement is gated behind `DEPRECATE_V3_CLI_DEPLOYS_ENABLED` (default `"0"`, off). When `"1"`, the server returns `400` with: > The trigger.dev CLI v3 is no longer supported for deployments. Please upgrade your project to v4: https://trigger.dev/docs/migrating-from-v3 The v3 CLI surfaces this verbatim as `Failed to start deployment: <message>` because `zodfetch` throws `ApiError` for non-retryable 4xx (400/422) and `deploy.js` in 3.3.7 prints `error.message`. ## Out of scope (intentionally) - `api.v1.deployments.$deploymentId.finalize.ts` / `FinalizeDeploymentService` / `createDeploymentBackgroundWorkerV3.server.ts` are V1-engine paths, not the v3 CLI gate. Leaving them alone per review. - Container-side `createDeploymentBackgroundWorker` call in `managed-index-controller.ts` is still used by v4's in-image indexer. Not touched. - v3 `trigger dev` flow (different code path) — separate deprecation if/when needed. ## Testing - Ran `pnpm run typecheck --filter webapp` locally — passes. - Verified v4 tarballs (4.0.0, 4.0.1, 4.0.5, 4.1.0, 4.2.0, 4.4.4) all include `type:` in the `initializeDeployment` call site, so none will be accidentally blocked. - Verified v3.3.7 tarball's `initializeDeployment` payload has no `type` field. Rollout plan after merge: 1. Deploy with `DEPRECATE_V3_CLI_DEPLOYS_ENABLED` unset → watch `Detected deploy from deprecated v3 CLI` log volume. 2. When comfortable, set `DEPRECATE_V3_CLI_DEPLOYS_ENABLED=1` to enforce. --- ## Changelog Detect v3 CLI deploys on `/api/v1/deployments` and, when `DEPRECATE_V3_CLI_DEPLOYS_ENABLED=1`, reject them with an upgrade message pointing at https://trigger.dev/docs/migrating-from-v3. v4 CLI deploys are unaffected. Link to Devin session: https://app.devin.ai/sessions/b242c11bd86e4099aeec8b59bab62143 Requested by: @ericallam |
||
|
Matt Aitken
|
6e6deb41e1 |
Admin endpoint to set concurrency burst factor (#3412)
Example cURL call using an admin user PAT (replace with a real one): ```sh curl -X PUT https://cloud.trigger.dev/admin/api/v1/environments/<environmentId>/burst-factor \ -H "Authorization: Bearer tr_pat_1234" \ -H "Content-Type: application/json" \ -d '{"burstFactor": 1.5}' ``` |
||
|
Iss
|
7d7ebdde52 | feat: Increase default project limit per org from 10 to 25 (#3409) | ||
|
nicktrn
|
9a988ab885 |
chore(webapp): clarify admin feature flags are global (#3408)
global flags are global. |
||
|
nicktrn
|
581db83f64 |
feat(webapp): highlight microVM regions on the regions page (#3407)
Adds a `MicroVM` badge next to the region name on the regions page. Uses the existing `small` badge variant for visual consistency with the `Default` badge already on this page. |
||
|
Eric Allam
|
69acdc2b32 |
fix(core): truncate large error stacks and messages to prevent OOM (#3405)
## Summary Large error stacks and messages can OOM the worker process when serialized into OTel spans or `TaskRunError` objects. This was reported when throwing an error with a massive `.stack` property from a chat agent hook. This adds frame-based stack truncation (similar to Sentry's approach) plus message length limits, applied consistently across all error serialization paths. ### What changed **`packages/core/src/v3/errors.ts`** - `truncateStack()` — parses `error.stack` into message lines + frame lines, caps at 50 frames (keep top 5 closest to throw + bottom 45 entry points, with "... N frames omitted ..." in between). Individual lines capped at 1024 chars. - `truncateMessage()` — caps error messages at 1000 chars - Applied in `parseError()` and `sanitizeError()` **`packages/core/src/v3/otel/utils.ts`** - `sanitizeSpanError()` now uses `truncateStack` and `truncateMessage` from `errors.ts` instead of duplicating truncation logic - Non-Error values (strings, JSON) capped at 5000 chars **`packages/core/src/v3/tracer.ts`** - `startActiveSpan` catch block now delegates to `recordSpanException()` instead of calling `span.recordException()` directly ### Limits | What | Limit | Rationale | |------|-------|-----------| | Stack frames | 50 | Matches Sentry's `STACKTRACE_FRAME_LIMIT` | | Top frames kept | 5 | Closest to throw site | | Bottom frames kept | 45 | Entry points / framework frames | | Per-line length | 1024 | Matches Sentry, prevents regex DoS | | Message length | 1000 | Bounded but generous | | Generic string (non-Error) | 5000 | Fallback for JSON/string errors in spans | ## Test plan - [x] 17 unit tests in `packages/core/test/errors.test.ts` - [x] E2E: threw a 300-frame / 5000-char-message error in the ai-chat reference app, verified truncated stack and message in span via `get_span_details` - [x] Verified the run survived the error (no OOM, continued waiting for next message) |
||
|
Matt Aitken
|
45ba398c80 |
Error page graph: for a time bucket don't fill zeros for a version with no errors (#3402)
This caused performance issues with large numbers of versions, and bad UX when hovering the graph (showing irrelevant versions) |
||
|
Eric Allam
|
9636e43567 |
fix(webapp): reduce error-level log noise for handled/benign cases (#3403)
Two changes to cut error volume from logs that represent handled
conditions, not real errors (combined ~1600/hr in prod):
1. api.v1.waitpoints.tokens.$waitpointFriendlyId.complete.ts
The route throws `json(..., { status: 404 })` when a waitpoint
isn't found, but the generic catch block caught that Response,
logged it as an error (with an empty {} body because Error fields
are non-enumerable), and rethrew as a 500 — so clients saw a 500
instead of the intended 404, and every stale-waitpoint request
produced a Sentry event.
Fix: re-throw Response objects unchanged so the correct status
propagates and we don't log user 404s as errors. Also serialize
remaining Error instances explicitly (name/message/stack) so the
logs are actionable when we do hit a real error.
2. v3/marqs/sharedQueueConsumer.server.ts:603
"Task run has invalid status for execution. Going to ack" — the
message itself says we're handling it gracefully. Benign race
between dequeue and completion/cancellation. Demote to warn.
|
||
|
devin-ai-integration[bot]
|
ff290dfe2f |
perf(run-engine): merge dequeue snapshot creation into taskRun.update transaction [TRI-8450] (#3395)
## Summary
Nests the `TaskRunExecutionSnapshot` creation inside the
`taskRun.update()` Prisma call in the dequeue flow, reducing **2 DB
commits → 1** per dequeue operation. This is the highest-volume of the
five unmerged flows identified in TRI-8450 (~9,200 commits/sec on the
engine service).
**Pattern**: Follows the same nested-write approach already used in the
completion path (`runAttemptSystem.ts:735`) and trigger path
(`engine/index.ts:674`).
**Changes**:
- `dequeueSystem.ts`: Moved snapshot creation into `executionSnapshots:
{ create: {...} }` within the existing `taskRun.update()`. Pre-generates
the snapshot ID via `generateInternalId()` (plain cuid, matching what
Prisma's `@default(cuid())` produces) so the event emission, heartbeat
enqueue, and return value can all be constructed from data already in
scope — **no extra DB read needed** after the merged write.
`SnapshotId.toFriendlyId()` is used only for the return value's
`friendlyId` field, matching the original `createExecutionSnapshot`
behavior.
- `executionSnapshotSystem.ts`: Added public
`enqueueHeartbeatIfNeeded()` method that exposes the heartbeat
scheduling logic (previously only available internally via
`createExecutionSnapshot`). This is needed because `PENDING_EXECUTING`
requires a heartbeat, unlike the `FINISHED` status in the completion
reference pattern. This method is reusable by future merge targets
(retry-immediate, checkpoint, cancel, requeue).
**Net DB change per dequeue**: eliminates 1 write transaction (the
separate `TaskRunExecutionSnapshot.create`). No extra reads added — the
snapshot ID is pre-generated and the `executionSnapshotCreated` event
payload is constructed inline from values already available in the
closure.
## Review & Testing Checklist for Human
- [ ] **Verify manually-constructed event payload matches DB state**:
The `executionSnapshotCreated` event is now built inline (not read back
from DB). Confirm the field values (`runStatus: "PENDING"`,
`attemptNumber`, `checkpointId`, `workerId`, `runnerId`,
`completedWaitpointIds`) match what Prisma actually writes. A mismatch
here would be silent — event consumers would get stale/wrong data.
- [ ] **Verify `attemptNumber` source is equivalent**: Old code used
`lockedTaskRun.attemptNumber` (post-update result). New code uses
`result.run.attemptNumber` (pre-update). The `taskRun.update()` data
payload does NOT include `attemptNumber`, so they should be identical —
but confirm this assumption holds for all dequeue scenarios (e.g.
retried runs).
- [ ] **Verify `isValid` defaults to `true` in schema**: The old
`createExecutionSnapshot` explicitly set `isValid: error ? false :
true`. The nested create omits `isValid` (no error in the dequeue happy
path). Confirm the Prisma schema default for
`TaskRunExecutionSnapshot.isValid` is `true`.
- [ ] **Verify `runStatus: "PENDING"` hardcoding matches the mapping**:
The old code passed `lockedTaskRun.status` ("DEQUEUED") to
`createExecutionSnapshot`, which mapped it to "PENDING" via `run.status
=== "DEQUEUED" ? "PENDING" : run.status`. The new code hardcodes
`"PENDING"` directly. This is correct but brittle if `status` ever
changes from "DEQUEUED" to something else upstream.
- [ ] **Spot-check `completedWaitpoints` connect + order logic**: The
nested create replicates the connect/order logic from
`createExecutionSnapshot` (lines 387-393). Verify the
`snapshot.completedWaitpoints` type provides `id` and `index` fields
compatible with this usage.
- [ ] **Verify `checkpoint` in return value**: The return now uses
`snapshot.checkpoint` (from the *previous* snapshot) instead of reading
the newly-created snapshot's checkpoint relation. Since `checkpointId`
is passed through unchanged, they should be identical — but worth a
sanity check.
**Recommended test plan**: deploy to staging, run the
`sample_pg_activity.py` sampler for a 5-minute window, and verify the
COMMIT count drop on the engine service + proportional `IO:XactSync`
reduction.
### Notes
- This only covers the **dequeue** flow (flow #1 from TRI-8450). The
remaining four flows (retry-immediate, checkpoint, requeue, cancel) are
separate follow-ups.
- The new `enqueueHeartbeatIfNeeded` method is deliberately designed for
reuse by those follow-up PRs.
- CI note: the `priority.test.ts` failure in shard 7 is a flaky ordering
assertion unrelated to this change (it compares `friendlyId` values in
dequeue order). The `audit` check is also pre-existing/unrelated.
Link to Devin session:
https://app.devin.ai/sessions/034fe0e7224f49278a2de260203e1377
Requested by: @ericallam
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Eric Allam <eallam@icloud.com>
|
||
|
Eric Allam
|
67d2025f33 |
feat(webapp): add 60s/60s SWR cache to getEntitlement (#3388)
Wraps getEntitlement in platform.v3.server.ts with the existing
platformCache (LRU memory + Redis) under a new `entitlement` namespace.
Eliminates a synchronous billing-service HTTP round trip on every
trigger.
Cache config: 60s fresh / 60s stale SWR. Cache key is the
organization id. Errors are caught inside the loader and return the
existing permissive { hasAccess: true } fallback, which is also
cached to prevent thundering-herd on billing outages.
Trade-off: plan upgrade/downgrade is now visible after up to ~120s
worst-case (60s fresh + 60s stale revalidation). Acceptable since
the existing limits and usage namespaces use 5min/10min, and the
defensive hasAccess: true fallback already exists.
|
||
|
Eric Allam
|
79b6053e13 |
feat(server): add TaskIdentifier registry to replace expensive distinct query (#3368)
Replace the expensive DISTINCT query for task filter dropdowns with a dedicated TaskIdentifier registry table backed by Redis. Environments migrate automatically on their next deploy, with a transparent fallback to the legacy query for unmigrated environments. Also fixes duplicate dropdown entries when a task changes trigger source, and adds active/archived grouping for removed tasks. Moves BackgroundWorkerTask reads in the trigger hot path to the read replica. |
||
|
Eric Allam
|
94abe97132 | fix(webapp): prevent dashboard crash when span accessory text is not a string (#3400) | ||
|
Eric Allam
|
02d2334c8a |
fix(webapp): fix Redis connection leak in realtime streams and broken abort signal propagation (#3399)
Pool Redis connections for non-blocking ops (ingestData, appendPart,
getLastChunkIndex)
using a shared singleton instead of new Redis() per request. Use
redis.disconnect()
for immediate teardown in streamResponse cleanup. Add 15s inactivity
timeout fallback.
Fix broken request.signal in Remix/Express by wiring Express
res.on('close') to an
AbortController via httpAsyncStorage. All SSE/streaming routes now use
getRequestAbortSignal() which fires reliably on client disconnect,
bypassing the
Node.js undici GC bug (nodejs/node#55428) that severs the signal chain.
|
||
|
Eric Allam
|
f5b4d34c4b | chore: allow Devin bot in claude-code-action workflows (#3401) | ||
|
nicktrn
|
93f2ca6bf4 |
feat(webapp): extend admin workers endpoint and unify admin api auth (#3390)
Extends the admin worker groups endpoint with a GET loader and more fields on POST (type, hidden, workloadType, cloudProvider, location, staticIPs, enableFastPath), and pulls the PAT + admin check that was inlined or locally duplicated across every admin.api route into a shared helper in personalAccessToken.server.ts. The generic authenticateAdminRequest returns a discriminated result; requireAdminApiRequest is the thin Remix loader/action wrapper that throws. The neverthrow-style route (platform-notifications.ts) now composes the generic helper instead of duplicating the check. Verified locally against GET (listing) and POST (new fields, invalid enum, minimal backwards-compat). |
||
|
Eric Allam
|
0c33de88f0 | chore: add Devin bot to vouch list and skip draft requirement (#3396) |