e0bf74bfaebdd6c45ee140d605c09bcb199e67fa
7559 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Katia Bulatova
|
e0bf74bfae |
docs: billing limits and alerts page (#4132)
New `/billing-limits` page covering the full [billing limits feature](https://trigger.dev/changelog/billing-limits) : the three limit options (plan / custom / no limit), billing alerts (% of limit or dollar thresholds), what happens when the limit is reached, the recovery flow, the soft-limits caveat, and the billing limit marker on the Usage page. |
||
|
DKP
|
00ee0751ec |
feat(webapp): proxy PostHog through a same-origin /ph path (#4183)
## Summary posthog-js sent product analytics to PostHog Cloud directly from the browser. This points `api_host` at a same-origin `/ph` path that forwards to PostHog Cloud EU server-side, following PostHog's standard first-party reverse-proxy setup. ## How it works A resource route forwards each request server-side, splitting by path: `/ph/static/*` and `/ph/array/*` go to the asset host, everything else (analytics events, feature flags) goes to the ingest host. It rewrites the `Host` header, strips the `/ph` prefix, and streams the response back. Only PostHog's own cookies are forwarded, so the app session cookie stays first-party. Upstream hosts default to PostHog Cloud EU, overridable via `POSTHOG_INGEST_HOST` / `POSTHOG_ASSETS_HOST`. It also sets `cross_subdomain_cookie` so a single PostHog session is shared across the marketing site and app. Verified locally: static assets return 200 from the EU asset host, and analytics events return 200 through the ingest host. |
||
|
Eric Allam
|
fbd86b6ee9 |
feat(sdk): onEvent observability callback on the chat transport (#4187)
## Summary
`sendMessage` from `useChat` gives no feedback about whether a message
actually reached the backend, and the `fetch` override is wire-level: it
requires knowing endpoint semantics, cannot attribute requests to
messages, and misses the headStart first-turn POST entirely. This adds a
typed `onEvent` observability callback to `TriggerChatTransport` /
`useTriggerChatTransport` so send-success metrics, time-to-first-token,
and "sent but never answered" watchdogs become a few lines of client
code.
## Example
```ts
const transport = useTriggerChatTransport({
task: "my-chat",
accessToken: ({ chatId }) => mintChatAccessToken(chatId),
onEvent: (event) => {
switch (event.type) {
case "message-sent":
// Durably acknowledged by the session's input stream, not just "request accepted".
metrics.increment("chat.message_sent", { source: event.source });
metrics.timing("chat.send_duration_ms", event.durationMs);
break;
case "message-send-failed":
metrics.increment("chat.message_send_failed", { status: event.status });
break;
case "first-chunk":
metrics.timing("chat.ttft_ms", event.sinceSendMs ?? 0);
break;
case "turn-completed":
metrics.timing("chat.turn_duration_ms", event.sinceSendMs ?? 0);
break;
}
},
});
```
## Design
One callback, one discriminated union (`ChatTransportEvent`):
- `message-sent` / `message-send-failed`: terminal send outcomes with
`messageId`, a `source` discriminator (submit, regenerate, steer,
action, stop, head-start), `durationMs`, `bodyBytes`, the append's
idempotency key (`partId`, also stored on the server-side record), and
error + HTTP status on failure. `message-sent` means the append was
durably acknowledged, after any internal token-refresh retries.
- `stream-connected` (with a `resumed` flag and the cursor it connected
from), `first-chunk` (chunk type plus `sinceSendMs` for
time-to-first-token), `turn-completed` (`sinceSendMs` full-turn latency
and the agent's committed input cursor), and `stream-error` follow the
response side, so a send can be paired with the answer that should
follow it. `messageId` on response events is client-side attribution
from the last turn-producing send on that chat.
Emissions sit at the transport's existing choke points, covering every
send path uniformly (including steering and headStart, which the fetch
override cannot observe). Exceptions thrown by the callback are
swallowed: observability can never break the chat. The React hook keeps
the callback live across renders instead of freezing the first-render
closure.
## Verification
Unit tests drive the transport directly with the `fetch` override as the
network stub (send success/failure per source, stream lifecycle, resumed
flag, field enrichment, callback exceptions swallowed). Verified
end-to-end against a realistic metrics setup in the ai-chat reference
app (counters, send-duration and TTFT histograms, and both watchdogs
built purely on these events): a healthy two-turn chat produces exactly
the expected event sequence and TTFT values; an oversized append records
`message_send_failed` with status 413; and killing the worker after a
durable send fires both `sent_but_no_stream` and `sent_but_unanswered`,
reproducing and detecting the "message disappeared" failure mode that
motivated this feature.
|
||
|
Eric Allam
|
fe07de4a2c |
fix(webapp): use provider-reported cost for AI generations when present (#4186)
## Summary The run page could show an AI generation cost well above what the provider actually charged, most visibly for OpenRouter and Vercel AI Gateway requests where a heavily cache-read prompt was priced at the full input rate. When the provider reports an exact per-request cost, we now use that instead of catalog pricing. ## Fix Gateway and OpenRouter include the exact per-request cost in `ai.response.providerMetadata` (`openrouter.usage.cost` / `gateway.cost`). That figure already reflects the cache-read discount and the real per-provider rate, which the catalog cannot reconstruct: cache-read counts do not arrive in `gen_ai.usage.*`, and per-model catalog prices drift from what the provider billed, in either direction. So provider-reported cost is now preferred, and the catalog is used only when no provider cost is present. Fallback routing is covered by the same change: when OpenRouter routes to a different model, `gen_ai.response.model` already carries the served model, so the cost follows the served model and the provider's own figure makes it exact. `extractProviderCost` now runs on every AI span, so it gets a cheap `"cost"` substring guard to skip the JSON parse on reasoning-model spans whose provider metadata carries large reasoning text and no cost field. Regression tests cover the cache-discount overcharge, fallback served-model pricing, gateway cost, and the catalog fallback path. |
||
|
James Ritchie
|
7c5f089d3d | feat(webapp): rework login page and SSO sign-in UI (#4182) | ||
|
Wes Mason
|
76c37ecd24 |
feat(sdk,core,webapp): offload large batch payloads to object storage (#4165)
## Summary `batchTrigger` and `batchTriggerAndWait` (and the by-id and by-task variants) now offload any per-item payload over 128KB to object storage before sending, the same way single `trigger`/`triggerAndWait` already do since [#3785](https://github.com/triggerdotdev/trigger.dev/pull/3785). A batch of large items no longer inflates the request body past the API limit. ## Demo A live local run: `batchTriggerAndWait` of 5 items × 300KB (1.5MB total). Each item offloads to object storage, so the receiver run rows hold a 65-byte `application/store` pointer instead of the 300KB body, and every item round-trips (received == sent). <img width="1000" height="494" alt="batch large-payload offload demo" src="https://github.com/user-attachments/assets/77ae3958-97d6-4b5c-ab25-39b217caefbc" /> ## Design Both the array and streaming batch paths funnel through `executeBatchTwoPhase`, so offloading happens once there: each item is measured, then offloaded through the existing `conditionallyExportPacket` when it crosses 128KB, with bounded concurrency so a big batch doesn't fire an unbounded number of presigned PUTs. Because items are offloaded before the request, SDK batches arrive as small `application/store` references, so the server-side inline offload during item ingest (parallelised in [#3777](https://github.com/triggerdotdev/trigger.dev/pull/3777)) mostly no longer fires for them. Every trigger and item also carries its pre-offload serialised size as `options.payloadSize`. The trigger span records that value, so an offloaded payload shows its real size instead of the size of the small object-store reference (previously the span measured the reference). |
||
|
nicktrn
|
94b30fc1a6 |
fix(webapp): reject deploy images with runtime-incompatible zstd layers (#4184)
Container runtimes (cri-o / containerd / podman) can't pull zstd-compressed layers carried in a Docker v2s2 manifest (`application/vnd.docker.image.rootfs.diff.tar.zstd`). A deploy built with an outdated CLI can produce exactly that combination - and today it's promoted to current and then fails every run at image-pull time. This extends the pre-promotion image check (#4049) to also inspect the manifest's layer media types. If any layer uses the unpullable zstd/v2s2 media type, the deploy is rejected at finalize with a clear message to upgrade the CLI and re-deploy, instead of silently shipping a version that can't start. The manifest is already returned by the existing ECR `BatchGetImage` call, so there's no extra registry request for single-arch images. Parsing is a lenient Zod schema and **fails open** - a manifest we can't read never blocks a deploy. Manifest lists / OCI indexes (no top-level `layers[]`) and OCI zstd (`...tar+zstd`, which runtimes support) pass unaffected. Also clarifies in the contributor docs that changesets and `.server-changes/` notes are user-facing and should be written for users, not maintainers. refs TRI-11702 |
||
|
claude[bot]
|
8bf5879b60 |
test(webapp): poll for replicated rows instead of fixed sleeps in runs replication tests (#4181)
<!-- ccr-slack-attribution --> _Requested by **Matt Aitken** · [Slack thread](https://triggerdotdev.slack.com/archives/C032WA2S43F/p1783430373189849?thread_ts=1783430373.189849&cid=C032WA2S43F)_ ## ✅ Checklist - [x] The PR title follows the convention. - [x] I ran and tested the code works (typecheck of the edited files is clean; see Testing) --- ## Testing **Before:** the webapp run-replication test shard failed on nearly every PR because assertions waited a fixed 1s for rows to replicate from Postgres → ClickHouse and intermittently checked before the row arrived under CI load. **After:** those assertions poll (up to 30s, 250ms interval) until the rows land, so they pass as soon as replication completes and stop flaking, without slowing the happy path. These tests are testcontainers-backed (need Docker + Postgres + ClickHouse), so the full suite is exercised in CI. Locally I confirmed the edited `runsReplicationService.part1..part8.test.ts` files type-check with no new errors. --- ## Changelog **How:** wrapped the ~21 present-row assertions across `runsReplicationService.part1..part8.test.ts` in `vi.waitFor`, matching the existing poll pattern in `part9.test.ts`. Left absence assertions (expecting 0 rows / no spans) on a fixed settle delay since there is nothing to poll for. Tests only — no production code changed. Note: this does NOT touch the `subscribe()` startup race in `internal-packages/replication/src/client.ts` (a riskier, separate follow-up). 💯 --- _Generated by [Claude Code](https://claude.ai/code/session_01KtUdSLKrK17eFVuRYXT6uj)_ --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Chris Arderne
|
aa74e68c71 |
feat(sdk): add bulk replay to api and sdk (#4105)
## Summary
Adds SDK and API support for run bulk actions. You can now create bulk
cancel or replay actions from `@trigger.dev/sdk` using run IDs or the
same filters as `runs.list()`, then retrieve, list, poll, or abort the
action by its `bulk_` handle.
Tests, docs, changesets added.
## Design
The dashboard bulk action service now accepts structured filters instead
of reading directly from a dashboard request, so the dashboard and API
share the same creation path. Replay actions created through the API are
attributed with the existing `api` trigger source, while
dashboard-created actions keep `dashboard`.
The SDK exposes the new surface under `runs.bulk.*`, including
`targetRegion` for replay region overrides and cursor pagination for
listing bulk actions.
## Filters and runIds
Nuance on filters. If `filter` is provided, it MUST have at least one
key. This is to remove the footgun of passing no filter and selecting
all runs.
```typescript
{ action: "cancel", runIds: ["run_1"] } // valid
{ action: "cancel", runIds: [] } // invalid, min(1)
{ action: "cancel", filter: { status: "FAILED" } } // valid
{ action: "cancel", filter: {} } // invalid
{ action: "cancel", filter: {}, runIds: ["run_1"] } // invalid
```
|
||
|
Daniel Sutton
|
d59743bd35 |
fix(webapp,run-ops-database): keep run-ops batch items co-resident with their batch (#4178)
## Summary Three fixes to the run-ops database split (the Cloud-only mode where run-lifecycle rows live on a dedicated Postgres). All are inert in the default single-database deployment. The main fix: on the batch trigger paths, a parentless batch's item runs chose their physical store from a fresh per-org mint-flag read at processing time, so flipping an org's flag mid-batch could land an item in a different store than its batch, breaking the `TaskRun.batchId` foreign key (or silently orphaning the item). The other two harden the split's safety nets: the schema-parity test now actually compares columns, and the read fan-out gate now signals when it has been silently disabled. ## Batch item residency `RunEngineBatchTriggerService` (api.v2) and the BatchQueue item callback (api.v3) now anchor each item's id mint on the batch's own friendlyId, mirroring the already-safe `BatchTriggerV3Service`. Residency is a pure id-shape check, so an item can no longer diverge from its batch across a mid-batch flag flip. The pre-failed-run fallback is anchored the same way (it also sets `batchId`), and the shared mint branch is consolidated into one helper so every mint path stays in lockstep. No new database queries; single-database mode is unchanged (a cuid-shaped batch friendlyId yields a cuid item). ## Schema parity test The parity test previously read only the dedicated schema and matched model headers with regexes, so it never compared columns and could not catch a run-subgraph column that diverged between the two physical schemas. It now parses both schemas and asserts bidirectional scalar-column parity (type, nullability, array-ness, default) across the run-subgraph models, and fails on any field line it can't parse. Scoped to the run-subgraph models so unrelated control-plane edits don't break it. ## Read fan-out signal The split read fan-out gate is decided by the object identity of the NEW vs control-plane clients. It now warns when both run-ops URLs are set but the NEW client isn't a distinct instance (fan-out silently off), and a new test exercises the real topology-into-gate wiring so a future refactor that aliases the clients can't disable fan-out unnoticed. ## Verification New unit and glue tests cover all three changes; the DB-backed residency, store-routing, and topology suites pass against real Postgres; `typecheck` is clean for both packages. |
||
|
Eric Allam
|
add0a7da0a |
fix(sdk,core): stop chat sessions dropping messages that arrive during a turn (#4176)
## Summary Sending a message to a chat whose run had ended could make the message vanish: the continuation run replayed already-answered messages, never processed the new one, and a page refresh lost it entirely. Chasing that report surfaced four composing message-loss bugs in the chat session runtime; this PR fixes all of them, each with a regression test. ## The fixes 1. **Stale resume cursor.** Records delivered while a run was suspended (the waitpoint path) advanced the SSE resume counter but not the committed-consume cursor, so the `session-in-event-id` header stamped on turn-completes went stale by one record per suspended turn. Continuation boots seed from that header, which is what made them replay already-processed messages. `session.in.wait()` now advances both cursors. 2. **Only the first buffered message dispatched.** Messages arriving during a turn are consumed into a buffer whose end-of-turn pickup dispatched only the first entry; the buffer was recreated each turn, so the rest were discarded, and since consuming a record commits the cursor the loss was permanent. A continuation boot's replay delivers several records back-to-back, which put the user's new message at index 1 or later. The buffer now outlives the turn and drains one message per turn in both `chat.agent` and `chat.createSession` (whose equivalent buffer was never read at all). 3. **Post-stop window in `chat.createSession`.** The turn's message listener stayed attached through the stopped turn's post-stream work, so a message sent shortly after stopping a turn was consumed into the dead steering queue and lost. The listener now detaches when the stream settles, matching the `chat.agent` loop. 4. **Handler leak on errored turns.** A turn that threw outside the streaming section (for example from an `onTurnStart` hook) leaked its message listener. Previously that silently lost mid-turn messages; with the loop-level buffer it would have duplicated them instead. The subscription handle is now detached by the turn's catch/finally, and `chat.createSession` defensively detaches its prior turn's listener when user code exits a turn without `complete()`/`done()`. ## Verification Reproduced end-to-end with the ai-chat reference project before the fix (message consumed but never answered, two replayed turns, gone on refresh) and verified after (single clean turn, survives refresh, turn-complete cursors strictly advancing). Regression tests in `packages/trigger-sdk/test/pending-message-drain.test.ts` cover all four, each verified red against the unfixed behavior. A smoke sweep of the standard chat scenarios (basic send, multi-turn, suspend/resume, mid-stream refresh, stop, steering, cancel + continue, and the `createSession` variant) passes on the final branch state. |
||
|
Matt Aitken
|
1a033b665b |
fix(webapp,core): retry run resume through transient database outages (#4161)
## Summary When the platform database is briefly unreachable while a run is resuming from a wait, the run no longer fails with `TASK_EXECUTION_ABORTED`. The worker now retries the resume through the outage instead of aborting on the first blip. ## Root cause Resuming a run calls the engine's `continue` worker-action endpoint. That route caught every error and returned a `422`, which the worker's HTTP client treats as non-retryable. So a transient Prisma infrastructure error (for example `P1001` "Can't reach database server") was flattened into a permanent failure: the worker gave up, force-killed the run process, and completed it with `TASK_EXECUTION_ABORTED`. ## Fix - The `continue` route now lets infrastructure errors propagate to the generic 500 handler (message scrubbed, and retryable by the worker's HTTP client), the same treatment the trigger path already gives them via `isInfrastructureError`. Genuine validation errors (snapshot mismatch, invalid state) still return `422`, so a stale retry stays non-retryable. Resuming is idempotent server-side (guarded by the snapshot id), so retrying is safe. - The worker's `continueRunExecution` calls (both the runner-to-supervisor and supervisor-to-engine hops) retry with a longer, jittered backoff so they can ride out an outage lasting tens of seconds, and the jitter keeps a fleet of resuming runs from stampeding the database the moment it recovers. Builds on #3960, which scrubbed the leaked message on these routes but left the status non-retryable. No changeset: this is a server-side behaviour fix recorded via `.server-changes`. The `@trigger.dev/core` edits are internal run-engine worker plumbing, not a public API change. |
||
|
Chris Arderne
|
018f445467 | chore: switch runners to warpbuild (#4175) | ||
|
Chris Arderne
|
f4af13981a | feat: auto-update dev and preview branch selector (#4171) | ||
|
Wes Mason
|
ce123c13e1 |
fix(llm-model-catalog): stop generated files showing as modified (#4170)
📚 Publish docs / publish (push) Has been cancelled
## Summary Running `pnpm run db:migrate` locally left `defaultPrices.ts` and `modelCatalog.ts` in `llm-model-catalog` showing as modified every time, a ~10k-line diff that only ever changed formatting. This stops the churn. ## Root cause The root `db:migrate` script ends in `&& turbo run generate`, which runs the `generate` script in every package that has one, including this one. The generator writes its output with `JSON.stringify` (quoted keys, no trailing commas), but the checked-in copies had been reformatted by oxfmt (unquoted keys, trailing commas). So the generator output never matched what was committed, even though the parsed data was identical. ## Fix Add the two generated files to `.oxfmtrc.json`'s ignore list and commit the raw generator output, matching how other codegen files in the repo are already handled (e.g. the tsql grammar). Generation is deterministic, so `generate` and `format` are both no-ops on a clean tree now. No changeset: internal package, dev tooling only, no runtime or public API change.docs-release-20260707-1007 |
||
|
Chris Arderne
|
5158ee8ec8 |
chore: use latest self-host compose images (#4140)
## Summary Depends on #4136. Docker Compose self-hosting now uses the maintained `latest` image tag by default instead of the frozen prerelease tag. The version-locking docs keep pointing production users at explicit versioned tags when they want pinned upgrades. |
||
|
github-actions[bot]
|
c584236937 |
chore: release v4.5.1 (#4126)
🚀 Publish Trigger.dev Docker / units (push) Failing after 22s
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 22s
🚀 Publish Trigger.dev Docker / publish-worker-v4 (push) Has been skipped
🚀 Publish Trigger.dev Docker / scan-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker (push) Has been skipped
🧭 Helm Chart Release / lint-and-test (push) Has been cancelled
🧭 Helm Chart Release / release (push) Has been cancelled
🚀 Publish Trigger.dev Docker / 📣 Dispatch main image (push) Has been cancelled
## Summary 1 improvement. ## Improvements - Extend the SSO plugin contract with WorkOS Directory Sync (SCIM) support. ([#4148](https://github.com/triggerdotdev/trigger.dev/pull/4148)) <details> <summary>Raw changeset output</summary> # Releases ## @trigger.dev/build@4.5.1 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.1` ## trigger.dev@4.5.1 ### Patch Changes - Updated dependencies: - `@trigger.dev/build@4.5.1` - `@trigger.dev/core@4.5.1` - `@trigger.dev/schema-to-json@4.5.1` ## @trigger.dev/python@4.5.1 ### Patch Changes - Updated dependencies: - `@trigger.dev/build@4.5.1` - `@trigger.dev/core@4.5.1` - `@trigger.dev/sdk@4.5.1` ## @trigger.dev/react-hooks@4.5.1 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.1` ## @trigger.dev/redis-worker@4.5.1 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.1` ## @trigger.dev/rsc@4.5.1 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.1` ## @trigger.dev/schema-to-json@4.5.1 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.1` ## @trigger.dev/sdk@4.5.1 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.1` ## @trigger.dev/core@4.5.1 ## @trigger.dev/plugins@4.5.1 ### Patch Changes - Extend the SSO plugin contract with WorkOS Directory Sync (SCIM) support. ([#4148](https://github.com/triggerdotdev/trigger.dev/pull/4148)) - Updated dependencies: - `@trigger.dev/core@4.5.1` </details> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>helm-v4.5.1 v4.5.1 v.docker.4.5.1 |
||
|
Chris Arderne
|
ef3eadab3c | chore: back to github runners (#4168) | ||
|
Daniel Sutton
|
4c2c25511b |
test(webapp): split triggerTask engine test into per-concern files (#4167)
The engine `triggerTask` suite was a single 2447-line file with 23 `containerTest` cases, each spinning its own Postgres + Redis. vitest shards by whole file, so all 23 container setups landed on one shard and dominated its wall-clock. The recorded entry in `test-timings.json` badly under-counts the real cost (it does not capture the per-`containerTest` container startup that dominates on CI), so the duration-sharding sequencer treated the file as light and stacked it, producing one ~21 minute shard. Splitting does not reduce the number of container setups; it lets those 23 cases distribute across shards instead of stacking on one. The webapp unit-test stage is gated by its slowest shard, so this cuts the stage's wall-clock roughly in half. ## CI timing (before vs after) Real CI wall-clock of the `Unit Tests: Webapp` shards (`--shard=i/10`). "Before" is sampled from recent runs on other branches (unsplit file, from `main`); "after" is this PR. | Shard | Before (s) | After (s) | |------:|-----------:|----------:| | 1 | 250 | 359 | | 2 | 444 | 411 | | 3 | 497 | 659 | | 4 | **1257** | 284 | | 5 | 545 | 641 | | 6 | 284 | 644 | | 7 | 244 | 214 | | 8 | 340 | 445 | | 9 | 188 | 395 | | 10 | 234 | 567 | | **Slowest shard (gates the stage)** | **~1247s (≈21m)** | **659s (≈11m)** | | Sum of all shards | 4283 | 4619 | Before: shard 4 is the long pole at 1237s / 1247s / 1257s across three sampled runs (the `triggerTask` file plus whatever else the packer put with it). After: the six pieces spread across shards, the slowest drops to 659s. The small rise in summed time is the extra per-file container startup, paid in parallel across shards, so the gating number still falls by about 10 minutes. ## Change Split into six per-concern files that share a `triggerTaskTestHelpers` module (the `vi.mock` calls stay per-file, since vitest hoists them): - `triggerTask.test.ts` (3): trigger + concurrencyKey coercion - `triggerTask.idempotency.test.ts` (4): idempotency + queue resolution - `triggerTask.debounce.test.ts` (4): retries + debounce validation - `triggerTask.mollifier.test.ts` (4): mollifier call-site behaviour - `triggerTask.metadataCache.test.ts` (4): DefaultQueueManager task metadata cache - `triggerTask.residency.test.ts` (4): child run residency inheritance All 23 cases are preserved. The file's `test-timings.json` entry is split across the new files so bin-packing stays balanced. While rewriting these files, cleanup was moved to `onTestFinished(() => engine.quit())` so an `engine`/`Redis` leaked on a failing assertion no longer persists on the worker-scoped Redis and cascades into later cases (`hookTimeout` raised to 60s so the after-cleanup gets the full budget). Prisma lookups switched from `findUnique` to `findFirst` to match the repo convention. Verified: all six files run green locally (23/23), oxlint and oxfmt clean. |
||
|
Daniel Sutton
|
e4ae8cbcd4 |
fix(run-engine,run-store,webapp): stop split-mode waits hanging on resume (#4164)
## Summary On the run-ops database split, a run that waits (`triggerAndWait`, `batchTriggerAndWait`, `wait.forToken`) could hang forever after its wait had already completed. The runner reads a resume from `/snapshots/since` exactly once: if that read returned the resume snapshot without its completed-waitpoints, the runner logged "executing without completed waitpoints", advanced its cursor, and never re-read it, so the awaiting run never continued. ## Root cause The resume snapshot and its completed-waitpoint rows were written as two separate commits. This regressed when the split replaced Prisma's atomic nested `connect` with an FK-free insert (in [#4163](https://github.com/triggerdotdev/trigger.dev/pull/4163)), and `/snapshots/since` is served from a read replica. A fetch landing in the sub-millisecond gap between the two commits, or a multi-reader replica serving the snapshot from a different point in time than its join rows, delivered an empty resume. Because the runner consumes each snapshot once and treats an empty resume as terminal, a single stale read was fatal and produced a permanent, nondeterministic hang. ## Fixes - Commit a snapshot and its completed-waitpoint links in one transaction, restoring the atomicity the split removed. - Repair the completed-waitpoints from the owning primary when a multi-reader replica serves the snapshot without its join rows. This covers single-waitpoint resumes, which carry no `completedWaitpointOrder` and so were missed by the count-based repair. - Read the primary in the checkpoint `WAIT_FOR_BATCH` pre-check, so a batch that already resumed is not re-suspended into a stall. - Fall back to the primary when a waitpoint token misses both read replicas, so a token completed immediately after it was minted no longer returns a spurious 404. - Route batch-item creation by `batchTaskRunId`, consistent with the batch-completion count and the row's foreign key. - Reject control-plane-only relation selects on the dedicated schema with a clear error instead of an opaque Prisma failure, and stop `createDateTimeWaitpoint` bypassing residency routing through a caller transaction. Verified against the deployed split topology: a resume snapshot and its completed-waitpoints are now always delivered together, so the runner can no longer drop a resume. |
||
|
Oskar Otwinowski
|
de65370fb9 |
feat(webapp): Directory Sync (SCIM) for Identity & Access (#4148)
Extend the SSO plugin contract for directory sync and apply membership effects from the accounts webhook worker: provision users in mapped groups (role from group mapping, else the org default role), deprovision on removal, and keep a sticky-removal tombstone so JIT never silently re-adds a removed user. JIT and Directory Sync coexist; roles default to Developer (the JIT default-role picker has no 'None'). Changing a group's role in the dashboard re-applies it to that group's current members immediately. The Directory Sync settings section (group→role mapping, external-domain + manual-membership policy, deferred Save) appears once a domain is verified — independent of SSO — gated by the hasSso flag. The settings page polls the whole page while entitled with override-aware drafts so in-progress edits are never clobbered. |
||
|
Daniel Sutton
|
f101983a70 |
fix(run-store,run-engine): fix run-ops split hangs from wrong-store reads on the resume path (#4163)
## Summary On the run-ops split, NEW-residency runs could hang. Time-based waits (`wait.for`, `wait.until`, `delay`, waitpoint tokens), `batchTriggerAndWait`, and attempt starts stalled and never resumed. Each was a run-ops read or update that hit the wrong database: either the owning store's read replica when it needed read-your-writes, or the wrong store entirely because it routed by an id that does not encode residency. ## Fixes **Waitpoint resume (the main hang).** The managed resume path reads a run's completed waitpoints by snapshot id (`findSnapshotCompletedWaitpointIds`). Snapshot ids are cuids, which always classify to the legacy store, so a NEW run's join rows (which live on the new store) were never found. The resumed run saw zero completed waitpoints and hung. It now fans out across both stores and merges, like its sibling readers. **Batch completion.** Batch item completion (`updateManyBatchTaskRunItems`) routed by the item id, which is also a cuid, so a NEW batch's items were updated on the wrong store, matched zero rows, and the batch was treated as already complete (its parent's `batchTriggerAndWait` then hung). It now routes by the batch id, which does encode residency, matching the sibling `countBatchTaskRunItems`. **Read-your-writes on the resume path.** The block-time pending-waitpoint check (`countPendingWaitpoints`) and the attempt-start lock check (`findRun` in `startRunAttempt`) both read the owning store's replica with no read-your-writes guarantee, so a just-committed waitpoint completion or dequeue lock could be missed under replica lag and strand the run. Both now read the owning primary. Each fix ships with a two-database store or engine test that reproduces the hang and passes with the fix. |
||
|
Daniel Sutton
|
712c7c3b1a |
feat(webapp): add RUN_REPLICATION_RUN_OPS_DATABASE_URL for the runs-replication source (#4160)
## Summary The run-ops runs-replication source now takes its connection URL from `RUN_REPLICATION_RUN_OPS_DATABASE_URL`, required whenever the run-ops split is enabled. The runs replicator speaks the Postgres streaming replication protocol, which cannot run through a transaction pooler, so it needs its own direct endpoint separate from the app's `RUN_OPS_DATABASE_URL` (which may point at a pooler). When the split is on and this is unset, boot fails via `SplitReplicationMisconfiguredError` rather than silently falling back to a wrong endpoint. |
||
|
Daniel Sutton
|
8257499d13 |
fix(supervisor): copy retry-prisma-generate.mjs into the image build (#4157)
## What Adds the missing `COPY scripts/retry-prisma-generate.mjs` to the supervisor `Containerfile` builder stage, before `RUN pnpm run generate`. ## Why The `generate` scripts in `internal-packages/database` and `internal-packages/run-ops-database` shell out to `scripts/retry-prisma-generate.mjs`. The supervisor build never copied that file into the image, so `pnpm run generate` failed: ``` @internal/run-ops-database:generate: Error: Cannot find module '/app/scripts/retry-prisma-generate.mjs' ``` This is the same failure class as #4156 (webapp Dockerfile). The supervisor `Containerfile` is the **only other** build file that runs `pnpm run generate` — the coordinator / docker-provider / kubernetes-provider Containerfiles don't, so this completes the fix. ## Verification Local `docker build` of the supervisor `Containerfile` builder target — result appended below once the build completes. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Daniel Sutton
|
b31ded7ce1 |
fix(docker): copy retry-prisma-generate.mjs into the image build (unbreaks publish) (#4156)
## Build-blocker hotfix `publish.yml` on `main` is failing to build the image after #4154 merged: ``` @trigger.dev/database:generate: Error: Cannot find module '/triggerdotdev/scripts/retry-prisma-generate.mjs' … ERROR: process "/bin/sh -c pnpm run generate" did not complete successfully: exit code: 1 ``` ## Cause #4154 (Windows-CI hardening) changed the `generate` scripts of `@trigger.dev/database` and `@internal/run-ops-database` to call `node ../../scripts/retry-prisma-generate.mjs`. But `docker/Dockerfile`'s `builder` stage does `COPY docker/scripts ./scripts` (replacing the scripts dir) and then copies back only the specific root scripts it needs (`updateVersion.ts`, `bundleSdkDocs.ts`) before `RUN pnpm run generate` — the new `retry-prisma-generate.mjs` wasn't copied, so `pnpm run generate` can't find it and the image build fails. `publish.yml` only runs on push to `main` (not on PRs), so #4154's PR CI never built the image and this slipped through. ## Fix One line — copy the retry script alongside the other root scripts before the generate step: ```dockerfile COPY --chown=node:node scripts/retry-prisma-generate.mjs scripts/retry-prisma-generate.mjs ``` ## Verification Built locally with `docker build --target builder` (the stage that runs `pnpm run generate`) to confirm the generate step now passes — result appended once it finishes. No changeset / `.server-changes` — Dockerfile/build-only change, no package or server-runtime change. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Daniel Sutton
|
092b9ef07a |
fix(run-ops): DNS-safe, sortable base32hex run id (replace base62 KSUID) (#4154)
## Problem
The run-ops split mints NEW-store run ids as **27-char base62 KSUIDs**.
The supervisor writes the run id into the Kubernetes pod name
(`runner-<id>`), and pod names must be DNS-1123 labels (lowercase
`[a-z0-9-]`) — so uppercase base62 ids make k8s reject the pod (422) and
**those runs never launch** (they loop in `PENDING_EXECUTING` until the
heartbeat-stall handler nacks them, forever). `.toLowerCase()` can't fix
it: base62 has both `A`(10) and `a`(36) as distinct symbols, so folding
collides distinct ids and destroys sort order.
## Fix: change the encoding, not the structure
Mint a **26-char lowercase base32hex** run id:
```
run_<24-char base32hex core><region char><version char>
[ 6-byte ms timestamp ][ 9 CSPRNG bytes ]
```
- **base32hex** (RFC 4648 §7, alphabet `0-9a-v`): lowercase,
order-preserving, DNS-safe; 15 bytes → exactly 24 chars, no padding.
Hand-rolled encode/decode (no new dependency).
- **48-bit ms timestamp** in the leading bytes → plain string sort ==
creation order at millisecond resolution.
- **72 bits CSPRNG** entropy; PK unique constraint is the backstop (no
retry loop).
- **region / version** are raw positional chars (read via one `charAt`
before decoding/routing), version = `"1"`.
DNS-safe from birth and hyphen-free, so **firekeeper is unchanged** —
`runner-<id>-attempt-N` → strip `runner-`, cut at first hyphen still
recovers the exact id incl. region+version.
## Residency discriminator: length → version char
`classifyKind`/`classifyResidency` (`runOpsResidency.ts`) previously
distinguished NEW vs LEGACY by **id length**. That gets ambiguous with a
third format. It now discriminates on the **version char at a fixed
position** (`isRunOpsIdBody`: 26 chars, `[25] === "1"`, base32hex
alphabet) → NEW; everything else → LEGACY. Total, never throws. The
`Residency` (NEW/LEGACY) contract the routing store consumes is
unchanged; the `"ksuid"` `ResidencyKind` label is retained only because
it's the persisted `runOpsMintKsuid` feature-flag value.
## Scope / verification
- Generator + discriminator in `@trigger.dev/core` isomorphic; mint path
+ all id-shape call sites swept (~40 webapp files); changeset added
(`@trigger.dev/core` patch).
- Core unit tests (encode/decode round-trip + property, generator shape,
ms sort-order incl. intra-second, parse partitioned-vs-legacy,
firekeeper round-trip): **24 pass**. `@trigger.dev/core` builds; webapp
typechecks; format/lint clean.
## Open decisions (flagged, not silently chosen)
1. **Backward-compat**: existing 27-char base62 KSUID runs now classify
LEGACY. On test cloud these are the broken/looping runs that never
completed, so this is acceptable — but worth a conscious call before
prod. No transitional length-recognition added (keeps the discriminator
clean).
2. **Storage collation**: the sort guarantee is byte-order — if the
run-ops id column is `TEXT` with default locale collation it's silently
not honored. Confirm whether `COLLATE "C"` / `BYTEA` is needed on the
run-ops schema.
3. **Region sourcing** wiring — see `regionCharForRegion` /
`REGION_CODES`.
---
## ⚠️ Required migration — deploy in lockstep
This PR renames a persisted feature-flag key/value and an env var. These
are **not** changed by the code alone and must be migrated when this
deploys, or affected orgs silently fall back to `cuid` minting (no crash
— `defaultValue: "cuid"`):
1. **Env var** (terraform): `RUN_OPS_MINT_KSUID_ENABLED` →
`RUN_OPS_MINT_ENABLED` (carry the value over).
2. **DB** `organization.featureFlags`: migrate both the key and value
together:
- key `runOpsMintKsuid` → `runOpsMintKind`
- value `"ksuid"` → `"runOpsId"`
Until an org's flag row is migrated, its `runOpsMintKind` lookup misses
and it mints `cuid` (legacy) — so no NEW-store ids for that org until
the data lands.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
Daniel Sutton
|
d977691219 | fix(run-store): route caller-passed read clients to the owning store's primary (#4153) | ||
|
Daniel Sutton
|
119189f31a |
fix(webapp): accept all valid run and batch IDs in dashboard filters (#4152)
## Summary The **Run ID** and **Batch ID** filters on the runs list, batches list, and logs view rejected valid IDs. The input showed an error and the **Apply** button stayed disabled, so filtering by an affected run or batch ID from the dashboard was impossible. The filter validators hard-coded exact friendly-id character lengths. Friendly IDs come in three generations that all still exist in the data (`<prefix>_` plus a 21-char nanoid, a 25-char cuid, or a 27-char ksuid), and the hard-coded lengths never covered all three at once. ## Fix All the ID filter validators (run, batch, waitpoint, schedule) now share one helper, `makeFriendlyIdValidator` (`apps/webapp/app/utils/friendlyId.ts`), which validates by prefix plus a base62 body of any known generator length (21 / 25 / 27). The cuid and ksuid lengths are sourced from core so the helper tracks any future change to those formats. Unit tests assert it accepts the output of the real id generators and rejects malformed input. Downstream was already unaffected: run/batch route params and URL-applied filters use unconstrained validation, so only the manual filter inputs needed the fix. |
||
|
Daniel Sutton
|
962bc48738 |
feat(run-ops): automatically migrate the dedicated run-ops database (#4150)
## What
Adds the ability to **automatically migrate the dedicated run-ops
database** (the NEW DB in the run-ops split), matching how every other
database in the system is migrated. Follow-up to the run-ops split
activation.
## Changes
- **Migrate runner** — new
`internal-packages/run-ops-database/scripts/migrate.mjs`, exposed as
`db:migrate:deploy` / `db:migrate:status`. Connects via
`RUN_OPS_DATABASE_URL` (the same var the app uses) and expands `${VAR}`
refs like Prisma's dotenv.
- **Self-host** — `docker/scripts/entrypoint.sh` runs the run-ops
migration on boot when the DB is configured, gated by
`SKIP_RUN_OPS_MIGRATIONS`. Single-DB installs never set the URL, so it's
a clean no-op.
- **Single env-var family** — the run-ops DB is now addressed by one
canonical `RUN_OPS_*` family, connect path and migrations resolving the
identical URL:
- `RUN_OPS_DATABASE_URL` (writer) — replaces `TASK_RUN_DATABASE_URL`
- `RUN_OPS_LEGACY_DATABASE_URL` — replaces
`TASK_RUN_LEGACY_DATABASE_URL`
- `RUN_OPS_DATABASE_READ_REPLICA_URL` — replaces
`TASK_RUN_DATABASE_READ_REPLICA_URL`
- the old `TASK_RUN_*` aliases, the `??` coalesce, the
`runOpsNewDatabaseUrl` indirection, and the migrate-only `directUrl` are
all removed (consumers read `env.RUN_OPS_DATABASE_URL` directly).
`directUrl` was dropped because it was only ever used by `prisma
migrate` (never the app runtime) to bypass a pooler for advisory locks —
premature here since the run-ops connection isn't wired to the app yet.
If a pooler is later introduced for the app, a direct URL can be
reintroduced then.
## Safety
- **Pure rename** — nothing deployed sets any `TASK_RUN_*` var (the
split isn't activated anywhere yet; `.env.example`, docker-compose, and
cloud already use `RUN_OPS_*`), so there is no config migration.
- **Single-DB / self-host** — no new required env var; entrypoint and
migrate are no-ops when `RUN_OPS_DATABASE_URL` is unset.
- **Cloud** — runs migrations as pre-deploy ECS tasks (companion cloud
PR), calling these same `db:migrate:deploy` / `db:migrate:status`
commands.
## Verification
- Live migration against a fresh scratch DB with only
`RUN_OPS_DATABASE_URL` set: both migrations applied, no `P1012`/`P1013`;
`${VAR}` expansion, idempotent re-run, `status`, and no-op skip all
pass.
- Schema parity 4/4; `typecheck --filter webapp` 18/18; affected
split/replication tests 34/34.
## Scope
This delivers automatic migrations only. Enabling the app to *use* the
new DB (setting `RUN_OPS_DATABASE_URL` + `RUN_OPS_SPLIT_ENABLED` on the
service) is a separate activation step.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
Daniel Sutton
|
c9f427e21f |
fix(replication): key logical replication leader lock on slot name (#4151)
## Problem
`LogicalReplicationClient` uses a Redlock leader lock to guarantee a
single active consumer per Postgres logical replication slot. The lock
resource was keyed on the client `name`:
```
logical-replication-client:${this.options.name}
```
A slot permits exactly one consumer, so the lock's job is to serialize
consumers **of a given slot**. Keying it on `name` breaks that whenever
two clients target the same slot with different names — most notably
across a rolling deploy where the client `name` changes but `slotName`
does not. Both acquire *distinct* locks, both consider themselves
leader, and the second to reach `START_REPLICATION` hits `replication
slot "<slot>" is active for PID <n>`. Because that query was
fire-and-forget and its failure was only logged (no retry), the consumer
stopped and replication stalled until the process was restarted.
## Fix
**1. Key the leader lock on `slotName`** — the actual single-consumer
resource:
```
logical-replication-client:${this.options.slotName}
```
Consumers of the same slot now contend on the same lock and hand off
cleanly across restarts/deploys; different slots stay independent.
`name` is kept for logging and the pg `application_name`.
**2. Self-healing resubscribe** (`resubscribeOnFailure`, opt-in) —
instead of logging-and-dying, a client re-subscribes with exponential
backoff after a lost election or a failed `START_REPLICATION`, so a
rolling deploy self-heals: the incoming pod retries until the draining
pod releases the slot, then takes over. Safety:
- `#cleanupAttempt()` unconditionally ends the pg client (freeing the
walsender) and releases the leader lock before rescheduling — retries
never leak connections/locks.
- `shutdown()` sets an intentional-stop latch re-checked after every
`await` in `subscribe()` (and aborts the lock-acquire spin), so a
resubscribe can never race or outlive an intentional shutdown.
- Backoff resets only on genuine stream start, so a permanently stuck
slot backs off to the ceiling and logs loudly rather than tight-looping;
an epoch guard neutralises stale `START_REPLICATION` catches.
Runs- and sessions-replication opt in and use `shutdown()` for all
intentional stops.
**3. Observability** — the admin runs-replication status route probed
the old name-keyed Redis key (would report `leader:false` for every
source after fix #1); now probes the slot-keyed key.
## Tests
`internal-packages/replication/src/client.test.ts` (real Postgres +
Redis containers):
- same-slot/different-name → second client must not double-lead or race
into "slot is active" (the regression)
- a failing `START_REPLICATION` retry loop must not leak connections or
locks
- `shutdown()` during an in-flight `subscribe()` must not leave a zombie
leader
- `subscribe()` after `shutdown()` re-arms `resubscribeOnFailure`
- self-heals once the leader releases the slot
Plus the multi-source wiring test updated to the slot-keyed lock keys.
## Rollout
With the self-healing resubscribe, this ships as a **plain rolling
deploy** — the incoming pods retry across the one-time lock-key
transition and take over once the old pods drain (a brief replication
stall that the durable slot replays on reconnect — no data loss). No
stop-before-start required.
|
||
|
Daniel Sutton
|
70bca82d84 | feat(run-ops): activation — drop cross-DB FKs, provision run-ops DB, enable split (#4124) | ||
|
Chris Arderne
|
1f4369d092 |
chore: make AGENTS.md canonical, point CLAUDE.md at it with @AGENTS.md (#4142)
Tested to confirm that Claude Code picks up the `@AGENTS.md` automatically with no agent turns. |
||
|
Daniel Sutton
|
618b921a51 | feat(run-ops): webapp routes — friendlyId reads, cross-seam token resolution, co-location writes (#4123) | ||
|
Daniel Sutton
|
0a51341347 | feat(run-ops): read presenters — de-join control-plane relations + read-through hydration (#4122) | ||
|
Daniel Sutton
|
5be6a4fe38 |
feat(run-ops): ClickHouse multi-source replication fan-in + admin ops (#4119)
## What Extends the ClickHouse runs-replication service to fan in from multiple Postgres sources (the control-plane DB and the run-ops DB) instead of a single source, plus the admin operations to run and observe it. - **Multi-source fan-in** (`services/runsReplicationService.server.ts`, new `runsReplicationInstance.server.ts`, `runsReplicationGlobal.server.ts`): factors the replication service into per-source instances and a coordinator so a single ClickHouse target is fed from more than one Postgres source. - **Admin ops** (`routes/admin.api.v1.runs-replication.status.ts`, `admin.api.v1.runs-replication.backfill.ts`, `v3/services/adminWorker.server.ts`): adds a status endpoint reporting per-source replication state and updates the backfill entrypoint for the multi-source shape. ## Why PR7 of the run-ops split stack, and the final piece: once run state can live in a separate run-ops DB (earlier PRs), the analytics replication into ClickHouse has to consume both sources so runs remain queryable regardless of residency. Behavior-changing for the replication service internals; the ClickHouse-facing output is unchanged (still one runs stream), and single-source operation is preserved when the split is not enabled. ## Tests New vitest coverage: `runsReplicationInstance.test.ts` (per-source instance behavior) and `runsReplicationService.part8`/`part9` suites exercising the multi-source coordinator. Testcontainers-backed (ClickHouse + Postgres); no mocks. ## Notes Draft, **stacked on #4118** (`runops/pr06-write-path`). Review that first; this diff is against it. Server-change / changeset note to be added at stack-assembly time. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Daniel Sutton
|
84f3e1b39c |
feat(run-ops): webapp write path — trigger/batch minting, idempotency routing, run lifecycle (#4118)
## What Routes the webapp write path through the run-ops split seam: trigger/batch minting, idempotency-key resolution, and the run-lifecycle services now determine residency and dispatch writes to the correct store. - **Trigger & batch** (`runEngine/services/triggerTask.server.ts`, `batchTrigger.server.ts`, `createBatch.server.ts`, `streamBatchItems.server.ts`, `v3/services/batchTriggerV3.server.ts`): mint ids with the run-ops-aware minting and route creation/streaming through the store; batch children inherit the parent's residency. - **Idempotency** (`runEngine/concerns/idempotencyKeys.server.ts` + new `idempotencyResidency.server.ts`): idempotency-key lookup/dedup is residency-aware so a keyed retrigger resolves against the store that owns the original run. - **Run lifecycle services** (`createCheckpoint`, `createTaskRunAttempt`, `enqueueDelayedRun`, `expireEnqueuedRun`, `finalizeTaskRun`, `resumeBatchRun`, `cancelDevSessionRuns`, `executeTasksWaitingForDeploy`, `triggerFailedTask`): resolve their target run through the store rather than a fixed client. - **Reads that fan out from writes** (`runsRepository` + `clickhouseRunsRepository`, `BulkActionV2` + batch read-through, realtime `sessions`/`runReader`, alerts `deliverAlert`/`performTaskRunAlerts`): route through the read-through resolver. - `9535ae63d` — resolves the parent run through an injectable run store in `TriggerFailedTaskService`. - `bf8f7c881` — drops the "known-migrated" concept from write-path and read repos; residency is id-shape only. - `515b897ea` — self-defaults `resolveWaitpointThroughReadThrough` to the safe run-ops clients. ## Why PR6 of the run-ops split stack. This is the write-path counterpart to the read foundation in the previous PRs: with it in place, both reads and writes route through the seam. Additive when the split is disabled (id-shape resolution collapses to the control-plane client); behavior-changing on the minting, idempotency, and lifecycle paths when enabled. ## Tests Large new/expanded vitest suite under `apps/webapp/test/` and colocated service tests: trigger-task and batch-trigger store routing, residency inheritance, idempotency dedup residency + legacy-authority, bulk-action read routing, cancel-dev-session routing, alerts store routing, runs-repository read-through, realtime session/run-reader read-through and stream-registration routing, and the waitpoint read-through default. Testcontainers-backed; no mocks. ## Notes Draft, **stacked on #4117** (`runops/pr05-webapp-foundation`). Review that first; this diff is against it. Server-change / changeset note to be added at stack-assembly time. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Daniel Sutton
|
8465ac5ac3 |
feat(run-ops): webapp db topology, flags, and split-mode resolver wiring (#4117)
## What Wires the run-ops split into the webapp: database topology, environment flags, split-mode gating, and the control-plane resolver/cache layer that the run-store and run-engine seams from the previous PR plug into. - **DB topology & env** (`apps/webapp/app/db.server.ts`, `env.server.ts`, `entry.server.tsx`): adds the run-ops database clients/topology and the environment variables that configure and gate the split. - **runOpsMigration module** (new `apps/webapp/app/v3/runOpsMigration/`): the webapp-side machinery — `splitMode.server.ts`, `controlPlaneResolver.server.ts` + `controlPlaneCache.server.ts`, `readThrough.server.ts`, `crossSeamGuard.server.ts`, `distinctDbSentinel.server.ts`, id-minting helpers (`mintBatchFriendlyId`, `runOpsMintKind`, `resolveInheritedMintKind`), `runOpsCascadeCleanup.server.ts`, the split read gate, and route/unblock catalogs. - **Store/engine wiring** (`app/v3/runStore.server.ts`, `runEngine.server.ts`, `runEngineHandlers.server.ts` + new `runEngineHandlersShared.server.ts`): points the webapp's store/engine construction at the resolver, and factors shared handler logic out so both seams use one path. - **Read-path touch-ups**: `runtimeEnvironment.server.ts`, `eventRepository/index.server.ts`, `taskRunHeartbeatFailed.server.ts`, `engineVersion.server.ts` route their run/environment lookups read-through the resolver. - `413a94511` — interlocks split mode against the native realtime backend so the two aren't enabled in an incompatible combination (see `.server-changes/run-ops-split-realtime-interlock.md`). - `dc74c57fd` — drops the earlier "known-migrated" read layer; residency is determined by id-shape only. ## Why PR5 of the run-ops split stack. This is the webapp foundation layer: it stands up the DB topology, flags, and resolver/cache the rest of the stack depends on, and repoints webapp read paths through the resolver. Additive when the split is not enabled (existing single-DB behavior preserved behind flags); behavior-changing on the read-through paths and the realtime interlock. ## Tests New vitest coverage across `apps/webapp/test/` and colocated `*.server.test.ts` files: db topology, split mode, split read gate, cross-seam guard, mint cutover / flip latency, control-plane cache, control-plane resolver, distinct-db sentinel, read-through loaders (route loaders, run-detail loaders, `findEnvironmentFromRun`), and the run-engine handlers. Testcontainers-backed; no mocks. `pnpm-lock.yaml` synced for the two new webapp deps. ## Notes Draft, **stacked on #4116** (`runops/pr04-store-engine`). Review that first; this diff is against it. Server-change / changeset note to be added at stack-assembly time. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Daniel Sutton
|
c266e96c87 |
feat(run-ops): run-store routing seam + run-engine read seams (#4116)
## What Introduces the run-store routing seam and the run-engine read seams that let run lifecycle operations be dispatched to either the control-plane database or a separately-generated run-ops database, depending on where a run/batch resides. - **run-store** (`internal-packages/run-store`): adds `runOpsStore.ts` and substantially expands `PostgresRunStore.ts` so the store can resolve residency and route reads/writes to the correct backing client. `types.ts` grows the routing/residency types; `NoopRunStore.ts` is removed. - **run-engine** (`internal-packages/run-engine`): adds `engine/controlPlaneResolver.ts` and routes the per-system read paths (dequeue, enqueue, waitpoint, checkpoint, run-attempt, ttl, delayed-run, execution-snapshot, pending-version, debounce, batch) through the resolver/store instead of talking to a single Prisma client directly. `engine/errors.ts`, `engine/types.ts`, and `engine/index.ts` are extended to support injecting the store/resolver. Three fixes are included on top of the seam work: - `c6cadd85f` — routes read-your-writes to the owning store's **writer**, not its lagging replica, so an operation immediately reading back what it just wrote sees a consistent result. - `05c912e05` — normalizes run-ops-generation Prisma errors to the control-plane error class at the store **write boundary**, so `instanceof` checks and the `P2002` → 422 handling continue to work across the separately-generated run-ops Prisma client. - `88d12907f` — resolves NEW-resident batches in `ApiBatchResultsPresenter` by routing the batch read through the store, so a dedicated-DB batch resolves instead of returning 404. The change is heavily test-first: the bulk of the diff is new unit/integration coverage for the store routing, residency, and each run-engine system's control-plane resolver path. ## Why PR4 of the run-ops split stack (PR1–PR3 land the ClickHouse test-container and earlier plumbing). This PR is the read-path foundation: it adds the seam and read-routing but leaves the write path to route through the same seam in a later PR. Behavior-changing where the three fixes above touch existing read-your-writes / error-normalization / batch-resolution paths; otherwise additive (new store module, new resolver, injectable dependencies with existing single-client behavior preserved when no dedicated store is configured). ## Tests Extensive new vitest coverage under `run-store/src/*.test.ts` (routing, residency, dual-schema select, cross-generation error normalization, read-after-write, idempotency dedup, mixed residency, waitpoint co-location) and `run-engine/src/engine/**/*.test.ts` (per-system `controlPlaneResolver` tests, injectability, block-edge residency, waitpoint read residency, trigger-create routing, lifecycle router). Testcontainers-backed; no mocks. ## Notes Draft, **stacked on #4114** (`runops/pr03-clickhouse-tc`). Review that first; this diff is against it. Server-change / changeset note to be added at stack-assembly time. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
nicktrn
|
ad74568955 |
ci: run oxfmt + oxlint on pre-push (lefthook) (#4147)
Adds a lefthook `pre-push` job that runs the same checks as CI code-quality (`oxfmt --check .` + `oxlint .`), so formatting/lint failures are caught locally before they reach a PR. Uses the existing lefthook setup - no new tooling. Caveat noted in the config: GitButler uses its own git implementation and only runs hooks when "Run hooks" is enabled in its per-project settings; with that off, this protects plain `git push`. Enable that setting to have it fire on `but push` too. |
||
|
Chris Arderne
|
86235e5cc7 | feat(webapp): add Abort button to bulk actions list rows (#4144) | ||
|
Daniel Sutton
|
2fd896ced6 |
fix(db:migrate): skip @internal/run-ops-database in local db:migrate (#4145)
`db:migrate` ran `prisma migrate deploy` for `@internal/run-ops-database`, which requires the dedicated run-ops DB (:5434) that isn't up in a default local — breaking local `db:migrate` for everyone; excluding it with `--filter=!@internal/run-ops-database` restores it. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
claude[bot]
|
b272529518 | chore: remove stale server-change file (#4143) | ||
|
Wes Mason
|
58b114f4c9 |
feat(webapp): seed a local CLI personal access token (#4135)
## Summary Getting the CLI talking to a local instance meant the browser magic-link login, which is no good when you're driving things headlessly (an agent, a container, or just no browser to hand). The seed already prints dev secret keys for the batch-limit orgs, so it now also mints a personal access token for the seeded `local@trigger.dev` user and prints a ready-to-run `export TRIGGER_ACCESS_TOKEN=...` next to them. Re-seeding stays idempotent: it decrypts and reprints the existing `local-dev-cli` token rather than piling up a new one on every run. <!-- GitButler Footer Boundary Top --> --- This is **part 2 of 2 in a stack** made with GitButler: - <kbd> 2 </kbd> #4135 👈 - <kbd> 1 </kbd> #4137 <!-- GitButler Footer Boundary Bottom --> --------- Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> |
||
|
claude[bot]
|
a1d14e7fef |
chore: remove non-user-facing changesets (#4141)
<!-- ccr-slack-attribution --> _Requested by **Eric Allam** · [Slack thread](https://triggerdotdev.slack.com/archives/C061L2MHW93/p1783083021892389?thread_ts=1783083021.892389&cid=C061L2MHW93)_ ## ✅ Checklist - [ ] I have followed every step in the [contributing guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md) - [x] The PR title follows the convention. - [ ] I ran and tested the code works --- ## Testing N/A — this change only removes two changeset markdown files; there are no code changes to test. --- ## Changelog ### What changed Removes two `@trigger.dev/core` changesets that were added for changes that are not user-facing package changes: - `.changeset/runops-core-residency.md` — internal run-ops residency classifier + ksuid mint/decode primitives - `.changeset/telnet-dev-logs.md` — dev-only `@trigger.dev/core/v3/telnetLogServer` module ### Why Per the convention that changesets should only be added when there are actual user-facing package changes, these two do not qualify. Removing them keeps the release (currently the automated v4.5.1 PR #4126) from bumping `@trigger.dev/core` for internal/dev-only additions. --- _Generated by [Claude Code](https://claude.ai/code/session_019xCdLwoozZm4Gr5ZHLYiws)_ Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Chris Arderne
|
2ad39443be |
chore: update self-host image release references (#4136)
## Summary Stable v4 Docker image builds now also publish `v4` and `latest` tags, giving Docker-based self-hosters a maintained floating tag to use after a stable release. The Kubernetes guide now uses the current Helm chart line and current pinned examples, so new installs and upgrades resolve to the 4.5 chart line instead of the 4.0 line. ## Design The publish workflows add the floating tags only for stable `v4.x.x` image tags. Prerelease and `main` builds keep their existing tags. |
||
|
Daniel Sutton
|
a93b101d44 |
feat(run-ops): cross-producer ClickHouse version helper + cross-Postgres-version compat tests (#4114)
## What - Adds `composeTaskRunVersion` to `@internal/clickhouse` (exported from the package index). It packs a small `originGeneration` epoch into the top 8 bits of the ReplacingMergeTree version and keeps the producer's own LSN in the low 56 bits, so `task_runs_v2` rows replicated from more than one Postgres producer become globally comparable while preserving in-producer ordering. Single-producer setups never call it and keep using the raw LSN version. - Adds unit coverage for the helper in `taskRuns.test.ts` (bit layout, ordering, epoch precedence, range validation). - Adds two run-engine tests that exercise the cross-Postgres-version testcontainer fixture: - `heteroPostgresFixture.test.ts` — a smoke test asserting byte-identity and identical `ORDER BY` (under a pinned ICU collation) across two different Postgres major versions. - `crossVersionCompat.test.ts` — mirrors the run-engine's real raw-SQL surfaces and asserts byte-identical, ordering-identical results across the two versions. Ships with an env-gated block (skipped by default) that can be pointed at a real dedicated database in CI. ## Why Third PR in the run-ops split stack. It is purely additive: it introduces one new exported helper plus tests and changes no runtime call sites, so on its own it has no runtime behavior change. It lays down the version-composition primitive and the cross-version compatibility proof that later PRs in the stack rely on. ## Tests - New unit tests for `composeTaskRunVersion` in `internal-packages/clickhouse/src/taskRuns.test.ts` (run against a real ClickHouse testcontainer). - New cross-version tests in `internal-packages/run-engine/src/engine/tests/` running against real Postgres containers of two different major versions (no mocks). The env-gated dedicated-database block is skipped unless its URL is set. ## Notes Draft, **stacked on #4113** (`runops/pr02-db-foundation`). Review that first; this diff is against it. Server-change / changeset note to be added at stack-assembly time. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Chris Arderne
|
20f9c781f6 |
fix(webapp): db seed script (#4137)
## Summary
`pnpm run db:seed` failed with `SyntaxError: The requested module
'./app/models/organization.server' does not provide an export named
'createOrganization'`, even though that export exists. Renaming the seed
entry
point from `seed.mts` to `seed.ts` runs it as CommonJS and fixes the
failure.
No seed logic changes.
## Root cause
`seed.mts` is an ES module, but the server modules it imports (`.ts`
files, and
no package declares `"type": "module"`) resolve as CommonJS. tsx
compiles those
to CommonJS using esbuild's getter-based export shape
(`Object.defineProperty(exports, name, { get })`), which Node's
`cjs-module-lexer` does not detect when it links the ESM importer. The
named
exports look absent, so linking throws before any code runs.
The seed only ever needed the `.mts` extension: it has no top-level
`await` and
no `import.meta`. Running it as `.ts` keeps the whole import chain
CommonJS to
CommonJS and never crosses the lexer boundary. Verified the seed runs to
completion after the change.
Possibly caused by Node 22 upgrade?
|
||
|
Daniel Sutton
|
84588bd236 |
feat(run-ops): dedicated run-ops database package + docker service + migration runner (#4113)
## What The **dedicated run-ops database** foundation for the split: a standalone Prisma package plus the infra to run and migrate it. - **`internal-packages/run-ops-database`** — a new Prisma package (`@internal/run-ops-database`) whose schema mirrors the run-execution tables that will live on the dedicated DB, with its own generated client, migrations, and migration runner. - **`prisma/schema.parity.test.ts`** — a parity test that guards the run-ops schema against drift from the control-plane schema for the mirrored tables. - **Docker** — a Postgres 17 service (`docker/Dockerfile.postgres17`, `docker/docker-compose.yml`) so the dedicated DB is available locally under the run-ops compose profile. - **Testcontainers** — hetero fixtures (PG14 legacy + PG17 dedicated) so later PRs can exercise cross-database behaviour with real containers rather than mocks. ## Why This is the **second PR in the run-ops split stack**, stacked on the core primitives. It stands up the dedicated database and its tooling. There is **no runtime wiring** into the webapp here — the app does not read or write this DB yet; that arrives in later PRs. On its own this PR only adds a package, a docker service, and test fixtures. ## Tests Schema-parity test for the run-ops schema; hetero testcontainer fixture smoke test. ## Notes - Draft, **stacked on #4112** (`runops/pr01-core-residency`). Review that one first; this diff is against it. - Server-change / changeset note to be added at stack-assembly time. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Daniel Sutton
|
4cfa596271 |
feat(core): id-shape residency classifier + ksuid mint primitives (run-ops split base) (#4112)
## What Foundation for the run-ops database split: an isomorphic **id-shape residency classifier** and the **ksuid mint primitives**, added to `@trigger.dev/core` under `v3/isomorphic`. - **`runOpsResidency.ts`** — classifies a run id by its shape: 25-char cuid → `LEGACY`, 27-char ksuid → `NEW`. Pure and environment-free (safe on both client and server). - **`friendlyId.ts`** — ksuid mint primitives and id helpers. - Both exported via `v3/isomorphic/index.ts`. ## Why This is the **base of a stacked series** implementing the run-ops DB split (routing run-execution data to a dedicated database by id-shape). Later PRs in the series consume this classifier and these primitives to route reads and writes across the two databases. On its own this PR is **purely additive** — new isomorphic helpers with unit tests, no runtime wiring, and no behaviour change to existing code paths. ## Tests Unit tests for the classifier (`runOpsResidency.test.ts`) and the id / mint primitives (`friendlyId.test.ts`). ## Notes - Draft, stacked on `main`; subsequent PRs in the series build on top of this one. - A changeset for `@trigger.dev/core` will be added before this is marked ready for review. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Eric Allam
|
eecb27c672 |
fix: deploy the dashboard agent one environment at a time (#4133)
The dashboard agent deploy runs a matrix leg per environment (staging, prod). Running them in parallel deployed the same Trigger.dev project twice at once, and one leg failed to create the background worker while the other succeeded. `max-parallel: 1` runs the legs sequentially so a deploy never races itself. Also aligns the workflow's Node version to the repo standard (`22.23.1`, per `.nvmrc`). Verified: re-running the staging leg on its own (no parallel prod) deployed cleanly. |