23016de17949710ecd8b4e591f5f172aa2c75635
7980 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Matt Aitken
|
23016de179 |
feat(cli): warn on createRequire packages missing from deployed images (#4851)
## Summary
A package loaded with `createRequire(import.meta.url)("pkg")` is
invisible to esbuild: the call is never resolved, so the package is
neither bundled nor collected as an external to install in the deployed
image. The deploy succeeds with zero diagnostics and the task fails at
runtime with a module-not-found error, which can surface as something
far more confusing when a library maps errors coarsely (a database
driver loaded this way can look exactly like a connection failure). It
also works fine in `trigger dev` because the local `node_modules`
exists, making the production-only failure extra misleading.
Both `deploy` and `dev` builds now warn about this, pointing at the
exact file and line, with a note showing the exact config that fixes it:
```
▲ [WARNING] "mssql" is loaded with createRequire() but won't be available in the deployed image, so loading it will fail at runtime. The bundler can't follow createRequire() calls, so "mssql" is neither bundled into your code nor installed in the image. [plugin create-require-collector]
src/db.ts:12:14:
12 │ const mssql = createRequire(import.meta.url)("mssql");
╵ ^
To fix this, install "mssql" into the image by adding the additionalPackages build extension to your trigger.config.ts:
import { additionalPackages } from "@trigger.dev/build/extensions/core";
export default defineConfig({
// ...
build: {
extensions: [additionalPackages({ packages: ["mssql"] })],
},
});
Alternatively, replace the createRequire() call with a static import so the package is bundled. Docs: https://trigger.dev/docs/config/extensions/additionalPackages
```
In `dev` the message instead explains that the code works locally but
deploys of it will fail, so the problem is caught while the code is
being written rather than after a deploy.
## How it works
An esbuild plugin scans the bundle's input files outside `node_modules`
for string-literal specifiers passed to `createRequire`-created require
functions: `createRequire(...)("pkg")`, `const req = createRequire(...);
req("pkg")`, `req.resolve("pkg")`, aliased imports, namespace access,
CJS destructuring, and dynamic `import("node:module")` bindings. Sources
are parsed with `@babel/parser` (already in the dependency tree), so
comments, strings, templates, regex literals and JSX can't confuse the
scan; a file that fails to parse is skipped. Relative paths and node
builtins never warn.
A usage only warns when the package will actually be missing from the
image. On deploys the resolved manifest externals are the source of
truth (extension-installed layers are already merged in when the warning
runs); `build.external` alone deliberately does not suppress, because
marking a package external installs nothing when nothing statically
imports it. In dev, which predicts a future deploy, suppression
additionally trusts what extensions declare they install, and stays
silent entirely when that can't be determined (an extension hook throws,
or an older `@trigger.dev/build`'s additionalPackages predates the
declaration hook), so dev never makes a false "deploys will fail" claim.
`additionalPackages` declares its packages via a new diagnostics-only
`BuildExtension` field, `installedPackagesForTarget`, which the bundler
ignores: bundling output is unchanged for existing projects.
Detection is name-based, module-level, and deliberately per-file:
computed specifiers, shadowed names, and require helpers imported from
other files are not followed (those degrade to today's behavior, an
unwarned runtime failure), and scanning is scoped to user code because
bundled libraries legitimately use optional-require patterns that would
drown real findings in noise. Packages named in build-layer install
commands (`RUN npm install ...`) are suppressed individually.
Deploys also now surface esbuild's own bundle warnings for user files
(for example `require()` with a non-literal argument), which were
previously discarded on the deploy path; `trigger dev` already showed
them.
|
||
|
Matt Aitken
|
ef7b3aaf38 |
fix(run-engine,webapp): guard run finalization against lost resume signals (#4849)
## Summary A run's finish commit and its follow-up side effects (completing the associated waitpoint, waking blocked parents, releasing the queue slot, nudging batch completion) are separate writes across Postgres and Redis. If a database error landed between them, the child run was already finished, so the runner's retries hit the "Run is already finished" guard and the completion signal was lost for good. A parent blocked on `triggerAndWait` or `batchTriggerAndWait` then stayed waiting forever. TTL expiry had the same shape: its worker retry returned early on a non-pending run, and the batch expiry path swallowed a failed waitpoint-job enqueue. ## Fix Every finalizing path (attempt success, permanent failure, cancellation, TTL expiry) now enqueues a durable `ensureRunFinalized` job before the finish commit, and acks it once the inline side effects all succeed. In steady state the guard never executes; the cost is one Redis enqueue and ack per completion. When the inline path dies in between, the guard fires after a short delay and re-derives everything from current state: it releases the run's queue message and concurrency slot, completes a still-pending associated waitpoint from the run row's output or error, re-runs the blocked-run fan-out (covering a lost unblock enqueue even after the waitpoint committed), and re-schedules the batch completion check. Every leg is idempotent, so racing the inline path is a no-op. The job retries with a capped backoff for roughly five weeks before dead-lettering, so it outlives any database outage while a genuinely poisoned item still becomes visible. Cancellation gets special handling: CANCELED is the only terminal run status where execution can still be in flight, so the guard only re-delivers for a canceled run once its execution snapshot is FINISHED, re-arming itself until then rather than resuming the parent while the child is still winding down. A `finalization_rederivations` counter increments whenever the guard actually re-delivers a lost signal; it should stay at zero in a healthy system. Tests cover six shapes: waitpoint completion lost after the finish commit, unblock fan-out lost after the waitpoint completed, a failed guard enqueue failing the completion request with nothing committed, a stale guard held back during an in-flight cancellation, waitpoint completion lost during TTL expiry, and the happy path where the guard is acked and never runs. Known accepted edge: a guard re-run after a partial inline completion can re-emit a cached-run completion event for the same span; this only happens during failure recovery and is bounded to duplicate trace events. |
||
|
github-actions[bot]
|
52848d8266 |
chore: release v4.5.15 (#4831)
## Summary
3 improvements.
## Improvements
- Stop shipping compiled test files in the published packages. The
`*.test.ts` sources were being emitted into `dist`, adding dead weight
to every install and leaving modules that `require("vitest")` (not a
dependency) inside the tarball, which tripped tooling that walks every
file in a package.
([#4833](https://github.com/triggerdotdev/trigger.dev/pull/4833))
- Named side channels on a Session: durable, two-way realtime streams
that outlive a single run and are shared across runs. Open a channel
with `sessions.open(id).channel(name)` (or `chat.channel(name)` inside a
`chat.agent`) to get an `.in`/`.out` pair addressed by name rather than
the reserved default pair. Writing a side channel's `.in` does not wake
or trigger a run, so a channel can carry out-of-band data (a stream of
frames, a control signal) that many clients read while the agent
produces it.
([#4815](https://github.com/triggerdotdev/trigger.dev/pull/4815))
```ts
// Inside a chat.agent: stream frames on a named channel, wakes nothing
const frames = chat.channel("screenshots");
await frames.out.append(frame);
frames.in.on((control) => {
/* client control, no suspend */
});
```
Declare channel record types once with `sessions.defineChannel(...)` and
infer them on both the producer and the consumer, including
`useSessionStreamChannel` in React. Channels get a default retention
that keeps them bounded, overridable per channel.
- Session `triggerConfig.tags` now accepts up to 10 tags, matching the
run tag limit. Previously it was capped at 5, which for `chat.agent`
left room for only 4 of your own tags after the automatic
`chat:{chatId}` tag.
([#4832](https://github.com/triggerdotdev/trigger.dev/pull/4832))
<details>
<summary>Raw changeset output</summary>
# Releases
## @trigger.dev/build@4.5.15
### Patch Changes
- Stop shipping compiled test files in the published packages. The
`*.test.ts` sources were being emitted into `dist`, adding dead weight
to every install and leaving modules that `require("vitest")` (not a
dependency) inside the tarball, which tripped tooling that walks every
file in a package.
([#4833](https://github.com/triggerdotdev/trigger.dev/pull/4833))
- Updated dependencies:
- `@trigger.dev/core@4.5.15`
## trigger.dev@4.5.15
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.15`
- `@trigger.dev/build@4.5.15`
- `@trigger.dev/schema-to-json@4.5.15`
## @trigger.dev/core@4.5.15
### Patch Changes
- Stop shipping compiled test files in the published packages. The
`*.test.ts` sources were being emitted into `dist`, adding dead weight
to every install and leaving modules that `require("vitest")` (not a
dependency) inside the tarball, which tripped tooling that walks every
file in a package.
([#4833](https://github.com/triggerdotdev/trigger.dev/pull/4833))
- Named side channels on a Session: durable, two-way realtime streams
that outlive a single run and are shared across runs. Open a channel
with `sessions.open(id).channel(name)` (or `chat.channel(name)` inside a
`chat.agent`) to get an `.in`/`.out` pair addressed by name rather than
the reserved default pair. Writing a side channel's `.in` does not wake
or trigger a run, so a channel can carry out-of-band data (a stream of
frames, a control signal) that many clients read while the agent
produces it.
([#4815](https://github.com/triggerdotdev/trigger.dev/pull/4815))
```ts
// Inside a chat.agent: stream frames on a named channel, wakes nothing
const frames = chat.channel("screenshots");
await frames.out.append(frame);
frames.in.on((control) => {
/* client control, no suspend */
});
```
Declare channel record types once with `sessions.defineChannel(...)` and
infer them on both the producer and the consumer, including
`useSessionStreamChannel` in React. Channels get a default retention
that keeps them bounded, overridable per channel.
- Session `triggerConfig.tags` now accepts up to 10 tags, matching the
run tag limit. Previously it was capped at 5, which for `chat.agent`
left room for only 4 of your own tags after the automatic
`chat:{chatId}` tag.
([#4832](https://github.com/triggerdotdev/trigger.dev/pull/4832))
## @trigger.dev/python@4.5.15
### Patch Changes
- Updated dependencies:
- `@trigger.dev/sdk@4.5.15`
- `@trigger.dev/core@4.5.15`
- `@trigger.dev/build@4.5.15`
## @trigger.dev/react-hooks@4.5.15
### Patch Changes
- Named side channels on a Session: durable, two-way realtime streams
that outlive a single run and are shared across runs. Open a channel
with `sessions.open(id).channel(name)` (or `chat.channel(name)` inside a
`chat.agent`) to get an `.in`/`.out` pair addressed by name rather than
the reserved default pair. Writing a side channel's `.in` does not wake
or trigger a run, so a channel can carry out-of-band data (a stream of
frames, a control signal) that many clients read while the agent
produces it.
([#4815](https://github.com/triggerdotdev/trigger.dev/pull/4815))
```ts
// Inside a chat.agent: stream frames on a named channel, wakes nothing
const frames = chat.channel("screenshots");
await frames.out.append(frame);
frames.in.on((control) => {
/* client control, no suspend */
});
```
Declare channel record types once with `sessions.defineChannel(...)` and
infer them on both the producer and the consumer, including
`useSessionStreamChannel` in React. Channels get a default retention
that keeps them bounded, overridable per channel.
- Updated dependencies:
- `@trigger.dev/core@4.5.15`
## @trigger.dev/redis-worker@4.5.15
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.15`
## @trigger.dev/rsc@4.5.15
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.15`
## @trigger.dev/schema-to-json@4.5.15
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.15`
## @trigger.dev/sdk@4.5.15
### Patch Changes
- Stop shipping compiled test files in the published packages. The
`*.test.ts` sources were being emitted into `dist`, adding dead weight
to every install and leaving modules that `require("vitest")` (not a
dependency) inside the tarball, which tripped tooling that walks every
file in a package.
([#4833](https://github.com/triggerdotdev/trigger.dev/pull/4833))
- Named side channels on a Session: durable, two-way realtime streams
that outlive a single run and are shared across runs. Open a channel
with `sessions.open(id).channel(name)` (or `chat.channel(name)` inside a
`chat.agent`) to get an `.in`/`.out` pair addressed by name rather than
the reserved default pair. Writing a side channel's `.in` does not wake
or trigger a run, so a channel can carry out-of-band data (a stream of
frames, a control signal) that many clients read while the agent
produces it.
([#4815](https://github.com/triggerdotdev/trigger.dev/pull/4815))
```ts
// Inside a chat.agent: stream frames on a named channel, wakes nothing
const frames = chat.channel("screenshots");
await frames.out.append(frame);
frames.in.on((control) => {
/* client control, no suspend */
});
```
Declare channel record types once with `sessions.defineChannel(...)` and
infer them on both the producer and the consumer, including
`useSessionStreamChannel` in React. Channels get a default retention
that keeps them bounded, overridable per channel.
- Session `triggerConfig.tags` now accepts up to 10 tags, matching the
run tag limit. Previously it was capped at 5, which for `chat.agent`
left room for only 4 of your own tags after the automatic
`chat:{chatId}` tag.
([#4832](https://github.com/triggerdotdev/trigger.dev/pull/4832))
- Updated dependencies:
- `@trigger.dev/core@4.5.15`
</details>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
docs-release-2026-08-31
helm-v4.5.15
v.docker.4.5.15
v4.5.15
|
||
|
Saadi Myftija
|
d43dfba73a |
chore(webapp): integration settings copy update (#4850)
Project → Settings → Integrations, build settings for GitHub deploys: - **Trigger config file**: description now says it is auto-detected by default and a path only overrides that. - **Use native build server**: rendered only for admins — GitHub deployments always use the native build server unless an admin opts a project out. The option to disable native builds in GitHub deployments will be removed entirely. |
||
|
claude[bot]
|
1d55693c0f | fix(webapp): move Queues search and pagination above the table (#4834) | ||
|
Matt Aitken
|
9cb5028ce1 |
fix(core,sdk,webapp): allow 10 session trigger tags, matching the run tag limit (#4832)
## Summary `SessionTriggerConfig.tags` was capped at 5, while runs (and the [tags docs](https://trigger.dev/docs/tags)) allow 10. Session trigger tags are forwarded verbatim as the run tags on every run a session schedules, so the lower cap was an inconsistency rather than a separate limit. For `chat.agent` it was worse in practice: the SDK prepends `chat:{chatId}` automatically and truncates, so users could only get 4 of their own tags through. The schema, the SDK truncation points, and the dashboard playground now all use 10. `chat.agent` users get 9 of their own tags plus the automatic `chat:{chatId}` tag. Docs updated to say so. |
||
|
Matt Aitken
|
054bb32249 |
chore(sdk,core,build): stop publishing compiled test files (#4833)
## Summary Fixes [#4825](https://github.com/triggerdotdev/trigger.dev/issues/4825). The published `@trigger.dev/sdk`, `@trigger.dev/core` and `@trigger.dev/build` tarballs included every `*.test.ts` file compiled into `dist`, plus their `.d.ts` and source maps. Those modules `require("vitest")`, which is not a dependency of any of the packages, so the tarballs contained modules that cannot resolve. That is dead weight on every install, and it trips tooling that walks or bundles every file in a package. ## Fix tshy supports an `exclude` list in its `package.json` config that applies to every dialect build, so each affected package now sets: ```json "tshy": { "exclude": ["src/**/*.test.ts"] } ``` Only `*.test.ts` files are excluded. The public test-helper entry points (`@trigger.dev/sdk/ai/test`, `@trigger.dev/core/v3/test`) live in `src/v3/test/` and are still built and exported. The CLI package already had an equivalent exclude. Type checking and vitest are unaffected because they run off the package `tsconfig.json`, not the tshy build config. Verified with clean builds of all three packages: zero `*.test.*` artifacts in `dist`, public entry points still present. |
||
|
Eric Allam
|
16352df366 |
feat(sdk,core,webapp,react-hooks): named side channels on a Session (#4815)
## Summary Adds **named side channels** to a Session: durable, two-way realtime streams that outlive a single run and are shared across every run of the session. Today a Session has exactly one reserved `.in`/`.out` pair (the chat transcript). This lets a session hold any number of *named* channels alongside it, each its own `.in`/`.out` pair, so an agent can stream out-of-band data (a feed of frames, telemetry, a control channel) on a stream separate from the transcript while many clients read it live. The two properties a named channel adds over the reserved pair: 1. It is addressed by a name that outlives a run and is shared across runs, not welded to the chat turn loop. 2. Writing its `.in` does **not** wake or trigger a run. A run observes it by subscribing; an external client writes it without spawning anything. This is the generalization half of the Momentic ask (stream browser screenshots from a `chat.agent` to the frontend on a channel separate from the chat). It builds directly on the start-from-latest / `useSessionStream` subscribe seam from #4811. ## Usage Declare the channel's record types once and infer them on both sides: ```ts // channels.ts (shared, client imports it type-only) import { sessions } from "@trigger.dev/sdk"; export const screenshots = sessions.defineChannel<{ out: ScreenshotFrame; in: ViewportControl }>( "screenshots" ); ``` Open a channel from a session handle (`sessions.open(id)` returns one for a known session id). Writing its `.out` is durable, cross-run, and wakes nothing; a run observes its `.in` by tailing, without suspending: ```ts import { sessions } from "@trigger.dev/sdk"; import { screenshots } from "./channels"; const channel = sessions.open(sessionId).channel(screenshots); await channel.out.append(frame); // frame: ScreenshotFrame (typed from the definition) channel.in.on((control) => { /* ... */ }); // control: ViewportControl, tail, no suspend ``` Passing the definition types `.out.append` / `.in.on` on the producer side; a bare name string also works, with records typed `unknown`. An external client writes the `.in` without waking a run, and reads the `.out` from React: ```ts sessions.open(sessionId).channel("screenshots").in.send({ paused: true }); const { records } = useSessionStreamChannel<typeof screenshots>("screenshots", { sessionId, accessToken, io: "out", from: "latest", maxRecords: 1, }); ``` `session.channel(name)` returns the same `{ in, out }` handle shape as the reserved pair, so `append` / `pipe` / `writer` / `read` / `writeControl` / `trimTo` on `.out` and `send` / `on` / `once` / `peek` on `.in` all carry over. Passing a name other than the declared one is a type error; a bare-string call without the generic stays valid with `records` typed `unknown`. ### With `chat.agent` This is the motivating case: a `chat.agent` answers on the reserved transcript as usual, and streams screenshot frames on a side channel in parallel. `chat.channel(name)` opens a channel on the current run's own Session, so there's no id to thread: ```ts import { chat } from "@trigger.dev/sdk/ai"; import { streamText } from "ai"; import { screenshots } from "./channels"; export const browserAgent = chat.agent({ id: "browser-agent", run: async ({ messages, signal }) => { const frames = chat.channel(screenshots); // client pause/resume arrives here without waking a turn frames.in.on((control: ViewportControl) => applyViewport(control)); // frames stream on their own channel, not the chat transcript driveBrowser({ signal, onFrame: (frame) => frames.out.append(frame) }); // the assistant reply still goes to the reserved transcript return streamText({ model: openai("gpt-4o"), messages, abortSignal: signal }); }, }); ``` `chat.channel(name)` is a shortcut for `chat.session().channel(name)`; `chat.session()` returns the current run's full `SessionHandle` if you need it. The frontend renders the transcript with `useChat` as before, and the screenshots with `useSessionStreamChannel<typeof screenshots>("screenshots", { sessionId: chatId, io: "out", from: "latest", maxRecords: 1 })`: a live view of the newest frame that survives across turns (each turn is a new run), because the channel is keyed on the session, not the run. ### From MCP An MCP client can observe and write a session's channels with two tools, built on the same apiClient surface as the hook and the dashboard viewer: - `read_session_channel` reads records from a channel (or the reserved pair). It is a point-in-time drain with cursor pagination (`afterEventId` / `nextCursor`, `maxRecords`); pass `timeoutInSeconds` to wait for the next record when none exist yet. - `write_session_channel` appends one record to a channel's `.in` (an object or a raw string), so an agent can send control input without waking a run. `.out` is producer-only, so it is not writable here. ### On the session page The session detail page lists a session's channels (via an S2 prefix list in the loader) and shows each as a tab beside `Rendered` and `Raw`. Selecting a channel renders its records in the same table as the Raw transcript view, sourced from that channel's `out` and `in` streams. ## How it works **Addressing.** A channel is a stream name segment: `sessions/{id}/channels/{name}/{io}`. The reserved pair keeps its two-part `sessions/{id}/{io}` name for back-compat, and the `channels/` segment means a user channel named `in`/`out` can never collide with it. The channel dimension is threaded through the session stream manager (keyed on `(session, channel, io)`, reserved = absent), `subscribeToSessionStream`, the session apiClient methods, and the `realtime.v1.sessions.$session.channels.$channel.$io.{ts,append,records}` routes. The reserved-pair routes are untouched. The start-from-latest tail path from #4811 is channel-agnostic, so `from: "latest"` and `maxRecords` compose unchanged. **No-wake.** The reserved `.in` append route ensures a run and drains waitpoints so a chat turn advances. The channel `.in` append route deliberately does neither: the record lands durably and a run picks it up when it next subscribes, so writing a side channel can't spawn or resume a run. A named channel's `.in` is therefore subscribe-only from the run side (`.on` / `.once` / `.peek`); `.wait()` / `waitWithIdleTimeout()` throw with a message pointing at the observe methods. **Auth.** Channel scope folds into the existing resource id (`sessions:<key>:channels:<channel>`), so no RBAC grammar change. A channel route authorizes both the channel-folded id and the bare session id, which means a session-wide token grants every channel while a channel-scoped token grants only its own. The per-io rule is preserved per channel: writing `.out` requires secret-key auth so a browser can't forge frames; `.in` is writable with the session token. **Retention.** Channel streams are created on demand on first write and inherit the org's stream retention (bounded age plus delete-on-empty from the store's default config), the same as the reserved chat streams. There is no per-channel control-plane call on the write path. Custom per-channel retention is deferred until the stream store can set config inline on the on-demand create, which avoids a control-plane round trip. **Spans.** Channel writes carry `channel` and `io` attributes, an accessory chip, and the session icon. Clicking a channel span in the run's span inspector renders the channel's actual records with the same viewer the run realtime streams use, rather than the raw properties JSON. ## Verification - **Unit (core):** the stream manager isolates channels: two channels on the same `(session, io)` never cross buffers, and a named channel is isolated from the reserved pair. - **Full-stack e2e** against a real stack (webapp, stream store, Postgres, real runs): - a named `.out` record is readable back **after the triggering run has gone terminal** (durable, cross-run); - a channel `.in` append creates **no** run, while a reserved `.in` append **does** wake one (the differential is the red/green); - `from: "latest"` on a named channel delivers the live record and does **not** replay the backlog from the start; - the span inspector renders a channel span's records, and the MCP read/write tools round-trip records on a real session; - an invalid channel name is rejected. ## Notes - **Channel listing works on the self-hosted store too.** The stream store's list operation is available on s2-lite, so the session page's channel list is an OSS feature. It is a control-plane call made once per session-page load (best-effort; a failure just hides the tabs), not on the write path. - **The ~1 MiB per-record cap is unchanged.** Large payloads (e.g. raw screenshots) still need object-store pointers on the channel rather than inline bytes; that's independent of this change. - Docs ride this branch: the side channels guide, the `useSessionStreamChannel` reference, and the MCP tools list are all updated here. ## Screenshots <img width="3444" height="1870" alt="CleanShot 2026-08-28 at 21 46 27@2x" src="https://github.com/user-attachments/assets/c192aaee-b946-4824-87b7-ca057514d25e" /> |
||
|
James Ritchie
|
6a87048432 | feat(webapp): polish the org Projects settings page (#4828) | ||
|
github-actions[bot]
|
f8aacacb8f |
chore: release v4.5.14 (#4813)
## Summary 4 improvements, 1 bug fix. ## Improvements - Native build server deploys now show a single updating build log line by default; pass `--build-logs full` to stream every line (always used in CI and when output is not a terminal). ([#4817](https://github.com/triggerdotdev/trigger.dev/pull/4817)) - Realtime stream subscriptions can now refresh an expired access token and reconnect, via a new optional `refreshAccessToken` option on the client configuration and the React hooks. ([#4811](https://github.com/triggerdotdev/trigger.dev/pull/4811)) - Subscribe to a realtime stream from its latest record instead of replaying the whole history. Pass `from: "latest"` to `useRealtimeStream`, `streams.read()`, or `fetchStream` to start at the current tail (the latest record, then live updates) instead of replaying (a live "last value" view), and `maxParts` to keep the accumulated `parts` array bounded. A reconnect or remount resumes from the last record it saw, so no records are missed and none are replayed. `from: "latest"` needs a server that supports it; older servers safely fall back to a full replay. ([#4811](https://github.com/triggerdotdev/trigger.dev/pull/4811)) `useRealtimeStream` also gains a `lastEventId` option and returns the `lastEventId` of the last part seen, so you can persist the cursor (for example across a page reload) and resume exactly where you left off. An `onParts` callback delivers each throttled batch of parts with their event ids. ```tsx const { parts, lastEventId } = useRealtimeStream<Frame>(runId, "frames", { from: "latest", // skip history, start at the current tail maxParts: 1, // keep only the most recent frame lastEventId: savedCursor, // resume from a persisted cursor onParts: (batch) => save(batch.at(-1)?.id), // track the cursor accessToken, }); ``` - Added a `useSessionStream` React hook for reading a session's output or input channel in realtime. It accumulates records with automatic resume from the last record you received, and supports `from: "latest"` (start at the current tail, only new records after you connect), `maxRecords` (keep a bounded number of records in memory), a `lastEventId` resume cursor, and an `onRecords` callback that delivers each throttled batch of records with their event ids. ([#4811](https://github.com/triggerdotdev/trigger.dev/pull/4811)) ## Server changes These changes affect the self-hosted Docker image and Trigger.dev Cloud: - Task retries that wait in the queue no longer count against the queue's internal redelivery limit, so runs with many long-delay retries are not wrongly failed with TASK_RUN_DEQUEUED_MAX_RETRIES. ([#4810](https://github.com/triggerdotdev/trigger.dev/pull/4810)) <details> <summary>Raw changeset output</summary> # Releases ## @trigger.dev/build@4.5.14 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.14` ## trigger.dev@4.5.14 ### Patch Changes - Native build server deploys now show a single updating build log line by default; pass `--build-logs full` to stream every line (always used in CI and when output is not a terminal). ([#4817](https://github.com/triggerdotdev/trigger.dev/pull/4817)) - Updated dependencies: - `@trigger.dev/core@4.5.14` - `@trigger.dev/build@4.5.14` - `@trigger.dev/schema-to-json@4.5.14` ## @trigger.dev/core@4.5.14 ### Patch Changes - Realtime stream subscriptions can now refresh an expired access token and reconnect, via a new optional `refreshAccessToken` option on the client configuration and the React hooks. ([#4811](https://github.com/triggerdotdev/trigger.dev/pull/4811)) - Subscribe to a realtime stream from its latest record instead of replaying the whole history. Pass `from: "latest"` to `useRealtimeStream`, `streams.read()`, or `fetchStream` to start at the current tail (the latest record, then live updates) instead of replaying (a live "last value" view), and `maxParts` to keep the accumulated `parts` array bounded. A reconnect or remount resumes from the last record it saw, so no records are missed and none are replayed. `from: "latest"` needs a server that supports it; older servers safely fall back to a full replay. ([#4811](https://github.com/triggerdotdev/trigger.dev/pull/4811)) `useRealtimeStream` also gains a `lastEventId` option and returns the `lastEventId` of the last part seen, so you can persist the cursor (for example across a page reload) and resume exactly where you left off. An `onParts` callback delivers each throttled batch of parts with their event ids. ```tsx const { parts, lastEventId } = useRealtimeStream<Frame>(runId, "frames", { from: "latest", // skip history, start at the current tail maxParts: 1, // keep only the most recent frame lastEventId: savedCursor, // resume from a persisted cursor onParts: (batch) => save(batch.at(-1)?.id), // track the cursor accessToken, }); ``` ## @trigger.dev/python@4.5.14 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.14` - `@trigger.dev/sdk@4.5.14` - `@trigger.dev/build@4.5.14` ## @trigger.dev/react-hooks@4.5.14 ### Patch Changes - Realtime stream subscriptions can now refresh an expired access token and reconnect, via a new optional `refreshAccessToken` option on the client configuration and the React hooks. ([#4811](https://github.com/triggerdotdev/trigger.dev/pull/4811)) - Subscribe to a realtime stream from its latest record instead of replaying the whole history. Pass `from: "latest"` to `useRealtimeStream`, `streams.read()`, or `fetchStream` to start at the current tail (the latest record, then live updates) instead of replaying (a live "last value" view), and `maxParts` to keep the accumulated `parts` array bounded. A reconnect or remount resumes from the last record it saw, so no records are missed and none are replayed. `from: "latest"` needs a server that supports it; older servers safely fall back to a full replay. ([#4811](https://github.com/triggerdotdev/trigger.dev/pull/4811)) `useRealtimeStream` also gains a `lastEventId` option and returns the `lastEventId` of the last part seen, so you can persist the cursor (for example across a page reload) and resume exactly where you left off. An `onParts` callback delivers each throttled batch of parts with their event ids. ```tsx const { parts, lastEventId } = useRealtimeStream<Frame>(runId, "frames", { from: "latest", // skip history, start at the current tail maxParts: 1, // keep only the most recent frame lastEventId: savedCursor, // resume from a persisted cursor onParts: (batch) => save(batch.at(-1)?.id), // track the cursor accessToken, }); ``` - Added a `useSessionStream` React hook for reading a session's output or input channel in realtime. It accumulates records with automatic resume from the last record you received, and supports `from: "latest"` (start at the current tail, only new records after you connect), `maxRecords` (keep a bounded number of records in memory), a `lastEventId` resume cursor, and an `onRecords` callback that delivers each throttled batch of records with their event ids. ([#4811](https://github.com/triggerdotdev/trigger.dev/pull/4811)) - Updated dependencies: - `@trigger.dev/core@4.5.14` ## @trigger.dev/redis-worker@4.5.14 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.14` ## @trigger.dev/rsc@4.5.14 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.14` ## @trigger.dev/schema-to-json@4.5.14 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.14` ## @trigger.dev/sdk@4.5.14 ### Patch Changes - Subscribe to a realtime stream from its latest record instead of replaying the whole history. Pass `from: "latest"` to `useRealtimeStream`, `streams.read()`, or `fetchStream` to start at the current tail (the latest record, then live updates) instead of replaying (a live "last value" view), and `maxParts` to keep the accumulated `parts` array bounded. A reconnect or remount resumes from the last record it saw, so no records are missed and none are replayed. `from: "latest"` needs a server that supports it; older servers safely fall back to a full replay. ([#4811](https://github.com/triggerdotdev/trigger.dev/pull/4811)) `useRealtimeStream` also gains a `lastEventId` option and returns the `lastEventId` of the last part seen, so you can persist the cursor (for example across a page reload) and resume exactly where you left off. An `onParts` callback delivers each throttled batch of parts with their event ids. ```tsx const { parts, lastEventId } = useRealtimeStream<Frame>(runId, "frames", { from: "latest", // skip history, start at the current tail maxParts: 1, // keep only the most recent frame lastEventId: savedCursor, // resume from a persisted cursor onParts: (batch) => save(batch.at(-1)?.id), // track the cursor accessToken, }); ``` - Updated dependencies: - `@trigger.dev/core@4.5.14` </details> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>v4.5.14 docs-release-2026-08-28-2 helm-v4.5.14 v.docker.4.5.14 |
||
|
Matt Aitken
|
1f8f23027d |
fix(run-engine): stop task retries consuming the queue nack budget (#4810)
## Summary A run whose task retries were delayed long enough to go back through the queue could end up failed with `TASK_RUN_DEQUEUED_MAX_RETRIES` and status `SYSTEM_FAILURE` even though every attempt had actually executed. The real failure from the final attempt was replaced by that placeholder error, and tasks configured for more retries than the queue redelivery limit never got them. ## Root cause Retries with a delay at or above the warm-start threshold are requeued via `tryNackAndRequeue`, which nacks the queue message. `nackMessage` increments the message attempt counter by default and dead-letters the message once it reaches the queue retry limit. That counter is meant to bound dequeues that never reach execution; a task retry after a completed attempt was being charged against it anyway, so a long-backoff retry schedule exhausted it. ## Fix `nackMessage` gains a `resetAttemptCount` option that zeroes the counter instead of incrementing it. `tryNackAndRequeue` exposes it as `resetQueueAttempts`, and the attempt-retry path passes it, since a completed attempt proves the run can start. The dequeue-failure and stalled `PENDING_EXECUTING` paths keep incrementing, as those are the genuine "could not start" cases the budget exists for. Tests cover the queue-level reset (no dead-letter at the limit) and an engine-level run that retries past the queue limit and finishes with its own error rather than a system failure. |
||
|
nicktrn
|
82ea72383c |
feat(webapp): emit workload auth gate metrics via opentelemetry (#4822)
## Summary `workload_auth_gate_total` records how each worker action authorizes: scoped by a verified environment header, grandfathered by the created-at gate, or suppressed by it. It was registered on the Prometheus registry served at `/metrics`, which is per-process. With `ENABLE_CLUSTER=1` every Node worker keeps its own registry, so a scrape returns whichever process happened to answer and the counter reads as a fraction of real traffic. This moves the counter onto the OpenTelemetry meter the webapp already uses for its other engine metrics. Each process exports under its own `service.instance.id`, so summing across them gives the true total no matter how many workers a deployment runs. ## Attributes The counter now carries `env_type` and `run_age_bucket` alongside `outcome` and `action`. `run_age_bucket` is the coarse age of the run behind an untokened worker action (`lt_1h`, `1h_1d`, `1d_7d`, `7d_30d`, `gt_30d`). It exists so an operator can size `WORKLOAD_TOKEN_CUTOFF` before committing to it: set the cutoff far in the future and every run is grandfathered, so the age distribution of untokened traffic is visible without anything being rejected. Both attributes come off the run row the gate already reads, so there is no extra query. |
||
|
Saadi Myftija
|
cca41a42f0 |
ci(publish): skip publish for packages not built into the images (#4821)
`publish.yml` builds the webapp and worker/supervisor images (and dispatches the enterprise-image build off the webapp). It currently triggers on all of `packages/**` and `internal-packages/**`, so a change confined to a package that never lands in those images still publishes new images. This adds `paths` negations for packages that are not built into either image: - **packages** (npm-only libraries / CLI): `cli-v3`, `build`, `python`, `react-hooks`, `rsc`, `schema-to-json` - **internal-packages** (test/tooling only): `testcontainers`, `sdk-compat-tests`, `observability-map` |
||
|
James Ritchie
|
f42c82091d |
feat(webapp): Improve the Usage page billing panels (#4820)
UI-only update of the **Usage** page (`/orgs/…/settings/usage`). **No logic or data changes**: the loader is byte-identical to `main` and the usage-bar calculations are unchanged. ### What changed - **Credits** and **Month-to-date** panels now sit in matching cards, with the big `$value` and title on one baseline-aligned row and the progress bar full-width beneath. - Added a **Set / Update billing limit** link (to the existing Billing limits page) on the Month-to-date panel. - The two progress bars share the same height/corners; the Month-to-date panel shrinks when there's no billing limit to show. - Removed the progress-bar load animation. - **Tasks**: moved the "dev environment runs are excluded…" note beside the title and switched the empty state to the standard `TableBlankRow`. <img width="3456" height="1364" alt="CleanShot 2026-08-28 at 15 37 52@2x" src="https://github.com/user-attachments/assets/cb69ec33-1521-47d1-ba0a-51b8afd7eb00" /> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Saadi Myftija
|
34529a4d7e |
feat(cli): compact build logs for native build deploys (#4817)
Native build server deploys (`--native-build`) now show compact build logs by default: one spinner line updated with the latest message, and the last 20 lines printed when the build fails. The previous timestamped line-by-line output is behind `--build-logs full`, and is used automatically in CI, with `--plain`, or when stdout is not a TTY. |
||
|
Daniel Sutton
|
63b8e6e1f5 |
fix(webapp): scan every run-ops store for the batches list (#4806)
## Summary Batches created on a run-ops store other than the two the list reads were missing from the Batches page. No error, nothing logged: the page just showed fewer batches than exist. This is only reachable once additional run-ops stores are configured, so nothing changes for anyone today. ## Fix The list scanned exactly two databases and merged them by keyset. It now covers one leg per configured store, in ascending precedence order, all issued together. The existing keyset merge generalises without change. Every leg runs the same query, with the same cursor predicate, ordering and over-fetch, so a row's rank within its own leg is never worse than its global rank, and the merged first page is still the true first page. That argument holds for any number of legs, not just two. The empty-state check keeps its existing sequential pair, since a project with no batches is the common case for that path, then issues the remaining checks in a single round trip. A store that declares itself an alias of another shares its client by reference, so it contributes no leg. Scanning it would query the same database twice for rows the other leg already returned. This matches how the routing store and the boot checks treat an alias. The fan-out deliberately fails the page if any store is unreachable, rather than returning a short page. A tolerant merge would recreate the same silent absence this change removes, with a wider blast radius. ## Verification Covered by container tests against real databases: gen-1, legacy and additional stores merged into one ordered page, paging forward and back across a boundary that spans stores, and the empty-state check. Also verified end to end against a live environment with a real corpus: the missing rows reproduce with the new leg removed and appear correctly with it present, ordering interleaves across stores as expected, paging across a store boundary loses and repeats nothing, and the page is byte-identical to before when no extra store is configured. Merge precedence is pinned by its own test: one id seeded on two stores, asserting the higher-authority copy is the one shown. Verified by mutation, since a union-only test passes regardless of leg order. ## Boot interlocks Two related boot checks changed alongside the read path, since configuring an extra store is what makes them reachable. A store configured while split reads are disabled is dropped in silence: no client is built, no leg is added, and rows already resident there disappear from every list with no error. The other two ways the split ends up disabled already refuse to start; this closes the one that did not, and names the stores it is refusing. A store that declares itself an alias of another owns no database, so it is exempt. The distinct-database probe fails closed, which meant one store being briefly unreachable collapsed the deployment to single-DB and then refused the boot entirely. Each target now gets a bounded number of attempts with a short backoff before the probe gives up. Failing closed is unchanged once that budget is exhausted, and a genuine duplicate is still a final answer that is never retried. |
||
|
James Ritchie
|
2e24c01ce0 |
fix(webapp): polish AI agent setup panel on tasks blank state (#4807)
Visual-only changes to the Tasks-page onboarding blank state (brand-new project, dev environment). Formatting, lint, and knip pass via the pre-push hooks; open the Tasks page for a new project to confirm the panel, copy button, and step 2 render as intended. --- ## Changelog Polished the "Set it up with your AI agent" onboarding panel: top-aligned the badge and switched it to the custom Ask AI sparkle icon, stopped the copy-prompt button from resizing when it swaps to "Copied prompt" (the bright check icon now sits beside the label), removed the sparkle from the button's idle state, removed the spinner next to "Start the dev server", and widened the gap between the panel text and the copy button. --- ## Screenshots <img width="800" height="643" alt="CleanShot 2026-08-27 at 19 04 43" src="https://github.com/user-attachments/assets/aede4ca1-30d5-4240-aa18-e1a20161973d" /> 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- conductor-workspace-link --> --- [Open workspace in Conductor](https://app.conductor.build/workspace/d4a21ab5-d6fa-4de2-b5f0-4f34abf0b8b9) Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Chris Arderne
|
9d3fedd7c5 | chore: fix email address in SECURITY.md (#4814) | ||
|
Eric Allam
|
1d13b7976a |
feat(realtime): start-from-latest streams and a useSessionStream hook (#4811)
## Summary
Realtime streams get a live "last value" mode: subscribe from the latest
record instead of replaying the whole history, keep memory bounded, and
resume across reloads. Plus a new `useSessionStream` hook for reading a
Session's channels from React.
## `useRealtimeStream`: start-from-latest, bounded, resumable
```tsx
const { parts, lastEventId } = useRealtimeStream<Frame>(runId, "frames", {
from: "latest", // skip history, only new records after connect
maxParts: 1, // keep just the most recent (bounded memory)
lastEventId: saved, // resume from a persisted cursor (survives reload)
onParts: (batch) => save(batch.at(-1)?.id), // per-batch event ids
accessToken,
});
```
`from`, `lastEventId` (option and return), and the batching also apply
to `streams.read()` and `fetchStream()`.
## `useSessionStream`: read a Session channel from React (new)
A read-only hook for a Session's `out` (default) or `in` channel, with
the same start / bound / resume options. `useSession` is reserved for
two-way (read and write).
```tsx
const { records, lastEventId } = useSessionStream<Frame>(sessionId, {
io: "out",
from: "latest",
maxRecords: 5,
onRecords: (batch) => {/* each throttled batch, with event ids */},
accessToken,
});
```
## Access-token refresh
Long-lived subscriptions can survive token expiry: pass
`refreshAccessToken` and a 401/403 triggers one re-mint and reconnect.
With no refresher, auth errors stay terminal exactly as before.
```tsx
const { parts } = useRealtimeStream<Frame>(runId, "frames", {
accessToken,
// called on a 401/403 to mint a fresh public token from your backend
refreshAccessToken: async () => {
const res = await fetch("/api/realtime-token");
return (await res.json()).token;
},
});
```
It is also available on `useApiClient` / `TriggerAuthContext`, so every
hook under a provider shares one refresher.
## Notes
Server support (S2 `tail_offset` / Redis `$`, and the start-position
header on the run and session SSE routes) ships here; a client passing
`from: "latest"` against an older server degrades safely to a full
replay. Resume, bounded memory, batched callbacks, and token refresh are
client-only.
Supersedes #4808 and #4809, folded in here. Verified end to end on an
isolated stack: `from: "latest"` on the run and session paths against
real S2, `lastEventId` resume across a reload, bounded memory, batched
callbacks, and a real 401 to token-refresh to reconnect.
|
||
|
Daniel Sutton
|
adcf0e7dc3 |
test(run-store,webapp): cover the run-ops router at three shards (#4805)
## Summary Several of the run-ops router's rules only apply above two stores, and the fake-slot suites only ever built two, so those rules were untestable by construction. `clearIdempotencyKey` is the sole caller of the "every other shard" helper, and with two stores that helper returns a single entry, which hides a take-the-first bug. The absent-id partition has the same blind spot: a gen-2 id and a cuid select the same store when only one other store exists. Three suites now run at two shards and at three, with the expected value indexed by topology wherever the rule genuinely changes. The fourth stays at two and says why in the file, because its N-shard behaviour is already pinned in `runOpsStore.shardMap.test.ts`. Two webapp tests defined their own local `RoutingRunStore`. They compiled against a two-store model whatever the real class did, and one described a routing rule the code never implemented. Both now build the real router over the two Postgres stores they already create. ## Validating a test-only change Every new assertion passed the first time it ran, which proves nothing. Each was checked by breaking the router in the way the test claims to guard, then confirming the failure lands in the three-shard arm while the two-shard arm still passes: - take-the-first fan-out in the "every other shard" helper - gen-2 keys moved to the front of the merge precedence order - the absent-id partition sending every id to the gen-1 pair, which fails as `expected +0 to be 1`, the shape a silently under-counted waitpoint takes - residency routing disabled entirely, caught by 3 of the 5 webapp tests Each mutation was reverted. No production code changes. One note for anyone extending these: the webapp resolves `@internal/run-store` to `dist/`, not to source, so a source edit without a rebuild makes those two tests assert against the previous router and pass. |
||
|
github-actions[bot]
|
aa0bfceff4 |
chore: release v4.5.13 (#4769)
## Summary 4 new features, 12 improvements, 5 bug fixes. ## Improvements - `trigger.dev deploy` now asks the server whether to build with Depot or the native build server unless `--native-build`, `--depot-build`, or `--local-build` is passed, so the native build server can be rolled out per organization without a CLI change. `--local-bundle` and `--detach` now require `--native-build`. ([#4803](https://github.com/triggerdotdev/trigger.dev/pull/4803)) - Add an experimental `--local-bundle` deploy flag that runs the install and bundling steps on your machine and uploads only the build output; the image is still built remotely. Useful when your project's install step needs tooling or credentials that only exist locally. ([#4331](https://github.com/triggerdotdev/trigger.dev/pull/4331)) - Send the CLI version header on all API requests so deployments are attributable to a CLI version ([#4778](https://github.com/triggerdotdev/trigger.dev/pull/4778)) - A message that arrives mid-turn and is not injected into that turn is now answered as the next turn, instead of being dropped. This is what the `pendingMessages` docs have always described, and it applies to the default too: configuring `pendingMessages` without a `shouldInject` declines every batch, which previously meant every mid-turn message was lost with no error at either end. ([#4795](https://github.com/triggerdotdev/trigger.dev/pull/4795)) ```ts chat.agent({ id: "my-chat", pendingMessages: { onReceived: ({ message }) => logger.info("arrived mid-turn", { id: message.id }), // Only interrupt once the agent has started calling tools. shouldInject: ({ steps }) => steps.length > 0, }, run: async ({ messages, signal }) => streamText({ model, messages, abortSignal: signal, // Required for injection. Without it nothing injects, and every // mid-turn message is answered as the next turn instead. ...chat.toStreamTextOptions(), }), }); ``` A declined message keeps its place in the queue, so it survives a crash and is answered by whichever run picks the conversation up. An injected one is consumed at the moment it is injected, so it is never also answered as a later turn. - Browser chats now keep the active turn open across page reloads when older completion records are replayed. ([#4643](https://github.com/triggerdotdev/trigger.dev/pull/4643)) - Add `chat.endAndContinue()` so fully hand-rolled custom chat agents can hand a conversation off to a fresh run on the latest deployed task version while preserving unconsumed Session input. ([#4647](https://github.com/triggerdotdev/trigger.dev/pull/4647)) - Custom chat agents now validate and parse client data declared with `chat.withClientData({ schema })` before passing it to agent code. ([#4646](https://github.com/triggerdotdev/trigger.dev/pull/4646)) ## Bug fixes - Fixes a case where a chat could silently lose a message. If a message arrived while the agent was between turns and a stop arrived after it, the cursor the next boot resumed from could point past that message, so it was never answered and no error was raised. This affected `chat.agent`, not just custom agents. ([#4644](https://github.com/triggerdotdev/trigger.dev/pull/4644)) Fixes a recovered answer being cut off. After a crash the agent replays the message it had not answered yet, but it was replaying the stop that arrived after that message too, so the turn answering it was aborted the moment it began. A stop is now only applied to the turn that was live when it arrived. That holds however the stop got there: sent after the last completed turn, or sent to a chat whose most recent turn was completed by an older version of the SDK. One limitation to know about: the recovered answer is persisted correctly, but a chat page that stayed open across the crash keeps showing the partial answer it had already received. Reload the page to see the full recovered answer. Also fixes a retried send being answered twice. When a send was retried and its idempotency claim was lost, the agent could consume the same message a second time. Custom agent loops can now inspect pending chat input without consuming it, and consume one record at a time, with `chat.messages.hasPending()` and `chat.messages.next()`. Records carry stable identifiers so a redelivery is recognisable. ```ts if (await chat.messages.hasPending()) { const record = await chat.messages.next({ timeoutInSeconds: 0 }); if (record) handle(record.payload); } ``` `hasPending()` answers for messages alone, so a message sitting behind a stop, or behind a record this version of the SDK does not recognise, still reports as pending and is still delivered. Anything the agent has no consumer for is discarded rather than left where it would make every message queued behind it undeliverable. `chat.messages.next()` returning `undefined` means no message became consumable before the timeout. `chat.writeTurnComplete()`'s `sessionInEventId` is the cursor that is safe to resume from, not the sequence of the record the turn answered. It is held back behind any message still waiting to be handled, so a value below the record you just handled is expected. - Fixed a chat agent hanging after an interrupted turn: when a run was killed mid-answer (out of memory, crash, or eviction) and only the one message it was answering was still outstanding, the new run never replied to it. That message is now re-answered on the new run. ([#4768](https://github.com/triggerdotdev/trigger.dev/pull/4768)) - Fix chat transport discarding the next turn after stopping generation. `skipToTurnComplete` is now reset when a new message or action is sent, so a message sent after `stopGeneration` streams normally instead of leaving the chat stuck in a streaming state. ([#4744](https://github.com/triggerdotdev/trigger.dev/pull/4744)) - Fixes a message sent while the agent was mid-answer being lost if the run then crashed. The cursor written at the end of each turn could point past a message that had arrived during that turn but had not been answered yet, so the next boot skipped it and no error was raised anywhere. Such a message is now held until a turn actually takes it. ([#4795](https://github.com/triggerdotdev/trigger.dev/pull/4795)) This also removes the in-memory buffer those messages used to sit in, on both `chat.agent` and `chat.createSession()`, so a message waiting for its turn is durable rather than only present in the worker that received it. ## Server changes These changes affect the self-hosted Docker image and Trigger.dev Cloud: - Self-hosted instances can now disable the admin dashboard and user impersonation entirely. See the self-hosting docs for the new setting. ([#4774](https://github.com/triggerdotdev/trigger.dev/pull/4774)) - The dashboard has two new themes, Black and White, plus appearance options for stronger colors and underlined links. ([#4547](https://github.com/triggerdotdev/trigger.dev/pull/4547)) - Deployment logs no longer jump to the bottom while you are reading earlier output. Scroll up to pause auto-scroll, and scroll back down or use the new scroll-to-bottom button in the log header to resume following. ([#4776](https://github.com/triggerdotdev/trigger.dev/pull/4776)) - Customize the runs list: show, hide, and reorder columns, and add smart columns that pull a value straight out of a run's payload, metadata, or output. Your column choices are saved in the page URL, so you can share a view, bookmark it, or save it straight to your favorites. ([#4652](https://github.com/triggerdotdev/trigger.dev/pull/4652)) - Stop the browser offering to autofill or save environment variable values as saved credentials. ([#4777](https://github.com/triggerdotdev/trigger.dev/pull/4777)) - Cut webapp CPU usage by about a quarter on the routes that workers call most, freeing headroom at the same request rate. Detailed event-loop blocking traces are no longer recorded by default, because producing them was itself a large part of that cost. ([#4746](https://github.com/triggerdotdev/trigger.dev/pull/4746)) - When a runs list or runs.list API request spans too much data to complete, it now returns a clear, actionable error asking you to narrow the time range, instead of failing with a generic error. ([#4773](https://github.com/triggerdotdev/trigger.dev/pull/4773)) - Improved the performance and reliability of the runs list and the runs.list API, especially for large projects and filtered views. ([#4763](https://github.com/triggerdotdev/trigger.dev/pull/4763)) - New Vercel connections now get version skew protection turned on automatically, so each run uses the task version its deployment shipped with. Automatic atomic deployments are deprecated and no longer offered when you connect a project, but stay available in your Vercel integration settings. ([#4741](https://github.com/triggerdotdev/trigger.dev/pull/4741)) - The Staging branch setting now shows an upgrade prompt on plans that don't include a Staging environment, instead of looking editable and then silently doing nothing when saved. ([#4784](https://github.com/triggerdotdev/trigger.dev/pull/4784)) <details> <summary>Raw changeset output</summary> # Releases ## @trigger.dev/build@4.5.13 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.13` ## trigger.dev@4.5.13 ### Patch Changes - `trigger.dev deploy` now asks the server whether to build with Depot or the native build server unless `--native-build`, `--depot-build`, or `--local-build` is passed, so the native build server can be rolled out per organization without a CLI change. `--local-bundle` and `--detach` now require `--native-build`. ([#4803](https://github.com/triggerdotdev/trigger.dev/pull/4803)) - Add an experimental `--local-bundle` deploy flag that runs the install and bundling steps on your machine and uploads only the build output; the image is still built remotely. Useful when your project's install step needs tooling or credentials that only exist locally. ([#4331](https://github.com/triggerdotdev/trigger.dev/pull/4331)) - Send the CLI version header on all API requests so deployments are attributable to a CLI version ([#4778](https://github.com/triggerdotdev/trigger.dev/pull/4778)) - Updated dependencies: - `@trigger.dev/core@4.5.13` - `@trigger.dev/build@4.5.13` - `@trigger.dev/schema-to-json@4.5.13` ## @trigger.dev/core@4.5.13 ### Patch Changes - `trigger.dev deploy` now asks the server whether to build with Depot or the native build server unless `--native-build`, `--depot-build`, or `--local-build` is passed, so the native build server can be rolled out per organization without a CLI change. `--local-bundle` and `--detach` now require `--native-build`. ([#4803](https://github.com/triggerdotdev/trigger.dev/pull/4803)) - Add an experimental `--local-bundle` deploy flag that runs the install and bundling steps on your machine and uploads only the build output; the image is still built remotely. Useful when your project's install step needs tooling or credentials that only exist locally. ([#4331](https://github.com/triggerdotdev/trigger.dev/pull/4331)) - A message that arrives mid-turn and is not injected into that turn is now answered as the next turn, instead of being dropped. This is what the `pendingMessages` docs have always described, and it applies to the default too: configuring `pendingMessages` without a `shouldInject` declines every batch, which previously meant every mid-turn message was lost with no error at either end. ([#4795](https://github.com/triggerdotdev/trigger.dev/pull/4795)) ```ts chat.agent({ id: "my-chat", pendingMessages: { onReceived: ({ message }) => logger.info("arrived mid-turn", { id: message.id }), // Only interrupt once the agent has started calling tools. shouldInject: ({ steps }) => steps.length > 0, }, run: async ({ messages, signal }) => streamText({ model, messages, abortSignal: signal, // Required for injection. Without it nothing injects, and every // mid-turn message is answered as the next turn instead. ...chat.toStreamTextOptions(), }), }); ``` A declined message keeps its place in the queue, so it survives a crash and is answered by whichever run picks the conversation up. An injected one is consumed at the moment it is injected, so it is never also answered as a later turn. - Fixes a case where a chat could silently lose a message. If a message arrived while the agent was between turns and a stop arrived after it, the cursor the next boot resumed from could point past that message, so it was never answered and no error was raised. This affected `chat.agent`, not just custom agents. ([#4644](https://github.com/triggerdotdev/trigger.dev/pull/4644)) Fixes a recovered answer being cut off. After a crash the agent replays the message it had not answered yet, but it was replaying the stop that arrived after that message too, so the turn answering it was aborted the moment it began. A stop is now only applied to the turn that was live when it arrived. That holds however the stop got there: sent after the last completed turn, or sent to a chat whose most recent turn was completed by an older version of the SDK. One limitation to know about: the recovered answer is persisted correctly, but a chat page that stayed open across the crash keeps showing the partial answer it had already received. Reload the page to see the full recovered answer. Also fixes a retried send being answered twice. When a send was retried and its idempotency claim was lost, the agent could consume the same message a second time. Custom agent loops can now inspect pending chat input without consuming it, and consume one record at a time, with `chat.messages.hasPending()` and `chat.messages.next()`. Records carry stable identifiers so a redelivery is recognisable. ```ts if (await chat.messages.hasPending()) { const record = await chat.messages.next({ timeoutInSeconds: 0 }); if (record) handle(record.payload); } ``` `hasPending()` answers for messages alone, so a message sitting behind a stop, or behind a record this version of the SDK does not recognise, still reports as pending and is still delivered. Anything the agent has no consumer for is discarded rather than left where it would make every message queued behind it undeliverable. `chat.messages.next()` returning `undefined` means no message became consumable before the timeout. `chat.writeTurnComplete()`'s `sessionInEventId` is the cursor that is safe to resume from, not the sequence of the record the turn answered. It is held back behind any message still waiting to be handled, so a value below the record you just handled is expected. ## @trigger.dev/python@4.5.13 ### Patch Changes - Updated dependencies: - `@trigger.dev/sdk@4.5.13` - `@trigger.dev/core@4.5.13` - `@trigger.dev/build@4.5.13` ## @trigger.dev/react-hooks@4.5.13 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.13` ## @trigger.dev/redis-worker@4.5.13 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.13` ## @trigger.dev/rsc@4.5.13 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.13` ## @trigger.dev/schema-to-json@4.5.13 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.13` ## @trigger.dev/sdk@4.5.13 ### Patch Changes - Fixed a chat agent hanging after an interrupted turn: when a run was killed mid-answer (out of memory, crash, or eviction) and only the one message it was answering was still outstanding, the new run never replied to it. That message is now re-answered on the new run. ([#4768](https://github.com/triggerdotdev/trigger.dev/pull/4768)) - Browser chats now keep the active turn open across page reloads when older completion records are replayed. ([#4643](https://github.com/triggerdotdev/trigger.dev/pull/4643)) - Add `chat.endAndContinue()` so fully hand-rolled custom chat agents can hand a conversation off to a fresh run on the latest deployed task version while preserving unconsumed Session input. ([#4647](https://github.com/triggerdotdev/trigger.dev/pull/4647)) - Fix chat transport discarding the next turn after stopping generation. `skipToTurnComplete` is now reset when a new message or action is sent, so a message sent after `stopGeneration` streams normally instead of leaving the chat stuck in a streaming state. ([#4744](https://github.com/triggerdotdev/trigger.dev/pull/4744)) - Custom chat agents now validate and parse client data declared with `chat.withClientData({ schema })` before passing it to agent code. ([#4646](https://github.com/triggerdotdev/trigger.dev/pull/4646)) - Fixes a message sent while the agent was mid-answer being lost if the run then crashed. The cursor written at the end of each turn could point past a message that had arrived during that turn but had not been answered yet, so the next boot skipped it and no error was raised anywhere. Such a message is now held until a turn actually takes it. ([#4795](https://github.com/triggerdotdev/trigger.dev/pull/4795)) This also removes the in-memory buffer those messages used to sit in, on both `chat.agent` and `chat.createSession()`, so a message waiting for its turn is durable rather than only present in the worker that received it. - A message that arrives mid-turn and is not injected into that turn is now answered as the next turn, instead of being dropped. This is what the `pendingMessages` docs have always described, and it applies to the default too: configuring `pendingMessages` without a `shouldInject` declines every batch, which previously meant every mid-turn message was lost with no error at either end. ([#4795](https://github.com/triggerdotdev/trigger.dev/pull/4795)) ```ts chat.agent({ id: "my-chat", pendingMessages: { onReceived: ({ message }) => logger.info("arrived mid-turn", { id: message.id }), // Only interrupt once the agent has started calling tools. shouldInject: ({ steps }) => steps.length > 0, }, run: async ({ messages, signal }) => streamText({ model, messages, abortSignal: signal, // Required for injection. Without it nothing injects, and every // mid-turn message is answered as the next turn instead. ...chat.toStreamTextOptions(), }), }); ``` A declined message keeps its place in the queue, so it survives a crash and is answered by whichever run picks the conversation up. An injected one is consumed at the moment it is injected, so it is never also answered as a later turn. - Fixes a case where a chat could silently lose a message. If a message arrived while the agent was between turns and a stop arrived after it, the cursor the next boot resumed from could point past that message, so it was never answered and no error was raised. This affected `chat.agent`, not just custom agents. ([#4644](https://github.com/triggerdotdev/trigger.dev/pull/4644)) Fixes a recovered answer being cut off. After a crash the agent replays the message it had not answered yet, but it was replaying the stop that arrived after that message too, so the turn answering it was aborted the moment it began. A stop is now only applied to the turn that was live when it arrived. That holds however the stop got there: sent after the last completed turn, or sent to a chat whose most recent turn was completed by an older version of the SDK. One limitation to know about: the recovered answer is persisted correctly, but a chat page that stayed open across the crash keeps showing the partial answer it had already received. Reload the page to see the full recovered answer. Also fixes a retried send being answered twice. When a send was retried and its idempotency claim was lost, the agent could consume the same message a second time. Custom agent loops can now inspect pending chat input without consuming it, and consume one record at a time, with `chat.messages.hasPending()` and `chat.messages.next()`. Records carry stable identifiers so a redelivery is recognisable. ```ts if (await chat.messages.hasPending()) { const record = await chat.messages.next({ timeoutInSeconds: 0 }); if (record) handle(record.payload); } ``` `hasPending()` answers for messages alone, so a message sitting behind a stop, or behind a record this version of the SDK does not recognise, still reports as pending and is still delivered. Anything the agent has no consumer for is discarded rather than left where it would make every message queued behind it undeliverable. `chat.messages.next()` returning `undefined` means no message became consumable before the timeout. `chat.writeTurnComplete()`'s `sessionInEventId` is the cursor that is safe to resume from, not the sequence of the record the turn answered. It is held back behind any message still waiting to be handled, so a value below the record you just handled is expected. - Updated dependencies: - `@trigger.dev/core@4.5.13` </details> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>v.docker.4.5.13 docs-release-2026-08-28 helm-v4.5.13 v4.5.13 |
||
|
Saadi Myftija
|
c7b04989b1 |
feat(cli): server-selected deploy build path (#4803)
The CLI now asks the server which build path to use before it builds or
uploads anything, so native builds can be rolled out per organization
and per environment type without a CLI release.
```
trigger.dev deploy
│
├─ explicit flag? (--native-build / --local-build / --depot-build)
│ └─ yes → use it, never ask the server
│
└─ GET /api/v1/projects/:ref/:env/deploy-settings (env API key, 5s timeout, one attempt)
│
│ server resolves: native unavailable → org[env type] → org → global[env type] → global → depot
│
├─ { "build_path": "native" | "native_local_bundle" } → that path
├─ { "build_path": "depot" } → Depot
└─ error / timeout / 404 → Depot (fail open)
```
The path comes from four enum feature flags, editable in the global and
per-org admin flag UIs: `deployBuildPath` and `deployBuildPathPreview` /
`Staging` / `Production`. Unset everywhere keeps current behaviour
unchanged; CLIs older than this release never call the endpoint and keep
their current behaviour.
|
||
|
Graham Tremper
|
acaa5ec227 |
feat(chat): runtime clientData validation for custom agents (#4646)
## Summary
`chat.withClientData({ schema }).customAgent()` now parses
`payload.metadata` before passing it to `run`, `chat.messages`, or
`chat.createSession`. Schema defaults and transforms are preserved.
Custom agents without a schema keep the existing pass-through behavior.
This does not change `chat.agent()`. Raw custom agents do not expose an
action schema, so `payload.action` remains `unknown`.
## Validation failures
Invalid client data is logged and never passed to user code. The client
receives a fixed `Invalid client data` error; validator details stay in
the task log and `onClientDataValidationError`.
- Submitted turns and async reads write the error followed by
`turn-complete`, then wait for the next valid frame. This settles the
invalid input before the raw read returns. Callers that need to
coordinate validation with their own persistence or settlement should
omit the schema and validate the full frame in their loop.
- Messageless preload and continuation boots call
`onClientDataValidationError` and wait without writing a terminal frame.
- Active `chat.messages.on()` subscriptions skip invalid frames and call
`onClientDataValidationError` without ending the response. `off()` stops
new frames. A valid frame accepted before `off()` finishes validation
and is delivered; an invalid pending frame is logged without invoking
user callbacks.
- `chat.messages.peek()` throws synchronously.
- Invalid head-start handovers fail closed. A skip ends the run. A real
handover writes the validation error after the warm output, writes
`turn-complete`, and ends the run.
Validation is automatic when a schema is declared. We can make it opt-in
or return a typed failure if maintainers prefer that contract.
## Testing
- `pnpm --filter @trigger.dev/sdk run test -- --run`
- `pnpm --filter @trigger.dev/sdk run typecheck`
- `pnpm run build --filter @trigger.dev/sdk`
- `pnpm run lint`
- Formatting checks pass
## ✅ Checklist
- [x] I followed the contributing guide
- [x] The PR title follows the convention
- [x] I ran and tested the change
## Changelog
Custom chat agents now validate and parse client data declared with
`chat.withClientData({ schema })` before passing it to agent code.
## Screenshots
Not applicable.
---------
Co-authored-by: Eric Allam <eallam@icloud.com>
|
||
|
claude[bot]
|
a3af29fd80 |
fix(sdk): re-dispatch a single in-flight user on recovery boot (#4768)
<!-- ccr-slack-attribution --> _Requested by **Matt Aitken** · [Slack thread](https://triggerdotdev.slack.com/archives/C061L2MHW93/p1787615162456839?thread_ts=1787615162.456839&cid=C061L2MHW93)_ **Before:** a `chat.agent` run is killed mid-answer (OOM, crash, eviction) while the message it was answering is the only one still outstanding. The new run boots, puts that message and the half-written reply into its context, and then waits for a message that already arrived. Nobody ever answers the user; the run sits idle until it times out. **After:** the new run re-runs that message as a fresh turn and replies to it. The half-written reply is dropped. When two or more messages are outstanding, nothing changes — the interrupted one still goes into context and the newer ones are re-run, exactly as before. ## ✅ Checklist - [x] I have followed every step in the [contributing guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md) - [x] The PR title follows the convention. - [x] I ran and tested the code works --- ## Testing New regression test in `packages/trigger-sdk/test/recovery-boot.test.ts` — seeds a partial assistant plus exactly one in-flight user, no `onRecoveryBoot`, and asserts one turn fires for that user with the orphan partial dropped from the chain. It fails on `main` (`turnCount` 0, no turn at all) and passes with this change. - `pnpm exec vitest run` in `packages/trigger-sdk` — 373 passed, 1 skipped (31 files passed, 1 skipped) - `pnpm exec oxfmt --check` on the changed files — clean - `pnpm exec oxlint packages/trigger-sdk/src packages/trigger-sdk/test` — clean - `pnpm run build --filter @trigger.dev/sdk` — clean **What it does:** with exactly one in-flight user on a recovery boot, re-dispatch that user as a fresh turn instead of splicing it into the seed chain, where it was never answered. **How:** the recovery-boot smart default made one decision in two halves — the seed chain and the recovered-turn list — both gated on `partialAssistant !== undefined && inFlightUsers.length > 0`. The splice consumes `inFlightUsers[0]` into the chain as "the question the partial was answering" and dispatches the rest. That only works when there *is* a rest: at n=1 `recoveredTurns` came out empty, the boot-injected queue stayed empty, the `session.in` cursor was advanced past the message anyway, and on a `preload` or continuation boot (no `message` on the wire payload) neither dispatch site fired. Both branches now require `length > 1`, so n=1 falls through to the documented default — chain = `settledMessages`, re-dispatch every in-flight user. The submit-message boot is unaffected: the existing dedup still drops a queued message identical to the one already on the wire payload. Also corrected alongside it: the two SDK docstrings and the `docs/ai-chat/patterns/recovery-boot.mdx` defaults section, which described the default as "re-dispatch every user" and never mentioned the splice. Follow-up (not in this PR): the webapp e2e OOM helper never streams a token before throwing, so it exercises the no-partial path only and would not have caught this. Worth a variant that emits a token first. --- ## Changelog Fixed a chat agent hanging after an interrupted turn: when a run was killed mid-answer and only the one message it was answering was still outstanding, the new run never replied to it. That message is now re-answered on the new run. Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: Matt Aitken <matt@mattaitken.com> Co-authored-by: Eric Allam <eric@trigger.dev> |
||
|
Daniel Sutton
|
15dd973f92 |
feat(core,webapp,run-engine): stamp a shard key onto run, batch and waitpoint ids (#4788)
## Summary Adds the id-minting half of sharding run data across several databases. Every entity that co-locates with a run now carries the run's shard key inside its own id, so its row is routable on its own instead of needing a directory table or a scatter across shards. Nothing changes for users yet. With no shard descriptors configured, every mint path produces exactly the ids it produces today, and the trigger path issues no extra query. ## Design A run's mint target travels as a single object carrying the kind and, when sharded, the shard character. The shard and the caller's region both occupy index 24 of a run-ops id, so passing them together makes it impossible for a caller to set two competing sources for one slot. A child run, a batch and a batch item read the shard from their parent's id rather than resolving a fresh one, so a run tree never splits across databases. Three services carried that branch separately, and one had already drifted, so it now lives in one function. Waitpoints mint through one shared pure function used by both the webapp and the run engine. They have to agree byte for byte, because the routing store refuses a waitpoint whose id is not stamped for the shard it is being written to: ```ts mintWaitpointIdForShard(key) // standalone token: the environment's shard mintWaitpointIdFor(anchorId) // co-located: the anchor's shard, or a cuid ``` The core is always freshly minted rather than derived from the anchor, since a derived body would be byte-identical to the run's own id. One latent bug fixed on the way: the failed-run path duplicated the mint branch inline and had drifted, so a child of a sharded parent would have been written to a different database from its parent. ## Guarding the create sites The expensive failure here is a waitpoint minted without its anchor's shard: one of the five create sites writes through a path that has no stamp check, so a miss there strands a blocked run with nothing logged. An enumerated census plus a source scan fails when a new create site appears, when an existing one stops passing its anchor, or when a site is added to a file the scan does not yet cover. The census was written before any site was converted, so it went red on the first commit and green as the last site landed. Both holes an earlier draft had, a file-granular count and a scan that missed the directory these mints used to live in, were confirmed closed by reintroducing them and watching the guard fail. ## Before enabling a shard Merging this is inert: with the mint list empty the resolver returns before it reads anything, and ids are identical to a measured `main` baseline. Verified against a live shard locally, including that the resolver issues no query across thirty triggers with no shard configured. Enabling is gated on two other pull requests, both open, both by the same author, each of which owns the file involved: - **#4781** adds the gen-2 shard arm to read-through. Without it a gen-2 run cannot wait on a token at all: the wait route resolves the waitpoint through read-through, which is shard-blind, so the wait fails. Do not set the mint list before it merges. - **#4780** generalises the distinct-database sentinel. Without it a shard pointed at the same physical database as the gen-1 store boots without complaint, which voids the disjointness the fan-out sums rely on. Testing also turned up a silent read-path gap that neither pull request covers: the paths that hydrate runs from ClickHouse through a fixed pair of Postgres clients drop gen-2 rows on the floor, so the runs list would show fewer rows than its own count with nothing logged. That needs its own change before a shard carries real traffic, and it is filed as such. ## Notes for reviewers Four commits in the middle of the stack do not typecheck in isolation: a signature change and its call-site repairs are separate commits, so bisecting inside the stack needs care. Commit `845ab06` also understates itself, since it rewrites the primary trigger path's mint alongside the failed-run path it names. No changeset and no server-changes entry: every path is inert while the feature is off, so there is nothing to tell users yet. --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com> |
||
|
Graham Tremper
|
4e006519de |
feat(chat): expose endAndContinue to custom agents (#4647)
## Summary Raw `chat.customAgent()` loops can now call `chat.endAndContinue()` to move the Session to a fresh run. The managed loop already used the same server operation through `chat.requestUpgrade()`, but raw loops could not call it directly. Call the method between turns after detaching input listeners from the old run. Await it and return immediately. Unconsumed `.in` records stay on the Session for the continuation run. I put this on the `chat` namespace next to the other raw chat primitives. Happy to move it if maintainers prefer a different API placement. ## Testing - `pnpm exec vitest run` in `packages/trigger-sdk` (374 tests) - Focused webapp Session E2E tests (3 tests) - `pnpm run build` in `packages/trigger-sdk` - Webapp typecheck - `pnpm run format` - `pnpm run lint` ## Checklist - [x] I followed the contributing guide - [x] The PR title follows the convention - [x] I tested the change ## Changelog Allow custom chat agents to rotate to a new task version without dropping unconsumed Session input. --------- Co-authored-by: Eric Allam <eallam@icloud.com> |
||
|
Eric Allam
|
d54bcaa29c |
fix(chat): stop losing a user message that arrived mid-turn (#4795)
Follow-up to [#4644](https://github.com/triggerdotdev/trigger.dev/pull/4644), now rebased onto main so the diff is just these three commits. ## Summary Two ways a chat could lose a user message, both pre-existing and both raised while reviewing #4644. A message arriving while a turn was streaming was handed to that turn's push handler and parked in an in-memory array. The router counts a record handed to a handler as terminally decided, so it stopped holding the resume floor behind it, and the turn boundary published a cursor past a message that existed only in that process. A crash before the next turn lost it, silently. Measured: with the message at sequence 1, the boundary published `session-in-event-id: 1`, so a resume skipped it. Separately, a message the agent declined to inject was discarded with the turn. Never injected, never written to the wire buffer, never answered. That was also the documented default, since a `pendingMessages` config without `shouldInject` declines every batch. ## Design Notification and consumption are now separate concerns on the router. `observe` reports that a record arrived without taking it, so the record stays queued and keeps holding the floor. It is rejected on an `at-arrival` route: an observer there would either have to count as a listener, which would stop an unconsumed stop being discarded and bring back a wedged mailbox, or watch records it cannot affect. `take` removes exactly one queued record. The managed loop and the `chat.createSession()` iterator now only subscribe when there is a steering config to feed, and injection is the point of consumption. A declined batch never reaches the take, so its records stay queued and become later turns. Both in-memory wire buffers are gone, so a message waiting for its turn is durable rather than living in whichever worker received it. The floor doubles as the wake cursor: `awaitWake` registers with it and the server completes the waitpoint immediately if anything sits after that sequence. An over-advanced floor was therefore also a missed wake. It is now recorded on the wait span so a run that never woke can be diagnosed from its trace. ## Verification Both fixes have a red and green pair, each checked against the unmodified source rather than only observed to pass: - the resume cursor test fails on the parent branch and passes here - the declined-message test fails without the second commit and passes with it Also 8 new router tests for `observe` and `take`. Suites green at 385 for the SDK and 886 for core. ## Not addressed A `pendingMessages` config with no `chat.toStreamTextOptions()` spread still swallows messages, because nothing drains the queue at all. Same shape, different trigger, tracked separately. |
||
|
Graham Tremper
|
1065251ca7 |
fix(chat): ignore stale turn completions after reconnect (#4643)
## Summary Reloading a browser chat mid-turn can replay a completion event for an older input and close the active turn too early. This persists the last browser-owned input sequence and reuses it on reconnect, so older completion events are ignored. The sequence is cleared after the matching boundary, and reconnect avoids the settled-peek shortcut while that sequence is active. The persisted field is optional, so sessions without it keep their existing behavior. ## Testing - `pnpm --dir packages/trigger-sdk run test ./src/v3/chat.test.ts ./test/chat-turn-correlation.test.ts --run` — 67 passed - `pnpm --dir packages/trigger-sdk run test --run` — 32 files, 379 tests passed - `pnpm run build --filter @trigger.dev/sdk` - `pnpm run format` - `pnpm run lint` ## Changelog Browser chats now keep the active turn open across page reloads when older completion records are replayed. ## ✅ Checklist - [x] I have followed every step in the [contributing guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md) - [x] The PR title follows the convention. - [x] I ran and tested the code works 💯 Co-authored-by: Matt Aitken <matt@mattaitken.com> |
||
|
Graham Tremper
|
c115f440bc |
feat(chat): custom agent mailbox helpers and session.in delivery fixes (#4644)
## Summary
Adds `chat.messages.hasPending()` and `chat.messages.next()` so a custom
agent
loop can inspect pending chat input without consuming it and take one
record at a
time, and fixes four ways a chat could mishandle input across a restart:
a
message silently lost, a recovered answer cut off by a stop the user had
already
pressed, a retried send answered twice, and a record the agent had no
consumer
for blocking every message queued behind it.
```ts
if (await chat.messages.hasPending()) {
const record = await chat.messages.next({ timeoutInSeconds: 0 });
if (record) handle(record.payload);
}
```
## Why the fixes came together
`session.in` carries records for consumers whose delivery needs differ.
A user
message must be delivered eventually, so it can wait arbitrarily long
for a turn
to take it. A stop only means anything to the turn that is live when it
lands.
Progress along the channel was tracked as one sequence number, and one
number
cannot say "control applied through 7, message 3 still owed" at the same
time.
Each of the bugs above is that mismatch surfacing somewhere different.
So instead of a rule per symptom, records are now classified once and
handed to
one route, and each route declares two things: whether it holds a record
when no
consumer is ready, and whether a record it never handled has to survive
into the
next boot. The resume cursor, the replay window and the
discard-the-unowned
behaviour are then derived from route state rather than maintained
beside it, and
`hasPending()` answers from the message queue instead of the head of a
buffer
shared with every other kind.
The wire is unchanged. Both cursors on the turn boundary keep their
meanings, so
existing chats resume as before and there is no webapp change.
## Behaviour worth calling out
`chat.writeTurnComplete()`'s `sessionInEventId` is the cursor that is
safe to
resume from, not the sequence of the record the turn answered. It is
held back
behind any message still waiting to be handled, so a value below the
record you
just handled is expected rather than a sign of a lost turn.
The stop fix also covers chats whose most recent turn was completed by
an older
SDK, by resolving the replay window from the channel when the boundary
does not
carry one. The trade there is deliberate: a stop that landed in the
moments
before boot and was never applied is dropped along with the replayed
ones,
because a stop the user can press again beats a stale one killing an
answer they
are waiting for.
## Verification
Thirteen reproductions against a local stack, each driving real runs
rather than
mocks, covering the documented `next()`/`hasPending()` loop, suspend and
resume, a
crash between consuming a message and writing turn-complete, a retried
send whose
idempotency claim is lost, and a continuation boot that must not replay
answered
messages. Where applicable each was also run against `main`, so the
fixes are
differences rather than assertions.
Five further legs on a deployed environment, which the earlier revisions
of this
branch did not cover at all: a message appended while the run is
genuinely
checkpointed, a message appended while the run is dead, the
stop-after-crash case
on the real crash path, and both version-skew directions (a newer worker
resuming
an older worker's turn boundary, and an older worker resuming a newer
one's).
Two of those restart fixes also have a browser-driven red and green pair
on a
deployed environment, staged identically on both sides and differing
only in the
SDK. For the lost-message fix, the unanswered message is replayed and
answered in
full here, and is never replayed at all on the released SDK. For the
stop fix,
both sides replay the message and diverge on the stop itself: it is
declined here
and the answer completes, while the released SDK re-applies it and the
recovered
answer dies before it streams.
The routing decision itself is a pure state machine, so it also has a
property
test over every interleaving of the record kinds crossed with each crash
point,
checked by mutation to confirm it fails when the cursor arithmetic or
the replay
window is broken.
## Known and not addressed here
The read of the woken record is unbounded, so a wake with nothing to
read makes
`wait()` outlive its own waitpoint. Tested and not a deadlock, since the
read
defers to the next record, but bounding it is a separate change with its
own
test.
Separately, and not caused by this branch: a run that crashes while a
message is
still queued is not replaced until the next inbound append, so that
message waits
rather than being recovered on its own. Worth its own issue.
Also not caused by this branch, but worth knowing when reading the
release note: a
chat page that stayed open across the crash keeps showing the partial
answer it
already received, so the recovered answer only appears after a reload.
The answer
itself is persisted correctly. The gap is on the client, which does not
apply a
re-delivered turn over a partial it already holds.
---------
Co-authored-by: Eric Allam <eric@trigger.dev>
Co-authored-by: Eric Allam <eallam@icloud.com>
|
||
|
wei-wei
|
c7f78e4853 |
fix(sdk): reset skipToTurnComplete when a new chat turn starts (#4744)
## ✅ Checklist - [x] I have followed every step in the [contributing guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md) - [x] The PR title follows the convention. - [x] I ran and tested the code works --- ## Testing Reproduced with `useTriggerChatTransport` + `useChat` and the stop pattern from the ai-chat frontend docs: 1. Send a message so a turn is streaming. 2. Call `transport.stopGeneration(chatId)`, then `useChat`'s `stop()`. 3. Send another message. Before this change the second turn never renders: no parts arrive, `status` stays `streaming`, and the session stays `isStreaming: true`, so a stop button stays on screen until the page is reloaded. The run itself is fine and everything persists, so a reload shows the full response. Cause: `stopGeneration` sets `state.skipToTurnComplete = true`, and the read loop only clears that when it sees a `TURN_COMPLETE` record. The abort closes the reader before that record arrives, so the flag survives into the next turn and every record of that turn is skipped, including its own `TURN_COMPLETE`. After this change the same sequence streams the second turn normally. Verified against 4.5.11 and 4.5.12 (both affected) with the equivalent patch applied to the built SDK. --- ## Changelog Reset `skipToTurnComplete` when a new chat turn or action is sent, so a message sent after `stopGeneration` streams normally instead of leaving the chat stuck in a streaming state. --------- Co-authored-by: Devin AI <devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com> |
||
|
Daniel Sutton
|
920892bc11 |
feat(webapp,run-store): gen-2 shard arms in read-through and idempotency (#4781)
Gives read-through and idempotency their gen-2 shard arms, so an id that names its own shard is read there and nowhere else. #4764 has landed, so this now targets `main` directly and no longer depends on an unmerged branch. It builds on what that PR supplied: `resolveShard`, `runOpsShardHandles` and the keyed router. TRI-13431 ## What changes **Read-through routes by `resolveShard`, not by the binary residency classifier.** A gen-2 id reads its own shard's replica once and probes no other store. A gen-1 v1 id still reads new only. **Callers now declare `idKind`.** A cuid gives no way to tell a run id from a waitpoint id, and the two must route differently: - a legacy-classified **run** id reads the legacy replica only — there is no cuid run migration, so the new-store probe cannot find it; - a cuid **waitpoint** keeps the new-first pair probe, which is load-bearing because a cuid waitpoint can be co-located with its run on the new store. There is no default, because a default would pick one of those arms silently. The field `runId` is renamed to `id`, since it carried both kinds already. **`ReadThroughResult` carries `found`.** `source` is an open-ended union once shards exist, so a consumer testing found-ness by listing the hit sources reads a gen-2 hit as a miss. One consumer did exactly that. Discriminating on `found` makes that class of bug a compile error rather than something a reviewer has to spot. **Idempotency resolves its client through one shard-keyed map.** Both call sites go through `clientForShardKey`, so they cannot disagree about which store owns an id. An absent key takes an explicit logged branch to the fallback, not a silent legacy default. The `classify` seam is retyped to return a `ShardKey`: `Residency` (`"NEW"`) and the reserved shard keys (`"new"`) differ only by case, and `ShardKey` collapses to `string`, so the compiler would not have caught feeding one into the other. The dead `isMigrated` branch is deleted. Nothing implemented it, and the one production comment recorded that omitting it was deliberate. **`PostgresRunStore._residency` widens to `ShardKey`.** Still unused; the store stays unaware of its siblings. ## Two behaviour fixes found while doing the above **An unconfigured shard key logs and returns not-found instead of throwing.** The waitpoint route takes the id from a URL parameter, and any base32hex core plus `[a-z0-9]` plus `"2"` parses as gen-2. The route turns a throw into a 500, so throwing here would let any authenticated client generate 500s and error logs by guessing shard chars, of which there are 36. An error-logged not-found is neither silent nor a misroute. Throwing stays correct on the router path, where ids are minted rather than received. **The two cross-seam batch hydration sites were gen-2 blind.** `hydrateRunsAcrossSeam` and `ApiBatchResultsPresenter` classified with the binary `ownerEngine`, so a gen-2 run id joined the gen-1 `new` group, missed there, and — classifying dedicated-family — never reached the legacy probe either. The id was dropped from a bulk-action page and from batch results with no error. Both now partition ids by shard key and read each configured shard once. Also: a gen-2 waitpoint that missed its shard replica fell back to the gen-1 new writer, a different database, silently disabling read-your-writes for the freshly minted token that fallback exists to serve. It now falls back to its own shard's writer. ## Merge safety Inert while `RUN_OPS_SHARDS` is unset: the shard maps are empty, so every gen-2 arm is unreachable, and gen-2 minting is not live yet. The one live change is the gen-1 run arm, and it removes work rather than adding it. `RoutingRunStore.findRun` never forwards the caller's client object — it routes by id and reads only the client's presence and replica brand — so `readRunForEvent`'s "new" closure already resolved a legacy-classified run id to the legacy store. The arm removes a duplicated read of the legacy replica. A test pins this, because a future caller passing a raw client and a run id would lose the pre-cutover 27-char case, which is new-resident but classifies legacy. ## Testing 14 tests added, testcontainers throughout, no mocks. 22 affected test files pass; typecheck, lint, format and knip are clean. Both arms were verified by neutralising them and confirming the new tests fail. The batch-results test needed rewriting after that check: the first version passed with the fix neutralised, because it used one container as both the gen-1 new client and the shard replica, so it was not testing what it claimed. Note for review: run testcontainer suites in small batches. Sixteen at once starves Docker and everything times out at 60 seconds. The run-ops legacy-guard baseline is refreshed in its own commit. The baseline is keyed by line number, so partitioning the batch-results read shifted four pre-existing entries and added one. Baselined violations in that file go from four to five, all reads; the new one is the shard read beside two gen-1 reads already there. No changeset and no `.server-changes` entry: a user notices nothing while the flag is unset. |
||
|
Oskar Otwinowski
|
4c16387426 |
fix(webapp): project integrations page — Staging gating, unreachable code, and follow-ups (#4784)
Three bugs on the project integrations page, one commit each for the two reported ones and four for the follow-ups found while fixing them. ## `chore`: remove unreachable code on the integrations page (TRI-12645) Two notification panels in `VercelSettingsPanel` could never render: 1. The **"Failed to load Vercel settings"** panel was gated on a `hasError` state whose setter is never called anywhere, so it was permanently `false`. 2. The **"connection expired"** banner *inside* the `connectedProject` branch was unreachable: `VercelSettingsPresenter` only populates `connectedProject` on its success exit, which hardcodes `authInvalid: false`, while both `authInvalid: true` exits return `connectedProject: undefined`. Removing them makes the surrounding `!showAuthInvalid` guards vacuous, and the `onboardingData?.authInvalid` disjunct redundant — the loader already folds onboarding auth state into `authInvalid` before it reaches the component. **No behaviour change.** An org with a connected project and an expired token still gets the banner, from the branch below (untouched). ## `fix`: gate Staging settings on plans without a Staging environment (TRI-12646) The ticket's premise was inverted, and I've corrected it there. In Git settings, **Preview** is the row that's correctly gated; **Staging** is the one with no gate at all: - Preview swaps its switch for an Upgrade button, and `projectSettings.server.ts` neutralises a forged `previewDeploymentsEnabled=on`. - Staging was a plain always-editable `Input`, and `validateStagingBranch` only checked the branch existed on GitHub. An org without a staging environment could type a tracking branch, hit Save, get a success toast, and have it silently do nothing. Staging and Preview environments are created together for projects on a plan that includes them, so gating one and not the other was an oversight. The Staging row now mirrors the Preview row. Server-side it ignores the submitted branch when there's no staging environment, but **preserves the stored branch rather than clearing it** — deliberately different from the Preview handling. Forcing a boolean off is harmless; forcing a *string* off would wipe a tracking branch the org had already configured the first time they saved after losing the environment. The Vercel write path had the same gap: `update-config` / `complete-onboarding` / `update-env-mapping` never re-derived available env slugs server-side, so `["stg","preview"]` could be persisted for a project with neither environment, and `createDefaultVercelIntegrationData` turned preview on unconditionally. Both now filter against the project's actual environments, via a pure `restrictConfigToAvailableEnvSlugs` helper that only touches keys present on the input. ## `fix`: show build settings when the GitHub app is disabled (TRI-13488) The page wrapped Git settings, the Vercel section **and** build settings in one `githubAppEnabled` guard, so with the GitHub app off it rendered an empty container. The Vercel section genuinely depends on GitHub — it can't sync environment variables or link deployments without a connected repo — so it stays gated. Build settings don't: they also apply to CLI deploys run with `--native-build-server`, exactly as the section's own description states. They now render regardless. ## `fix`: stop the Vercel onboarding modal spinning forever (TRI-13488) `computeInitialState` starts in `loading-projects` whenever the org has a Vercel integration but no onboarding data yet, and the effect that escapes it waits for `availableProjects !== undefined`. When `getOnboardingData` returns `null` — it does that on any thrown error, and when the org integration row is missing — nothing ever arrives. The empty-array case self-resolves (`[] !== undefined`), so this is specifically the null case. The route can tell "still loading" from "loaded nothing" because its fetcher always requests `?vercelOnboarding=true`; it now passes that down and the modal explains the failure with a retry and a link to check the integration's access on Vercel. ## `fix`: match staging and preview environments consistently (TRI-13488) The four places that ask "does this project have a staging / preview environment?" disagreed. `VercelSettingsPresenter` matched on type with no parent filter, so any preview *branch* row satisfied it — branches are `PREVIEW` rows too. `GitHubSettingsPresenter` and `ProjectSettingsService` matched on slug instead. Slug is the weaker key: it's derived at creation time and legacy rows can carry something else, which is why `memberDevelopmentEnvironmentWhere` deliberately avoids it. All four now match on `type` plus `parentEnvironmentId: null`, which excludes branches without depending on the slug being canonical. ## `fix`: explain when no Vercel environment can be mapped to Staging (TRI-13488) Reported while reviewing the branch. The Staging build settings show *"Set a Vercel environment for Staging first."* whenever the project has a staging environment and no mapping — but the control that sets the mapping only rendered when the Vercel project had at least one custom environment: ``` hint: hasStagingEnvironment && !configValues.vercelStagingEnvironment control: hasStagingEnvironment && customEnvironments.length > 0 ``` So a Vercel project with no custom environments, or one whose custom environments failed to fetch (the presenter swallows that error to `[]`), got an instruction with nothing to act on. Both conditions predate this PR. The mapping row now always renders alongside the hint and explains what to do when there's nothing to choose from, and the build-settings hint says the same thing. ## `chore`: remove the remaining dead code (TRI-13488) - The `"installing"` `OnboardingState` is unproducible — no `setState` call yields it — so its redirect effect, switch arm, `isLoadingState` conjunct and the `vercelAppInstallPath` import it was the only user of are all dead. - `(state as string) !== "completed"` sits in a branch where TypeScript has already narrowed `"completed"` out; the cast is what let it compile. - `hideSectionToggles` was only ever passed alongside `layout="settings"` but only read inside `layout="card"` blocks, so it could never take effect. Removed the prop entirely. - Unused bindings and the helpers only they referenced: `envSlugLabel`, `_formatSelectedEnvs`, `_CompleteOnboardingForm`, `_handleFinishOnboarding`, and the rest. No behaviour change in that commit. ## Not included The three overlapping modal-open effects in `settings.integrations/route.tsx` are left alone — they're defensive against a close-then-reopen race, and untangling them is a behavioural risk with no user-visible payoff. ## Verification `pnpm run typecheck --filter webapp`, `pnpm run lint` and `pnpm run knip` are clean. New `apps/webapp/test/vercelIntegrationConfig.test.ts` covers the slug restriction and the default-config seeding (both pure functions); 39 tests pass across it and the three existing Vercel/project-settings files. The new `projectId` + `slug` query is served by the existing `@@unique([projectId, slug, orgMemberId])` prefix — same access pattern as the preview check it mirrors. refs TRI-12645, TRI-12646, TRI-13488 |
||
|
Daniel Sutton
|
02e6157d12 |
feat(run-store): add an execution-snapshot store decorator behind an off-by-default dial (#4765)
## Summary Adds a `RunStore` decorator that mirrors execution snapshots into Redis alongside Postgres, plus the orphan-key sweep and the fault-injection suite that prove the write protocol converges after a crash. Nothing constructs it, so merging this changes no behaviour: the configuration, the production wiring and the Redis client all arrive in later work. The execution-state log is the hottest table in the run graph, and moving it out of Postgres has to happen without a big-bang cutover. This is the attachment point for that: a decorator that wraps the existing storage interface and intercepts only the methods that touch snapshots, so none of the many callers change. ## Design Write order is the correctness property, and the two orders differ on purpose. A transition writes Postgres first and Redis second. A crash in the gap leaves a run whose latest snapshot is stale, which is the state the heartbeat stall watchdog already heals in production today. A birth writes Redis first and Postgres second. A crash there leaves an unreachable key for a run that does not exist. Postgres first would instead leave a run with no snapshot at all, which the engine treats as a hard error, so the run would be stuck. Each order is chosen so the state a crash leaves behind is the harmless one. A lost cross-store write is never recovered by a transaction or an outbox; recovery is always the existing stall and repair job. A failed append retries, then hands the run to that job, and never rethrows, because Postgres has already committed and a throw would turn a healable gap into a caller-visible error. Inside a transaction the Redis half is staged and flushed only after the commit, so a rollback cannot leave Redis holding a transition that never happened. Reads are shape matched. Two of the snapshot reads take arbitrary Prisma arguments, and a key-value store cannot answer an arbitrary query, so the decorator recognises exactly the shapes the engine sends and delegates everything else. A miss falls back to Postgres, which is also how runs created before any cutover keep working. The sweep reaps under two rules, because neither can see what the other leaves behind. A finished run whose keyspace never received its completion expiry gets one applied. A keyspace with no run row at all, past an age threshold, is deleted; that is a crashed birth, which is non-terminal so it carries no expiry and has no run row, so the first rule can never match it. ## Inertness Three independent reasons this is a no-op if merged alone: - Nothing constructs the decorator or the Redis store outside tests. - No configuration reaches it, so the dial stays at its off position, which is a pass-through that makes no Redis call. - The existing Postgres store gains an off-by-default flag and two optional input fields. Both default to today's behaviour, and only the decorator would ever supply them. ## Notes for review The snapshot id and the creation instant are both minted by the decorator and written into both stores, so one snapshot has one identity and one timestamp wherever it is read. Without that, the two stores disagree on values that later tooling has to compare, and the cursor for a snapshot window resolved from one store misfilters the window walked in the other. Three defects in this work passed the full existing test suites before being found by review rather than by a test: the decorator wrote no wait cycle at all, the snapshot window dropped the ordering used to give each completed waitpoint its position in a batch, and the two stores stamped different creation times. The common cause was that no test drove a snapshot that actually carried waitpoints, and that the parity suite compared a timestamp against a value it had just read back from the row it was checking. Both gaps now have tests. |
||
|
Daniel Sutton
|
1801b0e80b |
feat(webapp,docker): run-ops boot interlocks and migrations at N databases (#4780)
## Summary
The run-ops boot interlocks and the migration entrypoint each assume
exactly two run-ops
databases. This generalizes them to any number, so a deployment that
configures
`RUN_OPS_SHARDS` gets the same safety guarantees it gets today with two
stores: no two stores
may point at one database, every store that owns its own database must
replicate to
ClickHouse, and every store must have its schema migrated.
With `RUN_OPS_SHARDS` unset, nothing changes. The distinctness check
over a two-element set is
the pairwise compare it replaces, replication coverage is the check it
was, and the entrypoint
runs the same two migration invocations.
A shard may declare `aliasOf: "new"`, which shares an existing store's
client by reference. An
aliased shard is not its own database, so it is exempt from the
distinctness check and needs no
replication slot of its own. Every check keys that exemption on the
declared field, never on
client object identity: two client objects can sit over one database,
which identity comparison
cannot see.
## Design
**Distinctness.** `probeDistinctDatabases` compared two URLs. It now
delegates to
`probeDistinctStores`, which reads every fingerprint in parallel and
groups them by system
identifier and database name. Any two stores under one key refuse the
boot. The old pairwise
entry point stays, so its existing container tests are the proof that
set uniqueness over one
pair gives the verdict it gave before. Fail-closed is unchanged: a probe
that cannot answer
returns not-distinct, because "distinct" is a positive claim a failed
probe cannot support.
**Co-residency.** The advisory runs once per store against the control
plane. The legacy
emission keeps its exact call shape and its untagged metric series, so
an existing dashboard
does not change. Each shard emits its own point carrying its shard key.
Every store emits
before any enforcement throw, so one offending store never costs another
store its metric.
**Replication.** `buildReplicationSources` appends one source per shard
that owns its own
database, taking the slot, publication and origin generation its
descriptor declares.
`assertReplicationCoversSplit` then requires a source per such shard.
That check also closes a hole it inherited. The descriptor parser
validates uniqueness among
shards only, so a shard could take the slot name, publication name or
origin generation of the
legacy or the new source. The replication service does validate this,
but it throws from its
constructor, and the caller reaches that constructor only after shutting
the bootstrap instance
down:
```ts
if (sources.length > 1) {
await service.shutdown(); // legacy stream stops here
service = new RunsReplicationService({ ... }); // throws: duplicate slotName
}
```
The throw was not a `SplitReplicationMisconfiguredError`, so the process
stayed up with no
replication at all, legacy included, behind one logged line. That is the
silent ClickHouse
under-count the error exists to prevent. The check now runs at the boot
gate, before anything is
torn down, and raises a subclass the existing exit path already
recognizes. A correct deployment
already satisfies it, because two consumers on one WAL slot is a data
race that cannot work.
**Migrations.** Every shard runs the identical schema, so a new shard is
the existing migrations
against a new DSN. The runner image has no `jq`, so a small node script
prints one DSN per line
and the entrypoint loops over them. The loop is a `for` and not a `while
read` pipeline: a
pipeline subshell swallows a failed migration on any iteration but the
last, which would let a
broken shard boot. Tracing stays off across the capture and the loop,
because `set -x` prints an
assignment and a DSN carries credentials.
Verified end to end against real Postgres containers for the fingerprint
probes, and against the
real shell block with a stubbed migration command: an aliased shard is
skipped, `directUrl` wins
over `url`, a failing shard stops the container on the first failure,
and a malformed descriptor
stops it before it migrates anything.
Stacked on #4764.
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
|
||
|
Saadi Myftija
|
8da393bf33 |
feat(webapp): add org slug and project name to deployment telemetry events (#4785)
Adds `$trigger.org.slug` and `$trigger.project.name` attributes to the `deployment.finished` / `deployment.initialized` events (follow-up to #4778). |
||
|
Saadi Myftija
|
38e78f8c7e |
feat(webapp): deployment lifecycle telemetry events (#4778)
Deployments currently leave little analytical trace. This PR makes every
deployment emit two analytics events to enable useful queries. It also
enables comparing deployments across build paths, CLI versions,
runtimes, and orgs.
### Where the events come from
```
trigger deploy
│
▼
initialize ─────────────────────────────▶ ✨ deployment.initialized
│ createdAt
▼
PENDING waiting for a build slot ┐
│ startedAt │ queue time
▼ ┘
INSTALLING build server installs deps ┐
│ installedAt (native paths only) │ install time
▼ ┘
BUILDING the image is built ┐
│ builtAt │ building time
▼ ┘
DEPLOYING indexing + registry push ┐
│ deployedAt / failedAt / canceledAt │ deploying time
▼ ┘
DEPLOYED · FAILED · TIMED_OUT · CANCELED
│
└───────────────────────────────────▶ ✨ deployment.finished
```
`deployment.finished` fires exactly once, whichever way the deployment
ends, and is backdated to cover the deployment's real lifetime. Not
every path visits every state (Depot deploys skip PENDING/INSTALLING,
for example) — a phase duration is simply omitted when its state was
never entered.
### What each event carries
- **Which path built it**: `depot`, `native`, or `native_local_bundle`
- **How it ended**: status, plus an error class and message when it
failed
- **How long each phase took**: queue, install, building, deploying, and
total — derived from the timestamps above
- **Who and with what**: org, project, environment, runtime, CLI
version, and how the deploy was triggered (CLI, GitHub, Vercel)
With that, one query gives failure rate per build path, duration
percentiles per phase, adoption per CLI version, or a per-org health
table.
### Fixes that ride along
- The old `deployment.outcome` span was silently dropped ~95% of the
time (it was subject to trace sampling). The new events opt out of
sampling explicitly, so every deployment is counted.
- The fail/timeout/finalize transitions were racy: a late timeout could
overwrite a successful deployment. They now use guarded writes, so
exactly one caller wins the terminal transition — and exactly one event
is emitted.
- Canceled deployments previously recorded nothing; they do now.
- The deployment's CLI version is now stored at initialization (new
nullable column), so even deploys that fail early are attributable to a
CLI release.
- Telemetry is flushed on shutdown (the last batch used to be lost on
every webapp deploy), and an optional second exporter can mirror just
these events into a dedicated dataset.
|
||
|
Daniel Sutton
|
00e3c151d4 |
feat(webapp): RUN_OPS_SHARDS config, topology and N-way store wiring (#4764)
Part of the RunOps N-way sharding work. This lets the webapp hold N run-ops stores, configured by a single `RUN_OPS_SHARDS` JSON descriptor, and routes to them through the existing keyed router. **Inert with `RUN_OPS_SHARDS` unset** — the topology, the wiring and `ROUTING_ENABLED` are byte-identical to today. ## What's here - **`RUN_OPS_SHARDS`** — a zod-validated JSON array of shard descriptors (`key`, `region`, `url`, `replicaUrl`, `directUrl`, `replication`, `knobs`, `aliasOf`), validated at boot in the `parseMachinePresetCsv` style. Unset or `[]` → no shards. - **One run-ops client factory** — `buildRunOpsWriterClient`/`buildRunOpsReplicaClient` collapse into one `buildRunOpsClient` parameterized by role and resolved pool knobs. The control-plane builders (`buildWriterClient`/`buildReplicaClient`) are a separate path and stay untouched; every resolved value matches the former builders. - **Shard loop in `selectRunOpsTopology`** — one client pair per descriptor; an `aliasOf: "new"` descriptor reuses the new store's clients by reference and opens no pool. - **N-way `buildRunStore`** — builds N dedicated stores + the keyed router via a new `RoutingRunStore.fromShards`, keeping the two-store compat router when no shards are configured. - **`UnknownShardKey`** — raised when an id resolves to an unconfigured key; never falls back to another store. `fromShards` injects `resolveShard` so a gen-2 id routes to its own shard. - **Per-shard transaction resilience** — each shard gets its own retry budget. - **Mint bound** — `computeMintShard` intersects the active mint list with the configured descriptor keys, so a key with no descriptor is never minted into. - **Boot table** — logs `key`, address fingerprint (host:port/db, no credentials), and role, only when shards are configured. ## Ordering constraint Do **not** configure a `RUN_OPS_SHARDS` descriptor in any environment until the routing-semantics change (TRI-13427) lands — three fan-out sites still truncate at N>2. Merging this PR alone is safe (inert with the var unset); configuring a descriptor is what must wait. ## Testing - Run-store corpus: green with zero test-file diffs (the bit-identical proof for the compat router). - `runOpsDbTopology.test.ts` 17/17, `runStore.server.test.ts` 4/4, `runOpsMigration` family 149/149. - New unit suites: descriptor validation, pool-knob value tables, `fromShards` routing + `UnknownShardKey`, boot-table formatter, mint bound. - typecheck (webapp + run-store), knip, lint, format: pass. ## Changelog Internal run-ops sharding infrastructure. No changeset or `.server-changes`: the change is inert with `RUN_OPS_SHARDS` unset and has no user-visible behaviour. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
DKP
|
ba57c1fc74 |
fix(webapp): disable browser autofill on environment variable inputs (#4777)
The environment variable key and value inputs did not set an autocomplete attribute, so browsers could offer to autofill or save typed values as saved credentials. This sets `autoComplete="off"` on those inputs in both the create and edit forms, matching the `autoComplete="off"` convention already used on the other credential-name inputs. `autoComplete="off"` is a best-effort hint. Browsers may still ignore it for password-typed fields, so this is defense-in-depth hardening, not a hard guarantee that a password manager cannot store the value. |
||
|
Saadi Myftija
|
6a6f0a4960 |
feat(webapp): pause deployment log auto-scroll on scroll-up (#4776)
Auto-scroll now only follows while you are at the bottom. Scrolling up pauses it; scrolling back to the bottom, or clicking the new scroll-to-bottom button in the log header, resumes it. When you are at the bottom the same button scrolls to the top. Switching to another deployment starts at the bottom again. |
||
|
Daniel Sutton
|
97d70b8906 |
feat(run-store): make the run-ops router correct at N shards (#4771)
## What
Makes `RoutingRunStore` correct when the run-ops layer routes across
more than two Postgres stores. Today it routes between a gen-1 `new`
dedicated database and a `legacy` control-plane database; this
generalizes every routing policy to N shards while keeping the two-store
behaviour byte-identical.
The change sets the four routing decisions that were implicit in code
order, and fixes one hazard that failed silently:
- **Id → shard key.** The router resolves a shard key with
`resolveShard` instead of the binary residency classifier, so a gen-2 id
reaches its own shard through the keyed map.
- **Membership vs routing.** `#distinctStores` (one entry per physical
database, aliases excluded by a declared `aliasOf`) drives every sum,
probe, and merge; `#shards` drives routing. An aliased shard can no
longer make a sum count one database twice.
- **Probe order.** A keyless lookup stays a sequential short-circuit at
two stores; above two it fans out in parallel, picks by precedence,
tolerates a single down leg, and keeps the canonical not-found throw on
the legacy leg.
- **Precedence and duplicates.** One merge helper across all four merge
sites. A duplicate id confined to `{new, legacy}` stays silent (the
known drain-mirror case); any other cross-shard duplicate increments
`runops_shard_duplicate_id_total` and logs at error level.
- **Disjoint sum (the silent hazard).** `countPendingWaitpoints` and the
waitpoint collector now partition absent ids by shard and **union by
id** rather than summing counts. A drain-mirrored waitpoint on both
gen-1 stores is counted once, so a blocked run can no longer hang
forever on a double-counted pending waitpoint.
- **Waitpoint completion.** A gen-2 waitpoint completes on its own
shard, overriding the legacy pins; a cuid waitpoint keeps its two-member
gen-1-pair probe unchanged.
- **Fail-loud creates.** A create with no shard key throws instead of
silently defaulting to `new`. An id resolving to an unconfigured shard
throws instead of being dropped.
Two new counters are exported: `runops_shard_duplicate_id_total` and
`runops_waitpoint_probe_fallback_total`.
## Why it is safe to merge
With only `{new, legacy}` configured every generalized rule reduces to
today's behaviour. `resolveShard` returns exactly what the old
classifier returned for every id shape that exists today, and no gen-2
id is minted yet. The only intentional behaviour change is the fail-loud
create throw; an enumeration of production call sites confirmed no
caller trips it.
## Testing
- New container-free algebra suite (50 cases) over probe order,
precedence, the duplicate alarm, the disjoint-sum partition, the
waitpoint probes, and the fail-loud paths.
- New `runOpsStore.nShardMatrix.test.ts` runs a four-store matrix
(legacy + new + two gen-2 shards) against real Postgres containers: the
disjoint-sum union, the alias topology, cross-tree completion,
pagination merges, and mixed-id hydration.
- New `makeNShardRunOpsPostgresTest(k)` fixture in
`@internal/testcontainers`.
- Full run-store corpus green: 71 files, 480 tests. Typecheck, lint,
format, and knip all clean.
## Notes
- Draft: opened for review; not marking ready yet.
- No changeset or `.server-changes` file: internal routing
infrastructure, no user-visible behaviour change.
- TRI-13427.
|
||
|
Saadi Myftija
|
ee29393862 |
perf(webapp): cache deployment logs across navigations (#4775)
Switching between deployments in the dashboard re-fetched the whole build log stream from record zero and re-rendered the list line by line every time. Logs are now cached per deployment for the lifetime of the tab: revisiting a deployment shows its logs immediately, and the stream is resumed from the next unread record rather than restarted. Finished deployments whose stream has been read through the `finalized` event are served entirely from the cache. ### Changes The stream/cache logic moved out of the route into a `useDeploymentLogs` hook. On each deployment switch it seeds state from the cache, resumes the S2 read session at `nextSeqNum`, and writes back on cleanup or natural session end. Completion is derived from the stream's own `finalized` event (plus a terminal deployment status), not from the session closing, so a session cut short by token expiry or a proxy cannot pin a truncated log in the cache. Memory is bounded by a small LRU (`deploymentLogsCache`): at most 20 deployments and 20,000 log lines in total, least recently viewed evicted first. The most recently viewed deployment is always kept, so a single very large log can temporarily exceed the line budget on its own. Records are batched into one state update per tick instead of one per line. |
||
|
Eric Allam
|
47ff76d727 |
feat(webapp,clickhouse): return an actionable error instead of a 500 when a runs list query is too expensive (#4773)
## Summary When a runs list query is too expensive to complete, it now fails with a clear, actionable error instead of a generic 500. Previously, a runs list query that exceeded ClickHouse resource limits threw an opaque error. On the public `runs.list` API that surfaced as a retryable 500, so a customer task calling it would keep retrying a query that could never succeed. On the dashboard it rendered as a generic error page with no hint about what to do. ## Fix The ClickHouse client now tags resource-limit failures (memory, time, rows, bytes) with their error type, and the runs repository maps those to a dedicated `RunsListQueryError` (HTTP 422). - `runs.list` API returns 422 with a message telling the user to narrow their `created_at` range, plus an `x-should-retry: false` header so the SDK does not retry it. - The dashboard runs list (and the errors, scheduled, standard-task, agents, and webhooks list views) render a shared error state with the same guidance, so a too-broad time filter is recoverable by the user. |
||
|
Saadi Myftija
|
1eda438a41 |
feat(webapp): put the admin dashboard behind an env var flag (#4774)
Adds an `ADMIN_DASHBOARD_ENABLED` env var (default: enabled) that turns the admin dashboard and user impersonation off for an entire instance. When disabled: - every admin dashboard page redirects away, and the admin navigation isn't rendered - existing impersonation cookies are ignored, and any lingering session is actively terminated with an audit record - every flow that could start an impersonation responds 404, and no impersonation tokens are minted Stopping an impersonation always works regardless of the flag, so nothing gets stuck. Machine-to-machine admin API endpoints are not affected. The variable is documented for self-hosters; instances that don't set it are unaffected. |
||
|
Daniel Sutton
|
45eaaa7bd7 |
feat(run-store,testcontainers): execution-snapshot read comparator and shared test utilities (#4772)
## Summary Adds the read comparator for the in-progress migration of the run execution-snapshot log from Postgres to Redis. The comparator samples a single read against both stores, normalizes the two results to one shape, and reports any per-field difference with a tagged metric. It never serves a read itself: the diff layer imports only types, so it cannot hold a store client, and a test enforces that by failing if any value import appears. Also adds a combined Postgres-and-Redis test fixture and two shared test utilities (a cluster-slot assertion and a generic fault-injection harness) that the parallel Redis-store work reuses. Everything here is inert. Nothing constructs the comparator, so merging changes no runtime behavior. It becomes active only when a later change turns on compare mode. ## Notes The divergence classes separate real differences (scalar, ordering, waitpoint id set, validity, missing on one side) from two expected classes that must not be driven to zero: a rotated idempotency key, and a Redis-only surplus at a since-cursor tie. The since comparison is direction sensitive: a Postgres-only entry at the cursor is always a lost write, never an expected tie. |
||
|
Oskar Otwinowski
|
036cf8d2c8 |
chore(webapp): admin endpoint to backfill Vercel deployment external ids (#4770)
Skew protection resolves a run's worker by (environmentId, externalId, status=DEPLOYED). A miss parks the run and then expires it, so deployments predating the feature — which already carry the same value in commitSHA — need externalId populated to stay reachable. Vercel instant-rollback is the sharpest case, which is why the scope is the current promotion plus a recent window rather than current alone. Follows the existing backfill shape: admin PAT, keyset cursor over environments, per-environment action results, pMap, dryRun defaulting to true. Reuses normalizeExternalDeploymentId so a backfilled id is byte-identical to what a build writes, and the update re-checks externalId IS NULL so a deploy landing mid-backfill keeps its own id. Refs TRI-13464. |
||
|
Eric Allam
|
11e1cd8174 |
feat(webapp): isolate the runs list ClickHouse read pool (#4763)
## Summary Improves the performance and reliability of the runs list and the `runs.list` API, especially for large projects and filtered views. ## What changed - **Filtered runs-list queries use `PREWHERE`.** Immutable and additive-only filters (tags, task identifier, version, queue, region, machine, and the rest) are applied in `PREWHERE` on the `task_runs_v2 FINAL` scan, so ClickHouse filters, and uses the tags skip index, before it reconciles versions and materialises the wide columns. Same results, far less memory per query. `status` stays in `WHERE`: it changes across a run's versions, so filtering it before `FINAL` could return stale rows. - **The runs-list ClickHouse pool gets per-query guardrails**, all env-configurable: a `max_execution_time` paired with the client request timeout, a per-query `max_memory_usage`, a `max_threads` cap, and `readonly`. Each bounds a single query to itself, so a heavy query can't affect other queries, and they are safe as pool-level settings only because this pool is read-only. - **Billing and bulk count reads move to the read pool**, off the write pool. Defaults are conservative for self-hosters; production values are set via env. |
||
|
nicktrn
|
2e87e93934 |
ci: run codeql on all prs via advanced setup (#4767)
Default setup doesn't run CodeQL on pull requests from forks, so external contributions are stuck on PR checks that never come. Advanced setup fixes this. Languages, categories and `main` coverage match the current default setup. The bare `pull_request` trigger (no `branches` filter) keeps stacked PRs scanned, whose base isn't `main`. Default setup has to be disabled in Settings -> Code security for these uploads to be accepted. Until it is, the CodeQL check here fails with `CodeQL analyses from advanced configurations cannot be processed when the default setup is enabled`. |
||
|
Saadi Myftija
|
f866210388 |
feat(cli): experimental --local-bundle deploy mode (#4331)
Adds an experimental `--local-bundle` flag to native build deployments: the project is installed and bundled on the local machine (exactly like in the depot path) and only the resulting build context is uploaded. The remote build then runs just the container image build. ### Design - The uploaded artifact is the same build context classic deploys produce: bundled output, a synthesized package.json with the resolved externals, build.json, and the generated Containerfile. The bundle is secret-free: build.json is deliberately scrubbed because it is copied into the image, and build-arg values never enter the bundle at all. - Build-arg values are sent with the deployment initialization request instead, stored encrypted (aes-256-gcm) in a new `WorkerDeployment.buildEnvVars` column, and cleared on every terminal status transition. They exist at rest only for the active build window, always encrypted. - A dedicated `GET /api/v1/deployments/:id/build-env-vars` endpoint returns the decrypted values to the same principals that can already read the environment's variables. It answers with an empty record for deployments without stored values or in a terminal state, keeping secret access to a single auditable route. - Size limits are enforced server side and pre-checked client side. If the server does not acknowledge storing the values, the CLI fails fast instead of letting the remote build run without them. - A `--from-bundle <dir>` mode builds a deployment image straight from such a bundle directory, skipping config loading and bundling entirely. In attach mode it fetches the stored build-arg values through the new endpoint. - Env var syncing (the `syncEnvVars` extension) happens client side, before the deployment initializes, since the remote side never sees the unscrubbed manifest. - Bundle artifacts use a distinct type and storage prefix so the server can always distinguish them from source uploads. |
||
|
Daniel Sutton
|
cc69ff4d26 |
feat(run-engine): Redis waitpoint store coordinator, Lua protocol, and waitpoint ids (#4761)
Builds the Redis-backed half of the waitpoint coordinator, beside the Postgres coordinator that #4753 extracted. Adds the coordination protocol as Lua scripts, the run-ops-format waitpoint id scheme, and the key layout. **No caller wires any of it up.** Refs TRI-13440. ## Inert by construction Merging this changes nothing observable. 3180 insertions, **zero deletions**, nine new or additively-edited files. - `WaitpointStoreCoordinator` is never constructed outside its own tests and the benchmark. - No env var, no config plumbing, no connection. It takes `redisOptions` as a constructor argument. - `waitpointSystem.ts` is untouched. Every live waitpoint operation still runs on Postgres through the coordinator merged in #4753. - No changeset and no `.server-changes` note — nothing here is user-facing yet. Deploying this needs no Redis or MemoryDB instance. That becomes a prerequisite when a later change routes traffic onto the store behind a per-organisation flag. ## What's here **Nine Lua scripts**, each atomic on one hash tag. Seven mutate state — create-if-absent, register-or-report, complete, idempotency reserve, absorb, deliver, clear. One reads state (`runReadBlockState`) and is separate because the pending, delivered and edge sets must be read as one consistent view. One discards an idempotency loser. **Two hash tags, deliberately.** `wp:{waitpointId}` holds a waitpoint's record, status, completion envelope and watcher hash. `wp:run:{runId}:*` holds one run's pending set, delivered set and edge set. A waitpoint has N watchers, so it cannot live under any single run's tag. **Waitpoint ids** reuse the run-ops body layout: a 24-char base32hex core, a type char (`r`/`b`/`d`/`m`), and version char `w`. RUN and BATCH ids derive from their anchor's core, so create-if-absent is idempotent with no lock. `parseWaitpointId` is total and never throws. **The single-slot guard.** Every script invocation goes through one private wrapper that asserts all keys share a hash tag. A single-node test server accepts what a real cluster rejects, so this assertion is the only enforcement — and it is mutation-tested: removing it fails a test. ## Measured Against the same population of real Postgres rows: | | store | postgres | |---|---|---| | pending count (the blocked/unblocked gate) | 0.13 ms p50 | 3.32 ms p50 | | full-payload read | 1.45 ms p50 | 7.70 ms p50 | Both are lower bounds: the benchmark charges Postgres a `COUNT(*)`, while the resume-time read is a join with a partial select plus filtering in JavaScript. Store-only paths, no Postgres counterpart: block+complete+deliver 0.88 ms p50; 100-watcher fan-out 13.8 ms; a 1001-edge fan-in 149.8 ms, flat at 0.15 ms per edge and round-trip bound rather than algorithmic. The benchmark lives in `*.bench.test.ts` and is excluded from the default suite. ## Review notes - **The type surfaces are not reconciled yet, on purpose.** `types.ts` (from #4753) carries the coordinator interface; `storeCoordinator.ts` declares its own operation types because this was built in parallel. The wiring change reconciles them. - **The read-time resolver is not here.** Another lane froze its contract while this was in flight, and its frozen types are not yet on main. Building a second copy would fork a just-frozen contract. - **Teardown is one-shard while registration is two-shard.** A terminal clear leaves a run registered as a watcher on the waitpoints it was blocked on, because the watcher hash is under a different tag and no script may span slots. Recorded, not fixed here — it needs a retention decision, and nothing observes it while the code is unwired. ## Verification 79 tests in the coordinator suite, 58 in the id suite. `typecheck` on run-engine and webapp, `build` on core, `knip`, `oxfmt` and `oxlint` all clean. The engine corpus passes 82/82. Every invariant is mutation-tested rather than merely asserted. A whole-branch review ran 14 mutants and killed 12; the two survivors were fixed with their own mutation checks. 🤖 Generated with [Claude Code](https://claude.com/claude-code) |
||
|
Daniel Sutton
|
f98e303292 |
feat(webapp): resolve which shard an environment mints run roots into (#4755)
## Summary Adds the shard-selection stage of run-id minting. `resolveMintShard(env)` returns which run-ops database an environment mints its new run roots into: the active shard list, then a fleet-wide override, then a per-environment or per-organization pin, then a rendezvous hash of the environment id. That half is inert. Nothing calls `resolveMintShard`, no deployment has any of the new flags set, and an empty active list returns the current answer without reading anything. **The other half is not inert, and it is where review effort belongs.** To stamp a grace window this needs a read-then-write under a lock, so it rewrites the global feature-flag write path that `runOpsMintKind` already depends on in production. See below. ## Placement Resolution reads the active list from a global flag, applies the grace window, and then picks: - a fleet-wide override if one is set, which is how a cutover completes without visiting each organization. `new` holds the whole fleet on the current id format. - otherwise a per-environment or per-organization pin. `new` holds one organization back while the rest move, which is how a canary works. - otherwise a rendezvous hash, so adding a shard moves only about 1/(N+1) of environments and removing one moves only its own. Two hash details are load-bearing. Scores are 64-bit `sha256(envId \0 key)`, because a 32-bit score collides at our environment count and an undetected tie would resolve by iteration order. The parsed key list is sorted, because otherwise two deployments listing the same shards in a different CSV order would place environments differently. A pin or override naming a shard that has left the active list falls through to the hash and reports once. Honouring it would leak the drain the active list exists to perform, and throwing would fail triggers whenever a pinned shard drains. ## Why the active list is a flag and not an environment variable A deploy rolls for hours, so two pods hold two different environment values at the same time. A list held in the environment therefore splits the fleet for the length of the rollout, with new pods placing an environment on one shard and old pods on another. A grace window measured in seconds cannot cover that, and the same knob times the existing mint-kind flip so it cannot simply be lengthened. An environment variable also cannot record its own flip time, and an operator cannot know a rollout's end in advance. So the list, its grace stamp and the override are global flags, written server-side against the control-plane clock under an advisory lock. This branch adds no environment variables. ## The write path, which is live Stamping generalises to any number of graced flag groups in one transaction under one lock. That has three consequences a reviewer should look at directly: - It closes a real bug. `runOpsMintKind` is an editable control on the global flags page, and that page previously wrote it with a bare upsert: no lock, no stamp. An operator flipping mint kind through the UI got an ungraced flip, so every pod crossed the cutover at a different moment. Verified against a running instance, before and after. - A graced group is all-or-nothing. Submitting its primary writes the group with a fresh stamp; omitting it deletes the primary and its stamp together, because a stamp left without its primary keeps being served and would mint into a shard just removed. - The advisory lock takes the previous id as well as the current one, in a fixed order, so writers on an older release still serialise during a rollout. The legacy id can be dropped one release after this ships. This folds with #4751 rather than replacing it: its `unlockLockedFlags` rule decides what the sweep may delete, and the graced groups keep their stamp under the lock. Both sets of tests pass. ## Notes for review Determinism is a property of the pure core for fixed inputs. The wrapper supplies the clock, the same split `effectiveMintKind` already uses. A failed read of the list falls back to the current id format rather than guessing. Six flags appear in the admin pages immediately. The two pins are per-organization, so they render read-only on the global page. The list, its stamp and the override are deployment-wide, so they render read-only in the organization dialog. Nothing bounds the active list against shards that actually exist. That is safe while nothing mints, but the change that carries a shard key into an id must land after the shard descriptors bound the list, or bound it itself. |