github-featured/triggerdotdev--trigger.dev
2
0
Fork 0
Code Issues Pull Requests Deployments Models Agent Notes Packages Projects Releases Wiki Activity

fix(sdk): re-dispatch a single in-flight user on recovery boot (#4768)

Browse Source 
<!-- ccr-slack-attribution -->
_Requested by **Matt Aitken** · [Slack
thread](https://triggerdotdev.slack.com/archives/C061L2MHW93/p1787615162456839?thread_ts=1787615162.456839&cid=C061L2MHW93)_

**Before:** a `chat.agent` run is killed mid-answer (OOM, crash,
eviction) while the message it was answering is the only one still
outstanding. The new run boots, puts that message and the half-written
reply into its context, and then waits for a message that already
arrived. Nobody ever answers the user; the run sits idle until it times
out.

**After:** the new run re-runs that message as a fresh turn and replies
to it. The half-written reply is dropped. When two or more messages are
outstanding, nothing changes — the interrupted one still goes into
context and the newer ones are re-run, exactly as before.

## ✅ Checklist

- [x] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [x] The PR title follows the convention.
- [x] I ran and tested the code works

---

## Testing

New regression test in `packages/trigger-sdk/test/recovery-boot.test.ts`
— seeds a partial assistant plus exactly one in-flight user, no
`onRecoveryBoot`, and asserts one turn fires for that user with the
orphan partial dropped from the chain. It fails on `main` (`turnCount`
0, no turn at all) and passes with this change.

- `pnpm exec vitest run` in `packages/trigger-sdk` — 373 passed, 1
skipped (31 files passed, 1 skipped)
- `pnpm exec oxfmt --check` on the changed files — clean
- `pnpm exec oxlint packages/trigger-sdk/src packages/trigger-sdk/test`
— clean
- `pnpm run build --filter @trigger.dev/sdk` — clean

**What it does:** with exactly one in-flight user on a recovery boot,
re-dispatch that user as a fresh turn instead of splicing it into the
seed chain, where it was never answered.

**How:** the recovery-boot smart default made one decision in two halves
— the seed chain and the recovered-turn list — both gated on
`partialAssistant !== undefined && inFlightUsers.length > 0`. The splice
consumes `inFlightUsers[0]` into the chain as "the question the partial
was answering" and dispatches the rest. That only works when there *is*
a rest: at n=1 `recoveredTurns` came out empty, the boot-injected queue
stayed empty, the `session.in` cursor was advanced past the message
anyway, and on a `preload` or continuation boot (no `message` on the
wire payload) neither dispatch site fired. Both branches now require
`length > 1`, so n=1 falls through to the documented default — chain =
`settledMessages`, re-dispatch every in-flight user. The submit-message
boot is unaffected: the existing dedup still drops a queued message
identical to the one already on the wire payload.

Also corrected alongside it: the two SDK docstrings and the
`docs/ai-chat/patterns/recovery-boot.mdx` defaults section, which
described the default as "re-dispatch every user" and never mentioned
the splice.

Follow-up (not in this PR): the webapp e2e OOM helper never streams a
token before throwing, so it exercises the no-partial path only and
would not have caught this. Worth a variant that emits a token first.

---

## Changelog

Fixed a chat agent hanging after an interrupted turn: when a run was
killed mid-answer and only the one message it was answering was still
outstanding, the new run never replied to it. That message is now
re-answered on the new run.

Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Matt Aitken <matt@mattaitken.com>
Co-authored-by: Eric Allam <eric@trigger.dev>
This commit is contained in:
claude[bot]
2026-08-27 17:34:54 +01:00
committed by GitHub
parent 15dd973f92
commit a3af29fd80
4 changed files with 93 additions and 12 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.changeset/brave-otters-recover.md
+5
View File
@@ -0,0 +1,5 @@
---
"@trigger.dev/sdk": patch
---
Fixed a chat agent hanging after an interrupted turn: when a run was killed mid-answer (out of memory, crash, or eviction) and only the one message it was answering was still outstanding, the new run never replied to it. That message is now re-answered on the new run.
docs/ai-chat/patterns/recovery-boot.mdx
+9 -2
View File
@@ -32,7 +32,7 @@ On a continuation boot, the runtime reads:
- **`session.out` tail past the snapshot cursor** — closed assistant turns plus, optionally, a `partialAssistant` (the trailing message whose stream never received a `finish` chunk). `cleanupAbortedParts` has already stripped streaming-in-progress fragments.
- **`session.in` tail past the last `turn-complete` cursor** — user messages the dead run hadn't acknowledged.
If both `partialAssistant` and `inFlightUsers` are non-empty, the runtime splices `[firstInFlightUser, partialAssistant]` onto the chain. The remaining in-flight users dispatch as fresh turns. The model sees:
If there's a `partialAssistant` and two or more `inFlightUsers`, the runtime splices `[firstInFlightUser, partialAssistant]` onto the chain. The remaining in-flight users dispatch as fresh turns. The model sees:
```
[ ...settledMessages, // chain through the last completed turn
@@ -138,10 +138,17 @@ type RecoveryBootResult<TUIM extends UIMessage = UIMessage> = {
};
```
- **`chain`** — replaces the seed chain. Defaults to `[...settledMessages, firstInFlightUser, partialAssistant]` when both partial and in-flight users exist, otherwise `settledMessages` alone.
- **`chain`** — replaces the seed chain. Defaults to `[...settledMessages, firstInFlightUser, partialAssistant]` when there's a partial **and two or more** in-flight users, otherwise `settledMessages` alone.
- **`recoveredTurns`** — user messages to dispatch as fresh turns after the chain is restored. Defaults to `inFlightUsers.slice(1)` when the smart default consumed the first user, otherwise `inFlightUsers`.
- **`beforeBoot`** — runs after the writer flushes and before the first recovered turn fires. Use for blocking persistence (write the partial to your DB so a later turn can reference it). Errors bubble — wrap your own try/catch if you want to soft-fail.
<Note>
The splice needs a follow-up user to answer, so it only applies with two or
more in-flight users. With exactly one — the plain OOM or crash-mid-answer
case — the orphan partial is dropped and that single user is re-dispatched as
a fresh turn, so the interrupted question still gets answered.
</Note>
## Examples
### Drop the partial — strict "cancel means discard"
packages/trigger-sdk/src/v3/ai.ts
+31 -10
View File
@@ -4554,8 +4554,11 @@ export type RecoveryBootEvent<TUIM extends UIMessage = UIMessage> = {
/**
* User messages that arrived on `session.in` past the cursor — i.e.
* the message(s) the predecessor was processing or had queued when
* it died. The runtime's default is to re-dispatch each as a fresh
* turn after the chain is restored. Return a different list via
* it died. The runtime's default re-dispatches each as a fresh turn
* after the chain is restored, except when a `partialAssistant` is
* present AND there are two or more of them: the first is then
* spliced into the chain (as the question the partial was answering)
* rather than dispatched. Return a different list via
* `recoveredTurns` to skip / reorder / collapse them.
*/
inFlightUsers: TUIM[];
@@ -4600,8 +4603,12 @@ export type RecoveryBootResult<TUIM extends UIMessage = UIMessage> = {
chain?: TUIM[];
/**
* The user messages to re-dispatch as fresh turns after the chain is
* restored. Default: `inFlightUsers` (re-process every in-flight
* user). Return `[]` to suppress all of them; return a filtered /
* restored. Default: `inFlightUsers.slice(1)` when a
* `partialAssistant` is present and there are two or more in-flight
* users (the first one is spliced into the chain instead), otherwise
* `inFlightUsers` — including the single-user case, where the
* interrupted user is re-dispatched and the orphan partial is
* dropped. Return `[]` to suppress all of them; return a filtered /
* reordered subset to skip specific ones.
*/
recoveredTurns?: TUIM[];
@@ -5168,8 +5175,13 @@ export type ChatAgentOptions<
* customer's DB.
*
* Defaults (returned when the hook is omitted or returns no field):
* - `chain` = `settledMessages` (drop the orphan partial)
* - `recoveredTurns` = `inFlightUsers` (re-dispatch every user)
* - With two or more in-flight users, the partial and the user it
* was answering are spliced into the chain:
* `chain` = `[...settledMessages, inFlightUsers[0], partialAssistant]`
* and `recoveredTurns` = `inFlightUsers.slice(1)`.
* - Otherwise `chain` = `settledMessages` (drop the orphan partial)
* and `recoveredTurns` = `inFlightUsers` (re-dispatch every user)
* — so a single interrupted user is answered on the new run.
*
* @example
* ```ts
@@ -6111,20 +6123,29 @@ function chatAgent<
}
}
// Default: splice partial + the user it was answering into
// the chain so follow-ups like "keep going" still have context.
// Default: splice partial + the user it was answering into the chain
// so follow-ups like "keep going" still have context, and re-dispatch
// the users that arrived after it.
//
// The splice needs a follow-up user to answer — it consumes
// `inFlightUsers[0]` into the chain instead of dispatching it. With
// exactly ONE in-flight user (the plain OOM / crash-mid-answer case)
// there is nothing left to dispatch, so splicing would strand that
// user unanswered and idle the run. Require `length > 1` on both
// branches: at n=1 the orphan partial is dropped and the interrupted
// user is re-dispatched as a fresh turn instead.
let seedChain: TUIMessage[];
let recoveredTurns: TUIMessage[];
if (hookChain !== undefined) {
seedChain = hookChain;
} else if (partialAssistant !== undefined && inFlightUsers.length > 0) {
} else if (partialAssistant !== undefined && inFlightUsers.length > 1) {
seedChain = [...settledMessages, inFlightUsers[0]!, partialAssistant];
} else {
seedChain = settledMessages;
}
if (hookRecoveredTurns !== undefined) {
recoveredTurns = hookRecoveredTurns;
} else if (partialAssistant !== undefined && inFlightUsers.length > 0) {
} else if (partialAssistant !== undefined && inFlightUsers.length > 1) {
recoveredTurns = inFlightUsers.slice(1);
} else {
recoveredTurns = inFlightUsers;
packages/trigger-sdk/test/recovery-boot.test.ts
+48
View File
@@ -295,6 +295,54 @@ describe("onRecoveryBoot — chat.agent recovery hook", () => {
}
});
it("smart default: a single in-flight user is re-dispatched, not swallowed by the splice", async () => {
// The plain OOM / crash-mid-answer shape: the run died while answering
// the only outstanding user message. Splicing that user into the chain
// would leave nothing to dispatch, so the run would boot and idle with
// the message unanswered. The default must re-dispatch it instead (and
// drop the orphan partial).
let observedChain: Array<{ role: string; idHead: string }> = [];
let turnCount = 0;
const model = new MockLanguageModelV3({
doStream: async () => {
turnCount++;
return { stream: textStream("ok") };
},
});
const partial = assistantMessage("partial answer in progress", "a-partial");
const u1 = userMessage("the question that OOM'd", "u-1");
const agent = chat.agent({
id: "recovery-boot.single-inflight-user",
// NO onRecoveryBoot — exercise the default path
onTurnStart: async ({ uiMessages }) => {
if (turnCount === 0) {
observedChain = uiMessages.map((m) => ({
role: m.role,
idHead: m.id.slice(0, 10),
}));
}
},
run: async ({ messages, signal }) => streamText({ model, messages, abortSignal: signal }),
});
const harness = mockChatAgent(agent, {
chatId: "single-inflight-user",
continuation: true,
previousRunId: "run_prior",
});
harness.seedSessionOutPartial(partial as never);
harness.seedSessionInTail([u1 as never]);
try {
await new Promise((r) => setTimeout(r, 100));
// One turn fires, for the interrupted user.
expect(turnCount).toBe(1);
// The orphan partial is dropped — the chain is just the re-dispatched user.
expect(observedChain.map((m) => m.role)).toEqual(["user"]);
expect(observedChain[0]!.idHead).toBe("u-1");
} finally {
await harness.close();
}
});
it("hook's recoveredTurns: [] suppresses re-dispatch of in-flight users", async () => {
let turnCount = 0;
const model = new MockLanguageModelV3({