A turn stores its messages before the model finishes, so the completed bodies arrived against ids that already existed and were skipped. Reopening a chat then replayed a tool call that never ends.
The system — contracts, storage, auth, the agent package and its webapp routes — lands first; the panel, the page-context marks and the entry points follow in their own PR.
📦 Preview packages (pkg.pr.new) / Build and publish previews (push) Has been cancelled
📚 Publish docs / publish (push) Has been cancelled
Implementing PlanetScale Insights improvement.
## Summary
Validating a schedule (creating or updating one through the API or the
dashboard, and deploying a project that declares schedules) looks up the
newest version of a task by slug. That lookup reads *every* version of
the task and sorts them to return one. A project gains a row per task on
every deploy, so the work grows with the project's age: the oldest
projects pay the most, and dev-mode redeploys make it worse. This was
picked because it was the largest single consumer of database time on
the schedules path, and the fix is a sort key with no index behind it.
## Fix
`BackgroundWorkerTask` is indexed on `(projectId, slug)`, which serves
the equality but not the `ORDER BY createdAt DESC`. Postgres seeks the
index, then bitmap-scans and top-N sorts the whole group to produce a
single row. Adding `createdAt` to the index lets it scan backward and
stop at the first row.
The same call site also selected all 21 columns, including five JSON
blobs, to read one field (`triggerSource`), so it now selects that field
alone.
## Benchmark
Local Postgres 17, 997,000 seeded rows / 748 MB, group sizes chosen to
match the distribution seen in production.
| Group size | Before | After |
| --- | --- | --- |
| 15,000 versions of one task | 11.118 ms, 1,510 buffers, 15,000 rows
scanned | 0.027 ms, 4 buffers, 1 row |
| 2,000 versions of one task | 2.081 ms, 1,455 buffers, 2,000 rows
scanned | 0.022 ms, 4 buffers, 1 row |
```
before: Limit -> Sort (top-N heapsort) -> Bitmap Heap Scan
after: Limit -> Index Scan Backward using BackgroundWorkerTask_projectId_slug_createdAt_idx
```
An ascending index scanned backward is enough here, so no descending
index is needed.
## Impact and risk
Real-world gain lands between the two rows above and scales with how
many deploys a project has accumulated. Projects with few deploys will
see little change, since there is barely anything to sort.
The new index costs noticeably more than the existing two-column one: 43
MB against 7.3 MB on the benchmark rig. Adding `createdAt` makes every
key unique, which defeats btree deduplication, so this is a real disk
and write cost rather than a rounding error. Writes to this table happen
at deploy time, not on the run path, so the write amplification is
acceptable. The existing `(projectId, slug)` index is now a redundant
prefix and could be dropped, but this PR keeps it so index usage can be
observed before removing it.
Behavior is unchanged: same predicate, same ordering, same row returned.
The narrowed select is the only code change, and the field it keeps is
the only one the caller read.
Deploy note: the migration is
`20260806100000_add_background_worker_task_project_id_slug_created_at_index`
and uses `CREATE INDEX CONCURRENTLY IF NOT EXISTS`, so it can be
pre-applied by hand before the deploy.
## Summary
`syncDeclarativeSchedules` runs on every background-worker creation
(every deploy, and every file save during `trigger dev`). It issued one
instance-delete per declarative schedule the current worker no longer
declares, in a loop, and the overwhelming majority of those deletes
matched zero rows. This collapses the loop into at most two set-based
statements and skips the instance delete entirely when the current
environment owns no instance of the schedule.
## Why so many, and mostly no-op
The loop runs once per entry in `missingSchedules`, which starts as
every DECLARATIVE schedule for the whole project across all its
environments (the query filters only by `projectId`). A schedule leaves
that set only when a declared task matches it by `taskIdentifier`
**and** the schedule already has an instance in the current environment.
That last clause is the amplifier. When a task's schedule has no
instance in the current environment, the create branch inserts a
brand-new `TaskSchedule` row with an instance for this environment
rather than adding an instance to the existing row. So the same
scheduled task, once it has run in dev and been deployed to prod, exists
as two separate schedule rows: one carrying a dev instance, one carrying
a prod instance.
On a dev worker sync of that project:
- the dev-instance row matches the declared task and is removed from the
set
- the prod-instance row has the same `taskIdentifier` but no dev
instance, so it stays in the set and gets `deleteMany(taskScheduleId =
prodRow, environmentId = dev)`, which matches zero rows
So every declarative task that has been synced in another environment
contributes one guaranteed no-op delete per sync, and the count scales
with (declarative tasks x environments), plus any leftover rows from
renamed or removed tasks. A project does not need to have dropped a
schedule to generate these; it just needs the same declarative tasks
present in more than one environment, which is the normal
develop-in-dev, deploy-to-prod case.
## Fix
The candidate schedules are already loaded with their instances, so the
branch is decided in memory:
- schedules with no instances (or only current-environment instances)
are removed in a single `taskSchedule.deleteMany`
- schedules that still have another environment's instance have only the
current environment's instance detached, in a single
`taskScheduleInstance.deleteMany`, and only when such an instance
actually exists
Behavior is unchanged (cascade delete still removes the instances of a
deleted schedule); the difference is statement count. A zero-row delete
writes no WAL and creates no dead tuples, so the removed work was pure
query and commit overhead.
Verified with a testcontainer test (red before, green after) counting
the emitted deletes across the no-op, batched-detach, and
schedule-delete cases, and end to end through `trigger dev`: three
declarative schedules created, surviving a re-sync, then two removed in
a single batched delete with the third preserved.
Debouncing with a `delay` longer than an hour did nothing at all.
The engine applied a server-side ceiling on how long a debounced run
could be pushed back, measured from the run's `createdAt` and defaulting
to one hour. A run is only pushed back while its new execution time
stays inside that ceiling, so a `delay` at or above it could never push
anything: the waiting run was released, the trigger started its own run,
and the next trigger repeated it. A `delay: "12h"` produced one run per
trigger, each correctly delayed by 12h, with no error raised and nothing
on the run to show the debounce key had been ignored.
The ceiling is now unset by default. A debounce key with no `maxDelay`
keeps collapsing triggers for as long as they keep arriving, which is
what the docs have always described. Self-hosters who want a bound can
still set `RUN_ENGINE_MAXIMUM_DEBOUNCE_DURATION_MS`.
That has a consequence worth stating plainly, so the docs now carry a
warning for it: with no `maxDelay`, a continuously triggered key never
executes. Set `maxDelay` when the work has to happen eventually.
**Failing fast on an unusable `maxDelay`.** A caller who sets `maxDelay`
no longer than their `delay` hits exactly the dead end described above,
so that pair is now rejected at trigger time instead of silently
behaving as if no debounce were set:
```
debounce.maxDelay (1h) must be longer than debounce.delay (12h). A debounced run is only
pushed back while it stays inside maxDelay, so with these values every trigger would create
its own run.
```
An unparseable `maxDelay` is rejected too, rather than quietly falling
back to no bound at all, and so is a `delay` given as a date rather than
a duration, which could never work because the value is re-applied on
every push.
The same check runs against a configured server ceiling, so a
self-hosted deployment that sets
`RUN_ENGINE_MAXIMUM_DEBOUNCE_DURATION_MS` gets the error rather than the
silent failure this PR is about. With no `maxDelay` and no configured
ceiling, which is the default, there is nothing to conflict with and
nothing is rejected.
The docs, the `TriggerOptions` JSDoc and the engine option all now state
that the room available to push is the gap between `delay` and
`maxDelay`. The run engine suite gains the case that motivated this:
four triggers on one key with a 12h delay now collapse to a single run.
The agent ships Chat and Investigate here; Watch — telling the user later —
follows in its own PR. The whole user-facing feature leaves: the watch card,
chips, wake banner and toast, the unread-wake badge and its poll, the watch
routes, checks, batches and sweeps, the watch alert email and its channel type,
and the watchMaintenance cron.
The agent no longer promises it either: schedule_watch and the alert tools are
gone from the tool set, and the Watches section is out of the system prompt.
Leaving that text in would have had the agent refuse to poll for something it
could no longer offer.
The datastore's watch tables stay. The migrations ship in this PR, so the drizzle
schema that describes them has to ship too — a schema that no longer matched the
migrated tables would make the next generate emit a drop.
The datastore's 21 migrations recorded the development history of a feature
that has never shipped, including a column added in 0019 and dropped in 0020.
Regenerated from the schema with drizzle-kit, so the single file is exactly
what `src/schema.ts` declares.
Replaying from an empty database gives the same 8 tables, 118 columns, 26
indexes and 9 constraints as the 21 files did; only the physical column order
differs, because a column added by ALTER lands at the end.
The malformed-message error carried 200 characters of the payload, which can be user text or tool output. It now names the shape only. A finalisation also verifies that the body's id is the row it targets, so the stored key and the payload cannot name different messages. The legacy-column guard missed a schema-qualified update, and now self-tests both spellings.
appendChatMessageOnce already verifies organizationId, but left it optional
because these call sites never threaded one — so the wake and the consented
investigation wrote durable user-facing messages with the organization check
skipped. Both paths now pass it on every append, the retry repair included, and
the chat id and the organization have to agree for a row to land.
The transcript moved out of `chats.messages` into `chat_messages`. TypeScript
already rejects a reference through the Drizzle schema, but a raw-SQL reference
compiles fine and only fails at runtime, and the earlier guard test went away
with the column. Zero hits today is the point: the test exists so a
reintroduction is caught rather than deployed.
`storeChatMessages` ended in `onConflictDoUpdate`, so `persistMessages` and
`persistTurn` — which are handed a whole snapshot — treated any differing body
under an existing message id as a deliberate finalisation. A stale snapshot
carrying `wake:watch_1:fired`, the watch consent record, the deterministic
confirmation or an investigation settlement card with a different body would
overwrite the durable row that was already recorded. The proxy caps body size
and metadata but does not rewrite message ids, so this was not an
internal-bug-only exposure. The same clause updated only the `message` JSONB and
never the `role` column, so `chat_messages.role` could end up disagreeing with
`message.role` — and the UI reads one while the quota query reads the other.
Ordinary transcript writes are now insert-only. Changing a stored message is its
own operation, `finalizeChatMessage`, guarded on chat id, message id and role.
`role` is verified rather than updated, and verified on both sides: the stored
column must match `expectedRole` and so must the incoming body's own `role`, so
the two cannot drift. A finalisation that matches nothing returns false; one
whose body contradicts `expectedRole` throws.
No production caller depended on the implicit finalisation. Every existing
finalisation-shaped path already writes through an insert-only append:
`settleInvestigationAndCloseCard`, `settleInvestigationStateAndCloseCard` and
the watch request/confirmation/refusal records all use
`appendChatMessageOnce(ByChatId)`.
Also: re-sending a snapshot no longer reserves positions for messages that are
already stored. The chat row is held, the missing ids are read under that lock,
and only those get slots. A 40-message chat grown one turn at a time used to
burn 1+2+…+40 = 820 slots for its 40 rows; it now burns 40. Deltas would be the
proper fix, but that reaches into the agent's turn hooks and is a larger change
than this pass.
Two smaller repairs in the same file: `messageIdOf`/`messageRoleOf` now fail
fast and name the chat and the offending message instead of casting unchecked
and surfacing a `NOT NULL` violation from the driver; and a batch carrying the
same message id twice throws instead of silently keeping the first, since that
is an impossible state and a silent pick is how the upstream bug would stay
invisible.
The comment on `reserveMessagePositions` claiming the row lock is "released with
the statement" was wrong — Postgres holds it to commit — and now says what is
true.
The wake and the consented investigation both stream their message before they
append the display copy, and the streamed copy is durable on session.out from
that moment. An append that failed therefore left the retry booting with the
message already in its history, taking the dedupe branch and never writing the
row: the model saw the message, the History panel didn't.
Both dedupe branches now re-append the message they found. The append is
id-deduped, so repairing when nothing is broken writes nothing.
Passing `debounce` in the per-item options of a batch trigger did
nothing when the items were an array. The option was accepted by the
types and by the API, then dropped before the request went out, so every
item created its own run instead of collapsing onto the debounce key.
Four public entry points were affected: `task.batchTrigger`,
`task.batchTriggerAndWait`, `tasks.batchTrigger`, and
`tasks.batchTriggerAndWait`. The streaming (async iterable) forms of the
same calls were already correct, as were `batch.trigger`,
`batch.triggerAndWait`, `batch.triggerByTask`, and
`batch.triggerByTaskAndWait`.
`useTaskTrigger` in `@trigger.dev/react-hooks` had the same silent drop
on the single-trigger path, so that is fixed here too. It also drops
`machine`, `priority`, `region`, `idempotencyKeyTTL`, and
`idempotencyKeyOptions`; those are left alone, since forwarding them is
a behaviour change beyond this bug.
Each batch item builder constructs its options field by field, which is
why one of them could fall behind without anything catching it.
TypeScript did not help: the literal is returned from a `.map` callback
inside `Promise.all`, so excess-property checking never fired against
the `BatchItemNDJSON[]` annotation, and the server's schema silently
strips unknown keys. A misspelled option name therefore reproduced this
bug with no compile error and no server error. Every builder now ends in
`satisfies BatchItemNDJSON`, which does catch it:
```
error TS2561: Object literal may only specify known properties, but 'debounceTYPO'
does not exist in type '{ ... debounce?: {...} | undefined; }'.
Did you mean to write 'debounce'?
```
The new test drives all six public batch surfaces in both array and
async-iterable form and asserts on the NDJSON that actually reaches the
wire. Each item carries a distinct debounce key so the test catches a
wrong item-to-option pairing, not just a wholesale drop.
Fixes#3304
## Summary
Adds a docs page for developers who already have a working Vercel AI SDK
chat app (`useChat` on the client, an `app/api/chat/route.ts` calling
`streamText`) and want to move it to `chat.agent`. There was no page
covering that path. `ai-chat/upgrade-guide` reads like it should be the
one, but it covers moving prerelease `chat.agent` code to the Sessions
release, which is a different reader.
The page is structured around what stays, what goes, and what is new,
because the reassuring part of this migration is how much is untouched:
the `streamText` call, model config, tool definitions, `useChat`, and
all message rendering carry over as-is. What gets deleted is the route
handler, the persistence glue wired into it, and any resumable-stream
setup. What is new is the agent task, two server actions, and
`useTriggerChatTransport`.
Covers moving tools onto the agent config so `toModelOutput` survives
past turn one, where existing database persistence goes
(`hydrateMessages` plus the turn hooks), a short section on what
durability you get once you are across, a note that
Hono/SvelteKit/Express follow the same shape, and a gotchas list built
from the mistakes this specific migration produces.
## Head Start
The one thing this migration makes worse is the opening response of a
new chat. The route handler answered out of a warm process; the agent
run has to be dequeued and booted first. That is the complaint the page
has to answer head on, so Head Start gets a full section rather than a
closing aside, plus a callout up top next to the "what changes" table so
nobody plans the migration without knowing it exists.
The section walks the four steps: splitting tool schemas away from tool
executes (the bundle-isolation constraint the whole feature rests on),
building the handler, mounting it back at `app/api/chat/route.ts` with
the original auth check wrapped around it, and the transport option.
Both server actions stay, because Head Start only owns the first turn.
Three gotchas go with it: a slow first turn without Head Start, Head
Start on but the route bundle still heavy, and the route timing out
because the handler holds the SSE response open for the whole turn
rather than just step 1.
The coding-agent prompt names Head Start as explicitly out of scope, so
an agent handed the migration does not attempt the tool split
unprompted.
Also fixes the `chat.headStart` example on `ai-chat/fast-starts`, which
set `stopWhen: stepCountIs(15)` after the spread.
`toStreamTextOptions()` pins `stopWhen` to `stepCountIs(1)`, so
overriding it makes the warm handler run steps the agent is supposed to
own (and `stepCountIs` was never imported in that snippet either).
## Migration prompt
The page also ships a copy-pasteable prompt for handing the migration to
a coding agent. It tells the agent to run `npx trigger.dev@latest
skills` first, so it picks up guidance version-pinned to the SDK
actually installed in the project, then read `quick-start.md`,
`frontend.md`, and `reference.md` (with `llms.txt` as the index) before
editing anything. The instructions are explicit about preserving the
existing model, prompt, and tool schemas rather than rewriting them.
Registered in `docs.json` under Agents, directly after Quick Start, so
it is picked up by the generated `llms.txt` and the per-page `.md`
variants.
## What
Rate-limit the API by **environment** rather than per API key.
Previously the limiter keyed its bucket on the hash of the full
`Authorization` header — one bucket per key. With additional environment
API keys (`tr_*_sk_*`), an environment can mint many keys and each got
its own full bucket, so more keys = higher effective rate limit. This
collapses all of an environment's keys onto a single shared
per-environment bucket, so the ceiling is exactly the configured limit
regardless of key mix.
## How
- `authorizationRateLimitMiddleware` now lets the override return `{
config?, identifier? }`. `identifier`, when present, is the rate limit
bucket key; otherwise it falls back to the hashed `Authorization` header
(unchanged legacy behavior, still used by `engineRateLimiter` and any
unauthenticated fallthrough).
- `apiRateLimiter`'s override resolves the environment id and uses it as
the identifier:
- **Additional keys** (`isAdditionalApiKey`) resolve via a new
`resolveAdditionalApiKeyRateLimitScope()` — a **scope-agnostic** keyHash
→ (environmentId, org limiter config) lookup. It is deliberately
permissive (restricted keys resolve too) because it's used **only for
bucketing, never as an auth decision** — request auth still goes through
the RBAC bearer controller, which enforces scopes. Revoked/expired keys
are excluded so they can't hold a bucket warm.
- **Root/legacy keys** reuse the environment already resolved by
`authenticateAuthorizationHeader` and key on `environment.id` too.
- The identifier is always the stable environment id, never the secret
key (which can rotate and would split the bucket).
- The whole override result is cached per key by the existing SWR cache,
so **no extra per-request lookup and no separate Redis mapping** is
added.
## Behavior notes
- Root + additional keys of the same environment now share one bucket
(ceiling = configured limit, not a multiple of it). Restricted
additional keys are included — they were the biggest gap, since they
authenticate via the RBAC controller and previously fell back to per-key
buckets.
- **Public JWTs** keep their existing fixed-window, per-token bucketing.
- One-time bucket reset on deploy (bucket keys change); harmless.
## Tests
- New: two tokens resolving to the same identifier share one bucket.
- New: with no identifier, bucketing stays per-key (legacy behavior
preserved).
- Updated existing override tests to the new `{ config }` return shape.
Base: `feat/multi-keys-surface`. Closes TRI-12888.
The concurrency test only exercised the single-message append, which
allocates its position inside one statement. Replacing the batch
allocator's atomic `next_message_position` bump with a read-then-write
left every test passing — so the invariant that concurrent writers get
disjoint ranges was not actually covered on the path a turn takes.
Four concurrent three-message batches now assert twelve distinct
positions and that each batch's own messages stayed contiguous and in
order. Against the read-then-write version this fails on
`chat_messages_chat_position_key`, which is the constraint doing the
work rather than the application code.
The feature is unreleased and this branch is local, so there is no rolling
deploy in which an old instance still selects `chats.messages` — the usual
reason to keep a replaced column for one more release does not apply, and
after release the drop would cost two deploys. So it goes now.
With no data to preserve there is no backfill, and with no backfill there
is no such thing as a stored message without an id: `message_id` is
`NOT NULL` and the writers read `message.id` with no fallback, so a
malformed message fails the insert instead of being given an invented
identity by untested code.
Removing the column from the drizzle schema is also the enforcement that
nothing writes it — a surviving TypeScript reference stops compiling. The
scan for raw SQL naming the column found nothing to guard, so there is no
test for it.
The four idempotency invariants each get their own test against a real
table: a repeated message id creates no row and keeps its position; eight
genuinely concurrent appends take eight distinct positions, with a raw
insert past the query layer proving it is `UNIQUE (chat_id, position)`
doing the work; a controlled update changes one body and leaves identity,
position and every other row alone; and a mid-turn append survives the
turn's write, landing where it happened rather than at the end.
`dashboardAgentTranscriptMerge.test.ts` becomes
`dashboardAgentTranscriptStore.test.ts`: it no longer tests a merge, and
the ordering it asserted has deliberately changed.
The whole transcript lived in one JSONB array on `chats` and every turn
rewrote it, so bytes written grew as N squared: a 40-message chat wrote
~3.2 MB plus comparable WAL for one appended message. Dedupe was a
`jsonb_array_elements` scan, the quota count scanned every chat the user
owns, and a wholesale snapshot write could delete a message another
process had appended — patched until now by reading the row `FOR UPDATE`
and merging in JS.
`chat_messages` replaces it: primary key `(chat_id, message_id)`, so a
redelivered message is a conflict rather than a duplicate, and
`UNIQUE (chat_id, position)`, so a lost or duplicated position is
impossible rather than unlikely.
Positions come from `chats.next_message_position`, bumped by the same
single statement that reads it. Concurrent writers therefore get disjoint
contiguous ranges, and the row lock is released with the statement instead
of being held across a JS merge — the lock was never the problem, reading
and rewriting a large blob under it was. A batch reserves one contiguous
range and keeps its incoming order; a message already stored keeps the
position it was first given, which is why a wake appended mid-turn now
sits before the turn's later messages rather than after them.
`role` is lifted out of the payload so `countUserMessages` is a `count(*)`
over an index instead of an array explosion. No counter column on `chats`:
that would be one more thing to keep in sync across insert, backfill and
controlled update, for no proven need.
The writers now split by intent. A new message is an INSERT. A repeat of
the same durable event — `appendChatMessageOnce`, the sweep's closing card
— is ON CONFLICT DO NOTHING and writes nothing at all, not the row, not
the position, not the chat's timestamps. A turn's batch is ON CONFLICT DO
UPDATE guarded by `is distinct from`, so re-persisting an unchanged
40-message transcript writes zero rows; when a message does need
finalising, only its body changes and its identity and position stand.
There is no wholesale replacement of a chat left anywhere.
`mergeStoredMessages` and `writeMergedMessages` are deleted rather than
left beside the new API, so there is no ready-made way back to the bug.
`appendChatMessage` goes with them: nothing outside its own test called
it, and a non-deduped append has no meaning now that identity is the key.
Migration 0019 is additive. It creates the table, backfills from
`jsonb_array_elements … WITH ORDINALITY` using each message's own id
(deriving a stable one from the chat and ordinal for a legacy message
without one, and keeping the first of a duplicated id exactly as the old
merge did), then advances each chat's allocator past what it wrote.
`chats.messages` stays for the transition and nothing reads or writes it;
dropping it is a separate later migration.
## Summary
Projects can create, inspect, expire, and revoke multiple API keys for
each environment. Plaintext values are shown only at creation; stored
credentials are hashed and the API keys page displays only an obfuscated
suffix afterward.
Self-hosted installations support full-access additional keys by
default. Authorization extensions can provide additional access presets
and optional task selection. Additional keys can also mint scoped public
access tokens through the Trigger.dev API without receiving the
environment signing key.
## Feature notes
- Only admin+ can create API keys (Developer can make in Development
branch).
- JWT self-signing will be a server call when used with new `_ak_` keys.
- JWTs with long expiry can keep working even with api key deleted (gets
priveleges from api key, signed with root key)
- Unfiltered session listings intentionally preserve the existing broad
task-read behavior. Filtered listings enforce task-level scopes for
every requested task.
- Buffered runs without a task identifier are not safely authorizable,
so cancel/replay requests fail closed rather than resolving an unscoped
run.
- Batch and waitpoint endpoints intentionally return server-minted,
narrowly scoped public tokens to all callers. These tokens have bounded
lifetimes and may remain valid until expiry after API-key revocation.
## Deployment notes
Deploy the management UI and public-token endpoint with new key creation
disabled. Enable creation for selected organizations after the
authentication path and released SDK have been verified, then expand
availability gradually.
Revoking an API key prevents new bearer requests and new token minting.
Public tokens already minted by that key remain valid until their own
expiration because they are signed by the environment signing key.
## TODO
- [x] Add "Created by" to the key table
- [x] Document that streamed batch ingestion is non-atomic and may
partially accept items before a validation or authorization error.
## Follow-ups
- [x] Add an organization-level feature flag for the API key management
UI and creation action.
- [x] Document rollout ordering: enable additional-key lookup before
enabling issuance.
- [x] Add a system-wide gate that can stop new key issuance without
disabling authentication for existing keys.
- [x] Replace the generic SDK compatibility warning with the first
published compatible version. Old SDK will mint an unusable token if
given an `_ak_` key.
- [x] Add public documentation covering creation, storage, expiration,
revocation, SDK compatibility, and public-token lifetime behavior.
- [x] Add observability for key creation, revocation, policy preparation
failures, and public-token mint failures.
- [ ] Exercise create, copy-once display, authenticate, mint, expire,
and revoke flows end to end before broad enablement.
Both places that roll the per-turn cache breakpoint onto the last message wrote
`anthropic: { cacheControl }` wholesale, replacing the namespace and silently
dropping every other Anthropic option that message carried.
`withCacheBreakpointOnLast()` now spreads the existing options and overwrites only
`cacheControl`. The agent's `prepareMessages` hook had the same write inlined, so
it goes through that one function instead, as `prepareTurnMessages()` — which also
gives the hook's behaviour a name a test can reach.
The system block's breakpoint in `chat.prompt.set` is left alone: it builds a fresh
options object, so there is nothing to merge into.
Nothing calls it. It settles investigation rows in their own writes and leaves the
caller to append the closing cards afterwards — the model that produced terminal
rows whose card never landed, which the stale sweep no longer selects. Keeping a
ready-made path back to that bug next to the atomic API is the whole risk.
`settlementCardMessages()` and the `SettledInvestigationCard` type stay: the
ordinary `persistTurn` path still uses both.
`settleInvestigationStateAndCloseCard()` bumped the revision before it looked at
the transcript, so a redelivered or replayed action settled the row a second time
while the append refused the duplicate card. The row then held revision 2 and the
transcript's terminal card revision 1: the live run rendered 2, a refresh rendered
1, for a tool advertised as idempotent on the action id.
The transaction now locks the investigation, checks the tenancy triple, locks the
chat, and looks for the message id before it writes anything. An action already in
the transcript returns the stored card and the current revision untouched.
Lock order stays investigation then chat, matching `persistTurn` and the sweep's
`settleInvestigationAndCloseCard`; taking the chat first here would deadlock
against them.
A missing or deleted chat was indistinguishable from an already-closed card —
both came back `closed: false` — and the settle committed anyway, which is exactly
the terminal-row-without-a-card the transaction exists to prevent. It is now
`{ ok: false, error: "chat_missing" }`, and the watch lane logs it as the race it
is rather than a fault.
The watch lane settled the investigation with `settleOpenInvestigations`, then
appended the terminal card as a separate write — and swallowed that write's
error, logging it and reporting success. So the row went terminal, the card
never arrived, the stale sweep stopped selecting the row because it was no
longer `in_progress`, and the user was left on "Working…" with nothing able to
repair it. Nothing in production calls the action again on its own, which is
exactly why the swallowed error mattered.
The lane now writes through `settleInvestigationStateAndCloseCard`: the terminal
revision and the closing card commit in one transaction, under the lane's own
message id, and the error propagates so the task's retry is a real retry. If the
card cannot be rendered the settle rolls back, leaving the `in_progress` row the
sweep still selects. The duplicated revision bump is gone with it — one outcome
is now one revision.
The regression test drives a failing close in one action; it no longer proves
recovery by calling the action a second time by hand.
`persistTurn` and `persistMessages` stored the whole `messages` array they were
handed. The array is the snapshot the turn started from, so anything another
process appended in between was deleted: a wake delivery, a watch consent
record, or the terminal card of an investigation the stale sweep had just
settled. That last one is unrecoverable — the row is already terminal, so the
sweep never selects it again, and the panel is back to "Working…" for ever.
Both writes now read the row under `select ... for update` inside the
transaction and merge by stable message id: incoming order is kept, a stored
message the snapshot does not have goes at the end, and no id appears twice. A
message with no id falls back to its content so it cannot be carried over twice.
An append-only `chat_messages` table is the better long-term shape; merging
under the lock is enough for this architecture and needs no migration.
Covers the two halves the earlier pass left open. Against a real database, a
stale card that cannot be rendered now proves the sweep rolls the settle back
with it: the row stays `in_progress` at revision 0, the chat stays empty, and
the row is still in the next run's selection.
The panel half is covered over `liveProgress`, the code that decides whether
"Working…" is shown: a mounted panel holding the unconcluded card re-reads the
stored transcript, merges by stable id, and the progress line goes away without
a reload. Re-reading repeatedly cannot add a second copy of the card.
The row settle, the transcript and the session state were three separate
operations. Once the row was terminal, a failed transcript write left a card
that read `in_progress` forever: the stale sweep only selects `in_progress`
rows, so nothing was left to repair it.
Both lanes now commit the pair atomically. The live turn hands its pending
settlements to `persistTurn`, which upserts the revisions, appends their cards
and writes the session in one transaction; the process-local entry survives
until that commits, so a retried `onTurnComplete` still settles. The sweep goes
through `settleInvestigationAndCloseCard`, whose rollback restores the
`in_progress` row the sweep already selects.
That also removes the last reader of the per-run `chatOwners` map, which had no
`delete` and grew for the life of the worker: the failure record now travels in
the transcript write, which needs no userId.
Separately, the consented watch investigation gets the rolling step cache. Its
ten-step `streamText` re-sent every accumulated tool output uncached; the
breakpoint helper and the per-step cache telemetry now live in `step-cache.ts`
and both lanes use them, wrapping any `prepareStep` the resolved options carry.
Binding a delegated token to its environment claim on the project-wide
routes also refused any token that carries no claim at all, which the
public PAT exchange used by MCP and the CLI is allowed to issue. Line the
project-wide helper up with its neighbours: a claimless dashboard-agent
token is still refused, everything else stays project-wide.
The sweep settled the row and appended nothing, so it visibly fixed nothing: the
chat kept rendering the last card it had, which was still "Working…". The settle
now returns the state and revision it wrote, and the sweep appends that as the
closing card revision on the chat — id-deduped on
`investigation-settlement:{id}:{revision}`, so a retried run can neither stack a
second card nor open a second investigation.
The append is scoped by chat id: a sweep runs off any session and has no user in
context, unlike the turn lane.
Settling the investigations row was invisible to the user. The panel builds the
winning revision from the transcript's own `tool-render_view` parts and never
reads that table, so a turn that ran out of steps left the card at
`in_progress` forever: the database believed the investigation had finished
while a refresh still showed "Working...".
`settleOpenInvestigations` now returns the revisions it committed, and
`onTurnComplete` appends each as one more card revision — after the transcript
write and id-deduped on `investigation-settlement:{id}:{revision}`, so a failed
append leaves the card visibly unclosed rather than silently lost, and a retry
can't stack a second card.
The card-building and the latest-revision reader move out of the watch lane and
into the runtime both lanes share, so there is one shape, not two. The watch
lane keeps its own message id: it dedupes on the action, not the revision.
Katia Bulatova