Migrations only, no code reads them yet. Postgres: nullable non-unique
externalId on WorkerDeployment plus a CONCURRENTLY-built (environmentId,
externalId) index in its own migration file. ClickHouse:
external_deployment_id String DEFAULT '' on task_runs_v2 (plain String,
not LowCardinality - commit SHAs are high-cardinality). Part of task run
version skew protection (TRI-12998).
## Summary
Move tree selection onto semantic tree items and use native expansion
buttons.
Dashboard and story tree rows now share mouse and keyboard selection
through `getNodeProps`. Expand and collapse affordances are named
buttons instead of clickable layout elements.
Base: [#4700](https://github.com/triggerdotdev/trigger.dev/pull/4700)
## Summary
Use native controls for sortable columns and selectable prompt versions.
Table headers keep filter actions separate from sort buttons, prompt
version rows expose pressed state, and a redundant deployment click
interceptor is removed.
Base: [#4699](https://github.com/triggerdotdev/trigger.dev/pull/4699)
## Summary
Make time-filter mode selection keyboard accessible.
Duration and exact-range modes now use native pressed buttons. Nested
date, duration, and quick-select controls no longer depend on click
propagation blockers.
Base: [#4698](https://github.com/triggerdotdev/trigger.dev/pull/4698)
## Summary
Replace mouse-only dashboard actions with native buttons.
Copy, remove, and stop-generation controls now expose keyboard focus and
accessible names. Hover-revealed actions remain mounted so keyboard
users can discover them, and a decorative clipboard icon no longer
captures clicks.
Base: [#4697](https://github.com/triggerdotdev/trigger.dev/pull/4697)
## Summary
Use native label and checkbox behavior for `CheckboxWithLabel` and
enforce `jsx-a11y/no-noninteractive-element-interactions`.
The component no longer simulates checkbox activation with click
handlers on non-interactive wrappers. Native change events now drive the
controlled checked state.
Base: [#4696](https://github.com/triggerdotdev/trigger.dev/pull/4696)
## Summary
Require accessible names for dashboard controls.
Filter menu action items and chart color controls now expose explicit
names. The chart legend action uses a native button, while lint depth
and spacer-cell configuration match the rendered control structure.
Base: [#4695](https://github.com/triggerdotdev/trigger.dev/pull/4695)
## Summary
Finish associating dashboard form labels with their controls and enforce
`jsx-a11y/label-has-associated-control`.
Repeated data store dialogs use unique generated IDs, story controls and
notification filters have explicit associations, and display-only status
text no longer uses label elements.
Base: [#4694](https://github.com/triggerdotdev/trigger.dev/pull/4694)
## Summary
Associate internal model administration labels with their form controls.
The model editor, creator, and tester now use explicit `htmlFor` and
`id` pairs. Section titles that do not label controls now use headings
instead of label elements.
Base: [#4693](https://github.com/triggerdotdev/trigger.dev/pull/4693)
## Summary
Enable foundational JSX accessibility checks for image text alternatives
and valid ARIA roles.
The avatar color picker now has an explicit accessible name and
decorative image alternative. Dashboard chat styling props no longer
reuse the reserved DOM `role` name.
Base: [#4692](https://github.com/triggerdotdev/trigger.dev/pull/4692)
## Summary
Add explicit types to native dashboard buttons and enforce
`react/button-has-type`.
This prevents action buttons from accidentally submitting a surrounding
form. Shared button primitives retain their caller-selected submit and
reset semantics with documented lint exceptions.
Base: [#4691](https://github.com/triggerdotdev/trigger.dev/pull/4691)
## Summary
Remove redundant React fragments from the dashboard and enforce
`react/jsx-no-useless-fragment`.
The cleanup returns existing nodes, arrays, and empty states directly
without adding wrapper elements.
Base: [#4689](https://github.com/triggerdotdev/trigger.dev/pull/4689)
## Summary
Keep component and renderer identities stable across dashboard renders.
Inline icon components, chart renderers, table cells, and select render
callbacks now use module-level implementations. Oxlint enforces the
pattern across the dashboard.
Base: [#4688](https://github.com/triggerdotdev/trigger.dev/pull/4688)
## Summary
Enforce stable React hook ordering in the dashboard and React hooks
package.
Conditional hook calls now keep a consistent order, and overloaded
realtime stream arguments are resolved before entering the shared hook
implementation.
Base: `main`
`@grpc/grpc-js` sat at 1.12.6 in the lockfile. `dockerode` is the only
consumer and already declares `^1.11.1`, so a scoped override is enough:
```json
"@grpc/grpc-js@>=1.12.0 <1.12.7": "1.12.7"
```
Pinned exactly to stay on the 1.12 line; a caret would pull 1.14.x.
## Summary
Listing schedules could block the event loop for seconds. A page of 100
timezone-aware schedules spent over two seconds on cron arithmetic
alone, after the database work was already done, which stalls every
other request on that process. The same page now resolves in tens of
milliseconds.
## Root cause and fix
`cron-parser` walks the calendar unit by unit, and under a named
timezone every step goes through luxon. Parsing an expression is cheap
(single-digit microseconds); *stepping* it is not, ranging from a couple
of hundred microseconds for a common expression to several milliseconds
for a sparse one like `0 0 29 2 *`. The presenter did three independent
walks per row, one backwards for "last run" and two forwards (re-parsing
each time) for the next run and the occurrence after it. At 100 rows
that is 300 calendar walks in one uninterrupted tick.
Run times now resolve for the whole page in one pass, in a new
`resolveScheduleTimings` that takes plain values rather than Prisma rows
so it can be tested and benchmarked on its own.
- **Nominal times are cached per `(cron, timezone)`** against a single
`now` pinned for the batch, so cost scales with the number of distinct
expressions instead of the number of rows. Rows in one response also
stop disagreeing about the current time.
- **The backwards walk is opt-in.** It is the most expensive of the
three and only the dashboard renders the column; the public API never
returned it at all.
- **Windowless schedules take one step instead of two.** The second step
only measures the interval to the following occurrence, and that
interval reaches the result solely through `min(intervalMs,
max(MINIMUM_SCHEDULE_RANGE_MS, windowMs))`. With no window `windowMs` is
0, and `CronPattern` rejects expressions with a seconds field, so
occurrences are always at least `MINIMUM_SCHEDULE_RANGE_MS` apart and
that `min` can never bind. It is also the costlier step, since it walks
a whole period rather than the remainder of the current one.
- **`nextScheduledTimestamps` steps one parsed expression** instead of
re-parsing per step, which also helps the single-schedule callers.
Behaviour is unchanged, error semantics included: a malformed expression
still throws for the next run and still degrades to an undefined last
run.
## Verification
Measured inside a real request against a live environment, 100
schedules: sparse expressions went from 2250-2652 ms to 23-30 ms, and
five distinct timezone expressions from 463-500 ms to 9.7-10.6 ms.
The new suite checks the optimized code against an inline copy of the
previous implementation across eleven cron and timezone combinations
plus five DST transitions, so the rewrite is verified as
behaviour-preserving rather than just faster. Separate tests pin the
invariant the single-step path depends on, so if sub-minute crons are
ever allowed they fail loudly instead of the timings quietly going
wrong.
Worth knowing for later: `cron-parser` v5 is a much faster rewrite on
exactly this workload (`prev()` under a timezone drops from roughly 2700
to 60 microseconds), but it is a breaking API change across several call
sites including the schedule engine, so it belongs on its own. The
differential test added here is the tool to de-risk it.
## Summary
Allow the logs search schema migration to run on ClickHouse versions
that require text index options to be literals.
## Root cause
The text index declared `lowerUTF8(search_text)` as a preprocessor
option. Some ClickHouse versions reject that column expression while
parsing index settings. The projected `search_text` is already
normalized to lowercase before insertion, so removing the redundant
preprocessor preserves search behavior.
Verified with the task events search integration tests.
## What
The one-off worker container boot is billed to whichever test resolves
the fixture first. This moves it into a `beforeAll` with its own
timeout.
## Why
vitest runs the fixture chain *inside* the test timer:
```js
// @vitest/runner 4.1.7
setFn(task, withTimeout(...withFixtures(handler)..., timeout, ...))
```
There is no `fixtureTimeout`. So booting Postgres (plus `CREATE
DATABASE`, schema push, ClickHouse and Redis) lands on the first test
and consumes a budget sized for test work.
That is why losing the image pre-pull on fork PRs was fatal rather than
merely slower: the extra ~10s crossed the 60s cap. Since fork time is
roughly internal + 10s and forks exceed 60s, internal runs were already
clearing that cap by under 10s — a latent flake regardless of forks.
## How
`withWarmup` wraps each fixture family and lazily registers a
`beforeAll` on first touch, with its own generous timeout. Registration
is lazy so only files that actually use a family pay for it —
`@internal/testcontainers` is imported by hundreds of test files, many
of which only need Redis. It registers once per file, since `isolate`
gives each file a fresh module registry.
Eight families are wrapped. `isolatedRedisTest`,
`replicationContainerTest` and `postgresAndRedisTest` are deliberately
untouched: they use per-test containers by design, so there is no
one-off boot to hoist.
No test file or CI changes, and it applies to every package using these
fixtures.
## Verification
Proven by mutation. `src/warmup.test.ts` runs container tests under a
deliberately tight cap:
| | Result |
| --- | --- |
| with the warm-up | passes |
| warm-up neutered | fails, `Test timed out` |
It is kept as a regression test — without it, unwrapping a fixture would
break nothing visibly.
`triggerFailedTask.call.test.ts`, one of the five shard casualties,
passes locally in 20.4s.
## Also here
`@internal/testcontainers` had no `test` script, so `turbo run test
--filter "@internal/*"` skipped the package and its existing
`heteroDedicated.test.ts` never ran in CI. Adding the script (matching
the sibling packages') runs both files; verified green through turbo
exactly as CI invokes it.
## What
Three corrections to the pre-pull lists, each verified against what the
suites actually use.
## Changes
**`ryuk:0.11.0` -> `0.14.0`** in `e2e-webapp.yml` and
`e2e-webapp-auth-full.yml`. The installed testcontainers hardcodes the
image it starts:
```js
// testcontainers@11.14.0 build/reaper/reaper.js
: ImageName.fromString("testcontainers/ryuk:0.14.0").string;
```
So those two lines were pre-pulling an image nothing starts, and the one
actually used was never pre-pulled. The other three workflows already
say 0.14.0.
**`postgres:17` added** to `unit-tests-webapp.yml`. The webapp suite
references `docker.io/postgres:17` across 10 files but only
`postgres:14` was pre-pulled. `unit-tests-internal.yml` already pulls
both.
**Electric pinned to its digest** in `unit-tests-webapp.yml`. The tests
run `electricsql/electric:1.2.4@sha256:20da...` while the pre-pull asked
for the bare tag, so the pre-pull did not necessarily populate the
manifest the tests then request.
## Not changed
The otel collector and s2 images are pulled by other workflows but are
not used by the webapp suite, so they are deliberately not added here.
`postgresAndRedisTest` uses per-test containers by design and needs
nothing pre-pulled.
## What
The `Pre-pull testcontainer images` step is gated on
`env.DOCKERHUB_USERNAME`. Fork PRs receive no repository secrets, so
that variable is empty and the step is skipped along with the DockerHub
login it was grouped with.
## Why
With the pre-pull skipped, testcontainers pulls images lazily — inside
the first test that resolves the fixture, against that test's
`testTimeout`. On PR #4534 that pushed five webapp shards past their 60s
cap across three runs, each failing as `Test timed out in 60000ms` while
42 of 43 files in the shard passed.
Measured cost of the missing pre-pull, comparing the delta from vitest
start to the first container fixture on the same runner class:
| Run | Delta |
| --- | --- |
| internal x2 | +139.9s, +139.4s |
| fork x2 | +149.7s, +149.4s |
A 10.0s penalty, bimodal to within 0.3s.
Note the pulls themselves succeed anonymously — there are no rate-limit
errors in any of the failing logs. Only the login needs credentials, so
the pre-pull can run unconditionally.
## Scope
Removes the `if:` from the pre-pull step in all five workflows that have
one. The DockerHub login stays gated, since it genuinely needs secrets.
## Summary
Memoize shared context values so provider renders do not unnecessarily
rerender every consumer. Oxlint now enforces this pattern for the rest
of the dashboard.
Base: [#4677](https://github.com/triggerdotdev/trigger.dev/pull/4677)
## Summary
Enable lint rules that prefer direct iteration and concise function
callback types.
The existing code now uses direct iteration where no index is needed,
and callback contracts use function types consistently.
Base: [#4675](https://github.com/triggerdotdev/trigger.dev/pull/4675)
## Summary
Enable JSX cleanup rules for shorthand fragments and self-closing
components.
The existing JSX is automatically simplified, and future components will
follow the same concise form.
Base: [#4673](https://github.com/triggerdotdev/trigger.dev/pull/4673)
## Summary
Enable small cleanup rules for redundant boolean expressions, object
ownership checks, assignments, and object construction.
The existing call sites now use the simpler equivalent forms, keeping
future code consistent without changing behavior.
Base: [#4672](https://github.com/triggerdotdev/trigger.dev/pull/4672)
## Summary
Enable additional lint rules that catch unsafe optional-chain
assertions, inherited-property iteration, anonymous symbols, and unsafe
external links.
The existing violations now use explicit values and own-property checks,
so the rules can prevent those patterns from returning.
Adds an optional priority class for run pods.
```
KUBERNETES_RUN_POD_PRIORITY_CLASS_NAME
```
When set, the value is applied as `priorityClassName` on the run pod
spec. When unset, pods are created exactly as before.
Off by default, and inert unless set. It sits beside the existing
`KUBERNETES_SCHEDULER_NAME` option and follows the same conditional
shape:
```ts
...(env.KUBERNETES_RUN_POD_PRIORITY_CLASS_NAME
? { priorityClassName: env.KUBERNETES_RUN_POD_PRIORITY_CLASS_NAME }
: {}),
```
## Verification
`typecheck --filter supervisor`, `format` and `lint` clean. No changeset
or `.server-changes/` note: off by default, no user-visible behaviour
change.
## What
Runs that carry no external trace context (schedules, task-to-task
triggers) fall back to a trace id generated once in the [`TracingSDK`
constructor](https://github.com/triggerdotdev/trigger.dev/blob/main/packages/core/src/v3/otel/tracingSDK.ts#L165).
With `experimental_processKeepAlive` the SDK outlives the run, so every
run on a warm process is exported to the external OTLP endpoint under
that one id.
Across our production traces, 80.3% contained spans from more than one
run, worst case 25. Per-run cost and latency attribution is unusable as
a result. This is the same warm-start hazard c043c4a6a fixed for the
external-context path, which left the fallback captured at construction.
## How
`FallbackExternalTraceIds` hands out one id per internal trace, shared
by the span and log wrappers so a run's spans and logs agree.
The id is keyed off the record's own internal trace id rather than
ambient state at export time, because batch processors drain
asynchronously and a run's records routinely export after the next run
has started. The map is bounded and evicts least-recently-used, so a run
that is still exporting can't lose its id.
Granularity follows the internal trace, so a run and the runs it
triggers stay on one trace.
**Risk:** the wrappers only exist when `exporters` / `logExporters` are
configured, so deployments that don't export externally are untouched.
Nothing outside `tracingSDK.ts` changes.
**Known gap (pre-existing):** sampling and id selection still branch on
ambient `getExternalTraceContext()`, so records draining across a run
boundary in mixed mode are misplaced in both directions. It can't use
the approach here — the external id comes from the run's incoming
`traceparent`, which isn't carried on the record — so closing it means
capturing `internalTraceId -> external context` in a span processor.
Happy to follow up separately.
---
## Testing
`packages/core` suite passes. `pnpm run format` and `pnpm run lint:fix`
produce no diff.
Six cases in `externalSpanExporterWrapper.test.ts`, each
mutation-checked rather than just observed passing: one id per run,
stability within a run, correct id when records drain after the next run
started (spans and logs together), external export stays off when
unconfigured, retention of a run still exporting while the map churns,
and the bound itself.
**CI:** the five failing `webapp` shards are the ones containing
`containerTest` suites. Fork PRs receive no repository secrets, so
`unit-tests-webapp.yml` skips the DockerHub login and the image pre-pull
(both gated on `env.DOCKERHUB_USERNAME`) and the container tests time
out at 60s. Same five shards across five runs, every failure a 60s
timeout, and those shards pass on internal PRs. Happy to be corrected if
you can run them with secrets available.
---
## Changelog
Unrelated runs are no longer merged into a single trace in your external
observability tool when they happen to execute on the same warm worker
process. A run and the runs it triggers still share one trace, so a run
tree stays together.
---
## Screenshots
_n/a_
---
_Supersedes #4526 (auto-closed before I was vouched) and #4533 (opened
ready rather than as a draft). GitHub won't reopen either._
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Iss <74388823+isshaddad@users.noreply.github.com>
Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
<!-- ccr-slack-attribution -->
_Requested by **Matt Aitken** · [Slack
thread](https://triggerdotdev.slack.com/archives/C0BKB98B84W/p1787045331358929)_
Copy-only reword of the banner shown to org admins who have not set a
billing limit yet.
**Before** — the banner read "Protect your organization from unexpected
usage spikes." with a button labelled "Configure billing limit".
**After** — it reads "Add a billing limit to your account to prevent
overspending" with a button labelled "Billing limit settings".
The new wording names the action up front and matches the destination it
sends you to, so the banner reads as a settings link rather than a
one-off setup step.
## ✅ Checklist
- [x] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [x] The PR title follows the convention.
- [x] I ran and tested the code works
---
## Testing
Formatting and linting pass (`oxfmt --check`, `oxlint`). No tests or
snapshots assert this copy. The change is two string literals in one
component, with no behaviour attached.
---
## Changelog
Reworded the billing-limit banner for organizations without a limit
configured, and relabelled its button to "Billing limit settings".
---
## How
Both strings live in `NoLimitConfiguredBanner` in
`apps/webapp/app/components/billing/OrgBanner.tsx`: the heading is the
`canManageBillingLimits` branch of the banner's children, and the label
is the `<span>` inside the `LinkButton`. Only those two literals
changed. The button still points at `v3BillingLimitsPath(organization)`
(`/orgs/{slug}/settings/billing-limits`), so routing, permissions and
the non-admin variant of the message are untouched.
Co-authored-by: Claude <noreply@anthropic.com>
Adds KUBERNETES_RUNNER_SECURITY_CONTEXT (off | baseline | restricted), selecting how constrained the run container is.
baseline drops the capability bounding set and blocks privilege escalation. restricted additionally pins the container to a non-root uid, chosen by runtime so bun images get their own.
Default is off, so this is inert on merge.
## Summary
A run triggered with a `ttl` could get permanently stuck showing as
queued. If the run started executing and was then requeued after a
failure (a stalled heartbeat, a worker dying mid-run) once its TTL had
already elapsed, the next dequeue pass silently dropped it from every
queue structure. The run stayed QUEUED in the database forever, and
nothing (dequeue, the TTL consumer, queue repair) could ever see it
again.
## Root cause
Enqueue registers a TTL entry for the TTL consumer, and the first
dequeue removes it ("the run is executing, not expired"). A nack rewrote
the message preserving the original `ttlExpiresAt` without
re-registering that entry. The next dequeue pass then took the
expired-TTL branch: remove the run from the queue sorted sets and leave
the message for the TTL consumer to finalize. But the consumer's entry
was gone, so nothing ever finalized the run.
The fix has two halves:
- `nackMessage` drops `ttlExpiresAt` from the rewritten message. TTL
only applies to runs that have never been dequeued (the same contract as
`includeTtl` on re-enqueues), so a requeued run stays dequeuable and is
never expired by its original deadline.
- The dequeue expired-TTL branches now (re-)register the TTL entry
instead of assuming it exists, so any message still carrying a lapsed
`ttlExpiresAt` with no TTL entry (including ones written before this
fix) finalizes as EXPIRED instead of orphaning.
## Verification
New engine test suite `ttlNackRequeue.test.ts` (testcontainers, real
Redis and Postgres). All four tests fail before the fix and pass after:
- a heartbeat-stalled EXECUTING run with a lapsed TTL is requeued and
dequeued again instead of orphaned (the full production failure chain)
- requeue-after-failure strips `ttlExpiresAt` so later dequeues do not
treat the run as expired
- a lapsed-TTL message whose TTL entry is missing is re-registered by
dequeue and finalized as EXPIRED, for both plain and concurrency-key
queues
Also ran the existing ttl, heartbeats, dequeuing and attemptFailures
engine suites plus the full run-queue suite (149 tests) against the
change.
Adds two optional env vars that rewrite the registry host of run pod images at pod creation, so a supervisor can pull from a registry in its own region. Off by default and inert unless both are set. Exact host-prefix matching, so look-alike hosts pass through untouched.
Replaces the hardcoded runner seccomp profile path with KUBERNETES_RUNNER_SECCOMP_PROFILE_PATH, and the node-24-only condition with KUBERNETES_RUNNER_SECCOMP_PROFILE_RUNTIMES (none | node-24-plus | all).
Both defaults reproduce current behaviour, so this is inert on merge. Widening the scope or turning attachment off becomes a config change rather than a deploy.
The supervisor now supports routing an organization's runs to specific
nodes. `KUBERNETES_ORG_PLACEMENT_OVERRIDES` takes JSON keyed by the
internal org ID, adding node selector entries and tolerations to that
org's run pods, e.g. to route an org onto a dedicated, tainted node
pool:
```json
{"<orgId>": {"nodeSelector": {"pool": "dedicated"}, "tolerations": "dedicated=runs:NoSchedule"}}
```
The node selector merges over the defaults (the override wins on key
collision, with a warning logged). Tolerations append to the existing
runner and scheduled-run sets. Overrides are validated at startup
similar to `KUBERNETES_RUNNER_TOLERATIONS`.
Exposed in the Helm chart as
`supervisor.config.kubernetes.orgPlacementOverrides`, where tolerations
can also be given as a list.
## What & why
The dashboard agent can now run its model calls through AWS Bedrock
instead of the direct Anthropic API, chosen by a single env switch. It's
**off by default** (`DASHBOARD_AGENT_MODEL_PROVIDER` unset ⇒
`anthropic`), so merging changes nothing at runtime — the Bedrock path
is a dormant branch until an operator sets the switch and AWS config.
The default Anthropic path is byte-for-byte unchanged.
This also carries a related tenant-isolation hardening for the agent's
delegated token (kept together deliberately — both land the agent on
Bedrock for HIPAA readiness). Refs: TRI-13251, TRI-11032.
## What's inside
**Provider seam** —
`internal-packages/dashboard-agent/src/model-provider.ts`: the registry
now holds both `anthropic` and `bedrock`; `resolveDashboardAgentModel()`
maps the canonical `"anthropic:<id>"` strings the managed prompts carry
to the active provider, and the cache-breakpoint helpers emit the active
provider's shape — Anthropic `cacheControl` vs Bedrock `cachePoint`.
Managed prompt strings stay canonical, so stored prompts don't change
meaning. Unmapped model ids throw rather than shipping a guaranteed-404
profile. All agent, watch, compaction and title callsites route through
the resolver; the `dashboardAgentModelKey` locals override (test mock
injection) is preserved.
**Cache telemetry** — `step-cache.ts`: cache token usage is read from
the active provider (Anthropic reports it on provider metadata; Bedrock
reports the write on metadata and the read via standard usage), so
`gen_ai.usage.cache_*` is populated on both. This also fixes a latent
ordering bug where step attributes could null-overwrite the prompt-cache
read count.
**Webapp callsites** — `dashboardAgentHeadStart.server.ts` and the
head-start route resolve the model and the cache breakpoint through the
same seam, so the warm-up prefix and the following turn share one
provider. The head-start firing gate is provider-aware: on Bedrock it
gates on `AWS_REGION` and lets the SDK resolve credentials (IAM role /
static keys / session token / bearer), so a role-based deploy still
warms; on Anthropic it stays `Boolean(ANTHROPIC_API_KEY)`.
`app/env.server.ts` gains the optional AWS vars and validates
`DASHBOARD_AGENT_MODEL_PROVIDER`. `ANTHROPIC_API_KEY` is untouched and
not required on a Bedrock deploy.
**Tenant-isolation hardening** —
`internal-packages/rbac/src/fallback.ts`: for a **scoped** context, the
OSS `authenticateUserActor` now applies the same membership floor as the
session path — a delegated user-actor token whose user is not a member
of the scoped org/project is denied (403). Unscoped tokens keep their
prior behavior (no tenant claim, no lookup). The user lookup falls back
replica→primary so replication lag can't spuriously 401 a just-joined
member. Members and admins are unaffected. Previously this invariant
held only through per-route discipline; this makes it structural.
## Enabling Bedrock (later, ops)
- Set `DASHBOARD_AGENT_MODEL_PROVIDER=bedrock` **identically** in both
the webapp and the agent task container — the webapp warms the cache
prefix and the task reads it, so a split would silently miss the cache.
- Set `AWS_REGION` and provide credentials the Bedrock SDK can resolve
(IAM role preferred). For v1 this runs **without** an Anthropic API key.
Note: with no Anthropic key set, rollback is "turn the agent off", not
"unset the switch" (unsetting falls back to the Anthropic provider,
which then has no key).
- Two things to confirm before rollout: the Sonnet inference-profile id
is validated against the SDK's own model-id union but still warrants a
live smoke test; and Bedrock prompt caching for Sonnet is a 5-minute
window (not Anthropic's 1h), so input-token cost rises when flipped.
## Testing
Unit tests cover both provider paths: the provider switch and
per-provider cache shapes, a structural regex asserting Bedrock ids are
real inference profiles (not an echo of the table), the split-metadata
cache telemetry, and real-Postgres RBAC tests — member allowed, scoped
non-member denied (org-only and project-only), missing user → 401, admin
non-member exempt, unscoped success. `typecheck --filter webapp` and the
dashboard-agent + rbac suites pass.
## Summary
When an AI SDK tool call failed inside a run, the span showed up under
the "Errors only" filter but the span inspector gave no hint of what
went wrong. The exception was recorded on the span all along; the
`ai.toolCall` and `ai.embed` inspector views just never rendered span
events. Failed tool call and embedding spans now show the standard error
block (message plus stack trace) below the Input section.
## Root cause
Generic spans render exception span events via the `SpanEvents`
component, but the AI-specific span entities replace the whole panel
with their own layout and dropped the events entirely. The span's events
are now passed into `AIToolCallSpanDetails` and `AIEmbedSpanDetails` and
rendered with the same `SpanEvents` component the generic view uses.
Errored generation spans (`ai.generateText` and friends) use a tabbed
view and still don't surface errors; that needs its own design pass and
is left for a follow-up.
## The bug
A concurrency key is an unrestricted client string
(`ConcurrencyKeySchema` is `z.union([z.string(),
z.number()]).transform(String)`), and `concurrencyKeySection` does no
escaping, so `*` reaches the queue raw. `queueKey` then renders it as
`...:queue:<q>:ck:*`, which is byte-identical to the wildcard member the
CK scripts keep in the master queue to mean "this base queue has
concurrency-key work".
Every CK script ends with the same pair:
```lua
-- Rebalance master queue with ck:* member
redis.call('ZADD', masterQueueKey, earliestIdx[2], ckWildcardName)
-- Remove old-format entry from master queue (transition cleanup)
redis.call('ZREM', masterQueueKey, queueName)
```
`ckWildcardName` is `toCkWildcard(message.queue)`, and for a `*`-keyed
run that returns the identical string, so the cleanup on the second line
deletes what the rebalance on the first line just wrote.
The master queue then has no entry for that base queue, while `ckIndex`
and the variant queues still hold the work. **Every concurrency key on
the queue stops being dequeued**, not just the `*` one. It is silent,
and it only recovers if some later write happens to re-add the member.
Reproduced before the fix:
```
master queue AFTER normal ck enqueue: ["{org:...}:queue:task/my-task:ck:*"]
master queue AFTER ck='*' enqueue: []
ckIndex members (work still queued): [":ck:user-1", ":ck:*"]
dequeued: []
```
Blast radius is bounded to the environment that triggers it, so it is
self-inflicted rather than cross-tenant, but a single trigger stalls the
queue for everything on it.
## The fix
Guard the cleanup so it never removes the wildcard member:
```lua
if queueName ~= ckWildcardName then
redis.call('ZREM', masterQueueKey, queueName)
end
```
Applied to all 10 CK scripts (4 enqueue, 6 ack/nack/dead-letter). No
key-format change and no migration: a queue already stranded in Redis is
repaired by its next write.
I considered rejecting `*` at the API boundary instead and rejected it.
Existing Redis state and `TaskRun.concurrencyKey` rows already hold raw
`:`-bearing and `*` keys, so changing key construction would orphan
in-flight messages and split concurrency accounting mid-deploy. Boundary
validation would still be reasonable as belt-and-braces later, but the
Lua guard alone fixes it including for state already out there.
## Testing
`ckWildcardKey.test.ts` covers the enqueue, ack and nack paths. All
three pass with the guard and **all three fail without it**, verified by
reverting. Full `src/run-queue/` suite is green (166 tests).
## Note for #4367
The virtual-time branch adds three more CK scripts with the same pattern
(`enqueueMessageCkVtimeTracked`, `enqueueMessageWithTtlCkVtimeTracked`,
`nackMessageCkVtimeTracked`). They do not exist on main so they are not
in this PR; the same guard needs applying there, and I will do that on
that branch.
Oskar Otwinowski