## What
Rate-limit the API by **environment** rather than per API key.
Previously the limiter keyed its bucket on the hash of the full
`Authorization` header — one bucket per key. With additional environment
API keys (`tr_*_sk_*`), an environment can mint many keys and each got
its own full bucket, so more keys = higher effective rate limit. This
collapses all of an environment's keys onto a single shared
per-environment bucket, so the ceiling is exactly the configured limit
regardless of key mix.
## How
- `authorizationRateLimitMiddleware` now lets the override return `{
config?, identifier? }`. `identifier`, when present, is the rate limit
bucket key; otherwise it falls back to the hashed `Authorization` header
(unchanged legacy behavior, still used by `engineRateLimiter` and any
unauthenticated fallthrough).
- `apiRateLimiter`'s override resolves the environment id and uses it as
the identifier:
- **Additional keys** (`isAdditionalApiKey`) resolve via a new
`resolveAdditionalApiKeyRateLimitScope()` — a **scope-agnostic** keyHash
→ (environmentId, org limiter config) lookup. It is deliberately
permissive (restricted keys resolve too) because it's used **only for
bucketing, never as an auth decision** — request auth still goes through
the RBAC bearer controller, which enforces scopes. Revoked/expired keys
are excluded so they can't hold a bucket warm.
- **Root/legacy keys** reuse the environment already resolved by
`authenticateAuthorizationHeader` and key on `environment.id` too.
- The identifier is always the stable environment id, never the secret
key (which can rotate and would split the bucket).
- The whole override result is cached per key by the existing SWR cache,
so **no extra per-request lookup and no separate Redis mapping** is
added.
## Behavior notes
- Root + additional keys of the same environment now share one bucket
(ceiling = configured limit, not a multiple of it). Restricted
additional keys are included — they were the biggest gap, since they
authenticate via the RBAC controller and previously fell back to per-key
buckets.
- **Public JWTs** keep their existing fixed-window, per-token bucketing.
- One-time bucket reset on deploy (bucket keys change); harmless.
## Tests
- New: two tokens resolving to the same identifier share one bucket.
- New: with no identifier, bucketing stays per-key (legacy behavior
preserved).
- Updated existing override tests to the new `{ config }` return shape.
Base: `feat/multi-keys-surface`. Closes TRI-12888.
## Summary
`batchTriggerAndWait()` could leave a parent run waiting forever. The
2-phase batch API blocks the parent on the batch's waitpoint as soon as
the batch is created, but the batch is only sealed at the end of item
streaming. If streaming never completed, nothing sealed the batch,
nothing completed the waitpoint, and the parent stayed suspended with no
timeout and no way to recover.
Supersedes #4016, which added the reaper alone.
## Fix
Admission for item streaming was being decided twice. Batch creation
passes its own rate limiter, which fixes `expectedCount` and blocks the
parent, and then the item stream had to pass the general API limiter as
well, competing with unrelated traffic. A second limiter could therefore
veto work the first had already committed the parent to. Creation now
mints a bounded grant that the item stream spends, so an admitted batch
can finish streaming. The grant is capped per batch rather than
exempting the path, and every failure mode (no grant, spent grant,
unreachable store) falls back to the normal limiter.
That makes stranding much rarer but not impossible, since a request
timeout or a crash can still end streaming for good. So a seal-timeout
reaper aborts any batch still unsealed after `BATCH_SEAL_TIMEOUT_MS` and
completes the parent's waitpoint with an error, letting
`batchTriggerAndWait()` reject instead of hang. It is race-safe against
a late seal, and it is only scheduled for batches that actually block a
parent, so fire-and-forget batches cost nothing.
Finally, the batches page used to report "Batch completion checked." for
these batches while doing nothing, because the completion path returns
early on an unsealed batch. It now says the batch cannot be resumed.
Rate limiting is no longer the reason a batch strands, so the reaper's
default stays at 30 minutes, comfortably above the SDK's worst-case
stream-retry budget.
## Verification
Unit and container tests cover the grant cap, the bypass ordering (it
runs after the authorization check, so it can never skip
authentication), and the reaper's abort, seal race, idempotency, and
no-waitpoint cases.
Also verified end-to-end against a running stack. With the general limit
exhausted, batch creation and other API calls returned 429 while a
granted batch still streamed and sealed; an ungranted batch id was rate
limited rather than bypassed; and the grant cut off exactly at its
configured attempt count. Reproducing the stranded state on a real
parent run, the batch was aborted at the timeout, the waitpoint
completed with an error, and the parent resumed and finished instead of
hanging. A parentless batch left unsealed was untouched well past the
reaper window.
## Verified against deployed runs
The reaper was proven end to end with a real deployed run (locally-run
supervisor, containerised
run) and a real network fault, rather than a simulated one: toxiproxy
severs the phase 2 item
stream mid-flight so every SDK stream retry genuinely fails, while phase
1 still succeeds. Only
the batch calls traverse the fault, so control-plane traffic is
untouched.
The reproduction is the shape that actually strands a parent: the task
catches the
`BatchTriggerError` the SDK throws and carries on, so the phase 1 block
outlives the thrown error
and the parent hangs at its next suspension point.
With the reaper disabled, the parent sat in `EXECUTING_WITH_WAITPOINTS`
for over 24 minutes holding
two blockers, and stayed stuck across a full infrastructure restart:
```
type | status | has_timeout
BATCH | PENDING | f <- orphan, completedAfter NULL
DATETIME | COMPLETED | t <- the wait already elapsed
```
With the reaper enabled the same task under the same fault completed in
about 75 seconds with zero
blockers left, the batch `ABORTED`, and its waitpoint completed carrying
the error.
Two conditions are required to observe this at all, which is worth
knowing for any future test:
the run must be deployed rather than `trigger dev` (dev runs execute in
process and finish while
still holding blocker rows), and the wait after the caught error must
exceed the checkpoint
threshold, or it is served in process and never suspends.
### Why completing the batch waitpoint is sufficient
`batchTriggerAndWait` runs create, then stream, then wait. A phase 2
failure throws before the wait
is ever reached, and the reaper only fires on an unsealed batch, so the
parent is never suspended
awaiting the batch when it runs. The parent therefore does not need a
synthetic result, only to stop
being blocked. Note this reasoning depends on that ordering: if the wait
were ever reached with an
unsealed batch, completing the batch waitpoint alone would not settle
the caller.
## Follow-ups
- Batches stranded before this ships still need a one-off recovery; the
reaper only schedules at creation time.
- That same property leaves a gap if the process dies between creating
the batch and scheduling the job. A periodic sweep would close it, but
wants a supporting index.
- When a partially streamed batch aborts, children already enqueued keep
running while the parent fails. Left as-is deliberately, since
cancelling triggered work is a bigger semantic call.
Once this is merged, oxlint is at a pretty sensible baseline.
**Enable `no-unused-vars`, `typescript/consistent-type-imports`, and
`import/no-duplicates` lint rules**
Turns on three previously-disabled oxlint rules across the monorepo and
fixes all violations:
- **`no-unused-vars`** – enabled as an error with standard ignore
patterns: unused function arguments are ignored by default (`args:
"none"`), variables/caught errors/destructured array elements prefixed
with `_` are allowed, and rest siblings are permitted.
- **`typescript/consistent-type-imports`** – enforced as an error; all
type-only imports now use the `import type` syntax.
- **`import/no-duplicates`** – enforced as an error; duplicate import
statements from the same module have been merged.
The remaining commits clean up the violations found across the codebase:
removing unused variables/imports/type aliases, adding `_` prefixes to
intentionally unused bindings, fixing duplicate imports, and converting
value imports to `import type` where appropriate.
* Remove unnecessary disabled org check
* Add support for separate redis servers for caching, rate limiter, marqs, and the pub/sub stuff (they all default to the previously used env vars for redis)
* Denormalize run tags, increase character limit to 128
* WIP realtime subscribing to runs
* extracted the stream stuff into core, made it more reusable
* WIP tags
* Remove tags for now because it’s not support in electric
* Support async iterables, readable stream, and callback style subscription styles
* Remove tags streaming endpoint
* Add realtime rate limits and scope them to the /realtime path
* WIP rate limt per org
* Introduce per org rate limits
* WIP JWT auth
* Move migrations into new internal db package
* Resolve pnpm lock file
* Authenticating to the realtime API with JWTs are working
* realtime in the client
* Created react-hooks package and starting to move stuff in there
* Improve types for hooks
* schema tasks
* Added useBatch hook
* build uploadthing/fal demo and change how run metadata is synced to the server
* tweaks
* WIL realtime concurrency tracking
* Implement test for realtime client using testcontainers
also updated electric to latest version
* Allow customizing the expiration time of the automatic JWT created after triggering a task
* Add support for subscribing to run tags
* Improve auth types and API
* finalize the realtime API
* Fixed some example stuff
* Allow up to 10 run tags
* Remove core from docker-provider tsconfig paths to prevent it from being typechecked
* do the same for the kubernetes provider
* Fixing some typecheck errors
* Fix webapp type errors
* Update @trigger.dev/platform to 1.0.13
* Fix attw error
* Remove from/to in subscribeToRuns query params
* Add tests for the rate limit middleware and add custom JWT rate limits
* turn off webapp test parallelism
* Finish renaming jwt -> publicAccessToken and automatically give the JWT read access to the tags when using trigger
* Add changeset
* Attempt to fix unit tests in CI
* Skip running the auth rate limit middleware tests for now
* Try a beefier machine
* Try and run webapp tests separately
* Setup env vars
* Make sliding window test more reliabile
Chris Arderne
