A static observability scorer for the webapp's route entry points,
Lighthouse-style. The idea comes from evlog's `map` command, but that
tool has no Remix adapter and checks for its own logging API, so the
idea is ported rather than the tool.
It scans all 427 loader/action entry points in `apps/webapp/app/routes`
with the TypeScript compiler API and scores each against five checks:
error-classification, auth-boundary, auth-scope, request-context and
audit-trail. Current output on the real tree is **19/100** over 412
measured entry points.
```
cd internal-packages/observability-map
pnpm exec tsx src/cli.ts # terminal report
pnpm exec tsx src/cli.ts --json # machine output
pnpm exec tsx src/cli.ts api/v1/token # one entry, per-check detail
```
The two findings at the top of the fix list are real: `/auth/sso` and
`/api/v1/authorization-code` mint or exchange credentials
unauthenticated, and `/_app/orgs/:organizationSlug/settings/team`
resolves its org from a URL slug and gates each mutating branch on an
RBAC check alone, which per `apps/webapp/CLAUDE.md` is not the tenant
floor on self-hosted.
Decisions worth knowing, all with the reasoning in the README:
- The score started at 83 during development and fell to 19. Every drop
was a perverse incentive being removed, not a regression: routes were
being paid for having no error handling, two checks were reading the
same fact, suppressing a failure raised the score, and a no-op `catch
(e) { throw e }` was worth 50 points a route.
- **A mutation corpus is the tool's main defence.** 44 entries apply
semantics-preserving edits to a copy of the real route tree and assert
the score cannot rise, per route as well as globally, because a mean can
hide one route going up by taking another down. One entry runs as a live
expected failure: `try { String(0); }` with a deciding catch is a known
open hole worth 19 to 44, and it is disclosed rather than quietly
excluded.
- `audit-trail` and `request-context` are reported as headline figures
rather than one finding repeated hundreds of times. Both still count in
full where they should.
- A cohort change moves the number without anything in the codebase
getting better. Widening the sensitive cohort from 26 to 67 took the
global from 15 to 19 with no webapp change at all, so the report prints
per-check applicability and what the global would be without each one.
CI: a report-only job posts a sticky comment when a PR moves the report,
and says nothing when it does not. The package's own tests gate through
`pr_checks.yml`. The diff-scoped merge gate is still deferred until the
report has been used in anger.
524 tests plus the corpus. No runtime or dependency changes to anything
that ships.
<!-- GitButler Footer Boundary Top -->
---
This is **part 1 of 4 in a stack** made with GitButler:
- <kbd> 4 </kbd> #4485
- <kbd> 3 </kbd> #4484
- <kbd> 2 </kbd> #4483
- <kbd> 1 </kbd> #4455👈
<!-- GitButler Footer Boundary Bottom -->
Adds a gitignore rule for `**/docs/superpowers/` so locally-generated
planning and design scratch docs under that path aren't committed;
preventive only, no-op for existing tree.
Add `.worktrees/` to the repo `.gitignore`.
The pre-push hook runs `oxfmt --check .` and `oxlint .` over the whole
tree, and those tools only read the in-repo ignore files (not a user's
global gitignore). Local git-worktree checkouts placed under
`.worktrees/` therefore got linted/formatted, failing the hook on
unrelated code. Ignoring the directory keeps both tools out of worktree
checkouts. No source changes.
## Summary
Adds an in-dashboard AI agent: a chat panel, reachable from any
environment
page, that answers questions about your runs, errors, tasks, and
analytics,
diagnoses why a run failed, charts your data, reads your connected
repo's
source, and answers product and how-to questions. It is gated behind the
`hasDashboardAgentAccess` feature flag (global or per-org, default off),
so
this PR ships disabled: the launcher is hidden unless the flag is
enabled.
## Design
The agent runs as a standalone `chat.agent` Trigger task in its own
internal
package, with no access to the webapp database, Prisma, or ClickHouse.
It reads
the user's data over the public API, acting as the user via a
short-lived
delegated user-actor token minted server-side each turn (never in the
browser),
building on
[#3997](https://github.com/triggerdotdev/trigger.dev/pull/3997). The
error and analytics tools use
[#4005](https://github.com/triggerdotdev/trigger.dev/pull/4005)
and the TRQL query API.
The first turn of a new chat streams from a warm webapp route (Head
Start) while
the durable agent boots in parallel. Structured answers (a run-failure
diagnosis
card, a live chart) render through a small typed view catalog rather
than
arbitrary markup. A knowledge lane forwards product and how-to questions
to the
support assistant.
Conversation history lives in a separate Drizzle-backed store on its own
Postgres schema, kept as a display read-model so it can never corrupt
the
agent's model context.
The SDK changes add an `apiClient` option to
`chat.createStartSessionAction` and
`chat.headStart`, and keep the Head Start tool-approval tail intact
across a
custom `prepareMessages` hook so prompt caching and Head Start compose.
Speeds up and de-flakes the unit-test suite: testcontainers booted once
per vitest worker (per-test isolation kept only where a test runs
background redis work that outlives it), a duration-weighted shard
sequencer so each shard does roughly equal work, the slowest suites
split, two genuine flakes fixed (`streamBatchItems` shared-redis leak;
run-engine waits that relied on fixed sleeps), and transient DockerHub
pulls retried.
**Timings (CI, per-shard wall):** worst unit-test shard ~771s → ~294s;
packages/webapp shards ~250-270s, most internal ~190-240s. All 25 shards
green.
A shard breaks down as ~70s fixed setup (install / image-pull /
generate) + ~70s cold `^build` + the actual container tests. So the
remaining cost is mostly the tests themselves plus that fixed setup.
**Next (separate, timings):**
- **typecheck (~6m24s)** — the slowest check overall; bound by
full-graph `tsc`, not the TS version (a TS6 branch is still ~6m17s). The
real lever is **tsgo** (the Go compiler).
- Possible later: turbo CI caching could trim the ~70s cold build on
*warm* runs, but it's conditional (cold runs rebuild anyway) and doesn't
touch setup or test time — secondary.
`cli-v3` e2e and `sdk-compat` are path-gated (don't run on test-infra
changes) and already comfortably fast.
Adds Sessions, a durable, run-aware stream primitive that scopes
session.in / session.out records to a session (not a single run).
Records survive run boundaries; reconnect-from-last-event-id is built in.
Server foundation:
- New /realtime/v1/sessions/:session/:io/append + /records routes
- sessionRunManager + sessionsRepository + clickhouseSessionsRepository
- mintRunToken for short-lived per-session tokens
- s2Append retry-with-backoff + undici cause diagnostics
- /api/v[12]/packets/* exempt from customer rate limits
- BackgroundWorker schema gains taskKind enum (TASK, AGENT, SCHEDULED)
- TaskRun.taskKind column + clickhouse 029_add_task_kind_to_task_runs_v2
Core types:
- new sessionStreams, inputStreams, realtimeStreams packages in @trigger.dev/core
- session-streams-api / realtime-streams-api surface
Sessions dashboard UI (the primitive's own viewer):
- /sessions index + detail routes
- SessionsTable, SessionFilters, SessionStatus, CloseSessionDialog
- AGENT/SCHEDULED filter in RunFilters + TaskTriggerSource
Includes the sessions-primitive changeset.
Input streams enable sending typed data to executing tasks from external
callers — backends, frontends, or other tasks. This unlocks interactive
use cases like approval UIs, cancel buttons, chat interfaces, and
human-in-the-loop AI workflows where the task needs to receive data
while running.
Three consumption patterns inside a task:
* `.wait()` — Suspend the task until data arrives (process freed, most
efficient)
* `.once()` — Wait for the next message (process stays alive)
* `.on()` — Subscribe to a continuous stream of messages
One send pattern from outside:
* `.send(runId, data)` — Send typed data to a specific run's input
stream
## User-facing API
### Define a typed input stream
```ts
import { streams, task } from "@trigger.dev/sdk";
const approval = streams.input<{ approved: boolean; reviewer: string }>({ id: "approval" });
```
### Consume inside a task
```ts
export const myTask = task({
id: "my-task",
run: async () => {
// Pattern 1: Suspend until data arrives (most efficient — frees the process)
const result = await approval.wait({ timeout: "5m" });
// Pattern 2: Wait for next message (process stays alive)
const data = await approval.once().unwrap();
// Pattern 3: Subscribe to multiple messages
approval.on((data) => { /* handle each message */ });
},
});
```
### Send from outside
```ts
// From a backend (using secret API key)
await approval.send(runId, { approved: true, reviewer: "alice" });
// From a frontend (using public JWT token from trigger response)
const { send } = useInputStreamSend("approval", runId, { accessToken });
send({ approved: true, reviewer: "alice" });
```
---------
Co-authored-by: Claude <noreply@anthropic.com>
Bundle superjson and its dependency (copy-anything) during build to
avoid
ERR_REQUIRE_ESM errors on Node.js versions that don't support
require(ESM)
by default (< 22.12.0) and AWS Lambda which intentionally disables it.
- Add scripts/bundle-superjson.mjs to bundle superjson with esbuild
- Update build script to bundle vendor files before tshy compilation
- Move superjson from dependencies to devDependencies
- Update imports to use vendored bundles
Fixes#2937
<!-- devin-review-badge-begin -->
---
<a
href="https://app.devin.ai/review/triggerdotdev/trigger.dev/pull/2949">
<picture>
<source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-open-in-devin-review-dark.svg?v=1">
<img
src="https://static.devin.ai/assets/gh-open-in-devin-review-light.svg?v=1"
alt="Open with Devin">
</picture>
</a>
<!-- devin-review-badge-end -->
---------
Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: Eric Allam <ericallam@users.noreply.github.com>
- Add CLAUDE.md providing Claude Code guidance and documenting the
Claude Code skill
- Add trigger-dev-tasks skill to assist writing Trigger.dev tasks
- Add SDK rules version 4.3.0 including batch trigger v2 and debouncing
features
New batch trigger system with larger payloads, streaming ingestion,
larger batch sizes, and a fair processing system.
This PR introduces a new `FairQueue` abstraction inspired by our own
`RunQueue` that enables multi-tenant fair queueing with concurrency
limits. The new `BatchQueue` is built on top of the `FairQueue`, and
handles processing Batch triggers in a fair manner with per-environment
concurrency limits defined per-org. Additionally, there is a global
concurrency limit to prevent the BatchQueue system from creating too
many runs too quickly, which can cause downstream issues.
For this new BatchQueue system we have a completely new batch trigger
creation and ingestion system. Previously this was a single endpoint
with a single JSON body that defined details about the batch as well as
all the items in the batch.
We're introducing a two-phase batch trigger ingestion system. In the
first phase, the BatchTaskRun record is created (and possibly rate
limited). The second phase is another endpoint that accepts an NDJSON
body with each line being a single item/run with payload and options.
At ingestion time all items are added to a queue, in order, and then
processed by the BatchQueue system.
## New batch trigger rate limits
This PR implements a new batch trigger specific rate limit, configured
on the `Organization.batchRateLimitConfig` column, and defaults using
these environment variables:
- `BATCH_RATE_LIMIT_REFILL_RATE` defaults to 10
- `BATCH_RATE_LIMIT_REFILL_INTERVAL` the duration interval, defaults to
`"10s"`
- `BATCH_RATE_LIMIT_MAX` defaults to 1200
This rate limiter is scoped to the environment ID and controls how many
runs can be submitted via batch triggers per interval. The SDK handles
the retrying side.
## Batch queue concurrency limits
The new column `Organization.batchQueueConcurrencyConfig` now defines an
org specific `processingConcurrency` value, with a backup of the env var
`BATCH_CONCURRENCY_LIMIT_DEFAULT` which defaults to 10. This controls
how many batch queue items are processed concurrently per environment.
There is also a global rate limit for the batch queue set via the
`BATCH_QUEUE_GLOBAL_RATE_LIMIT` which defaults to being disabled. If
set, the entire batch queue system won't process more than
`BATCH_QUEUE_GLOBAL_RATE_LIMIT` items per second. This allows
controlling the maximum number of runs created per second via batch
triggers.
## Batch trigger settings
- `STREAMING_BATCH_MAX_ITEMS` controls the maximum number of items in a
single batch
- `STREAMING_BATCH_ITEM_MAXIMUM_SIZE` controls the maximum size of each
item in a batch
- `BATCH_CONCURRENCY_DEFAULT_CONCURRENCY` controls the default
environment concurrency
- `BATCH_QUEUE_DRR_QUANTUM` how many credits each environment gets each
round for the DRR scheduler
- `BATCH_QUEUE_MAX_DEFICIT` the maximum deficit for the DRR scheduler
- `BATCH_QUEUE_CONSUMER_COUNT` how many queue consumers to run
- `BATCH_QUEUE_CONSUMER_INTERVAL_MS` how frequently they poll for items
in the queue
### Configuration Recommendations by Use Case
**High-throughput priority (fairness acceptable at 0.98+):**
```env
BATCH_QUEUE_DRR_QUANTUM=25
BATCH_QUEUE_MAX_DEFICIT=100
BATCH_QUEUE_CONSUMER_COUNT=10
BATCH_QUEUE_CONSUMER_INTERVAL_MS=50
BATCH_CONCURRENCY_DEFAULT_CONCURRENCY=25
```
**Strict fairness priority (throughput can be lower):**
```env
BATCH_QUEUE_DRR_QUANTUM=5
BATCH_QUEUE_MAX_DEFICIT=25
BATCH_QUEUE_CONSUMER_COUNT=3
BATCH_QUEUE_CONSUMER_INTERVAL_MS=100
BATCH_CONCURRENCY_DEFAULT_CONCURRENCY=5
```
* Add createdAt filter to realtime subscribing with tags
* Filter realtime colums and expose ability to skip some columns
* Add sharding support for electric
* Use unkey cache for the created at filter caching
* Remove 2 unused indexes on TaskRun
* Run list now filters by a single runtime environment
* Remove project ID indexes
* Use clickhouse in task list aggregation queries instead of pg (keep pg for self-hosters)
* WIP clickhouse powered runs list
stuff
* Improve the query to get the latest tasks for the task list presenter
* Update the usage task list to use clickhouse
* Implement next runs list powered by clickhouse
* Add new index for TaskRun for the runs list, by environment ID
* Add runTags gin index
* Handle possibly malicious inputs
* Ignore claude settings
* Better handling not finding an environment on the schedule page
* Use ms since epoch in test, not seconds
* Remove unused function
* Fix test
* Use an env var for the realtime maximum createdAt filter duration (defaults to 1 day)
* Fixed the query builder to correct the group by / order by order
* Make sure runs.list still works
* Create small-birds-arrive.md
* Added waitpoints/tokens to the sidebar
* Added indexes to the Waitpoint time for filtering
* Begun work on `WaitpointTokenListPresenter`, the pag is a copy of the Queues page for now
* MVP of waitpoint token page
* Added status
* Expiry of timeout/ttl
* Improvements to the waitpoint table
* Improved columns and icon
* Changes from the RunTag copy on hover branch
* Fix for nested button error
* Added waitpoint tags to the DB/table
* Applied Eric’s task run tag fix (it’s live on prod in the legacy run engine branch)
* Added tags to waitpoints
* Removed todos that have been done
* Added token support for releaseConcurrency. Also added a ton of JSDocs
* Added releaseConcurrency to the API token endpoint…
* WIP on waitpoint page filters
* Fix for tags filtering
* Waitpoint filters working
* Fix for badly named function
* WaitpointPresenter used from SpanPresenter
* Waitpoint detail panel WIP
* Fix for server client hydration issue with CodeBlock
* Selected waitpoint panel
* Added a blank state
* Added waitpoint docs link
* Fix for animated number going past the target
* Fix for the queue list pagination and upgrade status
* Engine version error for waitpoint token list
* RunTag component doesn’t get squished and hover behaviour is nicer
* Associating runs with waitpoints
* Added triggered icon
* Link directly to the waitpoint
* Fix for TS error on waitpoint retrieve
* Added CopyableText component, used for waitpoint id in the table
* Removed the confetti 🎊
* Deleted some old images
* Moved some schemas/types to core. Use `id` instead of `friendlyId`
* Added wait.listTokens() function. Made some changes to the types to make it nicer
* WIP wait.retrieveToken()
* wait.retrieveToken working
* Added data to retrieve token
* Separate ApiWaitpointPresenter completely
* Added completed time to the waitpoint detail panel
* Fix for the Avatar component having SSR issues. Specify the size in rems and removed the useLayoutEffect
* Fix for applied idempotency key filter dropdown showing the id field
* Use parentheses to make sure the token list query respects idempotency key correctly
* Use the proper logger, and have a decent message with info to track the bug down
* Pass the org title into the Avatar
* Better error when failing to creating a manual waitpoint after X attempts
* Ignore /packages/cli-v3/src/package.json
* Added more logs when resuming a dependency, added the runId
* A task for reproducing a race condition with checkpoints
* Fix for doing remote image build when not self-hosting
* Set team members, alerts and schedule limits to 100m for self-hosting
* Import fix
* Set the checkpointEventId in marqs when the checkpoint is created for batchTriggerAndWait
This should fix a horrible race condition when at max concurrency
* Support custom config file names & paths
* Fix entry point paths on windows
* Support custom conditions
Add support for custom conditions (for bundling and running), to support being able to import `ai/rsc` with the “react-server” condition.
- Fixed an issue where symlinking unresolvable externals after rebuilding caused the build to hang
- Fixed an issue with external not working with subpath exports (e.g. “ai” would not match “ai/rsc”)
- Protect better against build extensions breaking builds
* Add changeset
* Fix passing CLI process.env down to the task processes
* Remove unused import
* reviving the e2e CLI tests
* Another attempt at fixing windows
* yet another windows attempt (yawa)
* Output index child stdout and stderr (yawa)
* normalize import paths for windows
* Added some logging (yawa)
* normalize the loader path as well
* Added some logging to figure out why the entry points aren’t being found on windows
* Fix for entry point detection on windows
* Normalize runner import loader path
* Normalize import paths in dev and make sure rewritten build manifest paths are correct on windows as well
* Various cleanup after windows fixes
* Remove the webapp e2e for now
* Add node10 type resolution support for subpath exports (still does not actually import in Node 10 or Browserify/Parcel)
https://github.com/andrewbranch/example-subpath-exports-ts-compat
* init using templates again but downloaded from the repo this time
* Adding init schedule example
* Support for js init
* init now working with js
* Fix issues with links in terminals that don’t support them. Also skip update check of version starts with 0.0.0
* upgrade @opentelemetry packages to the latest versions
* remove v2 only packages, will be moved to a dedicated repo
* remove more v2 code and run pnpm install
* use the npm yalt package in the webapp
* convert @trigger.dev/core to tshy
* Switch from jest to vitest in @trigger.dev/core
* Fixed core test
* move core-backend code into core subpath export
* convert @trigger.dev/sdk to tshy
* Removed hono
* move core-apps to core/v3/apps, remove core-apps, start converting cli-v3
* Fix up some of the commands
* cli now building and loadable
* using package-json-from-dist to get package version now in core and cli
* dev command WIP
* cleaned up some repetition and structure of the entry point stuff
* bringing back the background worker stuff
* Indexing of the v3 catalog
* getting closer to executing dev runs...
* centralize dev logging using event emitter
* Move indexing to it’s own entry point, simplify code
* dev runs working
* Get instrumentation to work with openai
* debugging achieved internally
* provide worker files as part of the worker creation on the server
* support for cjs and esm javascript
* Fixed timeout
* worker manifest now has the config path
* auto-upgrade config to non-deprecated alternatives
* Adding package preview release
* deployment WIP
* improve the syncEnvVars output and adapt resolveEnvVars
* WIP bun runtime
* WIP bun support
* seed tasks with the machine preset if listed in the config
* deploy run executions WIP, extracted TaskRunProcess into 1 place
* deployed tasks running and executing 🎉
* support for waits and better flushing & process cleanup
* Fixed the heartbeating
* Better warning messages
* Improve and unify the indexing between dev and deploy
* Support for external deps that need node-gyp to build
* build extensions can now install custom packages and run instructions in the image. Also prisma extension now works and also works with multiple schema files
* Add back in the main/types/module to sdk
* dev no longer is Ink/React, grace period for disconnections in dev
* Fix the changeset config
* More changeset fixes
* Remove config packages
* More changeset fixes
* Fixed typescript issues (needed to revert back to zod 3.22.3
* Fix pr_checks workflow
* Remove the prepare script
* Fixed tests and package versions
* Remove cli test script
* Remove packages from tailwind watch paths
* Add repo to public packages
* Just commit the generated files and do the building at dev time
* Try and get pkg.pr.new working
* Try again
* Fix emitDecoratorMetadata importing named export from typescript
* config file backwards compat with export const config
* Fixed issue where import errors weren’t coming through
* p-retry is a prod dep
* typescript needs to be a prod dependency for emitDecoratorMetadata
* Add better debug logging to help track down import-in-the-middle bug
* An external is only considered resolvable if it resolves to the same path as the collected external
* Fix runtime checks to allow >=18.20
* Move extensions to a new build package
* Fixed building packages in dockerfile
* Remove the e2e test from publish workflow for now
* Don’t treat pkg.pr.new versions has needing upgrading
* making sure config handleError works, and discovered path aliases don’t work in config files
* Strip empty string env vars so they accidentally override real values
* Couple of things
* Update version to use preview instead of beta
* Hopefully fix re-attempts with >30s delay
* Match socket emit messages to current latest in main
* Initial guide
* Go back to beta
* Go back to the preview, and update guide to use pr preview tags
* Go back to beta
---------
Co-authored-by: Matt Aitken <matt@mattaitken.com>
* Implement manually invokable jobs through the invokeTrigger
Also implemented a job run notification system, that will POST details of a run on completion. This combines with the task callbackUrl system to implement the invokeAndWait
* Document the invoke trigger
* batch invoke and wait
* background fetch timeouts
* Use @whatwg-node/fetch instead of the polyfilled fetch
* Fix some outdated dependencies in webapp
* Improved subtask error propogation messages
* Document the OpenAI changes and the batch invoke stuff
* Fix dequeuing jobs
* Don’t retry the OpenAI completion background task
* Added OpenAI changesets
* Use the new ResumeTaskService in ProcessCallbackTimeout as well
- Improved starting and stopping pulsar locally
- Starting to explore how the connections stuff in workflows (including how the types will work)
- Authorization now returns an organization id (to properly scope the pulsar topics to a unique workflow/org pair)
- Better handling of closing host connections in the coordinator
- Upgrade all zod to 3.20
Wes Mason
