Closes this feature request:
[https://triggerdev.featurebase.app/p/isolated-dev-sessions-for-multiple-local-trigger-dev-instances](https://triggerdev.featurebase.app/p/isolated-dev-sessions-for-multiple-local-trigger-dev-instances)
### Feature notes:
- CLI `trigger dev` works as before
- `trigger dev --branch my-branch` to create a new branch and run
against it.
- `trigger dev archive --branch my-branch` to archive (or in webapp).
- New webapp page to manage and archive dev branches, currently feature
flagged.
### Implementation details:
- No changes to data model, no backfill. `isBranchableEnvironment`
column is ignored for dev branches, we use `parentEnvironmentId IS NULL`
instead.
- `x-trigger-branch` overloaded for preview and dev branches
- New `TRIGGER_DEV_BRANCH` env var available locally.
`TRIGGER_PREVIEW_BRANCH` overloaded for child runs.
- Lots of new glue code to sanitise the branch checks.
### Rollout
- Deploy webapp/API changes (all backwards compatible)
- Manual tests on some orgs
- Deploy docs, release CLI, flip feature flag for webapp feature
### NB
- `api.v1.projects.$projectRef.environments.ts` will return
`isBranchableEnvironment: true` for all dev environments.
### Prerequisites
- [x] Typecheck will not pass until we make a new release of
`@trigger.dev/platform` and bump it here
## Summary
Prisma infrastructure failures (P1xxx-class: database unreachable, timed
out, connection dropped, engine init/panic) carry the database hostname
in their `.message`. This captures them centrally for observability and
ensures they never reach API clients verbatim.
## Design
A `$allOperations` client extension on the writer and replica clients
logs infrastructure errors with the originating model and operation,
then rethrows the **original** error unchanged — call sites that branch
on `error.code` (unique-violation idempotency, not-found handling) and
transaction retries keep working. Only infrastructure errors are logged;
routine query/validation errors (P2xxx) are left alone.
`$allOperations` can't see the transaction boundary (`$transaction` is a
client method, not an operation), so infrastructure errors surfacing
from `$transaction()` without a Prisma code — e.g.
`PrismaClientInitializationError` — are logged separately at the
transaction wrapper, where the existing coded-error path would otherwise
miss them.
`clientSafeErrorMessage()` swaps an infrastructure error's message for
`"Internal Server Error"` at the API routes that previously returned
`error.message` raw. Status codes, headers, and every non-infrastructure
message are unchanged.
## Test plan
- [x] P2002 / P2025 rethrow with code intact and are not logged
- [x] Statement errors inside `$transaction` keep their code (retry
logic intact)
- [x] Raw queries wrapped without crashing on the undefined model
- [x] A genuine connectivity failure is logged with model/operation/code
- [x] `clientSafeErrorMessage` obfuscates infra messages, preserves all
others
- [x] `pnpm run typecheck --filter webapp` (12/12)
## Note
Overlaps with #3391 (Prisma 7 migration) on
`apps/webapp/app/db.server.ts` — coordinate rebasing.
## Summary
Stamp every Prisma span with `db.datasource: "writer" | "replica"` so
traces can distinguish which client the query went through.
Both `PrismaClient` instances share the same global
`@prisma/instrumentation`, so their spans come out with identical names
and attributes today. This makes them trivially filterable.
## How
Two pieces in `apps/webapp/app/`:
1. **`v3/tracer.server.ts`** — a `DatasourceAttributeSpanProcessor`
reads an OTel context key in `onStart` and calls
`span.setAttribute("db.datasource", value)`. Registered as the first
span processor.
2. **`db.server.ts`** — `tagDatasource(datasource, client)` wraps each
`PrismaClient` with `$extends({ query: { $allOperations } })`. The
middleware sets the context key around the query and directly tags the
active span (to catch `prisma:client:operation`, which Prisma creates
before the middleware fires).
### Context-propagation gotcha
`PrismaPromise` is lazy — `query(args)` returns a thenable that only
starts when someone `.then()`s it. The naive `context.with(ctx, () =>
query(args))` restores ALS synchronously, so when Prisma's internal code
awaits the thenable later, the engine spans fire with the original ALS.
Wrapping as `async () => await query(args)` forces the `.then()` inside
the `context.with` callback, so ALS stays on our context for the engine
spans.
### Coverage
- **Tagged**: all `prisma:engine:*` (`connection`, `db_query`,
`serialize`, `query`, etc.), `prisma:client:operation`,
`prisma:client:serialize`, `prisma:client:connect`
- **Not tagged**: `prisma:client:load_engine` — one-time startup, fires
before any query
Concurrent `Promise.all([writer.x, replica.y])` correctly tags each pool
separately (ALS isolates per-Promise chain).
### Performance
One `context.with` (~200ns) and one `setAttribute` per span (effectively
free per OTel JS benchmarks) per Prisma op. Negligible against a query
path measured in milliseconds.
## Test plan
- [ ] Verify `db.datasource` appears on `prisma:engine:connection` spans
after the webapp is restarted
- [ ] Spot-check a handful of real traces carry the attribute
Sets `application_name` on the Prisma writer and replica connection
strings using the existing `SERVICE_NAME` env var, so DB load can be
attributed by service.
* Initial work on upgrading to 6.14.0
Set the output to node_modules still to make it easier
* Use ./generated Prisma folder, update types to fix issues
* Docker compose restart Clickhouse
* Prisma instrumentation update
* Docker
* Removed database dockerignore file, add generated prisma client to the top-level one
* Delete v3-catalog package.json
* Resolved pnpm lock file
* Log errors for very slow queries
* Remove variables from dequeue log message
* Continue snapshot throw json
* Waiting for deploy error removed
* Realtime ECONNRESET is expected
* Redis worker logErrors changes, removed ids
* Preview branch without a branch shouldn't log an error, it's a user provided issue
* "Task run is not in a cancellable state" isn't an error, it's expected
* "CreateCheckpointService: Child run already resumed" is expected
* "CreateCheckpointService: Batch already resumed" is expected
* "Failed to insert events, will attempt bisection" changed to info, we have errors for complete failures
* Ignore "PrismaClient error"
* Don't log Redis worker DLQ errors if we're ignoring
* "Failed to parse machine config" is fine, sometimes a config is null or undefined
* "Failed to parse machine config" for v3
* MetadataTooLargeError shouldn't log an error
* prevent infinite retry loop for unicode errors
* structured logs for prisma events
* preserve all prisma event fields
* either use structured logs or stdout, never both
* split runs repo tests
* decrease test shards to 8
* Added isPrismaRetriableError()
* Retry completeBatchTaskRunItem if they fail because of a retriable Prisma error
* Retry using Redis worker
* Handle more retriable errors. Add special condition in for race condition error
* Added Postgres connection_timeout with default 20s
* Added a simple batchTriggerAndWait example
- remove calls to findUnique, especially when including/selecting relations
- add some missing indexes
- add spans to $transaction calls to help track down long running txs
* Some notes on the new run engine
* lockfile with setup for the run engine
* Documenting where TaskRun is currently mutated, to try figure out the shape of the new system
* Added notes about how triggering currently works
* Details about when triggering happens
* Lots of notes about waitpoints
* Started scaffolding the RunEngine
* Sketch of Prisma waitpoint schema while it’s fresh in my mind
* Got Prisma working with testcontainers
* Use beforeEach/afterEach
* Simple Prisma and Redis test
* Return Redis options instead of a client
* Simplified things
* A very simple FIFO pull-based queue to check the tests working properly
* Use vitest extend
* Separate redis, postgres and combined tests for faster testing
* Some fixes and test improvements
* Pass a logger into the queue
* A queue processor that processes items from the given queue as fast as it can
* Test for retrying an item that wasn’t processed
* First draft of waitpoints in the Prisma schema
* Remove the custom logger from the test
* Added a completedAt to Waitpoint
* Notes on the flow for an execution starting
* Added redlock, moved some files around
* Starting point for the TaskRunExecutionSnapshot table
* Added relationships to TaskRunExecutionSnapshot
* Change some tsconfig
* Moved some things around
* Added some packages
* WIP on the RunQueue
* Fix for some imports
* Key producer with some tests
* Removed the nv type from the keys… it’s not useful to do global queries
* Passing unit tests for all the public key producer functions
* Some basic tests passing for the RunQueue
* Simple enqueue test working
* Enqueue and dequeue for dev is working
* Don’t log everything during the tests
* Enqueuing/dequeuing from the shared queue is working
* Tests for getting a shared queue
* The key producer sharedQueue can now be named, to allow multiple separate queues
* The key producer uses the name of the queue as the input
* Extra info in the Prisma schema
* Dequeuing a message gets the payload and sets the task concurrency all in one Lua script
* Adding more keys so we can read the concurrency from the queue
* Setting the concurrency with dequeue and enquque is working
* Improved the tests and fixed some bugs
* Acking is resetting the concurrencies
* Check the key has been removed after acking
* Nacking is working
* Changed the package to CommonJS + Node10 so it works with Redlock
* Moved the database, otel and emails packages to be in internal-packages
* Moved some Prisma code to the database package
* Started using the RunEngine for triggering
* Progress on run engine triggering, first waitpoint code
* Create a delay waitpoint
* Moved ZodWorker to an internal package so it can be used in the run engine as well as the webapp
* Web app now uses the zod worker package
* Added parseNaturalLanguageDuration to core/apps
* internal-packages/zod-worker in the lockfile
* Pass in the master queue, remove old rebalance workers code
* Add masterQueue to TaskRun
* Fixed the tests
* Moved waitpoint code into the run engine, also the zod worker
* Completing waitpoints
* An experiment to create a new test container with environment
* More changes to triggering
* Started testing triggering
* Test for a run getting triggered and being enqueued
* Removed dequeueMessageInEnv
* Update dev queue tests to use the shared queue function
* Schema changes for TaskRunExecutionSnapshot
* First execution snapshot when the run is created. Dequeue run function added to the engine
* Separate internal package for testcontainers so they can be used elsewhere
* Remove the simple queue and testcontainers from the run-engine. They’re going to be separate
* Fix for the wrong path to the Prisma schem,a
* Added the testcontainers package to the run-engine
* redis-worker package, just a copy of the simple queue for now
* The queue now uses Lua to enqueue dequeue
* The queue now has a catalog and an invisible period after dequeuing
* Added a visibility timeout and acking, with tests
* Added more Redis connection logging, deleted todos
* Visibility timeouts are now defined on the catalog and can be overridden when enqueuing
* Dequeue multiple items at once
* Test for dequeuing multiple items
* Export some types to be used elsewhere
* Partial refactor of the processor
* First stab at a worker with concurrency and NodeWorkers
* Don’t have a default visibility timeout in the queue
* Worker setup and processing items in a simple test
* Process jobs in parallel with retrying
* Get the attempt when dequeuing
* Workers do exponential backoff
* Moved todos
* DLQ functionality
* DLQ tests
* Same cluster for all keys in the same queue
* Added DLQ tests
* Whitespace
* Redis pubsub to redrive from the worker
* Fixed database paths
* Fix for path to zod-worker
* Fixes for typecheck errors, mostly with TS versions and module resolution
* Redlock required a patch
* Moved the new DB migrations to the new database package folder
* Remove the run-engine package
* Remove the RunEngine prisma schema changes
* Delete triggerTaskV2
* Remove zodworker test script (no tests)
* Update test-containers readme
* Generate the client first
* Use a specific version of the prisma package
* Generate the prisma client before running the unit tests
* Add task run hierarchical relationships to the database
* Add depth and related runs to the retrieve run API response
* Remove prisma optimize
* restructure the migrations to create the index concurrently
* Delete these tsbuildinfo files
* Fix type error by adding depth to the run list presenter
* Cleanup the task hierarchy, share more code
* Remove some fields from the list run response
* Added traceId index to TaskEvent
* Get less data and get the trace from the run
* Use the read replica in the EventRepository
* Use the regular client to get the run because the redirect from test can be very fast
* Fix for “overridden” spelling mistake
* Make the task function names grey instead of yellow
* The task page now shows each task once
* Blankline
* WIP on activity bars
* Correct size of the graphs
* Add data for all days, even if there are no runs
* Activity graph is working and has a tooltip
* No activity label
* Got the bar bg working
* Activity chart now has a bg
* Added a compound id to make the activity graphs appear faster
* Improvements to the activity graph
* Tried to disable the bg from animating
* Added read replica support to the app, using Nick’s code from his batching PR
* Added the read replica to BasePresenter
* Added the queued and running columns to the Tasks list
* Added avg duration column
* Included completed but failed runs in the average duration
* Get rid of the old useDevEnvironment hook and remove API keys from the projects query
* Unified getting environment info to using a function and doing it in presenters
* Removed the path from the tasks table
* Onboarding for dev for 2nd+ user. Improved environment sorting.
* Removed log
* Improved the spacing
* Enable staging for new v3 projects
* add custom validation to db url env vars
* extract schema from db url and use in all raw queries
* use qualified names in scheduling raw queries
* cook a few raw queries
* Added missing raw query schema specifier to DeploymentListPresenter
---------
Co-authored-by: Matt Aitken <matt@mattaitken.com>
* Pass subscription status into the usage bar
* Page navigation spinner is now blue (was a bit subtle before)
* Better logging of db queries, this will be commented out before the PR is merged
* Select only the required fields
* We don’t need the member count for each org
* WIP redirecting with projectId in session
* Switching projects is now working, without duplicating the project query
* Removed logs in revalidate function
* Removes some unused imports
* ProjectPresenter: removed lots of unused db selects
* Root use defaultShouldRevalidate, not just true
* Simplified the job list query and separated the deleting job modal query
* Use requireUserId instead of requireUser wherever possible
* EventListPresenter query simplified
* Simplified the RunListPresenter query
* Disable query logging
* Use the latest updated version for the job list table
* We need to use the org presenter on the select plan page
* Implement manually invokable jobs through the invokeTrigger
Also implemented a job run notification system, that will POST details of a run on completion. This combines with the task callbackUrl system to implement the invokeAndWait
* Document the invoke trigger
* batch invoke and wait
* background fetch timeouts
* Use @whatwg-node/fetch instead of the polyfilled fetch
* Fix some outdated dependencies in webapp
* Improved subtask error propogation messages
* Document the OpenAI changes and the batch invoke stuff
* Fix dequeuing jobs
* Don’t retry the OpenAI completion background task
* Added OpenAI changesets
* Use the new ResumeTaskService in ProcessCallbackTimeout as well
- Use a better way of getting the latest job run number to increment
- Make the CreateRunService transaction more reliable
- Invoke dispatchers in parallel
- No longer swallow prisma errors in $transaction
* WIP job run performance improvements
- Added a `perf` tool to better measure job run performance under heavy load
- Removed `runFinished` job (not really needed)
- startQueuedRuns now uses a jobKey with replace
- Fixed an issue with ZodWorker when using jobKey
* Publish improvement docker images
* fixed the improvement docker publishing
* Downgrade back to prisma 4.16.0 because 5.1.x broke docker builds
* Changes to how queued runs work
- Split the worker into two different workers, one dedicated to performRunExecution
- Schedule performRunExecution in a single place, with a queue and using a round robin manually controlled concurrency
- Remove startQueuedRuns
- All runs are queued before they are started
- Setting the worker maxPoolSize to the same as the worker concurrency
- Starting to be able to split the docker image
* Remove queue name from startRun graphile job
* Make the prisma connection pool stuff configurable through env vars
* Hardcode (for now) the max concurrent runs limit
* Rewrite performRunExecution to be more performant
PerformRunExecutionV2:
- Does not create and manage jobRunExecution records
- Does not reimplement retrying, uses graphile worker retrying instead
I’ve kept around PerformRunExecutionV1 so this works when deploying. Definitely needs LOTS of testing
* Fix issues with cached tasks
- Limit the size of the cached tasks sent when executing a run, using the knapsack problem dynamic programming approach
- Actually USE the cached tasks in IO by using the idempotencyKey instead of the task ID
- Remove output from all logs
- Added a stress test job catalog
* Forgot to commit the logger updates
* Never log connectionString
* Login to docker hub to get around rate limits
* Add additional logging to the graphile workers
* Fix the *_ENABLED env vars
* Allow adding and removing jobs to be done from the webapp
* Don’t set the job to failed if it’s being retried
* Deprecated queue options in the job and removed startPosition. Now using the job/env combo as the job queue name
* Dequeung jobs doesn’t check if the runner is initialized
* Fixed issues with retrying a run getting stuck on a cancelled task, and errors from parsing the results of dequeing a job
* Remove queued round robin thing that isn’t used anymore
* Added slack to job catalog
* Better forwards compat
* Added long delay
* Fixed lock file
* Setup project-wide prettier
* Remove old workspace file
* Remove old debugging directives
* New top-level .prettierignore
* Updated Prettier config settings
* Contrubuting guide: Fix for some bad code blocks
* Added more ignores
* Improved the format script command
* printWidth set to 100
* Formatted entire repo (pnpm run format)
Chris Arderne