Single redis-worker changeset under its original filename.
references/stress-tasks, scripts/mollifier-challenge, _ops/ come out
of the tree — kept locally as working artefacts.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase 4's audit found two Zod drifts reactively (idempotencyKey: null
and parentId: undefined). This script proactively sweeps every public
SDK method with a buffered branch by calling them through the real
@trigger.dev/core apiClient — zodfetch's schemas execute against each
response, so any drift now fails the audit. The existing
mollifier-challenge shell scripts only do jq structural checks, which
miss schema-level drift like null-vs-undefined or
optional-vs-nullable mismatches.
Covers nine methods against a fresh buffered run each (separate runs
for destructive ones so they don't interfere): retrieveRun,
retrieveRunTrace, retrieveSpan, listRunEvents, addTags,
updateRunMetadata, replayRun, rescheduleRun, cancelRun.
Manually verified against the live local webapp — all nine pass with
no drift surfaced. The audit is reusable as a smoke-check before each
prod rollout.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Four scenarios that the unit-test stubs and the cold-gate burst (04)
don't exercise. All green against a live webapp with the claim system
wired in.
16 — claimant-crash recovery. Planted "pending" claim externally, fired
5 same-key triggers (all polling), DEL'd the claim mid-poll. Verifies
the retry-SETNX path: 1 waiter wins, 4 polling losers resolve to the
same runId.
17 — stale-runId recovery. Claim resolves to a runId that exists in
neither PG nor the buffer. IdempotencyKeyConcern logs a warn and falls
through; the trigger creates a fresh run. Validates the
"resolved-but-not-findable" branch.
18 — claim safety-net timeout. Long-lived "pending" claim with no
publisher; same-key trigger polls until safetyNetMs elapses, returns
503. Validates the wait/poll budget caps.
19 — burst → drain → re-burst with the same key. First burst converges
via the claim (drainer ON, materialises post-burst); second burst
resolves via PG-findFirst (existing IdempotencyKeyConcern behaviour),
bypassing the claim entirely. Validates that the new claim path
doesn't break the existing PG-cache resolution that takes over once
the run is in PG.
Closes the PG+buffer race during the mollifier gate-transition window.
Plan: _plans/2026-05-21-mollifier-idempotency-claim.md
redis-worker:
- New MollifierBuffer methods + atomic Lua: claimIdempotency
(SETNX-with-TTL returning claimed/pending/resolved), publishClaim,
releaseClaim, readClaim. Separate key namespace mollifier:claim:*
to keep isolated from the B6a buffered-side mollifier:idempotency:*
lookup.
webapp:
- New apps/webapp/app/v3/mollifier/idempotencyClaim.server.ts wraps
the buffer primitives with a wait/poll loop. Returns claimed /
resolved / timed_out. Fail-open on buffer outage so a transient
Redis blip doesn't 500 the trigger hot path.
- IdempotencyKeyConcern.handleTriggerRequest now consults the claim
after the existing PG-findFirst + buffer.lookupIdempotency cache
checks miss. Skipped for resumeParentOnCompletion (triggerAndWait
bypasses the mollifier gate via F4 and is PG-canonical anyway). When
we own the claim, the result's new `claim` field signals the caller
to publish on success / release on failure.
- RunEngineTriggerTaskService.callV2 wraps the trigger pipeline in a
try/catch that publishes the winning runId or releases the claim
depending on outcome. The publish updates the claim key so waiters
polling for our key resolve to our runId.
Validated end-to-end:
- scripts/mollifier-challenge/04-idempotency-collision.sh runs
cold-gate (no pre-warm) with 30 concurrent same-key triggers and
converges on 1 runId / 1 isCached:false. Before this fix the same
test produced 2 race-winners.
- 13 unit tests covering claimed/resolved/pending/timed_out paths,
fail-open behaviour, abort signal, publishClaim, releaseClaim.
- All 94 webapp mollifier tests still green.
- 03: assert tags via `.tags` (the retrieve API field) not `.runTags`.
- 04, 13: pre-warm the gate so the same-key burst all reaches the
buffer. Without the pre-warm, the first 1-2 same-key triggers land
in PG during gate-transition and create a second race-winner
(separate concern from B6's buffer-side dedup, surfaced for
follow-up).
The control trigger must land BEFORE the burst — once the burst trips
the gate, the hold-down (TRIGGER_MOLLIFIER_HOLD_MS) keeps mollifying
subsequent triggers in the same env until the marker expires. The
previous order (burst → control) caught the control in the hold-down
and false-positive-failed.
Also: clear mollifier:* Redis keys between runs. Stale LIST-typed
queue keys from before the B1 ZSET migration cause WRONGTYPE errors
on accept.
Script 15 exercises mutateWithFallback's safety-net cap: HSET-forces the
entry into each of the three busy-triggering states (DRAINING, FAILED,
materialised=true) and verifies the mutation API returns 503 within the
~2s safetyNetMs window. Also asserts the wait is bounded — fails if the
response comes back faster than 1s (would imply busy wasn't hit) or
slower than 5s (would imply the wait is unbounded).
The remaining uncovered slice of busy is the "drainer succeeds mid-wait
and pgMutation runs" branch, which requires injecting a PG row from
outside the webapp during the wait window. Documented as unit-test-only.
Adds per-endpoint contract checks beyond the status-only comparison:
- Read endpoints assert response shape (trace.traceId present;
events/attempts arrays; metadata-get { metadata, metadataType }
keys; retrieve-v3 carries id + taskIdentifier + status). The result
endpoint explicitly asserts 404 — its accidental-but-correct
pre-Phase-A behaviour is now the locked contract.
- Mutation endpoints get a read-back assertion: after PUT metadata,
re-read and confirm the snapshot reflects the patch. After POST
tags, retrieve and confirm runTags contains the new tag. Catches
the case where the API returns 200 but the snapshot didn't actually
patch.
- Replay asserts the response carries a new run_-prefixed id.
- New listing probe: hits /api/v1/runs and asserts the buffered runId
is present in the page. Locks in Phase E's listing-merge behaviour.
Script remains backwards-compatible — same exit codes, same env-var
contract. Drift count now reflects shape violations alongside status
divergences.
Master plan and five locked sub-design docs covering the API parity work:
- mollifier-api-parity.md — endpoint inventory, invariant, phased TDD plan.
- mollifier-listing-design.md (Q1) — ZSET buffer, compound cursor, no banner.
- mollifier-replay-design.md (Q2) — single code path, PG-or-buffer resolution.
- mollifier-mutation-race-design.md (Q3) — wait-and-bounce with safety net.
- mollifier-cancel-design.md (Q4) — mark_cancelled + drainer bifurcation.
- mollifier-idempotency-design.md (Q5) — keys in both stores symmetrically.
Plus the original phase-3 plan it builds on and the bash parity script
that surfaced the gaps and acts as the regression guard during
implementation.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary
Two papercuts new contributors hit running this repo locally:
1. Fresh clones default to v1 (Redis-only) realtime streams, so Sessions
and `chat.agent` error with `"S2 configuration is missing"`, even though
the `s2` service is already in `docker/docker-compose.yml` and pre-seeds
a `trigger-local` basin. Wire `REALTIME_STREAMS_S2_*` to it in
`.env.example` so the new-contributor flow just works. (Also drop the s2
healthcheck: the image is distroless, so the `wget` check always reports
unhealthy.)
2. Two clones can't both run `pnpm run docker` because ports, project
name, and container names are all hardcoded. Parameterize every host
port as `${VAR:-default}`, drive the project name via
`COMPOSE_PROJECT_NAME` (with a top-level `name:` field as the default),
prefix container names with `${CONTAINER_PREFIX:-}`, and pass
`--env-file .env` so compose reads the same root `.env` the webapp does.
The "Running multiple instances side by side" block in `.env.example`
lists every overridable knob.
Also split the optional services (`electric-shard-1`, `ch-ui`,
`toxiproxy`, `nginx-h2`, `otel-collector`, `prometheus`, `grafana`) into
`docker-compose.extras.yml` behind a new `pnpm run docker:full` script.
The core stack keeps everything the webapp actually needs to boot:
postgres, redis, electric, minio, clickhouse + migrator, s2-lite.
Defaults match every previous hardcoded value, so existing setups keep
working without touching `.env`.
## Test plan
- [x] `pnpm run docker` on a clean clone brings up the core services on
the standard ports under the `triggerdotdev-docker` project name.
- [x] Setting `COMPOSE_PROJECT_NAME=triggerdotdev-docker-alt` + the
`*_HOST_PORT` overrides in `.env` brings up a second stack alongside the
default one with no port or container-name clashes.
- [x] Webapp boots cleanly against the default `.env.example` values;
`/healthcheck` returns 200, no S2 errors.
- [x] s2-lite basin `trigger-local` accepts an append + read via the
same REST endpoints the webapp uses.
- [x] `pnpm run docker:full` brings up the optional services alongside
the core ones in the same project.
Follow-up to the v4.4.5 release incident where the release PR (#3406)
was merged with a stale lockfile and stale Chart.yaml, breaking npm +
helm releases. The two automation jobs (`update-lockfile`,
`bump-chart-version`) got cancelled mid-flight by `cancel-in-progress`
when the merge fired the workflow again on `main`.
This restructures `changeset:version` so all the post-version-bump
fixups happen in the same script and end up in a single atomic commit on
`changeset-release/main`, via `changesets/action`'s normal commit step.
Pattern borrowed from Cloudflare workers-sdk, Astro, shadcn/ui.
## Before
```
push: main
└── release-pr (changeset version → bumps package.jsons, opens PR)
└── update-lockfile (separate job, separate commit)
└── bump-chart-version (separate job, separate commit)
```
Three jobs, three commits to the release branch.
## After
```
push: main
└── release-pr
└── changesets/action runs:
changeset version
pnpm install --lockfile-only
node scripts/bump-helm-chart.mjs
node scripts/cleanup-server-changes.mjs
...all staged and committed as ONE commit by the action
```
One job, one commit.
- Add .server-changes/ convention for tracking server-only changes
- Create scripts/enhance-release-pr.mjs to deduplicate and categorize
changeset PR body
- Create scripts/generate-github-release.mjs to format unified GitHub
release body
- Change release.yml to create one unified GitHub release instead of
per-package releases
- Add update-release job to patch Docker image link after images are
pushed to GHCR
- Update changesets-pr.yml to trigger on .server-changes, enhance PR
body, and clean up consumed files
- Document server changes in CLAUDE.md, CONTRIBUTING.md, CHANGESETS.md,
and RELEASE.md
There’s an edge case that means runs can end up in the
currentConcurrency set when they’re not in the correct state for
execution. This means they will be permanently stuck in queued.
Given an environmentId this will fix those runs.
This is a temporary fix while we permanently fix the issue.
---------
Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: Eric Allam <ericallam@users.noreply.github.com>
## Summary
- Adds `IF NOT EXISTS` to the migration that adds
`idempotencyKeyOptions` column to prevent errors if the column already
exists
## Migration Checksum Fix
If you've already applied the previous version of this migration, you'll
need to update the checksum in your `_prisma_migrations` table to match
the new migration file.
**Previous checksum:**
`f8876e274e3f7735312275eb24a9c4b40f512ac12a286b2de3add47f66df5b27`
**New checksum:**
`0620a914ddbaf01279576274432e51c41f41502cd4c8de38621625380750e397`
### Fix instructions
Run this SQL command against your database:
```sql
UPDATE "_prisma_migrations"
SET checksum = '0620a914ddbaf01279576274432e51c41f41502cd4c8de38621625380750e397'
WHERE migration_name = '20260116154810_add_idempotency_key_options_to_task_run';
```
This updates the stored checksum to match the modified migration file,
allowing future migrations to proceed without checksum mismatch errors.
## Test plan
- [x] Verified migration applies cleanly on fresh database
- [ ] Verified checksum update works on database with previous migration
applied
🤖 Generated with [Claude Code](https://claude.com/claude-code)
* WIP clickhouse package with test containers setup
* More clickhouse client setup now with otel and real tests, and the v1 of raw run events
* Add some additional columns to raw_run_events_v1
* WIP runs dashboard service
* Create a new run engine event bus event for the runs dashboard to hook into
* Track run events in the run engine
* make sure engine v1 runs get synced to CH
* Update the attemptNumber of v3 task runs
* Restructure the run events to be more sparse
* emit more stuff
* Setup replication package
* scaffold the replication package
* replication wip
* resolve conflicts
* more replication stuff
* Add ability to drop the replication slot completely on teardown
* Use the new single replacingmergetree task events table for replication
* get it working
* insert payloads into their own table only on insert and then join
* prepare for using clickhouse cloud and now running ch migrations during boot in the entrypoint.sh
* Handover WIP and tests
* Testing the replication service
* Remove the runs dashboard stuff that we aren't using anymore
* Added a test for large payloads
* hacky typecheck fix
* Fix new internal package typecheck issues and start adding telemetry to the replication service
* tracing over spans, some other improvements
* Improvements to the runs replication service, now ready for testing
* Some fixes and cleanups
* Don't need this code anymore
* move transaction types into the runs replication service
* only send spans where there are transaction events
* A couple of suggested tweaks
* remove dead code
* rename managed to shared runtime manager
* rename to resolve waitpoint for clarity
* add resolver id helper
* store and correctly resolve waipoints that come in early
* fix ipc message type change
* branded type for resolver ids
* add fixme comments
* remove more unused ipc schemas
* fix entitlement validation when client doesn't exist
* restore hello world reference workspace imports
* runtime manager debug logs
* prefix engine run logs
* managed run logger accepts nested props
* runtime suspendable state and improved logs
* require suspendable state for checkpoints, fix snapshot processing queue
* add terminal link as cli module so we can more easily patch it
* apply cursor patch
* add license info
* remove terminal-link package and add deprecation notice
* remove old patch
* remove terminal-link from sdk
* rename snapshot module
* add cli test tsconfig
* add run logger base type
* add snapshot manager tests
* fix cli builds
* improve QUEUED_EXECUTING test
* changeset
* make testcontainers wait until container has stopped
* require unit tests for publishing again
* avoid mutation during iteration when resolving pending waitpoints
* improve debug logs and make them less noisy
* always update poller snapshot id for accurate logs
* detach task run process handlers
* check for env overrides in a few more places and add verbose logs
* log when poller is still executing when we stop it
* add supervisor to publish workflow
* always print full deploy logs in CI
* Revert "avoid mutation during iteration when resolving pending waitpoints"
This reverts commit 87b0ce1e5b.
* disable pre
* print prerelease script errors
* Revert "disable pre"
This reverts commit 9403409637.
* misc fixes
* better debug logs
* add snapshots since methods and route
* prep for snapshots since
* improve deprecated execution detection
* update supervisor and schema
* properly log http server errors
* detect restore after failed snapshot fetch
* run and snapshot id can be overridden
* fix restore detection
* fix deprecation checks, move into snapshot manager
* less logs
* rename snapshot manager stop
* restore detection was moved into snapshot manager
* fix notifier logs
* make runtime manager status a debug log
* no need to attach runtime status twice
* findUnique -> findFirst
* sort snapshots by created at everywhere
* Mutliple streams can be now consumed simultaneously
* Update prerelease script
* Add changeset
* Make it core
* Handle API error responses when streaming
* Fix resolving external packages that are ESM only by falling back to mlly resolvePathSync. This will fix mupdf
* when publishing a prerelease and aborting, clear the git stage
* Attempt to fix false package mismatch warnings
* Add changeset
* Add ability to test update checks in prerelease packages
* Resolve the trigger.dev package based on the package.json dir
* Try this
* Don’t use the version module, just resolve the packageJson
* One more dirname
* Comment
* Remove the version export because we aren’t using it anymore
* Fixed empty env vars overriding in dev runs
* Don’t import package.json anymore
* fix node10 moduleResolution in @trigger.dev/core
* Support self-hosters pushing to a custom registry when running deploy
* dev: Fixed stuck runs when a child run fails with a process exit
* Make some doc notes about known issues and docker hub private repos
* Fix --project-ref when running deploy
* Fix —config option when deploying
* Fixing the flushing/killing process with the new build system
* Add monorepo-react-email e2e test fixture
* Fix issue with emitDecoratorMetadata and tsconfigs with extends
* Got the emit decorator metadata fixture working
* Fixed typechecking yarn e2e CLI tests in monorepos
* Add remote forced externals system, in case we come across another package that cannot be bundled (spurred on by header-generator)
* Remote externals now powered by JSON Hero to be easier to update
* resolve config source files
* Add a —javascript option to init, defaults to typescript
* Add support for prisma typed sql
* Remove msw and retry.interceptFetch
* Add missing code to the openai retries example
* Don’t generate the v3 catalog prisma client during CI
* Fixed v3-catalog task imports
* Remove interceptor usage in task file
* Only import import-in-the-middle hook if there are instrumented packages
* Fix yarn.lock file
* upgrade @opentelemetry packages to the latest versions
* remove v2 only packages, will be moved to a dedicated repo
* remove more v2 code and run pnpm install
* use the npm yalt package in the webapp
* convert @trigger.dev/core to tshy
* Switch from jest to vitest in @trigger.dev/core
* Fixed core test
* move core-backend code into core subpath export
* convert @trigger.dev/sdk to tshy
* Removed hono
* move core-apps to core/v3/apps, remove core-apps, start converting cli-v3
* Fix up some of the commands
* cli now building and loadable
* using package-json-from-dist to get package version now in core and cli
* dev command WIP
* cleaned up some repetition and structure of the entry point stuff
* bringing back the background worker stuff
* Indexing of the v3 catalog
* getting closer to executing dev runs...
* centralize dev logging using event emitter
* Move indexing to it’s own entry point, simplify code
* dev runs working
* Get instrumentation to work with openai
* debugging achieved internally
* provide worker files as part of the worker creation on the server
* support for cjs and esm javascript
* Fixed timeout
* worker manifest now has the config path
* auto-upgrade config to non-deprecated alternatives
* Adding package preview release
* deployment WIP
* improve the syncEnvVars output and adapt resolveEnvVars
* WIP bun runtime
* WIP bun support
* seed tasks with the machine preset if listed in the config
* deploy run executions WIP, extracted TaskRunProcess into 1 place
* deployed tasks running and executing 🎉
* support for waits and better flushing & process cleanup
* Fixed the heartbeating
* Better warning messages
* Improve and unify the indexing between dev and deploy
* Support for external deps that need node-gyp to build
* build extensions can now install custom packages and run instructions in the image. Also prisma extension now works and also works with multiple schema files
* Add back in the main/types/module to sdk
* dev no longer is Ink/React, grace period for disconnections in dev
* Fix the changeset config
* More changeset fixes
* Remove config packages
* More changeset fixes
* Fixed typescript issues (needed to revert back to zod 3.22.3
* Fix pr_checks workflow
* Remove the prepare script
* Fixed tests and package versions
* Remove cli test script
* Remove packages from tailwind watch paths
* Add repo to public packages
* Just commit the generated files and do the building at dev time
* Try and get pkg.pr.new working
* Try again
* Fix emitDecoratorMetadata importing named export from typescript
* config file backwards compat with export const config
* Fixed issue where import errors weren’t coming through
* p-retry is a prod dep
* typescript needs to be a prod dependency for emitDecoratorMetadata
* Add better debug logging to help track down import-in-the-middle bug
* An external is only considered resolvable if it resolves to the same path as the collected external
* Fix runtime checks to allow >=18.20
* Move extensions to a new build package
* Fixed building packages in dockerfile
* Remove the e2e test from publish workflow for now
* Don’t treat pkg.pr.new versions has needing upgrading
* making sure config handleError works, and discovered path aliases don’t work in config files
* Strip empty string env vars so they accidentally override real values
* Couple of things
* Update version to use preview instead of beta
* Hopefully fix re-attempts with >30s delay
* Match socket emit messages to current latest in main
* Initial guide
* Go back to beta
* Go back to the preview, and update guide to use pr preview tags
* Go back to beta
---------
Co-authored-by: Matt Aitken <matt@mattaitken.com>
* WIP
* Allow marqsv2 and v2 graphile to run in parallel
* Fix missing GraphileLogger import
* Fixed heartbeat after rebase
* Replace postgres based run counters with redis ones with a backfill
* Add back in the graphile logger
* Remove duplicate visibility timeout calls
* Clamp simple weighted strategy to max of 5
Dan Sutton