Commit Graph

51 Commits

Author SHA1 Message Date
Eric Allam 5ba8557a51 chore(webapp,core): remove the end-of-life v3 (engine V1) execution stack (#4236)
## Summary

v3 (the engine that ran the SDK v3 era, internally
`RunEngineVersion.V1`) is end-of-life. Following the removal of the v3
execution apps
([#4194](https://github.com/triggerdotdev/trigger.dev/pull/4194)) and
the legacy dev websocket
([#4198](https://github.com/triggerdotdev/trigger.dev/pull/4198)), this
removes the remaining v3 execution stack from the server.

Clients still on v3 (an old SDK or CLI that has not upgraded) keep
getting a clear "upgrade to v4" response. Triggers, batch triggers,
reschedules, and deploys that resolve to v3 are rejected with a graceful
4xx pointing at the migration guide, never a 5xx, so a stale client
cannot affect server health. Self-hosted instances still running v3
should stay on the 4.5.x release line until they migrate.

## What is removed

- The MarQS queue and its shared/dev queue consumers.
- The v3 socket.io namespaces (coordinator, provider, shared-queue) and
the v3 run lifecycle services (attempt, checkpoint, and batch-resume).
- The graphile-worker background job system; all live jobs already run
on `@trigger.dev/redis-worker`.
- The `DEPRECATE_V3_ENABLED` flag: v3 is now rejected unconditionally,
so the flag is gone.
- Unused v3 exports from `@trigger.dev/core` (the `v3/zodNamespace`
subpath and the legacy socket message catalogs) and the now-dead MarQS
environment variables.

## What stays

The v4 engine is untouched. The graceful v3 rejection boundary stays,
`determineEngineVersion` still detects a v3 project so it can reject it,
and the batch service plus batch-completion worker stay for current
clients. Live queue concurrency limits and metrics now read from the v4
run engine instead of MarQS, and a brand-new dev environment now
defaults to v4.



## Dependency cleanup

Removes webapp dependencies left unused by this change: `seedrandom` and
`semver` (only the removed v3 code used them) plus a set that was
already dead, their orphaned `@types` packages, and two dead files. Adds
a `knip:deps` script and a `knip.json` config so unused dependencies can
be found the same way going forward.
2026-07-13 11:32:06 +01:00
claude[bot] 105f48927d fix(release): populate changelog and server-changes on release/dispatch (#4204) 2026-07-09 17:46:39 +01:00
Chris Arderne ef3eadab3c chore: back to github runners (#4168) 2026-07-06 16:22:31 +01:00
Daniel Sutton 092b9ef07a fix(run-ops): DNS-safe, sortable base32hex run id (replace base62 KSUID) (#4154)
## Problem

The run-ops split mints NEW-store run ids as **27-char base62 KSUIDs**.
The supervisor writes the run id into the Kubernetes pod name
(`runner-<id>`), and pod names must be DNS-1123 labels (lowercase
`[a-z0-9-]`) — so uppercase base62 ids make k8s reject the pod (422) and
**those runs never launch** (they loop in `PENDING_EXECUTING` until the
heartbeat-stall handler nacks them, forever). `.toLowerCase()` can't fix
it: base62 has both `A`(10) and `a`(36) as distinct symbols, so folding
collides distinct ids and destroys sort order.

## Fix: change the encoding, not the structure

Mint a **26-char lowercase base32hex** run id:

```
run_<24-char base32hex core><region char><version char>
      [ 6-byte ms timestamp ][ 9 CSPRNG bytes ]
```

- **base32hex** (RFC 4648 §7, alphabet `0-9a-v`): lowercase,
order-preserving, DNS-safe; 15 bytes → exactly 24 chars, no padding.
Hand-rolled encode/decode (no new dependency).
- **48-bit ms timestamp** in the leading bytes → plain string sort ==
creation order at millisecond resolution.
- **72 bits CSPRNG** entropy; PK unique constraint is the backstop (no
retry loop).
- **region / version** are raw positional chars (read via one `charAt`
before decoding/routing), version = `"1"`.

DNS-safe from birth and hyphen-free, so **firekeeper is unchanged** —
`runner-<id>-attempt-N` → strip `runner-`, cut at first hyphen still
recovers the exact id incl. region+version.

## Residency discriminator: length → version char

`classifyKind`/`classifyResidency` (`runOpsResidency.ts`) previously
distinguished NEW vs LEGACY by **id length**. That gets ambiguous with a
third format. It now discriminates on the **version char at a fixed
position** (`isRunOpsIdBody`: 26 chars, `[25] === "1"`, base32hex
alphabet) → NEW; everything else → LEGACY. Total, never throws. The
`Residency` (NEW/LEGACY) contract the routing store consumes is
unchanged; the `"ksuid"` `ResidencyKind` label is retained only because
it's the persisted `runOpsMintKsuid` feature-flag value.

## Scope / verification

- Generator + discriminator in `@trigger.dev/core` isomorphic; mint path
+ all id-shape call sites swept (~40 webapp files); changeset added
(`@trigger.dev/core` patch).
- Core unit tests (encode/decode round-trip + property, generator shape,
ms sort-order incl. intra-second, parse partitioned-vs-legacy,
firekeeper round-trip): **24 pass**. `@trigger.dev/core` builds; webapp
typechecks; format/lint clean.

## Open decisions (flagged, not silently chosen)

1. **Backward-compat**: existing 27-char base62 KSUID runs now classify
LEGACY. On test cloud these are the broken/looping runs that never
completed, so this is acceptable — but worth a conscious call before
prod. No transitional length-recognition added (keeps the discriminator
clean).
2. **Storage collation**: the sort guarantee is byte-order — if the
run-ops id column is `TEXT` with default locale collation it's silently
not honored. Confirm whether `COLLATE "C"` / `BYTEA` is needed on the
run-ops schema.
3. **Region sourcing** wiring — see `regionCharForRegion` /
`REGION_CODES`.


---

## ⚠️ Required migration — deploy in lockstep

This PR renames a persisted feature-flag key/value and an env var. These
are **not** changed by the code alone and must be migrated when this
deploys, or affected orgs silently fall back to `cuid` minting (no crash
— `defaultValue: "cuid"`):

1. **Env var** (terraform): `RUN_OPS_MINT_KSUID_ENABLED` →
`RUN_OPS_MINT_ENABLED` (carry the value over).
2. **DB** `organization.featureFlags`: migrate both the key and value
together:
   - key `runOpsMintKsuid` → `runOpsMintKind`
   - value `"ksuid"` → `"runOpsId"`

Until an org's flag row is migrated, its `runOpsMintKind` lookup misses
and it mints `cuid` (legacy) — so no NEW-store ids for that org until
the data lands.

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-05 10:05:54 +01:00
Chris Arderne c7861be520 chore: activate no-unused-vars and import linters (#4096)
Once this is merged, oxlint is at a pretty sensible baseline.

**Enable `no-unused-vars`, `typescript/consistent-type-imports`, and
`import/no-duplicates` lint rules**

Turns on three previously-disabled oxlint rules across the monorepo and
fixes all violations:

- **`no-unused-vars`** – enabled as an error with standard ignore
patterns: unused function arguments are ignored by default (`args:
"none"`), variables/caught errors/destructured array elements prefixed
with `_` are allowed, and rest siblings are permitted.
- **`typescript/consistent-type-imports`** – enforced as an error; all
type-only imports now use the `import type` syntax.
- **`import/no-duplicates`** – enforced as an error; duplicate import
statements from the same module have been merged.

The remaining commits clean up the violations found across the codebase:
removing unused variables/imports/type aliases, adding `_` prefixes to
intentionally unused bindings, fixing duplicate imports, and converting
value imports to `import type` where appropriate.
2026-07-02 11:37:05 +01:00
Chris Arderne 76833317d3 fix: blacksmith testbox test scripts (#4074)
🚀 Publish Trigger.dev Docker / units (push) Failing after 10m40s
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 10m41s
🚀 Publish Trigger.dev Docker / publish-worker (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker-v4 (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / scan-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / 📣 Dispatch main image (push) Has been cancelled
2026-06-29 17:13:17 +01:00
Chris Arderne 243d52bfd7 chore: add testbox workflows and scripts (#4071)
Needed to trigger Blacksmith Testbox workflows.
2026-06-29 15:08:04 +01:00
Chris Arderne b54201f986 chore: switch to oxfmt, oxlint - add ci checks (#3977) 2026-06-26 12:19:29 +01:00
Eric Allam 422d8da339 fix: keep all of a PR's changesets in the release PR summary (#4031)
## Summary

The auto-generated changeset release PR (`changeset-release/main`)
builds its `## Improvements` / `## Bug fixes` summary with
`scripts/enhance-release-pr.mjs`. The script deduplicated summary
entries by PR number, so when a single PR shipped more than one
changeset, only the first entry survived and the rest were silently
dropped from the summary. The dropped entries still appeared in the raw
`<details>` block, which is how the mismatch surfaced (for example in
[#3998](https://github.com/triggerdotdev/trigger.dev/pull/3998), where
one PR's four changesets showed up as a single summary line).

## Fix

Deduplicate on the full entry text rather than the bare PR number. The
entry text embeds the PR link, so:

- the same changeset rendered once per package section still collapses
to one,
- distinct changesets from the same PR are each kept,
- identical descriptions from different PRs stay separate.

Verified against the raw changeset output from
[#3998](https://github.com/triggerdotdev/trigger.dev/pull/3998): that
PR's changesets went from 1 to all 4 in the generated summary.
2026-06-25 09:20:10 +01:00
Daniel Sutton 65c545da4e refactor(run-store,webapp,run-engine): route Postgres TaskRun reads through the run store (#3990)
## Summary

Adds read methods to `RunStore` (`findRun`, `findRunOrThrow`,
`findRuns`) and routes every Postgres read of `TaskRun` through them,
mirroring how writes already go through the store. Behavior-preserving:
each relocated read keeps its exact query, field selection, and database
client (writer, replica, or transaction). This lets `TaskRun` reads be
retargeted to a different backing store later without touching call
sites.

Stacked on #3981 (the write adapter); that PR is the base of this one.

## Scope

In scope: the run engine, webapp services, presenters, and route
loaders. Three reads that pulled `TaskRun` in through a parent model's
relation `include` (alert delivery, batch results, attempt-dependency
cancellation) are decomposed to fetch the run(s) through the store and
stitch them back, since a relation include would not follow `TaskRun` to
a new table.

Left reading the existing table (out of scope): the legacy MarQS paths,
the legacy trigger idempotency read, and one raw-SQL recovery script
(commented for revisiting at cutover).

## Notes

Reads default to the read replica; callers pass the writer or a
transaction client wherever the original read did, so writer-vs-replica
behavior is unchanged.
2026-06-22 10:02:57 +01:00
Eric Allam 0c839e8566 feat(sdk,cli): namespace agent skills with trigger- and add cost-savings (#3970)
## Summary

Three improvements to the SDK-bundled agent skills (follow-up to the
skills installer):

- **`trigger-` namespace.** The installed skills (`authoring-tasks`,
`getting-started`, …) had generic names that collide with unrelated
skills in a shared agent skills directory. They're now prefixed —
`trigger-authoring-tasks`, `trigger-getting-started`, etc. — matching
the convention the public skills repo already uses.
- **New `trigger-cost-savings` skill.** An MCP-driven cost audit:
right-sizes machines, flags missing `maxDuration`, spots sequential
triggers that could batch, and reviews schedule frequency, using
`list_runs` / `get_run_details` for live analysis.
- **Bundle the full docs.** `@trigger.dev/sdk` now bundles the entire
"Documentation" section of the docs (157 pages) instead of a curated
55-page subset, so an agent has the complete, version-pinned reference
in `node_modules`.

## How the bundling works

`scripts/bundleSdkDocs.ts` now reads `docs/docs.json`, walks the
"Documentation" dropdown, and copies every page under it into the SDK.
The set tracks the docs navigation automatically — add a page to the nav
and it ships, no skill edits needed. The API reference and Guides &
examples dropdowns are intentionally excluded. A skill's `sources:`
frontmatter is now informational only.

The dropped idea of a dedicated `trigger-config` skill is replaced by
references to the bundled build-extension docs (`config/extensions/*`)
from the `trigger-authoring-tasks` config section and the chat-agent
skills.
2026-06-16 23:27:28 +01:00
Eric Allam 709477168f fix(release-pr): stop dropping changeset entries and stripping code blocks (#3954)
## Summary

The script that generates the changeset release PR description was
silently dropping some changelog entries and stripping code examples. In
[#3932](https://github.com/triggerdotdev/trigger.dev/pull/3932), entry
[#3937](https://github.com/triggerdotdev/trigger.dev/pull/3937) was
missing entirely from the Improvements list and
[#3952](https://github.com/triggerdotdev/trigger.dev/pull/3952)'s code
block was gone, even though both were present in the raw changeset
output.

## Root cause

`parsePrBody` parsed the raw changeset body line by line:

- The dependency-bump filter matched any entry whose text *began* with a
backticked package name, so a real changelog entry like ``
`@trigger.dev/sdk` now bundles... `` got thrown out along with the
genuine version-bump lines.
- Only the first line of each bullet was kept, so fenced code blocks,
sub-bullets, and continuation paragraphs were discarded.

## Fix

Group each top-level bullet with its indented continuation (code blocks,
sub-bullets, paragraphs), dedent it, and re-emit it intact. The
dependency filter is now anchored so it only matches lines that are
*entirely* a package bump, leaving real entries that merely start with a
package name.

Verified by replaying #3932's raw body through the script: #3937 returns
to the list, #3952's code block is preserved, and #3936's sub-bullets
nest correctly under their parent.
2026-06-15 16:33:09 +01:00
Daniel Sutton b7ef51d763 fix(webapp): make SDK bundle-docs build step work in pruned Docker image (#3947)
## Summary

The webapp Docker image build runs `pnpm run build --filter=webapp...`,
which builds `@trigger.dev/sdk` as a dependency. The SDK's `build`
script recently gained a `bundle-docs` step (`tsx
../../scripts/bundleSdkDocs.ts`), but the build couldn't run it in the
pruned image, breaking the image build.

Two things were missing:

- `docker/Dockerfile` copied `scripts/updateVersion.ts` into the builder
stage but not `scripts/bundleSdkDocs.ts`, so the step failed with
`ERR_MODULE_NOT_FOUND`.
- Even with the script present, the repo-level `docs/` tree it reads is
a separate workspace package that isn't in webapp's dependency graph, so
`turbo prune --scope=webapp` excludes it — the script's missing-docs
guard would then fail the build.

## Design

The Dockerfile now copies `bundleSdkDocs.ts` alongside
`updateVersion.ts`. `bundleSdkDocs.ts` skips gracefully when the repo
`docs/` tree is absent, which is exactly the pruned-dependency-build
case (the SDK is compiled there but never published). Publishing always
runs from the full monorepo where `docs/` exists, so the missing-docs
guard still protects releases — it only fires when `docs/` is present
but a cited doc is genuinely missing, rather than when the whole tree
was pruned away. This avoids dragging 27M of docs into a throwaway
builder stage.

## Test plan

- [x] `bundle-docs` from the full monorepo still bundles all cited docs
(exit 0)
- [x] Simulated pruned tree without `docs/` skips cleanly instead of
failing
- [ ] Webapp Docker image build succeeds in CI

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-15 10:59:39 +00:00
Eric Allam e092919c3f feat(sdk,cli): bundle agent skills + docs in the SDK for zero-drift (#3937)
## Summary

`@trigger.dev/sdk` now ships the Trigger.dev agent skills and a curated
snapshot of the docs those skills cite. The skills that `trigger skills`
installs into your coding agent are thin pointers that read this bundled
content from `node_modules`, so the guidance always matches the SDK
version installed in your project. Previously the full skill text was
copied into your repo at install time and went stale until you
reinstalled after an upgrade.

## How it works

The SDK's `files[]` now includes `skills/` (the full skill text) and
`docs/` (a curated snapshot generated at build time). The docs manifest
is derived from each skill's own `sources:` frontmatter, so a skill only
ships the docs it references, and a skill that cites a missing doc fails
the build.

The CLI installs thin skills whose body points the agent at
`node_modules/@trigger.dev/sdk/skills/<name>/SKILL.md` and
`node_modules/@trigger.dev/sdk/docs/`. They keep the high-value "Common
mistakes" anti-patterns inline so the trigger and the guardrails survive
even if the agent does not follow the pointer. `getting-started` stays
self-contained in the CLI because it runs before the SDK is installed.
2026-06-14 11:00:50 +01:00
Eric Allam 005d7e0a3f chore: add pkg.pr.new preview package releases (#3806)
## What

Adds [pkg.pr.new](https://pkg.pr.new) continuous preview releases. Every
push to a branch builds the public `@trigger.dev/*` packages and
publishes installable preview builds keyed by commit SHA — **without
touching the npm registry**. pkg.pr.new drops install instructions on
the associated PR:

```
npm i https://pkg.pr.new/@trigger.dev/sdk@<sha>
```

This lets reviewers and users try a branch (SDK, CLI, core, etc.) before
anything is released, separate from the changesets release, the manual
`--snapshot` prerelease, and the chat-prerelease flow.

## How

`.github/workflows/preview-packages.yml` (push trigger) → install →
generate Prisma → **stamp preview version** → build → `pkg-pr-new
publish`.

### The version stamp (the important part)

pkg.pr.new serves previews by SHA but does **not** rewrite the
package.json `version` field. If a preview shipped as `4.5.0-rc.4`, a
consumer who installed it would pin `4.5.0-rc.4` to the preview tarball
in their lockfile/cache — and a later `npm i
@trigger.dev/sdk@4.5.0-rc.4` from npm could resolve to the stale
preview. This is a known, by-design gap in the tool
(stackblitz-labs/pkg.pr.new#250, #390).

`scripts/stamp-preview-version.mjs` runs **before the build** and
rewrites every public package to a unique `0.0.0-preview-<sha>`. The
`0.0.0-` prefix can never satisfy a real semver range, so the collision
is structurally impossible (same convention React/Next canaries use).
Running before the build also means `scripts/updateVersion.ts` bakes the
preview version into the runtime `VERSION` constant, so previews are
self-identifying (`trigger --version`, the `x-trigger-cli-version`
header, the MCP server version) instead of all reporting the RC version.

Sibling `workspace:` specifiers are relaxed to `workspace:*` so `pnpm
pack` resolves them against the rewritten versions — `packages/python`
pins peerDependencies as `workspace:^4.5.0-rc.4`, which would otherwise
be unsatisfiable once the version changes. Non-public deps
(`@trigger.dev/database`, `@internal/*`) are left untouched. All
mutations happen on the ephemeral CI checkout; nothing is committed.

## GitHub App

The pkg.pr.new GitHub App is **already installed** on
`triggerdotdev/trigger.dev` (has been for a while), so no setup is
needed. Confirmed live — this branch's pushes published all 10 public
packages, e.g.

```
pnpm add https://pkg.pr.new/@trigger.dev/sdk@e4dfc59
```

## Fork limitation

pkg.pr.new authenticates with a GitHub Actions OIDC token, which GitHub
does not issue to `pull_request` workflows from forks. The `push`
trigger therefore covers branches pushed to this repo (core team), not
external fork PRs. Fork coverage would need a `workflow_run` two-stage
setup; left out for now.

## Notes

- Pinned `pkg-pr-new@0.0.75` (no Node engine constraint; Node 20 CI is
fine).
- pkg.pr.new
[#525](https://github.com/stackblitz-labs/pkg.pr.new/pull/525) adds a
built-in `--previewVersion` flag (still open). If it lands we can drop
the version-rewrite half of the script, but we'd keep a pre-build stamp
anyway so `updateVersion.ts` picks up the preview version (the flag
rewrites at pack time, too late for the baked `VERSION`).
2026-06-02 15:20:32 +01:00
Eric Allam 6c9f1f197e chore: parameterize docker host ports and wire s2-lite by default (#3642)
## Summary

Two papercuts new contributors hit running this repo locally:

1. Fresh clones default to v1 (Redis-only) realtime streams, so Sessions
and `chat.agent` error with `"S2 configuration is missing"`, even though
the `s2` service is already in `docker/docker-compose.yml` and pre-seeds
a `trigger-local` basin. Wire `REALTIME_STREAMS_S2_*` to it in
`.env.example` so the new-contributor flow just works. (Also drop the s2
healthcheck: the image is distroless, so the `wget` check always reports
unhealthy.)

2. Two clones can't both run `pnpm run docker` because ports, project
name, and container names are all hardcoded. Parameterize every host
port as `${VAR:-default}`, drive the project name via
`COMPOSE_PROJECT_NAME` (with a top-level `name:` field as the default),
prefix container names with `${CONTAINER_PREFIX:-}`, and pass
`--env-file .env` so compose reads the same root `.env` the webapp does.
The "Running multiple instances side by side" block in `.env.example`
lists every overridable knob.

Also split the optional services (`electric-shard-1`, `ch-ui`,
`toxiproxy`, `nginx-h2`, `otel-collector`, `prometheus`, `grafana`) into
`docker-compose.extras.yml` behind a new `pnpm run docker:full` script.
The core stack keeps everything the webapp actually needs to boot:
postgres, redis, electric, minio, clickhouse + migrator, s2-lite.

Defaults match every previous hardcoded value, so existing setups keep
working without touching `.env`.

## Test plan

- [x] `pnpm run docker` on a clean clone brings up the core services on
the standard ports under the `triggerdotdev-docker` project name.
- [x] Setting `COMPOSE_PROJECT_NAME=triggerdotdev-docker-alt` + the
`*_HOST_PORT` overrides in `.env` brings up a second stack alongside the
default one with no port or container-name clashes.
- [x] Webapp boots cleanly against the default `.env.example` values;
`/healthcheck` returns 200, no S2 errors.
- [x] s2-lite basin `trigger-local` accepts an append + read via the
same REST endpoints the webapp uses.
- [x] `pnpm run docker:full` brings up the optional services alongside
the core ones in the same project.
2026-05-18 09:28:58 +00:00
nicktrn cad8791859 chore: make changeset:version atomic (#3505)
Follow-up to the v4.4.5 release incident where the release PR (#3406)
was merged with a stale lockfile and stale Chart.yaml, breaking npm +
helm releases. The two automation jobs (`update-lockfile`,
`bump-chart-version`) got cancelled mid-flight by `cancel-in-progress`
when the merge fired the workflow again on `main`.

This restructures `changeset:version` so all the post-version-bump
fixups happen in the same script and end up in a single atomic commit on
`changeset-release/main`, via `changesets/action`'s normal commit step.

Pattern borrowed from Cloudflare workers-sdk, Astro, shadcn/ui.

## Before

```
push: main
└── release-pr (changeset version → bumps package.jsons, opens PR)
    └── update-lockfile (separate job, separate commit)
        └── bump-chart-version (separate job, separate commit)
```

Three jobs, three commits to the release branch.

## After

```
push: main
└── release-pr
    └── changesets/action runs:
          changeset version
          pnpm install --lockfile-only
          node scripts/bump-helm-chart.mjs
          node scripts/cleanup-server-changes.mjs
        ...all staged and committed as ONE commit by the action
```

One job, one commit.
2026-05-02 09:45:44 +01:00
Eric Allam 6f6523ff78 chore(repo): remove unnecessary "trigger.dev v4.4.2" header from the release PR description (#3183) 2026-03-06 14:07:32 +00:00
Eric Allam 5612383684 chore(repo): Improve formatting of server entries in release notes (#3134) 2026-02-26 11:59:47 +00:00
Eric Allam fe193418d0 chore(repo) auto-link server change entries to their PRs via GitHub API (#3129) 2026-02-25 16:02:32 +00:00
Eric Allam c05b30adfe chore(repo): fix enhanced release pr description to filter out dependency only updates (#3128) 2026-02-25 15:50:06 +00:00
Eric Allam 3c0644a3b8 feat: unified GitHub release, server change tracking, and enhanced release PR (#3085)
- Add .server-changes/ convention for tracking server-only changes
- Create scripts/enhance-release-pr.mjs to deduplicate and categorize
changeset PR body
- Create scripts/generate-github-release.mjs to format unified GitHub
release body
- Change release.yml to create one unified GitHub release instead of
per-package releases
- Add update-release job to patch Docker image link after images are
pushed to GHCR
- Update changesets-pr.yml to trigger on .server-changes, enhance PR
body, and clean up consumed files
- Document server changes in CLAUDE.md, CONTRIBUTING.md, CHANGESETS.md,
and RELEASE.md
2026-02-25 13:54:19 +00:00
Matt Aitken 6055c7d050 Recover runs that failed to dequeue (#2931)
There’s an edge case that means runs can end up in the
currentConcurrency set when they’re not in the correct state for
execution. This means they will be permanently stuck in queued.

Given an environmentId this will fix those runs.

This is a temporary fix while we permanently fix the issue.

---------

Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: Eric Allam <ericallam@users.noreply.github.com>
2026-01-23 16:25:58 +00:00
Eric Allam bd449f75dc fix(migrations): Add IF NOT EXISTS to 20260116154810_add_idempotency_key_options_to_task_run (#2923)
## Summary
- Adds `IF NOT EXISTS` to the migration that adds
`idempotencyKeyOptions` column to prevent errors if the column already
exists

## Migration Checksum Fix

If you've already applied the previous version of this migration, you'll
need to update the checksum in your `_prisma_migrations` table to match
the new migration file.

**Previous checksum:**
`f8876e274e3f7735312275eb24a9c4b40f512ac12a286b2de3add47f66df5b27`
**New checksum:**
`0620a914ddbaf01279576274432e51c41f41502cd4c8de38621625380750e397`

### Fix instructions

Run this SQL command against your database:

```sql
UPDATE "_prisma_migrations"
SET checksum = '0620a914ddbaf01279576274432e51c41f41502cd4c8de38621625380750e397'
WHERE migration_name = '20260116154810_add_idempotency_key_options_to_task_run';
```

This updates the stored checksum to match the modified migration file,
allowing future migrations to proceed without checksum mismatch errors.

## Test plan
- [x] Verified migration applies cleanly on fresh database
- [ ] Verified checksum update works on database with previous migration
applied

🤖 Generated with [Claude Code](https://claude.com/claude-code)
2026-01-21 14:43:37 +00:00
Eric Allam fc351cb6c4 chore(repo): remove GITHUB_TOKEN requirement for publishing prerelease packages (#2679) 2025-11-24 14:33:40 +00:00
nicktrn 885ae5e06f fix(runner): immediate poll to decrease restore time (#2516)
* fix(runner): immediate poll to decrease restore time

* chore: bump default prerelease tag

* fix(cli): s is not a function

* add changeset
2025-09-16 16:17:04 +01:00
Eric Allam 65da20c225 feat: replicate task runs to clickhouse to power dashboard improvements (#2035)
* WIP clickhouse package with test containers setup

* More clickhouse client setup now with otel and real tests, and the v1 of raw run events

* Add some additional columns to raw_run_events_v1

* WIP runs dashboard service

* Create a new run engine event bus event for the runs dashboard to hook into

* Track run events in the run engine

* make sure engine v1 runs get synced to CH

* Update the attemptNumber of v3 task runs

* Restructure the run events to be more sparse

* emit more stuff

* Setup replication package

* scaffold the replication package

* replication wip

* resolve conflicts

* more replication stuff

* Add ability to drop the replication slot completely on teardown

* Use the new single replacingmergetree task events table for replication

* get it working

* insert payloads into their own table only on insert and then join

* prepare for using clickhouse cloud and now running ch migrations during boot in the entrypoint.sh

* Handover WIP and tests

* Testing the replication service

* Remove the runs dashboard stuff that we aren't using anymore

* Added a test for large payloads

* hacky typecheck fix

* Fix new internal package typecheck issues and start adding telemetry to the replication service

* tracing over spans, some other improvements

* Improvements to the runs replication service, now ready for testing

* Some fixes and cleanups

* Don't need this code anymore

* move transaction types into the runs replication service

* only send spans where there are transaction events

* A couple of suggested tweaks
2025-05-12 22:12:36 +01:00
nicktrn 2b586c87a0 Fix controller waitpoint resolution, suspendable state, and snapshot race conditions (#2006)
* remove dead code

* rename managed to shared runtime manager

* rename to resolve waitpoint for clarity

* add resolver id helper

* store and correctly resolve waipoints that come in early

* fix ipc message type change

* branded type for resolver ids

* add fixme comments

* remove more unused ipc schemas

* fix entitlement validation when client doesn't exist

* restore hello world reference workspace imports

* runtime manager debug logs

* prefix engine run logs

* managed run logger accepts nested props

* runtime suspendable state and improved logs

* require suspendable state for checkpoints, fix snapshot processing queue

* add terminal link as cli module so we can more easily patch it

* apply cursor patch

* add license info

* remove terminal-link package and add deprecation notice

* remove old patch

* remove terminal-link from sdk

* rename snapshot module

* add cli test tsconfig

* add run logger base type

* add snapshot manager tests

* fix cli builds

* improve QUEUED_EXECUTING test

* changeset

* make testcontainers wait until container has stopped

* require unit tests for publishing again

* avoid mutation during iteration when resolving pending waitpoints

* improve debug logs and make them less noisy

* always update poller snapshot id for accurate logs

* detach task run process handlers

* check for env overrides in a few more places and add verbose logs

* log when poller is still executing when we stop it

* add supervisor to publish workflow

* always print full deploy logs in CI

* Revert "avoid mutation during iteration when resolving pending waitpoints"

This reverts commit 87b0ce1e5b.

* disable pre

* print prerelease script errors

* Revert "disable pre"

This reverts commit 9403409637.

* misc fixes

* better debug logs

* add snapshots since methods and route

* prep for snapshots since

* improve deprecated execution detection

* update supervisor and schema

* properly log http server errors

* detect restore after failed snapshot fetch

* run and snapshot id can be overridden

* fix restore detection

* fix deprecation checks, move into snapshot manager

* less logs

* rename snapshot manager stop

* restore detection was moved into snapshot manager

* fix notifier logs

* make runtime manager status a debug log

* no need to attach runtime status twice

* findUnique -> findFirst

* sort snapshots by created at everywhere
2025-05-07 13:02:32 +01:00
nicktrn 49a3f72e13 Publish redis-worker and add graceful shutdown manager (#1810)
* add shutdown manager

* update ai test instructions

* add shutdown timeout to redis-worker

* move redis worker to packages

* add unregister method

* prep for publishing package

* fix types

* update ai files

* fix cursor terminal links

* prevent overly friendly ids

* use structured logger

* use unique shutdown handler names

* rework suspend completion

* add trycatch util

* rework suspend restore

* add http server metrics

* add missing prom-client to core

* add prom metrics to redis worker

* bundle redis-worker

* fix esm/cjs interop

* remove proxy from changeset ignore and add supervisor

* add pause to prerelease script for any manual edits

* unregister the correct handler and add early detection

* small change to http handler return

* fix worker tests

* fix shutdown manager tests
2025-03-21 14:53:27 +00:00
nicktrn 4fda8a5ee1 Fix prerelease script (#1794)
* clean before building

* add main branch commit protection
2025-03-15 12:20:54 +00:00
Eric Allam 0a18e1d919 Additional dev queue consumer logging (#1606) 2025-01-13 16:37:19 +00:00
Eric Allam 67592ec2b4 Multiple streams can now be consumed at the same time (#1522)
* Mutliple streams can be now consumed simultaneously

* Update prerelease script

* Add changeset

* Make it core

* Handle API error responses when streaming
2024-12-02 17:58:10 +00:00
Eric Allam 40152af775 Load GITHUB_TOKEN env var in publish-prerelease, as it’s not required 2024-09-23 21:30:35 +01:00
Eric Allam d7a65f3e7a Fix resolving external packages that are ESM only (#1346)
* Fix resolving external packages that are ESM only by falling back to mlly resolvePathSync. This will fix mupdf

* when publishing a prerelease and aborting, clear the git stage
2024-09-23 16:21:04 +01:00
Eric Allam 3aa5811790 fix 3.0.0 update warning (#1308)
* Attempt to fix false package mismatch warnings

* Add changeset

* Add ability to test update checks in prerelease packages

* Resolve the trigger.dev package based on the package.json dir

* Try this

* Don’t use the version module, just resolve the packageJson

* One more dirname

* Comment

* Remove the version export because we aren’t using it anymore
2024-09-16 18:03:16 +01:00
Eric Allam 8578c9b281 v3: new build system fixes round 2 (#1283)
* Fixed empty env vars overriding in dev runs

* Don’t import package.json anymore

* fix node10 moduleResolution in @trigger.dev/core

* Support self-hosters pushing to a custom registry when running deploy

* dev: Fixed stuck runs when a child run fails with a process exit

* Make some doc notes about known issues and docker hub private repos

* Fix --project-ref when running deploy

* Fix —config option when deploying

* Fixing the flushing/killing process with the new build system

* Add monorepo-react-email e2e test fixture

* Fix issue with emitDecoratorMetadata and tsconfigs with extends

* Got the emit decorator metadata fixture working

* Fixed typechecking yarn e2e CLI tests in monorepos

* Add remote forced externals system, in case we come across another package that cannot be bundled (spurred on by header-generator)

* Remote externals now powered by JSON Hero to be easier to update

* resolve config source files

* Add a —javascript option to init, defaults to typescript

* Add support for prisma typed sql

* Remove msw and retry.interceptFetch

* Add missing code to the openai retries example

* Don’t generate the v3 catalog prisma client during CI

* Fixed v3-catalog task imports

* Remove interceptor usage in task file

* Only import import-in-the-middle hook if there are instrumented packages

* Fix yarn.lock file
2024-09-11 15:48:20 +01:00
Eric Allam f9ec66c562 v3: new build system (#1265)
* upgrade @opentelemetry packages to the latest versions

* remove v2 only packages, will be moved to a dedicated repo

* remove more v2 code and run pnpm install

* use the npm yalt package in the webapp

* convert @trigger.dev/core to tshy

* Switch from jest to vitest in @trigger.dev/core

* Fixed core test

* move core-backend code into core subpath export

* convert @trigger.dev/sdk to tshy

* Removed hono

* move core-apps to core/v3/apps, remove core-apps, start converting cli-v3

* Fix up some of the commands

* cli now building and loadable

* using package-json-from-dist to get package version now in core and cli

* dev command WIP

* cleaned up some repetition and structure of the entry point stuff

* bringing back the background worker stuff

* Indexing of the v3 catalog

* getting closer to executing dev runs...

* centralize dev logging using event emitter

* Move indexing to it’s own entry point, simplify code

* dev runs working

* Get instrumentation to work with openai

* debugging achieved internally

* provide worker files as part of the worker creation on the server

* support for cjs and esm javascript

* Fixed timeout

* worker manifest now has the config path

* auto-upgrade config to non-deprecated alternatives

* Adding package preview release

* deployment WIP

* improve the syncEnvVars output and adapt resolveEnvVars

* WIP bun runtime

* WIP bun support

* seed tasks with the machine preset if listed in the config

* deploy run executions WIP, extracted TaskRunProcess into 1 place

* deployed tasks running and executing 🎉

* support for waits and better flushing & process cleanup

* Fixed the heartbeating

* Better warning messages

* Improve and unify the indexing between dev and deploy

* Support for external deps that need node-gyp to build

* build extensions can now install custom packages and run instructions in the image. Also prisma extension now works and also works with multiple schema files

* Add back in the main/types/module to sdk

* dev no longer is Ink/React, grace period for disconnections in dev

* Fix the changeset config

* More changeset fixes

* Remove config packages

* More changeset fixes

* Fixed typescript issues (needed to revert back to zod 3.22.3

* Fix pr_checks workflow

* Remove the prepare script

* Fixed tests and package versions

* Remove cli test script

* Remove packages from tailwind watch paths

* Add repo to public packages

* Just commit the generated files and do the building at dev time

* Try and get pkg.pr.new working

* Try again

* Fix emitDecoratorMetadata importing named export from typescript

* config file backwards compat with export const config

* Fixed issue where import errors weren’t coming through

* p-retry is a prod dep

* typescript needs to be a prod dependency for emitDecoratorMetadata

* Add better debug logging to help track down import-in-the-middle bug

* An external is only considered resolvable if it resolves to the same path as the collected external

* Fix runtime checks to allow >=18.20

* Move extensions to a new build package

* Fixed building packages in dockerfile

* Remove the e2e test from publish workflow for now

* Don’t treat pkg.pr.new versions has needing upgrading

* making sure config handleError works, and discovered path aliases don’t work in config files

* Strip empty string env vars so they accidentally override real values

* Couple of things

* Update version to use preview instead of beta

* Hopefully fix re-attempts with >30s delay

* Match socket emit messages to current latest in main

* Initial guide

* Go back to beta

* Go back to the preview, and update guide to use pr preview tags

* Go back to beta

---------

Co-authored-by: Matt Aitken <matt@mattaitken.com>
2024-08-23 13:10:15 +01:00
nicktrn e23047f9ad update pre-release script 2024-06-12 09:49:47 +01:00
Eric Allam 8fc8f57b39 v2: MarQS powered job executions (#1149)
* WIP

* Allow marqsv2 and v2 graphile to run in parallel

* Fix missing GraphileLogger import

* Fixed heartbeat after rebase

* Replace postgres based run counters with redis ones with a backfill

* Add back in the graphile logger

* Remove duplicate visibility timeout calls

* Clamp simple weighted strategy to max of 5
2024-06-06 13:56:27 +01:00
Eric Allam c3f6557eb6 Create publish preprelease script 2024-04-25 11:26:14 +01:00
Eric Allam 17f6f29d05 Feature: Support multiple runtimes other than Node.js (#774)
🚀 Publish Trigger.dev Docker / e2e (push) Failing after 0s
🚀 Publish Trigger.dev Docker / units (push) Failing after 5s
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 5s
🚀 Publish Trigger.dev Docker / publish (push) Has been skipped
2023-12-06 10:23:37 +00:00
Eric Allam fd94a32b95 Removing a bunch of old stuff 2023-06-21 17:07:17 +01:00
Eric Allam 683f24d4e4 Getting ready to update all templates to the newest SDK after it’s released 2023-03-15 15:05:43 +00:00
Eric Allam 4e83ef642d Update prerelease template deps script 2023-03-14 09:15:12 +00:00
Eric Allam 5eeb275b32 Release template deps script 2023-03-13 15:33:23 +00:00
Eric Allam f39bc44eec Projects
- Deploy a new VM when a push event comes through
- Live updating project overview page
2023-03-07 15:12:42 +00:00
Eric Allam ab512157a5 Added ability to patch all template repos and push the changes 2023-03-03 08:34:15 +00:00
Eric Allam 6deabb21ba Bump notion 2023-03-02 22:30:43 +00:00
Eric Allam 408bd6ddc8 Added a script to automatically update template repo @trigger.dev packages instead of having to do it manually 2023-03-02 15:34:53 +00:00
Eric Allam 5ad386e306 Make it easier to start ngrok in different regions 2023-01-29 22:23:38 +00:00