59 Commits

Author SHA1 Message Date
Chris Arderne b33197691b chore: enforce no unused deps or code in ci (#4654) 2026-08-18 11:35:51 +01:00
nicktrn 6e7710282c ci: make the lefthook pre-push hook actually install (#4642)
## Summary

`lefthook.yml` has been in the repo since #4147, but nothing installs
lefthook and nothing runs `lefthook install`, so the pre-push hook it
describes has never fired for anyone. #3977 had removed the `lefthook`
devDependency a week before #4147 landed, and #4147 only added the
config file.

This supplies the missing half:

```diff
+    "prepare": "lefthook install",
+    "lefthook": "^2.1.10",
       "onlyBuiltDependencies": [
+      "lefthook",
```

With those in place, `pnpm install` wires the hook up on clone, and the
format and lint checks actually run before a push instead of first
failing in CI.

Also here: the pre-push jobs run in parallel rather than in sequence,
and `CONTRIBUTING.md` documents the hook, including how to skip it and
the fact that GitButler only runs hooks when "Run hooks" is enabled in
its settings.

`lefthook@2.1.10` is the current release.
2026-08-17 08:24:32 +01:00
Eric Allam 603c278687 docs: clarify when changesets and server-changes files are needed (#4617)
## Summary

Clarifies when to add a changeset or a `.server-changes/` file. The
friction that keeps coming up is treating these as "I touched a public
package or a server app, so I owe a note." They are user-facing release
notes that go straight into the changelog customers read, not a catalog
of every change.

The guidance now leads with the real test: would a user or customer care
about this change? Add a note when the change is something they would
notice, act on, or want to hear about. Skip it otherwise, even when a
public package or server app is touched, for example:

- internal-only or admin-only changes, refactors, test-only changes,
chores
- performance or query tuning with no user-visible behavior change
- public packages that are not consumed independently (e.g.
`@trigger.dev/redis-worker`), where a version bump means nothing to a
user

Anyone who wants the exact history reads the commits.

Updates every place that encoded the old "touched a package or app, so
add a note" rule so they agree: `AGENTS.md`,
`.server-changes/README.md`, `CONTRIBUTING.md`, `CHANGESETS.md`,
`.claude/rules/server-apps.md`, and `.claude/REVIEW.md` (the last drives
automated review flagging, so it stops flagging exactly the changes the
new guidance says to skip). Also handles the mixed-PR case where the
package change needs no changeset but the server change is user-facing.
2026-08-14 13:36:06 +01:00
Chris Arderne b902e65dfb chore: standardise internal node on 24.18.0 (#4254)
## Summary

Updates the internal development, CI, and runtime-image Node version to
24.18.0. SDK compatibility coverage continues to include Node 20, 22,
24, and 26.

The Node type definitions and the package-manager lockfiles now resolve
against Node 24 types.
2026-07-15 12:49:12 +01:00
Iss 9f4d8d8b0c docs: update schedule & test navigation for the new dashboard UI (#4252)
The dashboard was redesigned and two pages moved, but the docs still
described the old sidebar:

- **Schedules** no longer has its own sidebar page — schedules are
managed from the **Tasks** page (open a scheduled task to create / view
/ edit / enable-disable / delete them).
- The standalone list-based **Test** page is deprecated — you test a
task from its own **Test** button now.

## Changes

- `tasks/scheduled.mdx`: rewrote the "attaching schedules" and "testing
schedules" sections for the Tasks-based flow, added a "managing
schedules in the dashboard" section, and added explicit callouts noting
both pages moved (so readers — and search — aren't pointed at a page
that no longer exists). Re-shot the four schedule screenshots and fixed
a mislabeled alt text.
- `run-tests.mdx`, `snippets/step-run-test.mdx`,
`guides/examples/sentry-error-tracking.mdx`: replaced "select the Test
page in the sidebar" with the task-first flow plus a callout, and
refreshed `test-dashboard.png`.

TRI-11939
2026-07-13 20:01:15 -04:00
nicktrn a6bd370e42 chore: remove end-of-life v3 execution components (#4194)
v3 (engine V1) is end-of-lifed and the v3 clusters are gone, so this
removes the dead v3 execution code from the monorepo. It's the first
pass of TRI-11824 - the webapp v3 code paths are deliberately left
untouched and gated for a follow-up.

## Apps

Deletes the three v3-only execution apps and their build wiring:

- `apps/coordinator`, `apps/kubernetes-provider`, `apps/docker-provider`
- `.github/workflows/publish-worker.yml` - it built only those three;
the v4 worker publish is a separate workflow
- Their references in `.changeset/config.json`, `.cursorignore`,
`CHANGESETS.md`, `CONTRIBUTING.md`, `.server-changes/README.md`
- `pnpm-lock.yaml` regenerated to prune the apps and their app-only
dependencies (`socket.io`, `@kubernetes/client-node`, `p-queue`,
`execa`, `prom-client`, `tinyexec`)

## Core

Removes the helpers in `@trigger.dev/core` that only those apps used -
`ProviderShell`, `SimpleLogger`, the `Exec`/process helpers,
`isExecaChildProcess`, `getTextBody`, and `testDockerCheckpoint`. Each
was verified to have no remaining consumers anywhere in the repo.

Kept the helpers still used elsewhere: `ExponentialBackoff` (warm-start
client), `HttpReply`/`getJsonBody` (serverOnly http server),
`SimpleStructuredLogger` (widely used), and
`ZodNamespace`/`ZodSocketConnection` (still referenced by legacy v3
webapp code, hence the follow-up pass).

The `./v3/apps` and `./v3/serverOnly` export subpaths remain - only dead
members were trimmed from their barrels, so no `package.json` exports
changed.

## Verification

`@trigger.dev/core` builds, and `typecheck` passes for core, supervisor,
cli-v3, run-engine, redis-worker, and webapp.

refs TRI-11824
2026-07-08 19:47:20 +02:00
Chris Arderne 448443a024 chore: bump internal node to 22 and standardise (#4084)
Bumps the internal/toolchain Node version to the latest 22.x LTS
(`22.23.1`) and standardises it across the repo. Scope is the **platform
toolchain + the repo's own runtime images** (all `20 → 22` *upgrades*,
off the now-EOL node 20).

### Main changes
- Node `20.20.2 → 22.23.1` across all CI workflows, `.nvmrc`,
`CONTRIBUTING.md`, and the OSS `docker/Dockerfile` (digest-pinned).
- `@types/node → 22.20.0` (root dep + pnpm `overrides`, so the whole
workspace resolves to it); lockfile regenerated.
- `sdk-compat` matrix: adds Node 24 + 26 (keeps 20, still in `engines`).
- **App runtime images → node 22** (were on EOL node 20):
`apps/coordinator` → `node:22.23.1-bookworm-slim`;
`apps/docker-provider` + `apps/kubernetes-provider` → `node:22-alpine`
(reusing the exact digest `apps/supervisor` already runs, so all four
worker images are now identical). Stage aliases renamed off `node-20`.

### Possible issues / test notes
- `@types/node` 22.x can surface new TS errors — typecheck (now on 22)
is the gate.
- **Smoke-test the v3 worker path** — `coordinator` (`crictl`/CRI calls)
and the docker/kubernetes providers (talking to their daemons) now run
on node 22 (alpine/musl for the providers). Upgrade off EOL so low-risk,
but it's deployed runtime code with its own `publish-worker.yml`
pipeline.
2026-07-02 18:13:36 +01:00
Chris Arderne c4301958c2 chore: add format and lint to .md guides (#4057) 2026-06-26 14:02:57 +01:00
nicktrn a9f756b260 chore: move reference projects to their own repo (#3812)
The reference/example projects (`references/`) now live in their own
repo, https://github.com/triggerdotdev/references, so their heavy,
frequently-changing dependencies are no longer part of this repo's
lockfile and tooling. This removes them here and repoints everything
that referenced them.

- Deletes `references/`; updates the pnpm workspace + lockfile.
- Clears the references-only CI rules and `.vscode` configs.
- Repoints the docs (contributor/agent + one public page) to the new
repo.
- `seed.mts` keeps the local-dev projects (hello-world, d3-chat,
realtime-streams).
2026-06-02 22:19:28 +01:00
nicktrn 11631c19e1 chore: bump node to latest patch release (#3802)
Bumps Node to the latest 20.x patch.
2026-06-02 11:48:46 +01:00
Eric Allam 6c9f1f197e chore: parameterize docker host ports and wire s2-lite by default (#3642)
## Summary

Two papercuts new contributors hit running this repo locally:

1. Fresh clones default to v1 (Redis-only) realtime streams, so Sessions
and `chat.agent` error with `"S2 configuration is missing"`, even though
the `s2` service is already in `docker/docker-compose.yml` and pre-seeds
a `trigger-local` basin. Wire `REALTIME_STREAMS_S2_*` to it in
`.env.example` so the new-contributor flow just works. (Also drop the s2
healthcheck: the image is distroless, so the `wget` check always reports
unhealthy.)

2. Two clones can't both run `pnpm run docker` because ports, project
name, and container names are all hardcoded. Parameterize every host
port as `${VAR:-default}`, drive the project name via
`COMPOSE_PROJECT_NAME` (with a top-level `name:` field as the default),
prefix container names with `${CONTAINER_PREFIX:-}`, and pass
`--env-file .env` so compose reads the same root `.env` the webapp does.
The "Running multiple instances side by side" block in `.env.example`
lists every overridable knob.

Also split the optional services (`electric-shard-1`, `ch-ui`,
`toxiproxy`, `nginx-h2`, `otel-collector`, `prometheus`, `grafana`) into
`docker-compose.extras.yml` behind a new `pnpm run docker:full` script.
The core stack keeps everything the webapp actually needs to boot:
postgres, redis, electric, minio, clickhouse + migrator, s2-lite.

Defaults match every previous hardcoded value, so existing setups keep
working without touching `.env`.

## Test plan

- [x] `pnpm run docker` on a clean clone brings up the core services on
the standard ports under the `triggerdotdev-docker` project name.
- [x] Setting `COMPOSE_PROJECT_NAME=triggerdotdev-docker-alt` + the
`*_HOST_PORT` overrides in `.env` brings up a second stack alongside the
default one with no port or container-name clashes.
- [x] Webapp boots cleanly against the default `.env.example` values;
`/healthcheck` returns 200, no S2 errors.
- [x] s2-lite basin `trigger-local` accepts an append + read via the
same REST endpoints the webapp uses.
- [x] `pnpm run docker:full` brings up the optional services alongside
the core ones in the same project.
2026-05-18 09:28:58 +00:00
nicktrn 706a0b88c9 chore: upgrade pnpm to 10.33.2 with security hardening (#3489)
## Summary

- Upgrade pnpm from 10.23.0 → 10.33.2 (latest minor)
- Enable `blockExoticSubdeps: true` for supply-chain defense
- Update all version references across the repo

## Security improvements in 10.28.2+

- Path traversal protection in `directories.bin`
- Symlink-escape protection for `file:/git:` dependencies (prevents
reading `/etc/passwd`, `~/.ssh/...`)
- https://pnpm.io/settings#blockexoticsubdeps

## Files updated

- `package.json` — `packageManager` field
- `docker/Dockerfile` — 5 `corepack prepare` calls
- `apps/supervisor/Containerfile` — 1 `corepack prepare` call
- `pnpm-workspace.yaml` — added `blockExoticSubdeps: true`
- `CLAUDE.md`, `AGENTS.md`, `CONTRIBUTING.md`, `ai/references/repo.md` —
version references

## Verification

- `pnpm install --frozen-lockfile` succeeds (no lockfile regen needed)
- `pnpm install` (plain) produces zero lockfile diff
- All CI checks pass

Slack thread:
https://triggerdotdev.slack.com/archives/C061L2MHW93/p1777625600974279?thread_ts=1777622248.762639&cid=C061L2MHW93

https://claude.ai/code/session_01G759MUqmjsPh9k1qDxbdjG

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-05-01 16:24:26 +01:00
Eric Allam 4dced14ad5 chore: fix CONTRIBUTING.md setup steps and scope db:seed to webapp (#3450)
## Summary

Two fixes that together get a fresh-machine setup working from
`CONTRIBUTING.md` end-to-end with no manual workarounds:

### `CONTRIBUTING.md`

- Fix wrong path in the migration walkthrough: `cd packages/database` →
`cd internal-packages/database`. The current path doesn't exist; this
breaks step 2 for every contributor adding a migration.
- Renumber duplicate `4.` steps in **Adding migrations** and the skipped
`5.` in the hello-world **Running** section.
- Combine three sequential `pnpm run build --filter ...` calls into one
(Turbo parallelizes filters): `pnpm run build --filter webapp --filter
trigger.dev --filter @trigger.dev/sdk`.
- Add a `pnpm run db:seed` step after migrate. The seed creates the
local user, `References` org, and reference projects (including
`hello-world` with the stable `proj_rrkpdguyagvsoktglnod`). Removes the
manual instruction to edit the `externalRef` column in Postgres.
- Mention ClickHouse and the ClickHouse migrator alongside
Postgres/Redis in the Docker step (they're already part of `pnpm run
docker`, just invisible in the docs).
- Remove the V1-era **Add sample jobs** section.
`references/job-catalog` no longer exists; the hello-world flow above
replaces it.

### `turbo.json`

Scope `db:seed` to `webapp#db:seed → webapp#build`. The previous
root-level entry queued `build` for every workspace package — including
`references-*`, `docs`, `kubernetes-provider`, `coordinator`, etc. Only
`webapp` actually has a `db:seed` script, so the rest of those builds
were dead weight. Worse: a single broken reference (today,
`references-realtime-hooks-test` failing under Turbopack with
`node:fs/promises`) kills the whole seed pipeline.

After the change, `turbo run db:seed --dry-run` plan drops from 27 tasks
to 20 — only `webapp` and its real transitive workspace deps. Reference
projects no longer block seeding.

## Test plan

- [x] Fresh-machine setup followed end-to-end on a wiped Postgres +
ClickHouse: migrate → seed → build → webapp → CLI login → `trigger dev`
→ triggered `hello-world`, run completed with `{"message":"Hello,
world!"}`.
- [x] `turbo run db:seed --dry-run=json` confirms 20 tasks, all webapp
deps, no reference packages.
- [ ] CI green on the renamed turbo task name.
2026-04-27 18:13:29 +01:00
Eric Allam 2c688ca3d4 chore(repo): Add new contributing guidelines and auto-close outside PRs open in "ready to review" status (#3218) 2026-03-14 10:23:34 +00:00
Eric Allam 3c0644a3b8 feat: unified GitHub release, server change tracking, and enhanced release PR (#3085)
- Add .server-changes/ convention for tracking server-only changes
- Create scripts/enhance-release-pr.mjs to deduplicate and categorize
changeset PR body
- Create scripts/generate-github-release.mjs to format unified GitHub
release body
- Change release.yml to create one unified GitHub release instead of
per-package releases
- Add update-release job to patch Docker image link after images are
pushed to GHCR
- Update changesets-pr.yml to trigger on .server-changes, enhance PR
body, and clean up consumed files
- Document server changes in CLAUDE.md, CONTRIBUTING.md, CHANGESETS.md,
and RELEASE.md
2026-02-25 13:54:19 +00:00
Matt Aitken bfe417f679 docs: add note about single-issue PRs to contributing guide (#3044)
Add a prominent notice at the top of CONTRIBUTING.md clarifying that we
only accept PRs addressing a single issue, not multiple fixes.

https://claude.ai/code/session_011ZNQd5zkf38piSMhWbgiWt

Co-authored-by: Claude <noreply@anthropic.com>
2026-02-13 10:07:06 +00:00
Matt Aitken 170fde3498 Move vouch requirement to top of CONTRIBUTING.md (#3029)
Contributors need to be vouched before opening PRs, but this requirement
was buried far down in the document. This change:
- Adds mention of vouches in the intro paragraph
- Moves the "Getting vouched" section to right after the intro

This makes the requirement more visible to new contributors.

Slack thread:
https://triggerdotdev.slack.com/archives/C0A7Q6F62NS/p1770805895370749
https://claude.ai/code/session_01G6VVbgfUAeCpJfedELdqq1
<!-- devin-review-badge-begin -->

---

<a
href="https://app.devin.ai/review/triggerdotdev/trigger.dev/pull/3029"
target="_blank">
  <picture>
<source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-open-in-devin-review-dark.svg?v=1">
<img
src="https://static.devin.ai/assets/gh-open-in-devin-review-light.svg?v=1"
alt="Open with Devin">
  </picture>
</a>
<!-- devin-review-badge-end -->

Co-authored-by: Claude <noreply@anthropic.com>
2026-02-11 10:44:14 +00:00
Eric Allam bc63edd6bf chore(repo): adopt vouch with issue based workflow and require for PRs (#3022)
Adopting [https://github.com/mitchellh/vouch](vouch) so we can help
potential contributors by requiring a conversation before they can
submit a PR. Too many contributors have been skipping the conversation
part of contributing to an OSS repo and skipping right ahead to
submitting PRs
<!-- devin-review-badge-begin -->

---

<a
href="https://app.devin.ai/review/triggerdotdev/trigger.dev/pull/3022"
target="_blank">
  <picture>
<source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-open-in-devin-review-dark.svg?v=1">
<img
src="https://static.devin.ai/assets/gh-open-in-devin-review-light.svg?v=1"
alt="Open with Devin">
  </picture>
</a>
<!-- devin-review-badge-end -->
2026-02-10 12:03:24 +00:00
Dan 733894bb4f Impersonation log (#2896)
Closes #<issue>

##  Checklist

- [ ] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [ ] The PR title follows the convention.
- [ ] I ran and tested the code works

---

## Testing

_[Describe the steps you took to test this change]_

---

## Changelog

_[Short description of what has changed]_

---

## Screenshots

_[Screenshots]_

💯

---------

Co-authored-by: Matt Aitken <matt@mattaitken.com>
2026-01-15 16:54:45 +00:00
Eric Allam 936bddf198 fix: upgrade Node.js to 20.20.0 to address async_hooks DoS vulnerability (#2890)
## Summary

- Upgrades Node.js from 20.19.0 to 20.20.0 (and 22.12.0 to 22.22.0 for
supervisor) to address the async_hooks stack overflow DoS vulnerability
- Adds `maxDepth` parameter (default 128) to `flattenAttributes` and
`unflattenAttributes` to prevent stack overflow on maliciously deep
nested structures

## Details

The vulnerability (patched in Node.js 20.20.0, 22.22.0, 24.13.0, 25.3.0)
causes unrecoverable crashes (exit code 7) when stack overflow occurs
during async_hooks callbacks. Since the webapp uses `AsyncLocalStorage`,
it was theoretically vulnerable.

### Changes

**Node.js version updates:**
- `docker/Dockerfile`: 20.11.1 → 20.20.0
- `apps/supervisor/Containerfile`: 22-alpine → 22.22.0-alpine
- `.nvmrc`: 20.19.0 → 20.20.0
- `apps/supervisor/.nvmrc`: 22.12.0 → 22.22.0
- `references/prisma-7/.nvmrc`: 20.19.0 → 20.20.0
- All GitHub workflows: 20.19.0 → 20.20.0

**Defense in depth:**
- Added `maxDepth` parameter to `flattenAttributes()` and
`unflattenAttributes()` in `packages/core` to prevent stack overflow on
deeply nested user input

## Test plan

- [x] All existing `flattenAttributes` tests pass (50 tests)
- [x] New tests for depth limiting added
- [x] Verify Docker builds work with new base images
2026-01-15 10:47:44 +00:00
Eric Allam 61b338bea7 chore(repo): upgrade repo to pnpm@10 to prevent executing dep scripts on install (#2712)
* chore: migrate pnpm lockfile to v9 format via pnpm@9

* Upgrade to pnpm 10.23.0

* update the dockerfile and added a few deps to bundle in remix app
2025-11-27 16:26:19 +00:00
Eric Allam 688b108ec3 chore(references): remove v3-catalog (#2443) 2025-08-27 14:35:55 +01:00
nicktrn e7795a06ad Fix: fixes and prerequisites for v4 self-hosting (#2150)
* remove pgadmin

* remove V3_ENABLED

* v3 is always enabled

* enfore docker machine presets by default

* rename autoremove env var

* prefix more k8s-specific env vars

* same prefix for all docker settings

* improve profile switcher copy

* supervisor can load token from file

* optional webapp worker group bootstrap

* fix error message

* fix app origin fallback for otlp endpoint

* use pnpm cache for webapp docker builds

* increase default org and env concurrency limit to 100

* optional machine preset overrides

* improve s3 pre-signing errors

* fix DOCKER_ENFORCE_MACHINE_PRESETS bool coercion

* shard unit tests

* fix for s3-compatible services

* optional object store region

* Update apps/supervisor/src/workerToken.ts

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>

* fix DEPLOY_REGISTRY_HOST example

* fix platform mock

* remove remaining v3Enabled refs

* fix error type.. bad bot

---------

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
2025-06-04 18:02:38 +01:00
Eric Allam a69621bdcc Allow self-hosted deploys locally by pointing to docker.host.internal on macOS (#2064) 2025-05-16 15:20:48 +01:00
Eric Allam 67115ca880 Contributing guide updates (#1784) 2025-03-11 07:54:03 +00:00
nicktrn e97704d904 Run Engine 2.0 (WIP) (#1575)
* bump worker version

* Suggested glossary for the RunEngine, TBC

* Removed BatchTaskRun changes from this branch, they were done in main

* Set the BatchTaskRun status to completed when all runs are completed

* When dequeuing respect passed in maxResources

* Ported over the new run props: idempotencyKeyExpiresAt, versions, oneTimeUseToken, maxDurationInSeconds

* Didn’t hit save… the new props when triggering tasks passed through

* Idempotency expiration + waitpoint edge case

* WIP on creating checkpoint, parking for now

* fix worker routes

* upgrade webapp node types to support generic event emitter

* separate event bus handler singleton and run failure alerts

* duration waits

* fix execution snapshot debug spans

* task waits

* fix event bus types

* temporary fix for react hook run handle type

* disable run notifications for now

* convert any typecasts to expect errors to more easily fix later

* fix webapp types after node types upgrade

* updateEnvConcurrencyLimits across marqs and the runqueue

* Pass proper values into the run engine

* RunQueue settings and removed unused rebalancing workers

* Remove rebalancing prop

* Tidied more things up

* Update/remove queue limits for MARQS and RunQueue

* taskQueue/concurrencyLimit changes ported back into the RunEngine

* Reworked completing waitpoints to improve performance and reduce race conditions

* Improved test robustness

* Down to a single run lock only when a run is totally unblocked and ready to continue

* warm starts, worker notifications, wait fixes

* Fix for Run Engine poll interval env var

* Expect the waitpoint to be completed quickly

* If a run is locked then it’s too late to expire it

* Added VALKEY_ env vars and plugged them into the run engine

* Extracted and updated the guard queue function so it can be used when batching

* Added logging and universal concurrency changes to trigger task v1

* Added notes back in

* Bump @trigger.dev/worker to 3.3.7

* reportInvocationUsage for the runAttemptStarted event

* improve execution snapshot span debug span start times

* Unfriendly IDs

* update lockfile

* Created a shared determineEngineVersion function

* disable unfinished commands

* save new cli config to different location, misc fixes

* add basic engine version check via current deploy

* new run engine will default to node 22 runtime

* block some actions for projects on previous run engine

* fix worker group tests

* fix triggerAndWait test

* one typescript version to rule them all

* redlock type patch

* fix type issues caused by ts-reset

* improve cleanup scripts

* add missing socket.io dep

* fix run notification handler type

* fix worker group test again

* generate prisma client for e2e tests

* remove worker group tests for now

* prevent image pull rate limits during unit tests

* increase timeout for queue concurrency limit test

* generate prisma client for preview release

* same node types everywhere

* Updated engine readme, removed legacy system notes

* use default machine preset from platform package

* worker instances plural in schema

* disable pnpm update notifications

* return worker group details from connect call

* add workers admin route

* fix heartbeat route return type

* move deployment labels to core apps

* refactor run controller env schema

* Add firstAttemptStartedAt to TaskRun

* RunEngine 2.0 batch trigger support (#1581)

* Make it clear when BatchTriggerV2Service is used

* Copy of BatchTriggerV2Service

* WIP batch triggering

* Allow blocking a run with multiple waitpoints at once. Made it atomic

* Removed unused param

* New batch service

* Pass through the parentRunId and resumeParentOnCompletion

* Use the new batch service, and correct trigger task version

* Force V1 engine if using BatchTriggerV2Service, we’ve already done the check at this point

* Removed the $transaction and early exit if nothing changed

* Adedd a simple batch task to the hello world reference catalog

* Fix for batch waits not working

* Added parentRunId in a couple more places

* Removed waitForBatch log

* Added another parentRunId

* Expanded the example to include all the different triggers

* More changes to blocking to support continuing after idempotent completed runs

* Fix for the wrong type when blocking a run

* remove @map

* optimise worker auth query

* add engine version header to core api client requests

* remove unique constraint for default group id

* consolidate migrations

* the first managed worker becomes the global default

* Debug events off by default, added an admin toggle to show them

* worker group name can't be an empty string

* add exec helper to core

* move machine resources to core

* add pre-dequeue callback to determine max resources

* optionally skip dequeue

* bump worker package

* move worker to core

* fix ReadableStream type error

* fix another type issue

* update a few more tsconfigs

* add metadata changes introduced in #1563

* Run Engine 2.0 trigger idempotency (#1613)

* Return isCached from the trigger API endpoint

* Fix for the wrong type when blocking a run

* Render the idempotent run in the inspector

* Event repository for idempotency

* Debug events off by default, added an admin toggle to show them

* triggerAndWait idempotency span

* Some improvements to the reference idempotency task

* Removed the cached tracing from the SDK

* Server-side creating cached span

* Improved idempotency test task

* Create cached task spans in a better way

* Idempotency span support inc batch trigger

* Simplified how the spans are done, using more of the existing code

* Improved the idempotency test task

* Added Waitpoint Batch type, add to TaskRunWaitpoint with order

* Pass batch ids through to the run engine when triggering

* Added batchIndex

* Better batch support in the run engine

* Added settings to batch trigger service, before major overhaul

* Allow the longer run/batch ids in the filters

* Changed how batching works, includes breaking changes in CLI

* Removed batch idempotency because it gets put on the runs instead

* Added `runs` to the batch.retrieve call/API

* Set firstAttemptStartedAt when creating the first attempt

* Do nothing when receiving a BATCH waitpoint

* Some fixes in the new batch trigger service… mostly just passing missing optional params through

* Tweaked the idempotency test task for more situations

* Only block with a batch if it’s a batchTriggerAndWait… 🤦‍♂️

* Added another case to the idempotency test task: multiple of the same idempotencyKey in a single batch

* Support for the same run multiple times in the same batch

* Small tweaks

* Make sure to complete batches, even if they’re not andWait ones

* Export RunDuplicateIdempotencyKeyError from the run engine

* Latest lockfile

* Trigger with a machine (old run engine)

* RE2, allow setting machine when triggering

* Fix for new glob patterns

* add max run count to dequeue from version route

* add worker instance name env var and header

* queue consumer pre skip callback

* poll for more runs after final execution errors

* fix dequeue search param schema

* add shortcut to debug switch

* expose run engine timeouts as env vars

* make warm start durations configurable

* add optional status to json reply helper

* fix preSkip hook, add debug logs

* BLOCKED_BY_WAITPOINTS -> SUSPENDED

* exit controller when run suspended

* check if already replied before http reply

* run controller will wait for next run after the current one is suspended

* cancel run button shortcut

* minimal event repository environment type

* fix update metadata call

* run suspension and misc fixes wip

* change debug shortcut to shift + D

* Started work on the Dev supervisor

* Formatting

* Fix for bad imports

* Before rebuilding SSE

* Presence updating from the CLI working via SSE

* add worker notification debug logs

* send run:stop when exiting run phase

* skip current snapshot poll on worker notification

* add more logs and route to submit run debug logs

* add worker and runner ids to snapshots

* improve run notification debug logs

* add workload debug log route

* misc run controller fixes and refactor

* prevent parallel execution of critical functions

* update bun to 1.2.1

* WIP with dev dequeuing

* Method to convert friendlyIds to non-friendly, do nothing with actual ids

* Set the engine on BackgroundWorker, lazily upgrade projects to engine V2

* Runs with ttls were getting immediately expired… oops.

* Pass the Waiting for deploy reason through, so we have it on the execution snapshots

* Fixed the logic for getting the right background worker for a run

* Use the correct ID when dequeuing…

* determineEngineVersion is now fully functional

* Rate limiter ignores the dev endpoints

* Retrieving a batch gives you the runIds

* Set a unique version for the RE2 BatchTaskRun

* add provisional changeset

* The start of dev run execution is working

* First dev run working

* Moved the dev run controller closer to what Nick did with the managed one

* export exec output type

* Heartbeat fix: don’t heartbeat if _isHeartbeating == false

* Dev runs get notifications, some dev bug fixes

* Improved logging or dequeuing

* We need to dequeue runs from the latest version too, for triggerAndWait

* Ported Eric’s validateWorkerManifest with nicer errors

* When flattening an idempotency key if part is undefined, return undefined

* Dev logging fixes

* Remove sigterm listener

* Deprecating workers. Don’t specify a BackgroundWorker when dequeuing an environment

* Deleted some old files. Renamed “managed” to “deploy”

* When a build finishes, always copy the build dir (otherwise the first one gets trampled on by the 2nd)

* Dev master queues should work differently

* Deleting old workers

* Added debounce function to core

* Improvement to canceling

* WIP on debounce canceling on socket disconnection

* Added environment data to execution snapshots

* Dev runs that have stalled get “Canceled” with a reason explaining why

* Show CLI messaged when a connection to the platform is lost/restored

* Fix TriggerTask after merge

* Add trigger task v2 max attempts, replace some findUniques

* Port the new queue logic to the run engine

* More fixes post-merge

* We weren’t setting a `retryConfig` up for the tests… it’s now required

* Start the Redis worker inside the Run Engine… 🤦‍♂️

* Trying to make the testcontainers more reliable

* Added keyPrefix: "engine:”

* Badly placed bracket in trigger task

* Better Redis namespacing

* Fix for expired run not getting removed from the queue

* Don’t create a redis client in the testcontainers, return the redisOptions instead

* Cleanup redis client in the run lock tests

* Fix for the RunQueue not supporting keyPrefix

* Updated more of the RunQueue scripts rebalancing

* Trying to make Redis more robust in the tests…

* Improved test resiliciency more

* Fix for delays (checkpoint check)

* Increase the timeout slightly to fix ttl test

* Added priority support when triggering

* More wip trying to make test containers more reliable

* batchTriggerAndWait test is still failing… some wip to try fix it

* Fixed redis tests now we’re not providing a client

* Separate Redis clients for the run engine worker/queue/runlock

* Made the wait for duration test more resilient

* Added idempotencyKeyExpiresAt to Waitpoints

* Waitpoint timeouts and idempotency expiry

* Use finishWaitpoint, removed extra worker job

* Added waitpoint idempotency tests

* Creating resume tokens is working

* Some improvements to the resume tokens

* Moved resumeTokens to just be wait functions 🥳

* Delete old RuntimeManagers

* Wait for token is working

* Better test for the wait tokens

* Improved the test task some more

* Hide the accessories in the span inspector

* WIP on waitpoint inspector

* WIP on complete waitpoint form

* Span overview panel can be changed based on the entity type

* Improved the waitpoint display

* WIP on completing waitpoint form

* Use the existing CodeBlock for the tip

* Style improvements

* Complete waitpoint

* All waitpoint sidebar variants

* Waits now use a pause icon

* Durations waits use the API to create/block with a waitpoint, not the runtime

* Fix for engine.blockRunWithWaitpoint required org id

* Removed old wait code from the run controllers/task run process

* Form action for skipping a datetime waitpoint

* Move testDockerCheckpoint to a separate core package export (it can’t be bundled on the client)

* Fix for glitchy hourglass animation

* Completed waitpoints display better

* Increase Redis maxRetriesPerRequest to 20 (default)

* Completing and skipping waitpoints is working

* Remove the database prisma dev command, since we need to use create only now. Updated docs

* Added skip timeout, reworked the UI

* Tweaked spacing

* Added payload limit to waitpoint token completion from dashboard

* Test idempotency works on wait.for and wait.until

* Moved the worker-actions to /engine/ from /api/

* Moved dev engine endpoints to /engine/ from /api/

* Separate /engine/ rate limiter

* Added parallel wait prevention, it’s working for duration waits but not well for triggerAndWait yet

* WIP post-merge conflicts

* Set taskEventStore column in the new engine

* Remove duplicate keys

* Post-merge fixes

* Fix for span merge layout

* Use executedAt instead of firstAttemptStartedAt

---------

Co-authored-by: Matt Aitken <matt@mattaitken.com>
2025-03-05 14:40:14 +00:00
nicktrn bf05dac668 Fix last few internal packages paths 2024-10-09 12:52:52 +01:00
Eric Allam fd9a748553 Fixes #1337 - update docs for triggerdev -> trigger bin name change
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 0s
🚀 Publish Trigger.dev Docker / units (push) Failing after 0s
🚀 Publish Trigger.dev Docker / publish (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-infra (push) Has been skipped
2024-09-20 16:12:14 +01:00
Émile Ré c815f28c84 Update contributing instructions (#1121)
* Add OPEN_API_KEY env var in example file

* Update prerequisites and first-time setup
2024-05-22 16:12:40 +01:00
Eric Allam c9ebe7f114 Update contributing guide for local v3 development 2024-05-20 15:43:31 +01:00
Eric Allam 1c4f9ff1ff Add pgAdmin instructions to contributing guide 2023-10-09 17:14:40 +01:00
Eric Allam 486ed20ae7 Renamed the examples dir to references (true examples are in another repo and this was confusing) 2023-09-15 16:09:04 +01:00
Aniket Bindhani 363c74c6da Documentation Update: Added <github_username> instead of triggerdotdev to avoid confusion while cloning the repository (#477) 2023-09-15 12:55:56 +01:00
Eric Allam fef1670533 Improved the testing CLI contributing guide 2023-08-22 11:28:03 +01:00
Eric Allam 3897e6e695 @trigger.dev/cli ini: Make it more clear which API key the init command expects 2023-08-22 11:16:29 +01:00
Eric Allam c7c5fd7fac Fixed the examples job path 2023-08-22 11:06:32 +01:00
Arshad Jamal a998d6c505 Corrected clone link for Windows (#319) 2023-08-12 06:23:45 +01:00
nicktrn 603ffbc426 docs: add supabase guide (#297)
* docs: add supabase guide

* feat: add support for external connection poolers

* chore: use .env symlink in database package

* chore: remove database .env.example

* docs: finish supabase pooling section

* docs: fix small typo

* docs: replace prisma with app wording

* docs: fix up supabase pooling section

* fix: deleted too much in contributing docs

* docs: change pooling heading

* docs: reduce linux shaming

* docs: include direct connection url in fly.io
2023-08-11 15:51:54 +01:00
Eric Allam 9dd7fb2748 Improved the onboarding experience
- Automatically refresh the jobs page when there are no jobs
 - Celebrate 🎊
2023-08-11 14:08:39 +01:00
Sai Hari fa942fc9f5 feat: 🎸 add end-to-end test harness (#253)
* feat: 🎸 add end-to-end test harness

* Address PR comments
2023-08-08 13:55:25 +01:00
Matt Aitken 9ff4c0dbd5 Project-wide Prettier setup (#237)
* Setup project-wide prettier

* Remove old workspace file

* Remove old debugging directives

* New top-level .prettierignore

* Updated Prettier config settings

* Contrubuting guide: Fix for some bad code blocks

* Added more ignores

* Improved the format script command

* printWidth set to 100

* Formatted entire repo (pnpm run format)
2023-08-01 10:21:22 +01:00
Eric Allam 36ec16babe feat: encrypt values in SecretStore using an encryption key (#240)
* feat: encrypt values in SecretStore using an encryption key and `aes-256-gcm`

* Stopped using apiCors in the connectionId API endpoint
2023-08-01 10:11:47 +01:00
Matt Aitken 92233f2e90 Core package and move to @latest (#235)
* Changeset should ignore the example projects

* Removed note in Contributing instructions about not adding a changeset for internal

* Renamed @trigger.dev/internal to @trigger.dev/core. Set sdk and internal to be ES2020, so we don’t get errors about private identifiers

* Env vars in nextjs-example use square bracket syntax to avoid Turbo Repo errors

* Set the tsconfigs back for core and sdk

* @examples/nextjs compile error with undefined tasks

* Set the example projects to use ES2015 to avoid private modifier complaints

* package-tester example, which will use built packages

* package-tester package.json

* Created a readme for the package-tester

* Added all the packages to package-tester

* Create .env.local.example and readme instructions

* Added name to the package.json

* Upgraded @types/react and @types/react-dom everywhrre, so we can use server actions

* Upgraded @types/react and @types/react-dom everywhrre, so we can use server actions

* Reworked the react package build, so it generates separate files

* The SDK no longer bundles core

* package-tester setup with a server action and react hooks

* Added new SDK methods with logging

* Working tsup settings for client and server

* Explicit react hooks return types

* Added all the hooks for testing

* Added OpenAI step to the job to check types are still ok in integrations

* Get rid of rogue Changeset ignores

* Changeset: @trigger.dev/core is now a separate package

* Exited prerelease mode, added a changeset

* Changed @next to @latest

* Deleted seed.js, this shouldn’t be committed

* Ignore seed.js

* The webapp was importing @trigger.dev/core with a folder path rather package name…

* More webapp imports instead of @trigger.dev/core were a folder path

* Accidentally edited Stripe internal API url in the comment

* Added “sideEffects”: false so @trigger.dev/core is tree shaken by Remix
2023-07-31 18:47:34 +01:00
Matt Aitken b09ecb0b06 Update CONTRIBUTING.md 2023-07-31 16:06:39 +01:00
Matt Aitken 10fef0f667 Improved the migration instructions 2023-07-31 16:06:12 +01:00
Jemilu Mohammed 495acf4319 modify step 3 of migrations instructions 2023-07-27 10:17:40 +00:00
Jemilu Mohammed dce86119d4 add instructions for adding migrations to the project 2023-07-27 10:14:49 +00:00
alubọsa 8f050b40c1 chore(docs): add changeset guide to contributing.md (#219)
* chore(docs): add changeset guide to contributing.md

* chore: text changes to changeset guide

* Update CONTRIBUTING.md

---------

Co-authored-by: Eric Allam <eallam@icloud.com>
2023-07-26 17:30:58 +01:00
Ashutosh Bhadauriya 3cf01ff14c Add testing CLI changes guide to contributing.md (#220)
* Add testing CLI changes guide to contributing.md

* Update backticks
2023-07-26 17:18:42 +01:00
Hans Bignon K. Tognon 0eef5c2f01 Adds steps to contributing guide for running some jobs locally (#209)
* chore(contrib): added a few jobs

* chore(contrib): adds sample jobs for repo setup

* chore(contrib): refactor examples/jobs-starter

* docs(contrib): add steps to add starter jobs

* fix(contrib): typo

* chore(contrib): minor edits

* chore(contrib): update guide and starter code
2023-07-26 14:27:27 +01:00