Compare commits

...

857 Commits

Author SHA1 Message Date
Mihai Popescu e0998e6049 Merge branch 'main' into fix/tri-6732-hover-vertical-timeline-does-not-update-on-task-runs 2026-02-08 21:57:34 +02:00
nicktrn e536d35b17 fix(ci): fix docker image publishing and worker builds (#3013)
## Summary
- **Fix Docker publish automation**: The `v.docker.*` tags pushed by the
release workflow using `GITHUB_TOKEN` don't trigger the publish workflow
(GitHub Actions limitation to prevent infinite loops). Added a
`workflow_call` to `publish.yml` directly from the release job so Docker
images are built automatically after npm publish. Tags are still pushed
for reference.
- **Fix worker Containerfiles**: The coordinator, docker-provider, and
kubernetes-provider builds have been failing since the superjson
vendoring change in `@trigger.dev/core` (#2949). The Containerfiles now
run `bundle-vendor` before `build:bundle` to generate the vendor files
that esbuild needs.

### Context
- Docker images on GHCR have been stuck at v4.3.0 — v4.3.1, v4.3.2,
v4.3.3 tags existed on GitHub but never triggered publish runs
- The worker builds (publish-worker) have been failing on every push to
main since Jan 30

## Test plan
- [x] Verified kubernetes-provider Containerfile builds locally with the
fix
- [x] Manually dispatched publish workflow for v4.3.1 — all jobs
succeeded
<!-- devin-review-badge-begin -->

---

<a
href="https://app.devin.ai/review/triggerdotdev/trigger.dev/pull/3013"
target="_blank">
  <picture>
<source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-open-in-devin-review-dark.svg?v=1">
<img
src="https://static.devin.ai/assets/gh-open-in-devin-review-light.svg?v=1"
alt="Open with Devin">
  </picture>
</a>
<!-- devin-review-badge-end -->
2026-02-06 13:56:39 +00:00
DKP b96a0b70d4 Docs: Clarify AI tool compatibility and expand context snippet (#3000)
This pull request overhauls the "Building with AI" documentation
section. It includes a comprehensive restructuring of the main
building-with-ai page with new setup guides and troubleshooting
sections, reorganizes the navigation hierarchy to elevate
mcp-agent-rules as a top-level page, and updates multiple documentation
pages to clarify the relationships between three AI tools: Skills, Agent
Rules, and MCP Server. Changes also include formatting improvements,
such as replacing italicized text with inline code formatting, and
consistent additions of explanatory Note blocks and CardGroup components
across related pages.
2026-02-05 15:22:44 -08:00
James Ritchie 3bb9aac014 Fix(webapp): Prevent big numbers on Queue page from jumping around when animating (#3007) 2026-02-05 07:45:40 -08:00
Saadi Myftija 283f88b203 feat(webapp): add triggered via field to deployment details page (#2850)
Display the deployment trigger source (CLI, CI/CD, Dashboard, GitHub
Integration) with appropriate icons on the deployment details page. The
triggeredVia field was already in the database but not displayed.

Co-authored-by: Claude <noreply@anthropic.com>
2026-02-05 15:24:23 +01:00
James Ritchie c55af7bead fix(webapp): ask ai button missing tooltip (#2964)
Fixes
- the tooltip not displaying on the AskAI button in the side menu
- incorrect AskAI button heights
- Small UI tweaks
<!-- devin-review-badge-begin -->

---

<a
href="https://app.devin.ai/review/triggerdotdev/trigger.dev/pull/2964">
  <picture>
<source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-open-in-devin-review-dark.svg?v=1">
<img
src="https://static.devin.ai/assets/gh-open-in-devin-review-light.svg?v=1"
alt="Open with Devin">
  </picture>
</a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: Mihai Popescu <mihai.popescu.dev@gmail.com>
2026-02-05 03:17:18 +02:00
Iss db4fb9eeef docs: Add Hookdeck example (#3005)
Add documentation for integrating Hookdeck with Trigger.dev to receive
webhooks and forward them to Trigger.dev tasks.
<!-- devin-review-badge-begin -->

---

<a
href="https://app.devin.ai/review/triggerdotdev/trigger.dev/pull/3005">
  <picture>
<source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-open-in-devin-review-dark.svg?v=1">
<img
src="https://static.devin.ai/assets/gh-open-in-devin-review-light.svg?v=1"
alt="Open with Devin">
  </picture>
</a>
<!-- devin-review-badge-end -->
2026-02-04 16:50:59 -08:00
mpcgird f99c780c28 fixed on leave edge case 2026-02-05 00:34:49 +02:00
mpcgird 171c50f72a Added event for recalculating vertical line hover effect on timeline span 2026-02-04 23:19:31 +02:00
mpcgird dcd73f129e Merge branch 'main' into fix/tri-6732-hover-vertical-timeline-does-not-update-on-task-runs 2026-02-04 18:45:09 +02:00
Iss c0595700f8 docs: clarify .env.local loading and idempotency key reset scope (#2996)
Document that .env.local is automatically loaded during dev, and clarify
that backend-triggered idempotency keys should be reset with global
scope
2026-02-04 09:37:35 -05:00
Iss 6a45f5623b docs: multi-tenant applications and concurrency limits (#2961)
Adds an example of multi-tenant applications as alternative to
project/limit increase, and more info about queue times and concurrency
2026-02-04 09:33:05 -05:00
Iss 104f720f6f docs(troubleshooting): add COULD_NOT_FIND_EXECUTOR error and IPv4 support (#2950)
Document COULD_NOT_FIND_EXECUTOR error with dynamic imports and IPv4
database connection limitation in troubleshooting guide.
2026-02-04 09:30:51 -05:00
Iss e017913021 docs(self-hosting): added graphile worker troubleshooting to docs (#2883)
Add troubleshooting documentation for graphile worker schema migration
failures and PostgreSQL SSL certificate issues that prevent worker
initialization.
2026-02-04 09:29:01 -05:00
Matt Aitken 7781e2aad1 docs: usage function examples were missing the imports (#2830)
Closes #2828
2026-02-04 14:05:59 +00:00
Saadi Myftija 8e0034484c feat(supervisor): project-based scheduling affinity for image cache locality (#2995)
⚒️ Publish Worker (v4) / build (supervisor) (push) Has been cancelled
Adds optional pod affinity so pods from the same project prefer
scheduling on the same node. This can help improve image cache hit
rates; subsequent pods benefit from already-pulled image layers,
reducing startup time.

Complements the built-in ImageLocality scheduler plugin by helping
during burst scheduling scenarios. Pod affinity sees scheduled pods
immediately, while ImageLocality only sees images after they're fully
pulled.

Configuration:
- `KUBERNETES_PROJECT_AFFINITY_ENABLED` - Enable/disable (default:
false)
- `KUBERNETES_PROJECT_AFFINITY_WEIGHT` - Scheduler weight 1-100
(default: 50)
- `KUBERNETES_PROJECT_AFFINITY_TOPOLOGY_KEY` - Topology key (default:
kubernetes.io/hostname)

Uses soft (preferred) affinity so pods always schedule even if preferred
node is full.

<!-- devin-review-badge-begin -->

---

<a
href="https://app.devin.ai/review/triggerdotdev/trigger.dev/pull/2995">
  <picture>
<source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-open-in-devin-review-dark.svg?v=1">
<img
src="https://static.devin.ai/assets/gh-open-in-devin-review-light.svg?v=1"
alt="Open with Devin">
  </picture>
</a>
<!-- devin-review-badge-end -->
2026-02-04 14:39:47 +01:00
Eric Allam b72cacc671 feat(debounce): add maxDelay option to limit total debounce time (#2984) 2026-02-02 20:15:06 +00:00
Mihai Popescu 1ccb8c186f changed schema id (#2983)
Fixed duplicate schema id for clickhouse
2026-02-02 02:43:39 +02:00
nicktrn 279102c17c fix(cli): reject execute() immediately when child process is dead (#2978)
## Summary
- When a child process crashes and a retry (`RETRY_IMMEDIATELY`) is
attempted on the same `TaskRunProcess`, `execute()` hangs forever
because the IPC send is silently skipped and the attempt promise can
never resolve
- This caused runner pods to stay up indefinitely with no heartbeats or
polls
- Fix: reject the attempt promise immediately when the child is not
connected, so the controller can proceed to warm start or exit

## Test plan
- [x] Added `taskRunProcess.test.ts` — verifies `execute()` rejects
promptly instead of hanging when the child process is dead
- [x] Deploy and verify no more stuck runner pods accumulate over time
2026-01-30 16:44:46 +00:00
Eric Allam b221719c09 chore(repo): fixed missing dependency in pnpm lockfile (#2976)
<!-- devin-review-badge-begin -->

---

<a
href="https://app.devin.ai/review/triggerdotdev/trigger.dev/pull/2976">
  <picture>
<source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-open-in-devin-review-dark.svg?v=1">
<img
src="https://static.devin.ai/assets/gh-open-in-devin-review-light.svg?v=1"
alt="Open with Devin">
  </picture>
</a>
<!-- devin-review-badge-end -->
2026-01-30 10:02:30 +00:00
Matt Aitken e6861f4fe4 Fix: run page logs keep refreshing when a run finishes (#2971)
Closes #2798

When a run finished the logs UI could get stuck and so be pending and
never update again. If you did a hard reload it would be correct.

This happened because when we insert a log/span we ping Redis which
causes a reload of the UI. However there was a race condition – the
insert into ClickHouse can take a while so we were refreshing the UI too
early. Then never refreshing it again.
  
Changes
- Send refresh pings every 5s to keep run page logs live
- Throttle updates so the run UI is never updated more than once per
second
- Stop auto-reloading when a run has been completed for >= 30s
- Add type inference improvements for the throttle function
<!-- devin-review-badge-begin -->

---

<a
href="https://app.devin.ai/review/triggerdotdev/trigger.dev/pull/2971">
  <picture>
<source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-open-in-devin-review-dark.svg?v=1">
<img
src="https://static.devin.ai/assets/gh-open-in-devin-review-light.svg?v=1"
alt="Open with Devin">
  </picture>
</a>
<!-- devin-review-badge-end -->
2026-01-30 09:47:56 +00:00
Matt Aitken bc7ce78103 fix(sdk): export AnyOnStartAttemptHookFunction type (#2966)
Export AnyOnStartAttemptHookFunction type to allow defining
onStartAttempt hooks for individual tasks.

https://claude.ai/code/session_018jgSVcFtKVyv65ktGNQFFq
<!-- devin-review-badge-begin -->

---

<a
href="https://app.devin.ai/review/triggerdotdev/trigger.dev/pull/2966">
  <picture>
<source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-open-in-devin-review-dark.svg?v=1">
<img
src="https://static.devin.ai/assets/gh-open-in-devin-review-light.svg?v=1"
alt="Open with Devin">
  </picture>
</a>
<!-- devin-review-badge-end -->

Co-authored-by: Claude <noreply@anthropic.com>
2026-01-30 09:17:37 +00:00
Eric Allam 9937823a7f Standardize @types/node to version 20.14.14 across monorepo (#2970)
##  Checklist

- [ ] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [ ] The PR title follows the convention.
- [ ] I ran and tested the code works

---

## Description

This PR standardizes the `@types/node` dependency across the entire
monorepo to version `20.14.14`. Previously, different packages were
using different versions (ranging from 12.20.55 to 22.13.9), which could
cause type conflicts and inconsistencies.

### Changes Made

1. **tsconfig.json** - Added `"node"` to the `types` array in
`apps/webapp/tsconfig.json` to ensure Node.js types are properly
recognized
2. **package.json overrides** - Added `@types/node` version override to
`20.14.14` in the root `package.json`
3. **pnpm-lock.yaml** - Updated lock file to reflect the standardized
version across all packages and their dependencies
4. **Fixture package.json** - Updated
`packages/cli-v3/e2e/fixtures/emit-decorator-metadata/package.json` to
use the standardized version

This ensures consistent type definitions across the monorepo and
prevents version mismatches that could lead to type errors or unexpected
behavior.

---

## Testing

- Verified that all package references to `@types/node` now point to
version `20.14.14`
- Confirmed that the lock file properly reflects the override across all
transitive dependencies
- Ensured TypeScript configuration includes Node.js types for proper
type checking

---

## Changelog

- Standardized `@types/node` to version `20.14.14` across all packages
in the monorepo
- Added `"node"` to TypeScript compiler types in webapp configuration
- Updated all package dependencies to use the consistent version through
pnpm overrides

💯

https://claude.ai/code/session_018eqp2LvvErkFSN9oK5xBh1
<!-- devin-review-badge-begin -->

---

<a
href="https://app.devin.ai/review/triggerdotdev/trigger.dev/pull/2970">
  <picture>
<source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-open-in-devin-review-dark.svg?v=1">
<img
src="https://static.devin.ai/assets/gh-open-in-devin-review-light.svg?v=1"
alt="Open with Devin">
  </picture>
</a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: Eric Allam <ericallam@users.noreply.github.com>
2026-01-30 09:15:35 +00:00
Eric Allam 3925f8cc49 fix(core): vendor superjson to fix ESM/CJS compatibility (#2949)
Bundle superjson and its dependency (copy-anything) during build to
avoid
ERR_REQUIRE_ESM errors on Node.js versions that don't support
require(ESM)
by default (< 22.12.0) and AWS Lambda which intentionally disables it.

- Add scripts/bundle-superjson.mjs to bundle superjson with esbuild
- Update build script to bundle vendor files before tshy compilation
- Move superjson from dependencies to devDependencies
- Update imports to use vendored bundles

Fixes #2937
<!-- devin-review-badge-begin -->

---

<a
href="https://app.devin.ai/review/triggerdotdev/trigger.dev/pull/2949">
  <picture>
<source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-open-in-devin-review-dark.svg?v=1">
<img
src="https://static.devin.ai/assets/gh-open-in-devin-review-light.svg?v=1"
alt="Open with Devin">
  </picture>
</a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: Eric Allam <ericallam@users.noreply.github.com>
2026-01-30 09:15:02 +00:00
Iss 01208fde27 chore(docs): Add playwright workaround (#2973)
Adds a workaround to playwright to browser download failures based on
this GH issue: https://github.com/triggerdotdev/trigger.dev/issues/2440
<!-- devin-review-badge-begin -->

---

<a
href="https://app.devin.ai/review/triggerdotdev/trigger.dev/pull/2973">
  <picture>
<source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-open-in-devin-review-dark.svg?v=1">
<img
src="https://static.devin.ai/assets/gh-open-in-devin-review-light.svg?v=1"
alt="Open with Devin">
  </picture>
</a>
<!-- devin-review-badge-end -->
2026-01-30 08:43:23 +00:00
Eric Allam 5e049cde3a fix(run-engine): avoid NAPI string overflow in getExecutionSnapshotsSince by only fetching waitpoints for latest snapshot (#2972) 2026-01-29 19:03:54 +00:00
Matt Aitken 72c357125b Query enabled via feature flag (#2968)
- Renamed FeatureFlag functions to be singular where it makes sense.
- Added function to handle multiple feature flags
- canAccessQuery now checks the global feature flag and environment
variable as well
<!-- devin-review-badge-begin -->

---

<a
href="https://app.devin.ai/review/triggerdotdev/trigger.dev/pull/2968">
  <picture>
<source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-open-in-devin-review-dark.svg?v=1">
<img
src="https://static.devin.ai/assets/gh-open-in-devin-review-light.svg?v=1"
alt="Open with Devin">
  </picture>
</a>
<!-- devin-review-badge-end -->
2026-01-29 15:30:56 +00:00
DKP 0674d74bbb Added Trigger.dark theme to the docs (#2967)
<!-- devin-review-badge-begin -->

---

<a
href="https://app.devin.ai/review/triggerdotdev/trigger.dev/pull/2967">
  <picture>
<source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-open-in-devin-review-dark.svg?v=1">
<img
src="https://static.devin.ai/assets/gh-open-in-devin-review-light.svg?v=1"
alt="Open with Devin">
  </picture>
</a>
<!-- devin-review-badge-end -->
2026-01-29 13:59:27 +00:00
DKP 9e08712749 Add building with ai/skills pages and updated intro (#2962)
Closes #<issue>

##  Checklist

- [ ] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [ ] The PR title follows the convention.
- [ ] I ran and tested the code works

---

## Testing

_[Describe the steps you took to test this change]_

---

## Changelog

_[Short description of what has changed]_

---

## Screenshots

_[Screenshots]_

💯

<!-- devin-review-badge-begin -->

---

<a
href="https://app.devin.ai/review/triggerdotdev/trigger.dev/pull/2962">
  <picture>
<source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-open-in-devin-review-dark.svg?v=1">
<img
src="https://static.devin.ai/assets/gh-open-in-devin-review-light.svg?v=1"
alt="Open with Devin">
  </picture>
</a>
<!-- devin-review-badge-end -->
2026-01-29 13:55:43 +00:00
Matt Aitken f53db6fd16 Query: time limits, performance improvements, styling (#2953)
Summary
- Query: add time limits, performance improvements, and styling updates

Changes
- Add ClickHouse output_text and error_text columns with indexes
- Automatically use _text columns for JSON based on query pattern;
support JSON column data prefixes
- Add idempotency key and scope columns
- Add enforcedWhereClause for tenant and time restrictions, instead of
the old tenant stuff.
- Implement basic time filter limiting and set default time period based
on plan; show message when results are clipped
- UX: resizable code area (including vertical splits), collapsible
sidebar, fix table/chart vertical sizing, max height for chart legend in
fullscreen
- Styling and UI tweaks: improved chart legend styling, more chart
colours, thinner line chart stroke, pricing callout color, improved
layout for callouts
- Features: generate and save AI titles
<!-- devin-review-badge-begin -->

---

<a
href="https://app.devin.ai/review/triggerdotdev/trigger.dev/pull/2953">
  <picture>
<source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-open-in-devin-review-dark.svg?v=1">
<img
src="https://static.devin.ai/assets/gh-open-in-devin-review-light.svg?v=1"
alt="Open with Devin">
  </picture>
</a>
<!-- devin-review-badge-end -->
2026-01-29 13:02:47 +00:00
Oskar Otwinowski c0b86efbd3 feat(webapp): Add MiddleTruncate component for long task names (#2946)
Closes #

##  Checklist

- [ ] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [ ] The PR title follows the convention.
- [ ] I ran and tested the code works

---

## Testing

Tested the MiddleTruncate component in the TasksDropdown by:
1. Verifying that long task names (e.g.,
"namespace:category:subcategory:task-name") are truncated in the middle
2. Confirming the full text appears in a tooltip on hover
3. Testing responsive behavior - truncation adjusts when the container
is resized
4. Verifying that short task names that fit within the container are
displayed in full without truncation

---

## Changelog

Added a new `MiddleTruncate` primitive component that intelligently
truncates text in the middle while preserving the beginning and end
portions. This is particularly useful for long hierarchical identifiers
like task slugs.

**Key features:**
- Truncates text in the middle with an ellipsis (…) when it exceeds
available width
- Shows full text in a tooltip on hover when truncated
- Responsive - recalculates truncation on container resize using
ResizeObserver
- Maintains minimum character visibility (4 chars minimum on each side
for readability)
- Integrated into TasksDropdown to handle long task names

**Changes:**
- Created new `MiddleTruncate.tsx` component with binary search
algorithm for optimal character distribution
- Updated TasksDropdown to use MiddleTruncate for task slug display
- Increased TasksDropdown popover width from 240px to 360px to provide
better space for truncated text

---

## Screenshots
💯


https://github.com/user-attachments/assets/a7a2191a-2e36-437e-ab3f-517fe7620b93


<!-- devin-review-badge-begin -->

---

<a
href="https://app.devin.ai/review/triggerdotdev/trigger.dev/pull/2946">
  <picture>
<source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-open-in-devin-review-dark.svg?v=1">
<img
src="https://static.devin.ai/assets/gh-open-in-devin-review-light.svg?v=1"
alt="Open with Devin">
  </picture>
</a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-01-29 12:51:58 +01:00
Mihai Popescu e29e1c86d9 Fix/tri 7032 logs page feedback (#2947)
##  Changes
### UI & UX
- Normalized log level display across table and detail view
- Fixed table header scroll behavior and sidebar positioning
- Improved loading state with taller segment and disabled resizing
- Added "no more logs" message with count
- Enhanced keyboard shortcuts
### Filtering & Search
- Streamlined filters: RunId and Task only (removed run filters)
- Side panel closes when filters change
- Fixed logs from previous search remaining in table
- Fixed table scroll position when changing filters
### Backend
- Added performance indexes on message and attributes
(`014_add_task_runs_v2_search_indexes.sql`)
- Added DEBUG level logging by default
- Removed internal logs from display
- Fixed ServiceValidationError forwarding to frontend
  - Removed v1 logs API support
2026-01-29 12:47:59 +02:00
mpcgird 2066843998 Merge remote-tracking branch 'origin/main' into fix/tri-6732-hover-vertical-timeline-does-not-update-on-task-runs 2026-01-28 21:08:37 +02:00
James Ritchie 34203d6a6f Fix(webapp) prevent incidents url being called every frame (#2956) 2026-01-27 18:33:36 +00:00
James Ritchie d4e4fbd7fc fix(webapp): Truncate long branch names to prevent breaking the onboarding layout (#2954)
### Before
<img width="1736" height="964" alt="CleanShot 2026-01-21 at 20 11 08@2x"
src="https://github.com/user-attachments/assets/76c6ed42-d2a0-4212-aaad-6fb848025c85"
/>

### After
<img width="1776" height="1032" alt="CleanShot 2026-01-27 at 17 32
47@2x"
src="https://github.com/user-attachments/assets/030ba26e-6aad-4232-a9ff-f2fde9931f8c"
/>

<!-- devin-review-badge-begin -->

---

<a
href="https://app.devin.ai/review/triggerdotdev/trigger.dev/pull/2954">
  <picture>
<source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-open-in-devin-review-dark.svg?v=1">
<img
src="https://static.devin.ai/assets/gh-open-in-devin-review-light.svg?v=1"
alt="Open with Devin">
  </picture>
</a>
<!-- devin-review-badge-end -->
2026-01-27 18:26:43 +00:00
James Ritchie 49de105862 Feat(webapp): collapsible side menu (#2939)
- Better separation between main feature pages and manage pages
- Toggle via keyboard shortcut Cmd/Ctrl + B or click collapse button
- Collapsed state persisted to database (survives page refresh)
- All menu items show tooltips when collapsed
- Smooth animations using Framer Motion and CSS transitions
- Impersonation-safe (preferences not modified when impersonating)


https://github.com/user-attachments/assets/35827922-b80a-418e-8341-eb22c9bb5ed4


<!-- devin-review-badge-begin -->

---

<a
href="https://app.devin.ai/review/triggerdotdev/trigger.dev/pull/2939">
  <picture>
<source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-open-in-devin-review-dark.svg?v=1">
<img
src="https://static.devin.ai/assets/gh-open-in-devin-review-light.svg?v=1"
alt="Open with Devin">
  </picture>
</a>
<!-- devin-review-badge-end -->
2026-01-27 17:13:21 +00:00
DKP 5fb9cc36bc fix(security): upgrade CLI deps and add overrides (#2952)
- Upgrade @modelcontextprotocol/sdk 1.24.0 → 1.25.2 (CVE-2026-0621
ReDoS)
- Upgrade tar 7.4.3 → 7.5.4+ (CVE-2026-23950 race condition)
- Add pnpm overrides for transitive deps:
  - qs <6.14.0 → 6.14.0 (CVE-2025-15284 DoS)
  - systeminformation <5.27.14 → 5.27.14 (CVE-2025-68154 cmd injection)
  - lodash <4.17.23 → 4.17.23 (CVE-2025-13465 prototype pollution)

---------

Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
2026-01-27 16:33:04 +00:00
nicktrn 70c8d6d14b fix(react-hooks): prevent onComplete from firing prematurely when stream disconnects (#2929)
## Summary

Fixes #2856 - The `onComplete` callback in `useRealtimeRun` was firing prematurely

## Root Cause

The callback was triggered when the long-poll stream ended, regardless
of whether the run had actually completed. Reverse proxies often close
idle connections, causing the stream to end prematurely. In this case it
was caused by fetch abort due to React strict mode.

## Fix

Changed the condition from checking if `run` exists to checking if
`run?.finishedAt` exists, ensuring `onComplete` only fires when the run
has reached a terminal state.

---------

Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: nicktrn <nicktrn@users.noreply.github.com>
2026-01-27 16:21:56 +00:00
Eric Allam eeab6bdeac fix(run-engine): fix queue cache memory leak and replace MemoryStore with LRU cache (#2945)
- Fix memory leak in RunAttemptSystem queue cache - was keying by runId
instead of queue identifier
- Replace `@unkey/cache` MemoryStore with new LRUMemoryStore for O(1)
operations and better memory bounds

## Problem

### Cache Key Bug
The queue cache in `#resolveTaskRunExecutionQueue` was keyed by `runId`,
creating one cache entry per run instead of per queue. With 1-2 hour
TTLs and 5000 entry soft cap, these accumulated causing memory growth.

### MemoryStore Performance
The `@unkey/cache` MemoryStore uses O(n) synchronous iteration for
eviction, blocking the event loop at high throughput.

## Solution

### Cache Key Fix
Changed cache key from `params.runId` to queue identifier:

```typescript
const cacheKey = params.lockedQueueId ?? `${params.runtimeEnvironmentId}:${params.queueName}`;
```

LRU Cache

Created LRUMemoryStore adapter using lru-cache package:
- O(1) get/set/delete operations
- Strict memory bounds (hard max vs soft cap)
- No event loop blocking

Test Results:

| Metric | Before Fix | After Fix |
|---|---|---|
| Queue cache entries (per 1000 runs) | ~1000 | 1 |
| Old space growth | 32.27 MB | 5.45 MB |
| Heap growth | 7.21 MB (3.9%) | 4.81 MB (2.6%) |


<!-- devin-review-badge-begin -->

---

<a
href="https://app.devin.ai/review/triggerdotdev/trigger.dev/pull/2945">
  <picture>
<source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-open-in-devin-review-dark.svg?v=1">
<img
src="https://static.devin.ai/assets/gh-open-in-devin-review-light.svg?v=1"
alt="Open with Devin">
  </picture>
</a>
<!-- devin-review-badge-end -->
2026-01-26 22:23:23 +00:00
Matt Aitken 825219a2f4 Fix: requeue a run if the DB is unavailable during dequeuing (#2938)
In the DequeueSystem if the database is unavailable we were dequeuing
from Redis and then failing to requeue in the error catcher – this was
because the requeuing required DB access.

Now if in the `catch` we encounter a DB error we requeue directly using
Redis, putting it back in the queue.
<!-- devin-review-badge-begin -->

---

<a
href="https://app.devin.ai/review/triggerdotdev/trigger.dev/pull/2938">
  <picture>
<source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-open-in-devin-review-dark.svg?v=1">
<img
src="https://static.devin.ai/assets/gh-open-in-devin-review-light.svg?v=1"
alt="Open with Devin">
  </picture>
</a>
<!-- devin-review-badge-end -->
2026-01-25 19:23:04 +00:00
bharath kumar b143027d95 Fix/sdk stream root fallback (#2874) 2026-01-25 06:53:26 +00:00
Eric Allam 409388365e fix(fair-queue): ensure concurrency is released when a message reaches visibility timeout to prevent concurrency leaks (#2907)
## Summary

Fixes a concurrency leak in the batch queue where visibility timeout
reclaims do not release concurrency slots.

**The bug:** When a message visibility timeout expires (60s),
`reclaimTimedOut` puts the message back in the queue but does NOT
release the concurrency slot. The messageId stays in the concurrency set
(`engine:batch:concurrency:tenant:{envId}`), counting against the tenant
limit even though the message is no longer in-flight.

This causes:
1. Tenant appears at capacity when checking `SCARD >= limit`
2. New messages get released back to queue instead of being processed
3. Messages stuck in infinite loop, master queue grows indefinitely

**The fix:**
- Modified `reclaimTimedOut` to capture message data (including
tenantId) BEFORE releasing from in-flight
- Returns `ReclaimedMessageInfo[]` with messageId, queueId, tenantId,
and metadata
- `#reclaimTimedOutMessages` now iterates over reclaimed messages and
calls `concurrencyManager.release()` for each

## Test plan

- [x] Added test: `should return reclaimed message info with tenantId
for concurrency release`
- [x] Added test: `should return empty array when no messages have timed
out`
- [x] Added test: `should reclaim multiple timed-out messages and return
all their info`
- [x] Updated `raceConditions.test.ts` for new return type
- [x] All tests passing
- [ ] Monitor production after deploy for concurrency leak recurrence

refs TRI-7049
<!-- devin-review-badge-begin -->

---

<a
href="https://app.devin.ai/review/triggerdotdev/trigger.dev/pull/2907">
  <picture>
<source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-open-in-devin-review-dark.svg?v=1">
<img
src="https://static.devin.ai/assets/gh-open-in-devin-review-light.svg?v=1"
alt="Open with Devin">
  </picture>
</a>
<!-- devin-review-badge-end -->
2026-01-24 13:06:35 +00:00
Gautam Singh fe5178f3e8 feat(webapp): add priority to task test and replay options (#2936)
Closes #2934

##  Checklist

- [x] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [x] The PR title follows the convention.
- [x] I ran and tested the code works

---

## Testing

_[Describe the steps you took to test this change]_

1. make sure there are no worker connected or the queue is paused.
2. run hello world tasks and assign priority
3. notice the priority value in task context.
4. try running a replay for task with priority
5. notice the context priority
---

## Changelog

_[Short description of what has changed]_
Added option to set priority on task test and replay dialog
---

## Screenshots

_[Screenshots]_
<img width="953" height="804" alt="image"
src="https://github.com/user-attachments/assets/d452b872-6c05-4a9e-a1e3-f6a283bb363d"
/>
<img width="958" height="886" alt="image"
src="https://github.com/user-attachments/assets/8f9f3096-da99-4d81-a9c5-2f3004b89ceb"
/>
<img width="1022" height="736" alt="image"
src="https://github.com/user-attachments/assets/233e01ac-a42b-4b4f-ac3e-e9b18638b971"
/>


💯

<!-- devin-review-badge-begin -->

---

<a
href="https://app.devin.ai/review/triggerdotdev/trigger.dev/pull/2936">
  <picture>
<source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-open-in-devin-review-dark.svg?v=1">
<img
src="https://static.devin.ai/assets/gh-open-in-devin-review-light.svg?v=1"
alt="Open with Devin">
  </picture>
</a>
<!-- devin-review-badge-end -->
2026-01-24 09:10:14 +00:00
Iss a3f1eb2361 chore(docs): Admin requirements for creating worker groups (#2925)
Adds documentation for creating additional worker groups via the admin
API endpoint, including how to make users admin (new vs existing users),
and clarifies that ADMIN_EMAILS only applies on signup.
2026-01-24 09:00:50 +00:00
Iss ab4b50b95a chore(docs): Add clarifications for delay semantics and payload handling (#2928)
Adds documentation notes clarifying that:
- delayed runs execute on the current deployment version
- guidance for Date objects in payloads
- static IP availability
- version locking behavior for delayed runs
2026-01-24 08:59:55 +00:00
Eric Allam 1859fd0283 chore(docs): Improvements to the idempotency key docs for the user expored itempotency key changes (#2910) 2026-01-23 18:07:12 +00:00
github-actions[bot] 700fe91bec chore: release v4.3.3 (#2893)
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 0s
🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker-v4 (push) Has been skipped
🚀 Publish Trigger.dev Docker / units (push) Failing after 0s
This PR was opened by the [Changesets
release](https://github.com/changesets/action) GitHub action. When
you're ready to do a release, you can merge this and publish to npm
yourself or [setup this action to publish
automatically](https://github.com/changesets/action#with-publishing). If
you're not ready to do a release yet, that's fine, whenever you add more
changesets to main, this PR will be updated.


# Releases
## @trigger.dev/build@4.3.3

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.3.3`

## trigger.dev@4.3.3

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.3.3`
    -   `@trigger.dev/build@4.3.3`
    -   `@trigger.dev/schema-to-json@4.3.3`

## @trigger.dev/core@4.3.3

### Patch Changes

- Add support for AI SDK v6 (Vercel AI SDK)
([#2919](https://github.com/triggerdotdev/trigger.dev/pull/2919))

    -   Updated peer dependency to allow `ai@^6.0.0` alongside v4 and v5
- Updated internal code to handle async validation from AI SDK v6's
Schema type

- Expose user-provided idempotency key and scope in task context.
`ctx.run.idempotencyKey` now returns the original key passed to
`idempotencyKeys.create()` instead of the hash, and
`ctx.run.idempotencyKeyScope` shows the scope ("run", "attempt", or
"global").
([#2903](https://github.com/triggerdotdev/trigger.dev/pull/2903))

- Fix batch trigger failing with "ReadableStream is locked" error when
network failures occur mid-stream. Added safe stream cancellation that
gracefully handles locked streams during retry attempts.
([#2917](https://github.com/triggerdotdev/trigger.dev/pull/2917))

- Add a maxDepth to flatten/unflattenAttributes to prevent possible
issues ([#2890](https://github.com/triggerdotdev/trigger.dev/pull/2890))

## @trigger.dev/python@4.3.3

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/sdk@4.3.3`
    -   `@trigger.dev/core@4.3.3`
    -   `@trigger.dev/build@4.3.3`

## @trigger.dev/react-hooks@4.3.3

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.3.3`

## @trigger.dev/redis-worker@4.3.3

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.3.3`

## @trigger.dev/rsc@4.3.3

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.3.3`

## @trigger.dev/schema-to-json@4.3.3

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.3.3`

## @trigger.dev/sdk@4.3.3

### Patch Changes

- Add support for AI SDK v6 (Vercel AI SDK)
([#2919](https://github.com/triggerdotdev/trigger.dev/pull/2919))

    -   Updated peer dependency to allow `ai@^6.0.0` alongside v4 and v5
- Updated internal code to handle async validation from AI SDK v6's
Schema type

- Expose user-provided idempotency key and scope in task context.
`ctx.run.idempotencyKey` now returns the original key passed to
`idempotencyKeys.create()` instead of the hash, and
`ctx.run.idempotencyKeyScope` shows the scope ("run", "attempt", or
"global").
([#2903](https://github.com/triggerdotdev/trigger.dev/pull/2903))

-   Updated dependencies:
    -   `@trigger.dev/core@4.3.3`

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2026-01-23 17:36:11 +00:00
Matt Aitken 6055c7d050 Recover runs that failed to dequeue (#2931)
There’s an edge case that means runs can end up in the
currentConcurrency set when they’re not in the correct state for
execution. This means they will be permanently stuck in queued.

Given an environmentId this will fix those runs.

This is a temporary fix while we permanently fix the issue.

---------

Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: Eric Allam <ericallam@users.noreply.github.com>
2026-01-23 16:25:58 +00:00
Matt Aitken cf1c311dea Use the replica for API auth queries (#2932)
When we auth API keys we get the environment, project and org. This is a
very hot path so even though these queries are fast they contribute a
significant percentage of total load.

This moves them to use the read replica instead.
2026-01-23 16:23:51 +00:00
Eric Allam fb94e1741f fix(dashboard): display correct batch rate limit current token value (#2927) 2026-01-23 16:13:29 +00:00
Eric Allam 022f69c8c8 fix(v3): redesign of batch completion to prevent heavy row-level contention on the BatchTaskRun (#2930) 2026-01-23 15:52:52 +00:00
Eric Allam d893b26ed2 fix(engine): store costInCents and usageDurationMs on the TaskRun table via existing run engine updates (#2926)
Moving usage updates into the run engine to prevent inefficient &
additional incremental updates to the TaskRun table. Read/Modify/Write
pattern is safe inside of the run engine because of the run lock. We can
also now cap the usageDurationMs value from overflowing and causing an
error.

## Why?

This is preventing at least one update per TaskRun and instead updating
these values piggybacking on other updates.

## Aurora PostgreSQL Reader Consistency Notes

### TL;DR
Aurora readers share the same storage as the writer, but maintain
separate in-memory page caches. This means:
- **Storage is always consistent** - writes are synchronously committed
to shared storage
- **Page cache can lag** - typically <100ms, but can cause stale reads
if data is cached

### How It Works
1. Writer commits to shared storage (synchronous 4/6 quorum)
2. Writer sends cache invalidation messages to readers (asynchronous)
3. If reader has data in cache → returns cached (potentially stale)
value
4. If reader has cache miss → fetches from shared storage (always
current)

### Monitoring
```sql
SELECT server_id,
       CASE WHEN session_id = 'MASTER_SESSION_ID' THEN 'Writer' ELSE 'Reader' END AS role,
       replica_lag_in_msec
FROM aurora_replica_status();
```
2026-01-23 12:34:04 +00:00
Eric Allam 6f26acb581 fix(run-engine): use reader for pending version queries (#2924)
Move expensive findMany queries for PENDING_VERSION and
WAITING_FOR_DEPLOY
runs to read replicas to avoid blocking migrations on the primary
database.

Changes:
- Add readOnlyPrisma to SystemResources type
- Pass readOnlyPrisma to systems in RunEngine constructor  
- Update pendingVersionSystem to use readOnlyPrisma for findMany
- Update executeTasksWaitingForDeploy to use _replica for findMany
2026-01-21 15:37:21 +00:00
Eric Allam bd449f75dc fix(migrations): Add IF NOT EXISTS to 20260116154810_add_idempotency_key_options_to_task_run (#2923)
## Summary
- Adds `IF NOT EXISTS` to the migration that adds
`idempotencyKeyOptions` column to prevent errors if the column already
exists

## Migration Checksum Fix

If you've already applied the previous version of this migration, you'll
need to update the checksum in your `_prisma_migrations` table to match
the new migration file.

**Previous checksum:**
`f8876e274e3f7735312275eb24a9c4b40f512ac12a286b2de3add47f66df5b27`
**New checksum:**
`0620a914ddbaf01279576274432e51c41f41502cd4c8de38621625380750e397`

### Fix instructions

Run this SQL command against your database:

```sql
UPDATE "_prisma_migrations"
SET checksum = '0620a914ddbaf01279576274432e51c41f41502cd4c8de38621625380750e397'
WHERE migration_name = '20260116154810_add_idempotency_key_options_to_task_run';
```

This updates the stored checksum to match the modified migration file,
allowing future migrations to proceed without checksum mismatch errors.

## Test plan
- [x] Verified migration applies cleanly on fresh database
- [ ] Verified checksum update works on database with previous migration
applied

🤖 Generated with [Claude Code](https://claude.com/claude-code)
2026-01-21 14:43:37 +00:00
Saadi Myftija cd2f536620 feat(docker): enable skipping db migrations on container startup (#2922)
Adds support for skipping Postgres migrations on container startup via
the new `SKIP_POSTGRES_MIGRATIONS` environment variable.

Set `SKIP_POSTGRES_MIGRATIONS=1` to skip migrations, matching the
existing behavior of `SKIP_CLICKHOUSE_MIGRATIONS`.
2026-01-21 13:34:00 +00:00
Matt Aitken 3056a51b82 Query improvements (#2905)
What changed
- Upgraded recharts to 2.15.2
- Added multiple chart types and components: big number, line, stacked,
bar (including zoomable & reference line), big dataset bar, and usage
graph
- Implemented custom legend with animated values, tooltip showing x-axis
data, and hover/highlight behaviors for stacks and legend
- Added loading, no-data, and invalid chart states plus loading spinners
and improved loading animations/layout
- Storybook integration: initial charts setup, separate chart files,
alphabetized menu, chart state toggles, and story updates
- Interaction & UX improvements: zooming (drag/select), crosshair
pointer, show/select dates while zooming, prevent text selection on
drag, hide mouse wheel zoom, capped legend items, axis/legend styling
tweaks, better spacing, and min-height for charts
- Data & state handling: moved date data to route for unified zooming,
moved chartState to main Chart component, moved hard-coded/mock data out
of components, and set chart data when zooming to start/end dates
- Performance & animation: turned off/reduced chart animations, sped up
animated numbers, removed hover transitions for bars
- New UI primitives and layout: Card component, small card updates, SVG
icons, improved segmented control and popover variants, table
improvements (resizable columns, filtering, sorting, scrolling fixes)
- Various fixes and polish: tooltip style fixes, legend value updates,
hover/leave state resets, bar width fixes for small datasets,
type/import fixes, and numerous small style/typo tweaks

---------

Co-authored-by: James Ritchie <james@trigger.dev>
2026-01-21 13:07:07 +00:00
Eric Allam 23ec5ff8fa feat(sdk): add support for AI SDK v6 (#2919)
## Summary
- Add support for Vercel AI SDK v6 as a peer dependency
- Update internal code to handle async validation from AI SDK v6's
Schema type

Closes #2918

Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: Eric Allam <ericallam@users.noreply.github.com>
2026-01-20 22:31:41 +00:00
Eric Allam 31e4753122 fix(sdk): handle locked ReadableStream when retrying batch trigger (#2917)
When fetch crashes mid-stream during batch item upload (e.g., connection
reset, timeout), the request stream may remain locked by fetch's
internal reader. Attempting to cancel a locked stream throws 'Invalid
state: ReadableStream is locked', causing the batch operation to fail.

Added safeStreamCancel() helper that gracefully handles locked streams
by catching and ignoring the locked error. The stream will be cleaned up
by garbage collection when fetch eventually releases the reader.

Fixes customer issue where batchTrigger failed with ReadableStream
locked error during network instability.
2026-01-20 15:55:23 +00:00
Eric Allam 8bc6b99285 fix(batch-queue): allow batch queue consumers to run independently from the run engine worker (#2916)
new environment variable `BATCH_QUEUE_WORKER_ENABLED` now can be used
independently from `RUN_ENGINE_WORKER_ENABLED`
2026-01-20 15:26:55 +00:00
Eric Allam 87167524cc fix(api): prevent null idempotency keys in responses, only undefined accepted (#2912) 2026-01-20 14:06:37 +00:00
Eric Allam 36168b3eb6 feat(sdk): expose user-provided idempotency key and scope in task context (#2903)
## Summary
- Store the original user-provided idempotency key and scope alongside
the hash
- Expose `ctx.run.idempotencyKey` as the user-provided key (not the
hash)
- Add `ctx.run.idempotencyKeyScope` to show the scope ("run", "attempt",
or "global")

<img width="539" height="450" alt="CleanShot 2026-01-19 at 11 40 46"
src="https://github.com/user-attachments/assets/b6f42991-697e-4314-a164-aef77b8fd25c"
/>

  ## Problem
Idempotency keys were hashed (SHA-256) before storage, making debugging
difficult since users couldn't see the value they originally set or
search for runs by idempotency key.

  ## Solution
Attach metadata to the `String` object returned by
`idempotencyKeys.create()` using a Symbol, extract it in the SDK before
the API call, and store it in the database alongside the hash.

  ```typescript
const key = await idempotencyKeys.create("my-key", { scope: "global" });
  await childTask.triggerAndWait(payload, { idempotencyKey: key });

  // In child task:
  ctx.run.idempotencyKey      // "my-key" (previously showed the hash)
  ctx.run.idempotencyKeyScope // "global"
```

  Test plan

  - Trigger task with idempotencyKeys.create() using different scopes (run, attempt, global)
  - Verify ctx.run.idempotencyKey returns user-provided key
  - Verify ctx.run.idempotencyKeyScope returns correct scope
  - Verify PostgreSQL stores idempotencyKeyOptions JSON
  - Verify ClickHouse receives idempotency_key_user and idempotency_key_scope via replication

---------

Co-authored-by: James Ritchie <james@trigger.dev>
2026-01-20 11:23:07 +00:00
mintlify[bot] c859be9c53 Document new Limits page in dashboard (#2908)
⚒️ Publish Worker (v4) / build (supervisor) (push) Has been cancelled
Added documentation for the new Limits page feature that allows users to
view their current limits, quotas, and rate limit usage in real-time
from the dashboard. The page displays rate limit token availability,
quota usage, and plan features for organizations.

## Files changed
- `docs/limits.mdx` - Added introductory paragraph about the new Limits
page in the dashboard

Generated from [feat(webapp): New limits
page](https://github.com/triggerdotdev/trigger.dev/pull/2885) @samejr

Co-authored-by: mintlify[bot] <109931778+mintlify[bot]@users.noreply.github.com>
Co-authored-by: James Ritchie <james@trigger.dev>
2026-01-19 11:30:02 +00:00
Saadi Myftija b7f7d88623 feat(supervisor): add per-machine-preset resource request ratios (#2906)
Adds support to configure CPU/memory request ratios per machine preset.
Falls back to the global request ratio configs if no specific override
is specified.

Runs across different machine presets have different usage patters, so
this enables use to manage the available capacity better.
2026-01-19 12:26:05 +01:00
Eric Allam 72594a46ee fix(dashboard): properly cleanup trace pubsub redis clients to redis/memory/elu leaks in the dashboard (#2901) 2026-01-16 15:20:30 +00:00
Matt Aitken 5504e7f8cc Fix schedule limit counting (#2899)
Fix schedule counting
- Inactive TaskSchedule not counted
- Inactive environments (archived preview branches) not counted

Clarified per-project limits descriptions.
2026-01-16 13:56:49 +00:00
James Ritchie 7a7c4b1a82 feat(webapp): New limits page (#2885)
<img width="1381" height="1362" alt="CleanShot 2026-01-14 at 13 41 02"
src="https://github.com/user-attachments/assets/0537dccf-60c7-4ab7-a0e4-3164eac1e97d"
/>

---------

Co-authored-by: Matt Aitken <matt@mattaitken.com>
2026-01-15 18:03:06 +00:00
Dan 733894bb4f Impersonation log (#2896)
Closes #<issue>

##  Checklist

- [ ] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [ ] The PR title follows the convention.
- [ ] I ran and tested the code works

---

## Testing

_[Describe the steps you took to test this change]_

---

## Changelog

_[Short description of what has changed]_

---

## Screenshots

_[Screenshots]_

💯

---------

Co-authored-by: Matt Aitken <matt@mattaitken.com>
2026-01-15 16:54:45 +00:00
Matt Aitken b696bbb1df Add TaskScheduleInstance projectId (#2897)
Summary
- Add nullable projectId field to TaskScheduleInstance.
- Create an index for TaskScheduleInstance.environmentId (added only if
it doesn’t exist, handled concurrently).
- Ensure TaskScheduleInstance.projectId is set everywhere in the
codebase.

Backfilling projectIds, once this is live

```sql
UPDATE "TaskScheduleInstance" tsi
SET "projectId" = ts."projectId"
FROM "TaskSchedule" ts
WHERE tsi."taskScheduleId" = ts."id";
```
2026-01-15 15:06:52 +00:00
Eric Allam aa69b9027d fix(repo): undo node.js supervisor upgrades and use the multiplatform node.js digest in Dockerfile (#2895) 2026-01-15 11:49:37 +00:00
Mihai Popescu 0b0df071bf logs-page-fixes (#2889)
* Removed EVENT_REPOSITORY_CLICKHOUSE_ROLLOUT_PERCENT
* Added hasLogsPageAccess featureFlag for logs page
* Replaced attributes with attributes_text for logs to reduce memory
usage and improve query performance
* Added support for event_v1 for logs, now depending on the settings the
logs are fetched either from `task_events_v1` or `task_events_v2`
* Show an error in the interface in cast the repository store is
`postgres`
2026-01-15 11:07:01 +00:00
Eric Allam 936bddf198 fix: upgrade Node.js to 20.20.0 to address async_hooks DoS vulnerability (#2890)
## Summary

- Upgrades Node.js from 20.19.0 to 20.20.0 (and 22.12.0 to 22.22.0 for
supervisor) to address the async_hooks stack overflow DoS vulnerability
- Adds `maxDepth` parameter (default 128) to `flattenAttributes` and
`unflattenAttributes` to prevent stack overflow on maliciously deep
nested structures

## Details

The vulnerability (patched in Node.js 20.20.0, 22.22.0, 24.13.0, 25.3.0)
causes unrecoverable crashes (exit code 7) when stack overflow occurs
during async_hooks callbacks. Since the webapp uses `AsyncLocalStorage`,
it was theoretically vulnerable.

### Changes

**Node.js version updates:**
- `docker/Dockerfile`: 20.11.1 → 20.20.0
- `apps/supervisor/Containerfile`: 22-alpine → 22.22.0-alpine
- `.nvmrc`: 20.19.0 → 20.20.0
- `apps/supervisor/.nvmrc`: 22.12.0 → 22.22.0
- `references/prisma-7/.nvmrc`: 20.19.0 → 20.20.0
- All GitHub workflows: 20.19.0 → 20.20.0

**Defense in depth:**
- Added `maxDepth` parameter to `flattenAttributes()` and
`unflattenAttributes()` in `packages/core` to prevent stack overflow on
deeply nested user input

## Test plan

- [x] All existing `flattenAttributes` tests pass (50 tests)
- [x] New tests for depth limiting added
- [x] Verify Docker builds work with new base images
2026-01-15 10:47:44 +00:00
Eric Allam b1e21cf03b feat(api): add admin endpoint for updating org feature flags (#2891) 2026-01-15 10:41:34 +00:00
Eric Allam c3f2d07708 chore(repo): remove claude code review (keep claude bot) (#2887) 2026-01-15 09:32:30 +00:00
SHIVA REDDY VANJA 4dc504d9d0 docs: fix waitpoint token completion request body field (#2888)
Fixed documentation examples to use correct 'data' field instead of
'output' for the waitpoint token completion endpoint.

The API schema expects 'data' in the request body, but all code examples
(curl, Python, Ruby, Go) incorrectly showed 'output', causing waitpoints
to complete with empty/undefined output when users followed the docs.

Fixes #2872

Closes #<issue>

##  Checklist

- [x] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [x] The PR title follows the convention.
- [x] I ran and tested the code works

---

## Testing

_[Describe the steps you took to test this change]_
This is simple doc fix
---

## Changelog

_[Short description of what has changed]_
Documentation example for the specific api route had a wrong field,
fixed that to have correct field

---

## Screenshots

_[Screenshots]_
<img width="780" height="252" alt="Screenshot 2026-01-14 at 10 49 50 PM"
src="https://github.com/user-attachments/assets/50a03f2c-edfb-4bd4-bb72-a0fd79e77216"
/>
The above image shows the correct request format, but docs previously
had incorrect payload.
<img width="723" height="307" alt="Screenshot 2026-01-14 at 11 24 01 PM"
src="https://github.com/user-attachments/assets/b9096225-3f7c-4511-b8c2-e8144c896900"
/>
This is the exact wrong field in docs, that was fixed
https://trigger.dev/docs/wait-for-token#from-another-language


💯

Co-authored-by: appdevelopers9a <appdeveloper@s9alabs.com>
2026-01-14 19:33:39 +00:00
Eric Allam a5339a1bac fix(dashboard): remove span ID to original span ID cache and use a different approach to link to the cached runs (#2886) 2026-01-14 14:36:29 +00:00
Matt Aitken 5b07bd11a0 chore: FeatureFlag add createdAt and updatedAt (#2880)
It’s useful to know when they were modified for debugging and auditing.

For existing rows createdAt and updatedAt are set to now() during the
migration, to avoid a nullable column.
2026-01-14 14:28:06 +00:00
nicktrn 260fb7cd70 fix(helm): support secrets.existingSecret for core secrets (#2860)
When users set `secrets.enabled=false` to use an external secret via
`secrets.existingSecret`, the environment variables `SESSION_SECRET`,
`MAGIC_LINK_SECRET`, `ENCRYPTION_KEY`, and `MANAGED_WORKER_SECRET` were
not being populated from the secret.

Fixes #2859

Also adds automatic helm prereleases for PRs

---------

Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: nicktrn <nicktrn@users.noreply.github.com>
2026-01-14 10:44:08 +00:00
Copilot 9934627c0b Revert "Set EVENT_REPOSITORY_DEFAULT_STORE default to clickhouse_v2" (#2881)
Reverts commit 495a2531f0.

Restores `EVENT_REPOSITORY_DEFAULT_STORE` default from `"clickhouse_v2"`
back to `"postgres"`.

##  Checklist

- [x] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [x] The PR title follows the convention.
- [x] I ran and tested the code works

---

## Testing

Single-line env schema default change; no runtime testing required.

---

## Changelog

- Reverted `EVENT_REPOSITORY_DEFAULT_STORE` default value from
`"clickhouse_v2"` to `"postgres"` in `apps/webapp/app/env.server.ts`

---

## Screenshots

N/A - configuration change only

💯

<!-- START COPILOT CODING AGENT SUFFIX -->



<!-- START COPILOT ORIGINAL PROMPT -->



<details>

<summary>Original prompt</summary>

> Can you revert the latest commit
495a2531f0 and create a PR for the revert


</details>



<!-- START COPILOT CODING AGENT TIPS -->
---

 Let Copilot coding agent [set things up for
you](https://github.com/triggerdotdev/trigger.dev/issues/new?title=+Set+up+Copilot+instructions&body=Configure%20instructions%20for%20this%20repository%20as%20documented%20in%20%5BBest%20practices%20for%20Copilot%20coding%20agent%20in%20your%20repository%5D%28https://gh.io/copilot-coding-agent-tips%29%2E%0A%0A%3COnboard%20this%20repo%3E&assignees=copilot)
— coding agent works faster and does higher quality work when set up for
your repo.

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: ericallam <534+ericallam@users.noreply.github.com>
2026-01-14 10:38:08 +00:00
Matt Aitken 495a2531f0 Set EVENT_REPOSITORY_DEFAULT_STORE default to clickhouse_v2 (#2879)
Change EVENT_REPOSITORY_DEFAULT_STORE default from "postgres" to
"clickhouse_v2" so new deployments favor the ClickHouse v2 event store
by default. This updates runtime behavior to use the newer store
implementation unless explicitly overridden.
2026-01-13 18:06:00 +00:00
Matt Aitken 1bca378000 Query fixes (#2876)
Don’t allow aliased columns to be queried – it was actually safe but
confusing. We call `created_at` -> `triggered_at` but we still allowed
created_at which was confusing.

Now we have nice errors if you try select columns that aren’t
selectable.

Also removed a ClickHouse setting `allow_experimental_object_type` which
worked fine locally but stopped all queries working on ClickHouse Cloud
🤦‍♂️
2026-01-13 17:43:25 +00:00
nicktrn b042b0b2f9 fix(webapp): cancel button in API key regeneration modal (#2878)
The Cancel button was missing an onClick handler to close the modal
dialog. This caused confusing behavior where clicking Cancel would not
dismiss the dialog. Also added type="button" to prevent form submission
since the button is inside a form.

Co-authored-by: Claude <noreply@anthropic.com>
2026-01-13 18:26:13 +01:00
Dan 3be55db8f8 Add smart spreadsheet docs (#2877) 2026-01-13 17:21:00 +00:00
Eric Allam bb253400a2 perf(runs-replication): Improve the CPU efficiency and throughput of the runs replication to clickhouse (#2866)
## Summary

Optimizes the runs replication service for better CPU efficiency and
throughput when inserting task runs into ClickHouse.

### Key Changes

- **Switch to compact array format** - Uses
`JSONCompactEachRowWithNames` instead of `JSONEachRow` for ClickHouse
inserts, reducing JSON serialization overhead
- **Type-safe tuple arrays** - Introduces `TaskRunInsertArray` and
`PayloadInsertArray` tuple types with compile-time column order
validation
- **Pre-sorted batch inserts** - Sorts inserts by primary key before
flushing for better ClickHouse insert performance
- **Programmatic index generation** - `TASK_RUN_INDEX` and
`PAYLOAD_INDEX` are generated from column arrays to prevent manual
synchronization errors

### Files Changed

- `runsReplicationService.server.ts` - Core optimization to use compact
array inserts
- `@internal/clickhouse` - Added `insertCompactRaw` method and tuple
types
- `taskRuns.ts` - Column definitions, index constants, and insert
functions

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-13 12:18:30 +00:00
Mihai Popescu c8686b5f1c feat: tri-6738 Create aggregated logs page (#2862)
Closes #<issue>

##  Checklist

- [x] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [x] The PR title follows the convention.
- [x] I ran and tested the code works

---

## Testing
- Verified log detail view displays correctly with message, metadata,
and attributes
- Tested search highlighting functionality in log messages (escapes
special regex characters)
- Confirmed tabs (Details/Run) switch properly with keyboard shortcuts
(d/r)
  - Verified run information loads via async fetcher in Run tab
  - Tested close button and Escape key for dismissing the panel
- Verified log details display correct information: level badges, kind
badges, timestamps, trace IDs, span IDs
  - Confirmed links to parent spans and run pages work correctly
- Tested with various log levels (ERROR, WARN, INFO, DEBUG, TRACE) and
kinds (SPAN, SPAN_EVENT, LOG_*)
- Verified admin-only fields display correctly when user has admin
access
- Tested data loading states and error states (log not found, run not
found)


---

## Changelog

Created new Logs page. 
The information shown is gathered from the spans from each run.
The feature supports all run filters with two new filters for level and
logs text search.


---

## Screenshots

<img width="2059" height="1196" alt="Logs page preview"
src="https://github.com/user-attachments/assets/70b667b4-98cc-4728-855a-2766dd5c1aa5"
/>

💯

---------

Co-authored-by: James Ritchie <james@trigger.dev>
2026-01-13 11:21:13 +00:00
Iss dfb46d8847 chore(docs): add WASM external configuration documentation (#2871)
Adds a WASM block to the External section documenting that WASM packages
must be added to the `external` array in `trigger.config.ts`.
2026-01-13 11:12:54 +00:00
Matt Aitken 9942518e49 TRQL/Query improvements (#2870)
Summary
- Improve query experience and safety across ClickHouse and TSQL.

Changes
- Display JSON columns when in non-pretty mode (no longer show [Object
Object]).
- Sanitize ClickHouse errors originating from TSQL.
- Remove tenant details from errors.
- Add AI-assisted error-fixing for queries.
- Improve code quality and readability.
- Provide autocomplete support for enum values.
- Enforce limits on ClickHouse queries (10s query limit).
- Add org-level and global concurrency limits.
- Warn and train AI to avoid SELECT *; when used, only return core
columns and show info.
- If AI suggests no time range, default to past 7 days.
- Format the default query for readability.
- Add an admin-only EXPLAIN button.
- Prevent impersonation queries from being saved to history.
2026-01-13 11:12:15 +00:00
Saadi Myftija a3c387697e feat(supervisor): add node affinity rules for large machine worker pool scheduling (#2869)
⚒️ Publish Worker (v4) / build (supervisor) (push) Has been cancelled
**Background**
Runs with `large-1x` or `large-2x` machine presets are disproportionally
affected by scheduling delays during peak times. This is in part caused
by the fact that the worker pool is shared for all runs, meaning large
runs compete with smaller runs for available capacity. Because large
runs require significantly more CPU and memory, they are harder for the
scheduler to bin-pack onto existing nodes, often requiring a node with a
significant amount of free resources or waiting for a new node to spin
up entirely. This effect is amplified during peak times when nodes are
already densely packed with smaller workloads, leaving insufficient
contiguous resources for large runs. Also, large runs make up a small
percentage of the total runs.

**Changes**

This PR adds Kubernetes node affinity settings to separate large and
standard machine workloads across node pools.

- Controlled via `KUBERNETES_LARGE_MACHINE_POOL_LABEL` env var (disabled
when not set)
- Large machine presets (large-*) get a soft preference to schedule on
the large pool, with fallback to standard nodes
- Non-large machines are excluded from the large pool via required
anti-affinity
- This ensures the large machine pool is reserved for large workloads
while allowing large workloads to spill over to standard nodes if needed
2026-01-13 10:54:47 +01:00
James Ritchie 7a9490893c fix(webapp): Fix for missing table rows divider lines in safari (#2873) 2026-01-12 21:14:11 +00:00
James Ritchie 768206c91b feat(dashboard): Upgrade to the dateTime filter UI (#2864)
UI/UX improvement to the date/time picker:

- You can now choose a custom duration
- Adds a new DateTimePicker.tsx component, using a new shadcn
Calendar.tsx component
- Clear UI separation between the 2 actions, choosing a duration or
choosing date range
- Adds new quick select options for picking a date range quickly


https://github.com/user-attachments/assets/6b59b49d-2a56-4354-ad72-d8426437e56e

---------

Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: Eric Allam <ericallam@users.noreply.github.com>
2026-01-12 13:01:17 +00:00
mpcgird 12f508a949 Merge branch 'main' into fix/tri-6732-hover-vertical-timeline-does-not-update-on-task-runs 2026-01-12 14:35:32 +02:00
Eric Allam 2c30c2defb chore(claude): add CLAUDE.md and claude skill for writing trigger.dev tasks + add 4.3.0 rule set (#2867)
- Add CLAUDE.md providing Claude Code guidance and documenting the
Claude Code skill
- Add trigger-dev-tasks skill to assist writing Trigger.dev tasks
- Add SDK rules version 4.3.0 including batch trigger v2 and debouncing
features
2026-01-12 11:01:06 +00:00
Matt Aitken 839d5e8eba Allow query access when impersonating (#2863)
Co-authored-by: Mihai Popescu <mihai.popescu.dev@gmail.com>
2026-01-09 14:57:09 +00:00
Eric Allam 42dfb87ebb chore(ci): pre-pull test container images to prevent dockerhub rate limiting when running tests (#2861) 2026-01-09 12:15:53 +00:00
Matt Aitken 49df40cb11 TRQL and the Query page (#2843)
TRQL (pronounced Treacle like the delicious British dark sweet syrup) is
the TRiggerQueryLanguage. It allows users to safely write queries on
their data. The queries are safely turned into ClickHouse queries which
are tenant-safe and not SQL injectable.


https://github.com/user-attachments/assets/bbfca473-b3fc-4150-8fe6-79e8840a2d29

This started out as a translation of HogQL by PostHog from Python to
TypeScript.

Features
- Tenant safe queries.
- Many underlying ClickHouse features including functions and
aggregations.
- Virtual columns, which are exposed to users as real columns but are
actually expressions.
- Transformations of data types and where clauses.
- Simple JSON path querying.
- Limits on execution time.
- Reporting of query statistics.

## Query page

There’s a new Query page (currently behind a feature flag) where you can
write TRQL queries and execute them against your environment, project or
organization.

Features
- Executing TRQL queries
- Syntax highlighting and errors
- Autocomplete
- AI generation/editing of queries
- Help and examples
- Table with auto-inferred data types from the table schema
- Table cell renderers for our special types like Run ids, environments,
machines, tasks, queues, etc.
- Copy/export as CSV/JSON
- Line and bar graphs with grouping and stacking
- History of queries
2026-01-09 11:39:36 +00:00
Eric Allam cf0aa9b3ca core(claude): correctly setup repo and deps for claude code agent (#2858) 2026-01-09 10:27:23 +00:00
Eric Allam a8024afd0a chore(claude): add some allowed tools to the claude github workflow agent (#2857) 2026-01-09 10:17:29 +00:00
Eric Allam 57ba2528b2 feat(runs): use metrics instead of spans in the Runs Replication service (#2851) 2026-01-08 15:56:44 +00:00
Eric Allam 7d29f5aac9 fix(webapp): use useFetcher for env var edits to preserve scroll/toggle state (#2847)
Replaces `redirectDocument` with `useFetcher` for editing environment
variables. This allows background form submission without full page
reload, which preserves:

- Scroll position in the env vars list
- "Reveal values" toggle state
- Search filter state

Fixes #2845

Generated with [Claude Code](https://claude.ai/code)

Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: Eric Allam <ericallam@users.noreply.github.com>
2026-01-08 12:51:07 +00:00
Eric Allam 36b0762100 feat(metrics): add observable gauge for batch queue worker length (#2848) 2026-01-08 12:50:46 +00:00
Eric Allam f5b2ccb494 chore(docs): add docs for handling batch trigger errors (#2838) 2026-01-08 11:33:43 +00:00
github-actions[bot] f73138fe6c chore: release v4.3.2 (#2811)
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 4s
🚀 Publish Trigger.dev Docker / units (push) Failing after 0s
🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker-v4 (push) Has been skipped
This PR was opened by the [Changesets
release](https://github.com/changesets/action) GitHub action. When
you're ready to do a release, you can merge this and publish to npm
yourself or [setup this action to publish
automatically](https://github.com/changesets/action#with-publishing). If
you're not ready to do a release yet, that's fine, whenever you add more
changesets to main, this PR will be updated.


# Releases
## @trigger.dev/build@4.3.2

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.3.2`

## trigger.dev@4.3.2

### Patch Changes

- fix(cli): update command should preserve existing package.json order
([#2810](https://github.com/triggerdotdev/trigger.dev/pull/2810))
-   Updated dependencies:
    -   `@trigger.dev/build@4.3.2`
    -   `@trigger.dev/core@4.3.2`
    -   `@trigger.dev/schema-to-json@4.3.2`

## @trigger.dev/python@4.3.2

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/sdk@4.3.2`
    -   `@trigger.dev/build@4.3.2`
    -   `@trigger.dev/core@4.3.2`

## @trigger.dev/react-hooks@4.3.2

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.3.2`

## @trigger.dev/redis-worker@4.3.2

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.3.2`

## @trigger.dev/rsc@4.3.2

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.3.2`

## @trigger.dev/schema-to-json@4.3.2

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.3.2`

## @trigger.dev/sdk@4.3.2

### Patch Changes

- Improve batch trigger error messages, especially when rate limited
([#2837](https://github.com/triggerdotdev/trigger.dev/pull/2837))
-   Updated dependencies:
    -   `@trigger.dev/core@4.3.2`

## @trigger.dev/core@4.3.2

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2026-01-08 11:26:59 +00:00
Eric Allam 7c2e78c9de fix(batch): more high cardinality metric attribute fixes (#2846) 2026-01-08 10:07:49 +00:00
mintlify[bot] 6dfbe1d762 Document spike alerts feature (#2844)
Updated the billing alerts documentation to reflect the new spike alerts
feature added in PR #2829. The documentation now explains both standard
alerts (75%, 90%, 100%, 200%, 500%) and spike alerts (10x, 20x, 50x,
100x) that help catch runaway usage from bugs or errors.

## Files changed
- `docs/how-to-reduce-your-spend.mdx` - Added section explaining the two
types of billing alerts

Generated from [Chore(webapp): Adds additional billing
alerts](https://github.com/triggerdotdev/trigger.dev/pull/2829) @samejr

Co-authored-by: mintlify[bot] <109931778+mintlify[bot]@users.noreply.github.com>
2026-01-07 16:40:04 +00:00
James Ritchie 3f19b98f9f Fix(webapp): Dynamically load spline on 404 on page (#2834)
Remove the spline dependency and load it dynamically on the 404 page.

Uses a web component with a workaround for React.
2026-01-07 15:50:49 +00:00
Eric Allam 09460ab37d feat: add event loop block notifications and env flag (#2842)
Add colored console warnings when the event loop is blocked and wire
a feature flag to enable/disable notifications- Introduce
notifyEventLoopBlocked() in eventLoopMonitor.server.ts to
  log a colored warning with blocked and async type.
- Call notifyEventLoopBlocked() when an event-loop stall is detected.
- Add EVENT_LOOP_MONITOR_NOTIFY_ENABLED to env schema with a default of
  "0" so notifications are off by default.
- Will notify when over the `EVENT_LOOP_MONITOR_THRESHOLD_MS` env var

This makes it easier to spot long event-loop stalls during development
or when notifications are explicitly enabled.

<img width="840" height="132" alt="CleanShot 2026-01-07 at 15 03 24@2x"
src="https://github.com/user-attachments/assets/be20fa6a-be2b-46a1-aa89-d0913ed8b5b3"
/>
2026-01-07 15:04:50 +00:00
Eric Allam 062766e974 fix(batch): optimize processing batch trigger v2 (#2841)
This PR fixes some issues with the new BatchQueue by implementing the
full two-phase dequeue process in the FairQueue, and moving the
responsibility of consuming the worker queue to the BatchQueue and
independently enabling it via the `BATCH_QUEUE_WORKER_QUEUE_ENABLED` env
var. We've also introduced the `BATCH_QUEUE_SHARD_COUNT` env var to
control the count of master queue shards in the FairQueue. We can also
control how many queues are considered in each iteration of the master
queue consumer via the `BATCH_QUEUE_MASTER_QUEUE_LIMIT` env var.

This PR will also now skip trying to dequeue from tenants that are at
concurrency capacity, which should lead to fewer issues with low
concurrency tenants blocking higher concurrency tenants from processing.
2026-01-07 15:00:09 +00:00
James Ritchie c68eb2d951 fix(webapp): concurrency limits modal cancels and resets limit on enter (#2804)
The Override concurrency limit modal has 2 type="submit" buttons. The
first one in the DOM was firing when the "enter" key is hit which
canceled and reset the limit instead which is a bad UX.

### The fix
This fix adds a hidden button above in the DOM order which mirrors the
Update Override button. Having a double submit button is rare in our
modals so feels safe to add this to the specific modal that needs it.

### Alternative solution
Switching the order of the buttons in the main FormButton component,
then using `flex-row-reverse` to flip them back in CSS works, but it
reverses the tab order. Adding a `tabIndex` to fix that issue didn't
seem to work reliably.
2026-01-07 14:14:23 +00:00
James Ritchie 583d299891 chore(webapp): Concurrency page UI improvements (#2825)
### UI Improvements to the Concurrency page: 

- Truncates long branch names and includes a tooltip
- The Tables have a new variant if you don't want the rows to highlight
on hover
- Small fix to pluralize some words in the purchase modal
- Fix to prevent tooltip buttons being `type=submit`
- Updates the /limits docs page to include purchasing more concurrency
- Adds a clear banner when you have a positive balance of unallocated
concurrency



https://github.com/user-attachments/assets/54d927c3-84e3-4d55-8f42-726098f4daf0
2026-01-07 14:13:36 +00:00
James Ritchie 17338050b1 Chore(webapp): Adds additional billing alerts (#2829)
- Adds 4 additional alert thresholds to ensure customers are emailed if
they have runaway usage.
- Separated these into a new section called "Spike alerts" with a
tooltip so it's clear what they are.
- Tooltip message is: "Catch runaway usage from bugs or errors. We
recommend keeping these enabled as a safety net."
- A billing service PR now returns all orgs to populate the email list,
rather than oldest 5.
- Adds `defaultChecked` logic to honour existing orgs who have
configured alerts in the DB. New orgs get all alerts checked on by
default.

<img width="1316" height="1560" alt="CleanShot 2026-01-05 at 09 43
56@2x"
src="https://github.com/user-attachments/assets/ce749407-2b7f-4864-9c09-9333c5ac495a"
/>
2026-01-07 14:10:10 +00:00
James Ritchie c7a238c664 Chore(webapp): Queues page UI improvements (#2826)
Very small UI improvement to increase the spacing between the right hand
table columns. Also a tooltip wording improvement.
2026-01-07 14:07:50 +00:00
Eric Allam 8ba7526d51 fix(batch): rate limiting by token bucket no longer incorrectly goes negative (#2837)
Also improves the BatchTriggerError when a result of getting rate
limited.
2026-01-07 14:02:19 +00:00
nicktrn 47bed15255 docs: fix self-hosting section in deploy docs (#2836)
Fixes #2835

There were still some flags in here we removed, deploying is a lot
simpler now for self-hosters.

Also updates the github actions guide.

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-01-06 20:20:20 +00:00
Eric Allam 752ad32293 Add Claude Code GitHub Workflow (#2839) 2026-01-06 19:26:31 +00:00
James Ritchie db0df17a6a chore(webapp): Run navigation UI improvements (#2802)
**Improvements to the run ID copy button and run navigation buttons for
consistency**

- Adds some x-padding and layout adjustment to the copy ID button.
<img width="664" height="114" alt="CleanShot 2025-12-19 at 16 05 21@2x"
src="https://github.com/user-attachments/assets/ebc8e0de-011b-419c-bdcc-eb4157553d1c"
/>

- New custom navigation icons that work better at tiny sizes
<img width="330" height="196" alt="CleanShot 2025-12-19 at 16 06 38@2x"
src="https://github.com/user-attachments/assets/bfd8d6b8-8a65-4eac-9ce1-d70acf0ad265"
/>

Some other small improvements/fixes:
- Fixes a browser html error where there was a <button> inside a
<button>
- Updates the shortcut description to match the tooltip text for
consistency
- Made the hover states more consistent
- The shortcut bar at the bottom snaps to the list sooner because there
are more items now
2026-01-06 18:20:54 +00:00
Eric Allam edf5b142fc fix(fair-queue): Prevent unbounded memory growth from metrics cardinality explosion (#2819) 2025-12-25 08:51:40 +00:00
Eric Allam 4c3dfac223 fix(fair-queue): prevent unbounded cooloff states growth (#2818) 2025-12-24 16:33:38 +00:00
nicktrn 7ccbbdb368 docs: tidy up sync env vars note blocks (#2817) 2025-12-24 13:02:26 +00:00
Eric Allam 71279a7b12 fix(fair-queue): prevent unbounded memory growth by cleaning up queue descriptor and cooloff state cache (#2816) 2025-12-24 10:40:32 +00:00
Eric Allam 2eba36c086 chore(redis-worker): add otel spans to fair queue processing pipeline (#2815) 2025-12-24 00:37:24 +00:00
Eric Allam 29827e96f2 fix(batch): add batch queue back into master queue after visibility timeout (#2814) 2025-12-24 00:25:46 +00:00
Dan d416f340ad Added example projects link (#2812)
Closes #<issue>

##  Checklist

- [ ] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [ ] The PR title follows the convention.
- [ ] I ran and tested the code works

---

## Testing

_[Describe the steps you took to test this change]_

---

## Changelog

_[Short description of what has changed]_

---

## Screenshots

_[Screenshots]_

💯
2025-12-23 18:37:19 +00:00
nicktrn 7a54a843e2 fix(cli): update command should preserve existing package.json order (#2810)
This fixes a regression introduced in #2778 - stable sort is required
for deterministic builds, but we can safely preserve order for the user
package.json during package updates
2025-12-23 12:52:06 +00:00
nicktrn 52e9baede5 fix(webapp): don't rate limit deployment finalization (#2809)
turns out we also had v2 and v3 routes
2025-12-23 09:22:00 +00:00
Eric Allam deb80890fe chore(otel): add spans to the batch queue processing pipeline (#2808) 2025-12-23 08:40:33 +00:00
Eric Allam d82089686c fix(batch): extract the queue name out of an already nested queue option (#2807) 2025-12-22 21:40:14 +00:00
Oskar Otwinowski acc10e847c chore(docs): add more cost saving tips (#2806)
Co-authored-by: Eric Allam <eallam@icloud.com>
2025-12-22 15:36:54 +00:00
Eric Allam f1a83cffc4 chore(docs): debounce options (#2797) 2025-12-22 15:35:51 +00:00
Eric Allam 61fee91830 chore(docs): upgrade for new batch trigger limits and functionality (#2787) 2025-12-22 15:35:43 +00:00
github-actions[bot] caa40ce925 chore: release v4.3.1 (#2788)
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 0s
🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker-v4 (push) Has been skipped
🚀 Publish Trigger.dev Docker / units (push) Failing after 4s
This PR was opened by the [Changesets
release](https://github.com/changesets/action) GitHub action. When
you're ready to do a release, you can merge this and publish to npm
yourself or [setup this action to publish
automatically](https://github.com/changesets/action#with-publishing). If
you're not ready to do a release yet, that's fine, whenever you add more
changesets to main, this PR will be updated.


# Releases
## @trigger.dev/build@4.3.1

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.3.1`

## trigger.dev@4.3.1

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.3.1`
    -   `@trigger.dev/build@4.3.1`
    -   `@trigger.dev/schema-to-json@4.3.1`

## @trigger.dev/core@4.3.1

### Patch Changes

- Added support for idempotency reset
([#2777](https://github.com/triggerdotdev/trigger.dev/pull/2777))

## @trigger.dev/python@4.3.1

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/sdk@4.3.1`
    -   `@trigger.dev/core@4.3.1`
    -   `@trigger.dev/build@4.3.1`

## @trigger.dev/react-hooks@4.3.1

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.3.1`

## @trigger.dev/redis-worker@4.3.1

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.3.1`

## @trigger.dev/rsc@4.3.1

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.3.1`

## @trigger.dev/schema-to-json@4.3.1

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.3.1`

## @trigger.dev/sdk@4.3.1

### Patch Changes

- feat: Support for new batch trigger system
([#2779](https://github.com/triggerdotdev/trigger.dev/pull/2779))
- feat(sdk): Support debouncing runs when triggering with new debounce
options
([#2794](https://github.com/triggerdotdev/trigger.dev/pull/2794))
- Added support for idempotency reset
([#2777](https://github.com/triggerdotdev/trigger.dev/pull/2777))
-   Updated dependencies:
    -   `@trigger.dev/core@4.3.1`

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2025-12-22 15:29:18 +00:00
Dan ba9b0e17c1 Added Claude Agent SDK guide and examples (#2803) 2025-12-19 17:10:32 +00:00
Oskar Otwinowski 469808cf09 fix(webapp): Make spans fluid again while task is executing (#2801)
Closes #<issue>

##  Checklist

- [x] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [x] The PR title follows the convention.
- [x] I ran and tested the code works

---

## Testing

Tested on executing tasks and on moving between tasks

---

## Changelog

Small UI fix for spans on the task runs page

---

## Screenshots



https://github.com/user-attachments/assets/72095b03-c74f-4472-afde-9c63e6e7c224



💯
2025-12-19 15:45:54 +01:00
Mihai Popescu 7574c69c2d feat(webapp): Add support for resetting idempotency keys (#2777)
Add support for resetting idempotency keys both from ui and sdk

##  Checklist

- [x] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [x] The PR title follows the convention.
- [x] I ran and tested the code works

---

## Testing
- Created a new run with a idempotency idempotencyKey.
- Started a new run with the same task and got redirected to the first
run.
- Deleted the key from the UI on the run details
- Started a new run with the same task  and it created a new one
- Did the above steps using the SDK


---

## Changelog

- Add new action route for resetting idempotency keys via UI
- Add reset button in Idempotency section of run detail view
- Added API and SDK for resetting imdepotency
- Updated docs page for this feature


---

## Screenshots

_[Screenshots]_
<img width="438" height="363" alt="Screenshot 2025-12-11 at 11 56 37"
src="https://github.com/user-attachments/assets/30b8ef5e-8aac-4d04-b57a-9bf30d085dcb"
/>
2025-12-19 11:55:04 +00:00
Eric Allam 06cbe6e3ca chore(docs): upgrade sentry guide to use new global init and global hooks (#2799) 2025-12-19 00:00:02 +00:00
Eric Allam 3875bb292a feat(engine): run debounce system (#2794)
Adds support for **debounced task runs** - when triggering a task with a
debounce key, subsequent triggers with the same key will reschedule the
existing delayed run instead of creating new runs. This continues until
no new triggers occur within the delay window.

## Usage

```typescript
await myTask.trigger({ userId: "123" }, {
  debounce: {
    key: "user-123-update",
    delay: "5s",
    mode: "leading", // default
  }
});
```

- **key**: Scoped to the task identifier
- **delay**: How long to wait before executing (supports duration
strings like `"5s"`, `"1m"`)
- **mode**: Either `"leading"` or `"trailing"`. Leading debounce will
use the payload and options from the first run created with the debounce
key. Trailing will use payload and options from the last run.

### "trailing" mode overrides

When using `mode: "trailing"` with debounce, the following options are
updated from the **last** trigger:

- **`payload`** - The task input data
- **`metadata`** - Run metadata
- **`tags`** - Run tags (replaces existing tags)
- **`maxAttempts`** - Maximum retry attempts
- **`maxDuration`** - Maximum compute time
- **`machine`**  - Machine preset (cpu/memory)

## Behavior

- **First run wins**: The first trigger creates the run, subsequent
triggers push its execution time later
- **Idempotency keys take precedence**: If both are specified,
idempotency is checked first
- **Max duration**: Configurable via `DEBOUNCE_MAX_DURATION_MS` env var
(default: 10 minutes)

Works with `triggerAndWait` - parent runs correctly block on the
debounced run.
2025-12-18 16:04:43 +00:00
Saadi Myftija ff80742ab7 fix(ci): use workflow_dispatch instead of repository_dispatch for publish.yml (#2791)
Mixed up the trigger in #2790
2025-12-16 16:53:22 +01:00
Saadi Myftija 11366e658c chore(ci): add manual trigger to the image publishing workflow (#2790)
Useful to retry failures manually.
2025-12-16 16:34:57 +01:00
Eric Allam e751f8832e fix(app): incorrectly duplicated env vars for new batch trigger system (#2789) 2025-12-16 15:27:13 +00:00
Eric Allam a999d9ea3f feat(engine): Batch trigger reloaded (#2779)
New batch trigger system with larger payloads, streaming ingestion,
larger batch sizes, and a fair processing system.

This PR introduces a new `FairQueue` abstraction inspired by our own
`RunQueue` that enables multi-tenant fair queueing with concurrency
limits. The new `BatchQueue` is built on top of the `FairQueue`, and
handles processing Batch triggers in a fair manner with per-environment
concurrency limits defined per-org. Additionally, there is a global
concurrency limit to prevent the BatchQueue system from creating too
many runs too quickly, which can cause downstream issues.

For this new BatchQueue system we have a completely new batch trigger
creation and ingestion system. Previously this was a single endpoint
with a single JSON body that defined details about the batch as well as
all the items in the batch.

We're introducing a two-phase batch trigger ingestion system. In the
first phase, the BatchTaskRun record is created (and possibly rate
limited). The second phase is another endpoint that accepts an NDJSON
body with each line being a single item/run with payload and options.

At ingestion time all items are added to a queue, in order, and then
processed by the BatchQueue system.

## New batch trigger rate limits

This PR implements a new batch trigger specific rate limit, configured
on the `Organization.batchRateLimitConfig` column, and defaults using
these environment variables:

- `BATCH_RATE_LIMIT_REFILL_RATE` defaults to 10
- `BATCH_RATE_LIMIT_REFILL_INTERVAL` the duration interval, defaults to
`"10s"`
- `BATCH_RATE_LIMIT_MAX` defaults to 1200

This rate limiter is scoped to the environment ID and controls how many
runs can be submitted via batch triggers per interval. The SDK handles
the retrying side.

## Batch queue concurrency limits

The new column `Organization.batchQueueConcurrencyConfig` now defines an
org specific `processingConcurrency` value, with a backup of the env var
`BATCH_CONCURRENCY_LIMIT_DEFAULT` which defaults to 10. This controls
how many batch queue items are processed concurrently per environment.

There is also a global rate limit for the batch queue set via the
`BATCH_QUEUE_GLOBAL_RATE_LIMIT` which defaults to being disabled. If
set, the entire batch queue system won't process more than
`BATCH_QUEUE_GLOBAL_RATE_LIMIT` items per second. This allows
controlling the maximum number of runs created per second via batch
triggers.

## Batch trigger settings

- `STREAMING_BATCH_MAX_ITEMS` controls the maximum number of items in a
single batch
- `STREAMING_BATCH_ITEM_MAXIMUM_SIZE` controls the maximum size of each
item in a batch
- `BATCH_CONCURRENCY_DEFAULT_CONCURRENCY` controls the default
environment concurrency
- `BATCH_QUEUE_DRR_QUANTUM` how many credits each environment gets each
round for the DRR scheduler
- `BATCH_QUEUE_MAX_DEFICIT` the maximum deficit for the DRR scheduler
- `BATCH_QUEUE_CONSUMER_COUNT` how many queue consumers to run
- `BATCH_QUEUE_CONSUMER_INTERVAL_MS` how frequently they poll for items
in the queue

### Configuration Recommendations by Use Case

**High-throughput priority (fairness acceptable at 0.98+):**

```env
BATCH_QUEUE_DRR_QUANTUM=25
BATCH_QUEUE_MAX_DEFICIT=100
BATCH_QUEUE_CONSUMER_COUNT=10
BATCH_QUEUE_CONSUMER_INTERVAL_MS=50
BATCH_CONCURRENCY_DEFAULT_CONCURRENCY=25
```

**Strict fairness priority (throughput can be lower):**

```env
BATCH_QUEUE_DRR_QUANTUM=5
BATCH_QUEUE_MAX_DEFICIT=25
BATCH_QUEUE_CONSUMER_COUNT=3
BATCH_QUEUE_CONSUMER_INTERVAL_MS=100
BATCH_CONCURRENCY_DEFAULT_CONCURRENCY=5
```
2025-12-16 14:32:49 +00:00
mpcgird fd46381c8b optimized the animation frame loop 2025-12-15 23:28:27 +02:00
Oskar Otwinowski 28a66ac021 fix(ui): respect rootOnlyDefault, disable adjacent run navigation in inspector views (#2781)
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 0s
🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker-v4 (push) Has been skipped
🚀 Publish Trigger.dev Docker / units (push) Failing after 0s
## Changelog

- Add disableAdjacentRows prop to TaskRunsTable component to control
table state encoding
- Pass rootOnlyDefault prop from loader to TaskRunsTable for proper
state management
- Disable adjacent run navigation in schedule, waitpoint, and other
inspector views
- Preserve adjacent run navigation on main runs list page with rootOnly
filter support
2025-12-12 12:16:29 +00:00
github-actions[bot] 7d34817473 chore: release v4.3.0 (#2770)
This PR was opened by the [Changesets
release](https://github.com/changesets/action) GitHub action. When
you're ready to do a release, you can merge this and publish to npm
yourself or [setup this action to publish
automatically](https://github.com/changesets/action#with-publishing). If
you're not ready to do a release yet, that's fine, whenever you add more
changesets to main, this PR will be updated.


# Releases
## trigger.dev@4.3.0

### Minor Changes

- feat(cli): deterministic image builds for deployments
([#2778](https://github.com/triggerdotdev/trigger.dev/pull/2778))
- feat(cli): enable zstd compression for deployment images
([#2773](https://github.com/triggerdotdev/trigger.dev/pull/2773))

### Patch Changes

- The new `triggeredVia` field is now populated in deployments via the
CLI. ([#2767](https://github.com/triggerdotdev/trigger.dev/pull/2767))
- fix(dev): stop max listeners exceeded warning messages when running
more than 10 runs concurrently
([#2771](https://github.com/triggerdotdev/trigger.dev/pull/2771))
- Upgrade @modelcontextprotocol/sdk to 1.24.3
([#2768](https://github.com/triggerdotdev/trigger.dev/pull/2768))
-   Updated dependencies:
    -   `@trigger.dev/core@4.3.0`
    -   `@trigger.dev/build@4.3.0`
    -   `@trigger.dev/schema-to-json@4.3.0`

## @trigger.dev/core@4.3.0

### Minor Changes

- feat(cli): deterministic image builds for deployments
([#2778](https://github.com/triggerdotdev/trigger.dev/pull/2778))

### Patch Changes

- The new `triggeredVia` field is now populated in deployments via the
CLI. ([#2767](https://github.com/triggerdotdev/trigger.dev/pull/2767))

## @trigger.dev/build@4.3.0

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.3.0`

## @trigger.dev/python@4.3.0

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.3.0`
    -   `@trigger.dev/build@4.3.0`
    -   `@trigger.dev/sdk@4.3.0`

## @trigger.dev/react-hooks@4.3.0

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.3.0`

## @trigger.dev/redis-worker@4.3.0

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.3.0`

## @trigger.dev/rsc@4.3.0

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.3.0`

## @trigger.dev/schema-to-json@4.3.0

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.3.0`

## @trigger.dev/sdk@4.3.0

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.3.0`

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2025-12-12 13:11:56 +01:00
mpcgird a2277d5a37 Merge branch 'main' into fix/tri-6732-hover-vertical-timeline-does-not-update-on-task-runs 2025-12-12 09:33:13 +00:00
Saadi Myftija 6d6ed471d1 feat(cli): deterministic image builds for deployments (#2778)
⚒️ Publish Worker (v4) / build (supervisor) (push) Has been cancelled
This PR makes our image builds deterministic and reproducible by
ensuring that identical source code always produces the same image
layers and image digest. This means that deployments where nothing has
changed will no longer invalidate the image cache in our worker cluster
nodes, thus avoid making the cold starts for runs worse.

**Context**
New deployments currently increase the cold start times for runs, as
they generate a new image which needs to be pulled in the worker cluster
where runs are executed. It happens also when the source code for the
deployment has not changed due to non-deterministic steps in our build
system. This addresses the latter issue by making builds reproducible.

**Main changes**
- Avoided baking `TRIGGER_DEPLOYMENT_ID` and
`TRIGGER_DEPLOYMENT_VERSION` in the image, we now pass these via the
supervisor instead.
- Used `json-stable-stringify` for consistent key ordering in the files
we generate for the build, e.g., `package.json`, `build.json`,
`index.json`.
- Removed `metafile.json` from the image contents as it is not actually
used in the container. This is only relevant for the `analyze` command.
- Added `SOURCE_DATE_EPOCH=0` and `rewrite-timestamp=true` to Docker
builds to normalize file timestamps.
- Removed some `timings` and `outputHashes` from build outputs and
manifests.

The builds are now reproducible for both native build server and Depot
paths. This should also lead to better image layer cache reuse in
general.
2025-12-12 09:48:04 +01:00
mpcgird b4d6d7859e Fix for mouse hover on a timline not being updates after resize
Refactor MousePositionProvider to improve reactivity during resize, scroll, and layout shifts.
2025-12-11 22:53:49 +00:00
Oskar Otwinowski 7f7f993587 feat(webapp): improve adjacent runs navigation and timeline performance (#2776)
- Add replace prop to LinkButton to use history replacement for adjacent
run navigation
- Preserve span and tab params when navigating between adjacent runs
- Disable animations for completed spans in timeline to improve
performance
- Include spanId in runs list navigation for better context preservation
- Direct link to task test page when filtering by single task with no
runs
- Fix minor styling issue with run friendlyId display padding
2025-12-11 17:26:15 +00:00
Oskar Otwinowski d28707826c feat(webapp): add GitHub onboarding flow to empty Tasks and Deployments pages (#2775)
##  Checklist

- [x] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [x] The PR title follows the convention.
- [x] I ran and tested the code works

---

## Testing

Set up the local github application, and tested its connection with
trigger.dev

Checked:
- Change is backwards compatible
- Actions/ux are uniform across the pages (empty Tasks, Deployments,
project settings)
- Connecting GH, Connecting Repo, disconnecting Repo, modifying settings

---

## Changelog

- Create new resource route for GitHub settings management with loader
and actions
- Add GitHubSettingsPresenter to fetch connected repos and installations
- Implement GitHubSettingsPanel component for reusable GitHub
configuration UI
- Refactor project settings page to use shared GitHubSettingsPanel
component
- Integrate GitHub connection flow into empty state onboarding for Tasks
and Deployments
- Add support for GitHub repo connection, disconnection, and branch
tracking settings
- Include redirect URL support for seamless navigation after GitHub
actions
- Remove duplicate GitHub connection code from project settings route

---

## Screenshots


https://github.com/user-attachments/assets/8fc24699-640b-4f9e-afd8-b26edc945218

🐐

---------

Co-authored-by: James Ritchie <james@trigger.dev>
2025-12-10 17:03:16 +00:00
Max Strübing 8b00198f99 docs(kubernetes): use v4 helm chart instead of beta (#2671) 2025-12-10 14:52:32 +00:00
nicktrn 74e9246bfa feat(cli): enable zstd compression for deployment images (#2773)
This will speed up ice cold starts (*) for two reasons:
- better compression ratio
- faster decompression

This is a minor release because zstd compression will now be enabled by
default for all deployments.

(*) ice cold starts happen when deploy images are not cached on the
worker node yet. These cold start durations are highly dependent on
image size and as it turns out, also the type of compression used.
2025-12-10 14:49:44 +00:00
Mihai Popescu 07a31d3732 fix(webapp) : fixed Cmd+Left Arrow was intercepted by the TreeView component (#2772)
Fixed the issue where Cmd+Left Arrow was being intercepted by the
TreeView component on task runs screen.

Solution:
Added a check in the getTreeProps keyboard handler to detect when
metaKey (Cmd on macOS) is pressed with Left Arrow. When detected, the
handler returns early without preventing the default browser behavior,
allowing Chrome's native back navigation to work.


Closes #<issue>

##  Checklist

- [ x] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [ x] The PR title follows the convention.
- [ x] I ran and tested the code works

---

## Testing

Went to the task details page and confirmed that Cmd + Left Arrow will
navigate back

---

## Changelog

Modified TreeView.tsx to check for e.metaKey before handling Left Arrow
key events
When Cmd+Left is pressed, the event is no longer prevented, allowing
browser default behavior
---

Co-authored-by: Mihai Popescu <mihaipopescu@Mihais-MacBook-Pro.local>
2025-12-10 12:05:16 +00:00
Saadi Myftija da111e220f fix(api): whitelist deployment endpoints from the general API rate limits (#2774)
Deployments are affected by general API rate limits, this is just a
quick fix by whitelisting the deployment related endpoints. In a follow
up PR we'll add a separate rate limiter for this group of endpoints.
2025-12-10 12:03:56 +01:00
Oskar Otwinowski 2c3cb4a43a feat(webapp): UX improvements for TaskRun page and TaskRun table (#2760)
##  Checklist

- [x] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [x] The PR title follows the convention.
- [x] I ran and tested the code works

---

## Testing

Manual testing of the task run pages

---

## Changelog

- Add previous/next run navigation buttons to run detail page header
- Support [ and ] keyboard shortcuts to jump between adjacent runs
- Preserve runs table state (filters, pagination) when navigating
- Preload adjacent page runs at boundaries for seamless navigation
- Add actions prop to PageTitle component
- Document shortcut in keyboard shortcuts panel

- Store current filter state from runs table as `tableState` search
param when navigating to individual run pages
- Restore filters when navigating back from run detail view to runs list
- Update `v3RunPath` and `v3RunSpanPath` helpers to accept optional
searchParams
- Use `useOptimisticLocation` to capture current search params in
TaskRunsTable
- Parse `tableState` param in run detail route and pass filters to back
button

- This improves UX by remembering filter selections (task, status, date
range, etc.) when users click into a run and then navigate back to the
runs list

- Add new text-below variant that shows "Click to copy" tooltip on hover
and "Copied" on click. Also add controlled open/onOpenChange props to
SimpleTooltip for managing tooltip visibility.

---

## Screenshots


https://github.com/user-attachments/assets/5067bbe0-1bcd-4e75-80a7-f56dabd5ed69
2025-12-09 13:10:11 +00:00
Saadi Myftija b71bf89444 chore(releases): minor improvements to the release workflow (#2764)
Changes in this PR:
- Arbitrary refs are now allowed when triggering the release workflow
manually (ref must be on the main branch).
- Release summary is now displayed in the GH job output; makes for a
nicer experience when approving the release workflow.
2025-12-09 11:29:48 +01:00
Eric Allam 28c0c78257 fix(api): triggering a batch on a v4 project via the v3 SDK no longer results in errors (#2752)
<!-- CURSOR_SUMMARY -->
> [!NOTE]
> Updates `BatchTriggerV3Service` to send `batch.id` (not `friendlyId`)
as `batchId` to `TriggerTaskService.call`.
> 
> <sup>Written by [Cursor
Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit
68bf8df4a29bdcac5bfb2806bc417541f69ffd6e. This will update automatically
on new commits. Configure
[here](https://cursor.com/dashboard?tab=bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
2025-12-09 09:54:39 +00:00
Eric Allam c021d1db63 fix(dev): stop max listeners exceeded warning messages when running more than 10 runs concurrently (#2771)
<!-- CURSOR_SUMMARY -->
> [!NOTE]
> Centralizes SIGTERM handling in `DevSupervisor` and removes per-run
SIGTERM listeners in `DevRunController` to avoid
MaxListenersExceededWarning under high concurrency.
> 
> - **Dev runtime**:
> - **SIGTERM handling**: Add centralized handler in
`packages/cli-v3/src/dev/devSupervisor.ts` to gracefully stop all run
controllers; unregisters on `shutdown()`.
> - **Cleanup**: Remove per-controller `SIGTERM` listener and handler
from `packages/cli-v3/src/entryPoints/dev-run-controller.ts` to reduce
event listeners and warnings.
> - **Changeset**: Add patch note in
`.changeset/fuzzy-ghosts-admire.md`.
> 
> <sup>Written by [Cursor
Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit
5ad2f5341829cebf6fd37a3c616a2db5e4ad936a. This will update automatically
on new commits. Configure
[here](https://cursor.com/dashboard?tab=bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
2025-12-09 09:54:28 +00:00
Eric Allam f62cdfe00e feat(dashboard): login with google and "last used" indicator (#2746)
<img width="568" height="513" alt="CleanShot 2025-12-05 at 14 27 16"
src="https://github.com/user-attachments/assets/1f44d8b9-8791-4b44-96d5-4a0960a1ab36"
/>

<!-- CURSOR_SUMMARY -->
---

> [!NOTE]
> Adds Google OAuth login and a cookie-based “last used” indicator on
the login page, with supporting backend, routes, and schema updates.
> 
> - **Auth/Backend**:
> - **Google OAuth**: Integrates `remix-auth-google` via new
`addGoogleStrategy` and enables when `AUTH_GOOGLE_CLIENT_ID/SECRET` are
set (`services/googleAuth.server.ts`, `services/auth.server.ts`).
> - **User handling**: Implements `findOrCreateGoogleUser` with
linking/upsert logic and conflict logging (`models/user.server.ts`).
> - **MFA + session**: Google/GitHub/Magic callbacks now set session,
handle MFA, and set a "last-auth-method" cookie
(`routes/auth.google*.tsx`, `routes/auth.github.callback.tsx`,
`routes/magic.tsx`, `services/lastAuthMethod.server.ts`).
> - **GitHub strategy**: Safer email check
(`services/gitHubAuth.server.ts`).
> - **Routes/UI**:
> - **Login page**: Adds "Continue with Google" button and animated
"Last used" badge based on cookie; keeps GitHub/Email options
(`routes/login._index/route.tsx`).
> - **Redirect safety**: Sanitize redirect paths and persist redirect
via cookies in auth actions (`routes/auth.github.ts`,
`routes/auth.google.ts`).
>   - **Assets**: Adds `GoogleLogo` SVG.
>   - **Avatar**: Set `referrerPolicy="no-referrer"` on profile image.
> - **Config/Schema**:
> - **Env**: Adds `AUTH_GOOGLE_CLIENT_ID`/`AUTH_GOOGLE_CLIENT_SECRET`
(`env.server.ts`).
> - **DB**: Extends `AuthenticationMethod` enum with `GOOGLE` (Prisma
schema + migration).
> - **Dependencies**:
>   - Adds `remix-auth-google` in `package.json`.
> 
> <sup>Written by [Cursor
Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit
9f84f974bd6f21f1699c4f69a6aa91616842d1b1. This will update automatically
on new commits. Configure
[here](https://cursor.com/dashboard?tab=bugbot).</sup>
<!-- /CURSOR_SUMMARY -->

---------

Co-authored-by: James Ritchie <james@trigger.dev>
2025-12-09 09:51:42 +00:00
nicktrn 66c6da7114 security: dependabot alert triage (#2768)
- Upgrade @modelcontextprotocol/sdk to 1.24.3
- Override jws to 3.2.3
2025-12-08 21:02:13 +00:00
Saadi Myftija 7fba9e9f6b feat(deployments): add build server meta and trigger source info (#2767)
This PR applies a small change to the deployments table to keep track
of:
- where the deployment was triggered from
- build server metadata, if the build server was involved
2025-12-08 16:29:26 +00:00
Saadi Myftija cf63fc9cd2 chore: update version requirement hint in build settings (#2757) 2025-12-08 09:27:28 +01:00
Saadi Myftija d1c3bfb9c9 fix(deployments): ecr repo exists check (#2762)
We recently upgraded the ECR sdk version. Our ECR repo exists check
relies on the type of the error thrown and the new ECR sdk version seems
to have broken that behavior. This PR adds a workaround to the issue.
2025-12-05 19:16:28 +01:00
Saadi Myftija d8f5853457 fix(releases): use contents: write permission in the release workflow (#2761)
Needed to push tags
2025-12-05 17:22:24 +01:00
Saadi Myftija a52566d9cc feat(releases): add a manual trigger for the package release workflow (#2759)
The manual trigger is currently enabled only for the prerelease job.
This PR adds it for the normal release flow too, as it is useful to
retrigger failed release workflows.
2025-12-05 17:00:02 +01:00
Saadi Myftija 07a1d04d52 fix(releases): add missing npm upgrade step (#2758)
Accidentally removed this in a previous PR.
2025-12-05 16:41:45 +01:00
github-actions[bot] 185f4ecaf9 chore: release v4.2.0 (#2713)
🚀 Publish Trigger.dev Docker / units (push) Failing after 0s
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 19s
🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker-v4 (push) Has been skipped
This PR was opened by the [Changesets
release](https://github.com/changesets/action) GitHub action. When
you're ready to do a release, you can merge this and publish to npm
yourself or [setup this action to publish
automatically](https://github.com/changesets/action#with-publishing). If
you're not ready to do a release yet, that's fine, whenever you add more
changesets to main, this PR will be updated.


# Releases
## trigger.dev@4.2.0

### Minor Changes

- feat(cli): upgrade bun deployments to v1.3.3
([#2756](https://github.com/triggerdotdev/trigger.dev/pull/2756))

### Patch Changes

- fix(otel): exported logs and spans will now have matching trace IDs
([#2724](https://github.com/triggerdotdev/trigger.dev/pull/2724))
- The `--force-local-build` flag is now renamed to just `--local-build`
([#2702](https://github.com/triggerdotdev/trigger.dev/pull/2702))
- fix(cli): header will always print the correct profile
([#2728](https://github.com/triggerdotdev/trigger.dev/pull/2728))
- feat: add ability to set custom resource properties through
trigger.config.ts or via the OTEL_RESOURCE_ATTRIBUTES env var
([#2704](https://github.com/triggerdotdev/trigger.dev/pull/2704))
- feat(cli): implements content-addressable store for the dev CLI build
outputs, reducing disk usage
([#2725](https://github.com/triggerdotdev/trigger.dev/pull/2725))
- Added support for native build server builds in the deploy command
(`--native-build-server`)
([#2702](https://github.com/triggerdotdev/trigger.dev/pull/2702))
-   Updated dependencies:
    -   `@trigger.dev/build@4.2.0`
    -   `@trigger.dev/core@4.2.0`
    -   `@trigger.dev/schema-to-json@4.2.0`

## @trigger.dev/build@4.2.0

### Patch Changes

- syncVercelEnvVars to skip API and read env vars directly from
env.process for Vercel build environments. New syncNeonEnvVars build
extension for syncing environment variablesfrom Neon database projects
to Trigger.dev. The extension automatically detects branches and builds
appropriate PostgreSQL connection strings for non-production, non-dev
environments (staging, preview).
([#2729](https://github.com/triggerdotdev/trigger.dev/pull/2729))
-   Updated dependencies:
    -   `@trigger.dev/core@4.2.0`

## @trigger.dev/core@4.2.0

### Patch Changes

- fix: prevent ERR_IPC_CHANNEL_CLOSED errors from causing an unhandled
exception on TaskRunProcess
([#2743](https://github.com/triggerdotdev/trigger.dev/pull/2743))
- Added support for native build server builds in the deploy command
(`--native-build-server`)
([#2702](https://github.com/triggerdotdev/trigger.dev/pull/2702))

## @trigger.dev/python@4.2.0

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/build@4.2.0`
    -   `@trigger.dev/sdk@4.2.0`
    -   `@trigger.dev/core@4.2.0`

## @trigger.dev/react-hooks@4.2.0

### Patch Changes

- fix: prevent infinite useEffect when passing an array of tags to
useRealtimeRunsWithTag
([#2705](https://github.com/triggerdotdev/trigger.dev/pull/2705))
-   Updated dependencies:
    -   `@trigger.dev/core@4.2.0`

## @trigger.dev/redis-worker@4.2.0

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.2.0`

## @trigger.dev/rsc@4.2.0

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.2.0`

## @trigger.dev/schema-to-json@4.2.0

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.2.0`

## @trigger.dev/sdk@4.2.0

### Patch Changes

- fix(sdk): Re-export schemaTask types to prevent the TypeScript error
TS2742: The inferred type of 'task' cannot be named without a reference
to '@trigger.dev/core/v3'. This is likely not portable.
([#2735](https://github.com/triggerdotdev/trigger.dev/pull/2735))
- feat: add ability to set custom resource properties through
trigger.config.ts or via the OTEL_RESOURCE_ATTRIBUTES env var
([#2704](https://github.com/triggerdotdev/trigger.dev/pull/2704))
-   Updated dependencies:
    -   `@trigger.dev/core@4.2.0`

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2025-12-05 16:27:21 +01:00
nicktrn 0d764d4c46 feat(cli): upgrade bun deployments to v1.3.3 (#2756)
New deployments with `runtime: "bun"` will now use Bun v1.3.3

Link to Bun release: https://bun.com/blog/bun-v1.3.3
2025-12-05 13:31:27 +00:00
Saadi Myftija beb52b9800 chore(releases): adjust gh workflow title (#2755)
Just a tiny workflow label change
2025-12-05 13:33:35 +01:00
Saadi Myftija 3401a1d0a9 feat(releases): require approval for package releases (#2753)
**Background**
Currently the changeset PR creation and the publishing is handled by the
same workflow. This is not ideal:
- The build steps are executed on every run of the pipeline, even though
they're only needed for the publish case.
- The PR creation workflow does not need permissions to publish to npm,
only the release path needs them.
- Adding an approval step is painful as we'd need to also approve each
changeset PR creation workflow run.

**Changes in this PR**
- Separated the changeset PR creation into its own workflow and minimum
permission set.
- Added a GH environment with an approval step for the package
publishing workflow (also for prereleases).
- New publish workflow runs will not cancel in-progress runs; helps
avoid partial failures in publishing.

These changes also enable hardening the npm OIDC setup by tying it to a
GH environment that requires approval.
2025-12-05 13:25:25 +01:00
nicktrn 3f982ed366 docs: update email regex examples (#2749)
The previous examples lacked start and end anchors
2025-12-04 17:58:25 +00:00
Saadi Myftija 4dc956470d fix(deployments): misc fixes for the native build server deployment flow (#2748)
Improved a couple of error messages.

Also fixed an issue with the s2 token caching.
2025-12-04 17:01:47 +01:00
Oskar Otwinowski 7fddadcce9 chore: Improve Vercel/Neon syncEnvVars build extensions docs (#2747)
chore: Improve Vercel/Neon `syncEnvVars` build extensions docs
2025-12-04 16:34:57 +01:00
Oskar 6e038d4d1e chore: Improve Vercel/Neon syncEnvVars extensions docs 2025-12-04 16:32:15 +01:00
Oskar Otwinowski 05b53967ea feat(build): Add NeonDB branch resolution for Vercel preview environments (#2729)
Vercel's NeonDB integration renders database connection environment
variables at runtime, which means Trigger.dev cannot directly sync these
values during the build process. This change adds support for fetching
branch-specific NeonDB connection strings via the Neon API.

feat(build): Add syncNeonEnvVars extension and improve Vercel env var
syncing

Add a new `syncNeonEnvVars` build extension for syncing environment
variables
from Neon database projects to Trigger.dev. The extension automatically
detects
branches and builds appropriate PostgreSQL connection strings for
non-production
environments (staging, dev, preview).

Features of `syncNeonEnvVars`:
- Fetches branch-specific database credentials from Neon API
- Generates all standard Postgres connection strings (DATABASE_URL,
POSTGRES_URL,
  POSTGRES_PRISMA_URL, etc.) with both pooled and unpooled variants
- Supports custom database name, role name, and env var prefix options
- Skips automatically in Vercel environments (Neon's Vercel integration
handles this)
- Skips for production environments (designed for preview/staging/dev
branches)

Improvements to `syncVercelEnvVars`:
- When running in a Vercel build environment (detected via VERCEL env
var),
values are now read from process.env instead of the Vercel API response
- This ensures the build uses the actual runtime values Vercel provides
- Removed embedded Neon-specific logic (now handled by separate
extension)
- Simplified and cleaned up the extension code

Documentation updates for both extensions with usage examples and
configuration
options.

Closes #2714

##  Checklist

- [x] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [x] The PR title follows the convention.
- [x] I ran and tested the code works

---

## Testing

Set up Vercel + Trigger.dev envs, used Vercel's chat-bot-ai template.
2025-12-04 16:26:35 +01:00
Oskar 702f3b4bca chore: Improve changeset message 2025-12-04 16:26:24 +01:00
nicktrn 8fcd93001d chore: insert missing spaces in changeset 2025-12-04 13:42:32 +00:00
Eric Allam b01b8740cc fix(engine): prevent overriding a queue concurrency limit resuming a paused queue (#2745) 2025-12-04 13:33:12 +00:00
Oskar 357aa99309 Add changeset 2025-12-04 14:12:42 +01:00
Oskar 7cbf82a4ae fix: PR feedback, doc improvements for neon/vercel syncEnvVars 2025-12-04 14:07:31 +01:00
Eric Allam e7fec4097f fix(dev): CLI now properly cleans up the store dir on dev CLI exit (#2744) 2025-12-04 13:13:01 +01:00
Eric Allam d279988e38 fix(workers): prevent ERR_IPC_CHANNEL_CLOSED errors from causing an unhandled exception on TaskRunProcess (#2743) 2025-12-04 12:02:30 +00:00
Saadi Myftija 748ae658f7 fix(releases): use npm 11.x for OIDC support in the release workflow (#2742)
Support for OIDC requires npm v11.5.1 or newer.
2025-12-04 12:21:21 +01:00
Oskar Otwinowski 249878ed92 feat(webapp): link to task-specific test page from filtered runs table (#2741)
When viewing runs filtered to a single task, the "Create a test run" and
"Run a test" buttons now navigate directly to the task-specific test
page instead of the generic test page.

This improves UX by pre-populating the test form with the filtered task,
saving users from having to manually select it again.


##  Checklist

- [x] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [x] The PR title follows the convention.
- [x] I ran and tested the code works

## Screenshots

<img width="1706" height="1392" alt="image"
src="https://github.com/user-attachments/assets/d8b5d445-73b5-426c-83a4-90ac2a95b955"
/>
2025-12-04 10:56:24 +00:00
Eric Allam 117b1d5a53 chore(dependabot): upgrade next.js for CVE-2025-66478 in d3-chat example project (#2740) 2025-12-04 10:30:37 +00:00
Eric Allam 04173a93b9 fix(replication): detect misconfigered run replication publication and output helpful error messages (#2736)
Add validation for logical replication publication configuration. Helps
diagnose an issue where runs are no longer replicated to clickhouse
because of a configuration issue with the replication publication.

## Problem

The `LogicalReplicationClient` only checked if a publication existed,
not if it was correctly configured. This caused a silent failure where:

- Replication would start successfully
- Transaction boundaries (begin/commit) were received
- **But no actual data changes were replicated**

This happened when a publication existed but:
1. Had no tables associated with it
2. Was missing required actions (e.g., `delete`)

## Solution

Added `#validatePublicationConfiguration()` method that validates:
-  Publication includes the expected table
-  Publication has all required actions configured

When validation fails, error messages include the exact SQL command to
fix the issue:

**Missing table:**
```
Publication 'task_runs_to_clickhouse_v1_publication' exists but has NO TABLES configured. 
Expected table: "public.TaskRun". 
Run: ALTER PUBLICATION task_runs_to_clickhouse_v1_publication ADD TABLE "TaskRun";
```

**Missing actions:**
```
Publication 'task_runs_to_clickhouse_v1_publication' is missing required actions. 
Expected: [insert, update, delete], Current: [insert, update], Missing: [delete]. 
Run: ALTER PUBLICATION task_runs_to_clickhouse_v1_publication SET (publish = 'insert, update, delete');
```

This prevents silent data loss and makes debugging configuration issues
much easier.
2025-12-04 10:29:13 +00:00
Saadi Myftija da4c753b68 fix(releases): add missing GH token in prerelease workflow (#2739)
The `changeset version` command needs the GH token too.
2025-12-04 11:27:11 +01:00
Eric Allam 6ae3b69745 chore(cursor): add deslop command (#2722) 2025-12-04 10:24:09 +00:00
Saadi Myftija 652d95c7eb feat(releases): add a prerelease workflow (#2737)
Adds a manual trigger to the `release.yml` workflow for publishing
prerelease versions. Needs to be in the same workflow file due to a NPM
limitation on how OIDC claims are checked.

Currently there is a validation step on the ref for the prerelease: it
must be merged to the main branch. We can revisit this in the future in
case we find it too limiting.
2025-12-04 10:17:05 +01:00
Saadi Myftija 341e27d213 chore(deployments): lazily update ECR repo cache settings (#2734)
To avoid doing a migration for the ECR repo cache settings, we lazily do
it on the next deployment for that project. Failures to update the repo
settings are just logged and will not cause the deployment to fail.
2025-12-03 17:15:59 +01:00
Eric Allam 9821d02af7 fix(sdk): Re-export schemaTask types to prevent the TypeScript error TS2742 (#2735)
Fixes this type of error when exporting a `schemaTask` in a monorepo:

```
error TS2742: The inferred type of 'helloWorldSchema' cannot be named without a reference to '@trigger.dev/core/v3'. This is likely not portable.
```
2025-12-03 16:12:23 +00:00
Saadi Myftija 255a73a2fe feat(deployments): --native-build-server support for the deploy command (#2702)
This PR adds support for CLI deployments using the native build server.

**Background**

The deployment command currently does the following:
- bundles the code
- submits the build context to our external build provider and waits for
the build
- triggers deployment state transitions using the platform API

Upstream build provider outages cause issue with deployments,
potentially blocking deployments entirely. We recently introduced the
`--force-local-build` flag as a fallback to enable deployment without a
dependency on the upstream build provider, though it requires users to
have docker in their systems. This PR continues that work by providing a
remote build path which uses our own build server and does not rely on
the external provider.

**Changes in this PR**

Introduced the new `--native-build-server` flag, which does the
following:
- scans all files relevant for the Trigger deployment and evaluates
ignore rules
- packages it up in an archive and uploads it as a deployment artifact
- queues the deployment and triggers the build
- streams logs from the build server

This no longer relies on external build services. Also deployment state
transitions happen on the server-side, giving us more flexibility to
evolve the flow and schemas of related deployment API endpoints. In
general it gives us better control of the whole build and deployment
process. This path will eventually become the default.

The `--detach` flag is also new, allowing to trigger deployments without
waiting for the result.

The deployment artifacts are uploaded via pre-signed URLs to avoid
unnecessary load on the platform. The new `/artifacts` endpoint
generates the pre-signed URLs; size limits are enforced on s3. This
endpoint is deliberately generic, we could extend it in the future to
upload other artifacts client-side in a similar way, e.g., large payload
packets.
2025-12-03 16:40:21 +01:00
Saadi Myftija c5f7a8daf7 fix(releases): explicitly use no frozen lockfile in the install step (#2733) 2025-12-03 16:28:06 +01:00
Saadi Myftija 8b0f51b317 chore(releases): automatically update the lockfile in changeset PRs (#2732)
This is a step which we currently need to do manually and it's rather
painful. The lockfile update is necessary due to cross references in our
packages.

Added it as a separate job instead of a step to start from fresh
workspace, as the state that the `changeset` step leaves the workdir is
not explicitly clear to the reader.
2025-12-03 16:04:30 +01:00
Oskar 53f21e1330 feat(build): Add syncNeonEnvVars extension and improve Vercel env var syncing
Add a new `syncNeonEnvVars` build extension for syncing environment variables
from Neon database projects to Trigger.dev. The extension automatically detects
branches and builds appropriate PostgreSQL connection strings for non-production
environments (staging, dev, preview).

Features of `syncNeonEnvVars`:
- Fetches branch-specific database credentials from Neon API
- Generates all standard Postgres connection strings (DATABASE_URL, POSTGRES_URL,
  POSTGRES_PRISMA_URL, etc.) with both pooled and unpooled variants
- Supports custom database name, role name, and env var prefix options
- Skips automatically in Vercel environments (Neon's Vercel integration handles this)
- Skips for production environments (designed for preview/staging/dev branches)

Improvements to `syncVercelEnvVars`:
- When running in a Vercel build environment (detected via VERCEL env var),
  values are now read from process.env instead of the Vercel API response
- This ensures the build uses the actual runtime values Vercel provides
- Removed embedded Neon-specific logic (now handled by separate extension)
- Simplified and cleaned up the extension code

Documentation updates for both extensions with usage examples and configuration
options.
2025-12-03 16:03:41 +01:00
Saadi Myftija 331882f59c chore(releases): add package version in changeset PRs (#2730)
* Add the release version to changeset PRs

* Add missing id-token permission, needed for oidc

* Remove a couple of unnecesary steps

* Reference the `changeset-release/main` branch explicitly
2025-12-03 14:42:40 +01:00
nicktrn 1276491a83 docs: clarify slack channel on pro (#2731) 2025-12-03 13:23:38 +00:00
Eric Allam 5b7dfe23b5 feat(cli): implements content-addressable store for the dev CLI build outputs, reducing disk usage (#2725)
* feat(cli): implements content-addressable store for the dev CLI build outputs, reducing disk usage

* fix a few things
2025-12-03 10:17:35 +00:00
Oskar 49b2f683f4 feat(build): Add NeonDB branch resolution for Vercel preview environments
Vercel's NeonDB integration renders database connection environment
variables at runtime, which means Trigger.dev cannot directly sync
these values during the build process. This change adds support for
fetching branch-specific NeonDB connection strings via the Neon API.

Changes:
- Discover NEON_PROJECT_ID from incoming Vercel environment variables
- Call NeonDB API to search for branches matching the git branch name
- Filter branches to find exact matches with Vercel environment prefix
  (e.g., "preview/branch-name") to avoid false positives from partial
  string matches
- Retrieve branch endpoints and select the write endpoint (or first
  available)
- Build connection strings (DATABASE_URL, POSTGRES_URL, etc.) using
  the branch endpoint host while preserving user/password credentials

Safety measures for non-production environments:
- Filter out all Neon-related env vars (DATABASE_URL, PGHOST, etc.)
  before calling the Neon API to prevent accidental use of production
  database credentials
- Only add branch-specific database env vars if a matching Neon branch
  is found and the API call succeeds
- If neonDbAccessToken is not provided or the API fails, non-production
  environments will not receive any database connection env vars

Usage:
Users must provide a NEON_ACCESS_TOKEN (via options or env var) to
enable automatic branch resolution for preview deployments. Production
environments continue to use Vercel's standard env var sync without
modification.
2025-12-02 19:52:45 +01:00
nicktrn af9b3e1c99 fix(cli): header will always print the correct profile (#2728) 2025-12-02 17:45:35 +00:00
Eric Allam df4ab97d59 fix(otel): fix broken schedule run spans (#2727)
schedule spans can sometimes show as generic spans when using the 
task_events_v2 table because of the inserted_at filter. Increasing the 
buffer for the start time does the trick and doesn’t cause any perf 
Issues (and is in general just more robust)
2025-12-02 17:16:08 +00:00
Eric Allam 9f27422472 fix(otel): exported logs and spans will now have matching trace IDs (#2724)
When using custom OTLP exporters via `telemetry.exporters` and 

This occurred when tasks were triggered **without** a parent trace 
context (e.g., via API or dashboard). In this scenario: - Spans were 
correctly rewritten to use the generated `externalTraceId` - Logs kept 
their original internal trace ID due to a bug in the early return logic

### Root Cause

In `ExternalLogRecordExporterWrapper.transformLogRecord()`, the early 
return condition incorrectly included `!this.externalTraceContext`:

```typescript
if (!logRecord.spanContext || !this.externalTraceId || 
!this.externalTraceContext) {   return logRecord;  // Bug: Returns early
when externalTraceContext is undefined }

// This fallback logic was never reached:
const externalTraceId = this.externalTraceContext
  ? this.externalTraceContext.traceId
  : this.externalTraceId;
```

### Fix

1. **Reordered logic in `transformLogRecord()`**: Move the 
1. `externalTraceId` calculation before the early return, and check the 
1. culated value instead of `this.externalTraceContext`:

```typescript
const externalTraceId = this.externalTraceContext
  ? this.externalTraceContext.traceId
  : this.externalTraceId;

if (!logRecord.spanContext || !externalTraceId) {
  return logRecord;
}
```

2. **Clarified `_isExternallySampled` logic**: Updated both 
2. `ExternalSpanExporterWrapper` and `ExternalLogRecordExporterWrapper` 
2. explicitly handle the case where there's no external trace context 
2. a generated `externalTraceId` exists:

```typescript
this._isExternallySampled = externalTraceContext
  ? isTraceFlagSampled(externalTraceContext.traceFlags)
  : !!externalTraceId;
```

### Impact

Logs and spans from the same task run will now have matching trace IDs 
when exported to external observability tools, enabling proper trace correlation regardless of whether the task was triggered with or without a parent trace context. 
`telemetry.logExporters` in `trigger.config.ts`, logs and spans were 
exported with **different trace IDs**, breaking trace correlation in
external observability tools like Datadog.
2025-12-02 14:16:06 +00:00
nicktrn 2f1a72b109 security: remedy dependabot alerts (#2723)
* security: override js-yaml

* security: upgrade vite

* security: update nodemailer
2025-12-02 12:02:48 +00:00
Eric Allam 3c326a4b4a fix(clickhouse): ensure start_time is never older than X ms to prevent old partition merge issues (#2721) 2025-12-01 15:43:10 +00:00
Saadi Myftija 6ae1317b69 chore(clickhouse): enable dropping run debug events (#2720)
Added a new env var (`EVENT_REPOSITORY_DEBUG_LOGS_DISABLED`) that allows disabling writing run debug logs in the event repository.
2025-12-01 15:42:35 +01:00
Eric Allam 2e1c4f6df6 fix(clickhouse): partition by insertion date to prevent "Too many parts" errors when partitioning by start time (#2719) 2025-12-01 12:01:06 +00:00
nicktrn 2bf86dc20e fix(supervisor): image builds with pnpm v10 (#2718) 2025-12-01 11:28:05 +00:00
nicktrn 485782cae1 feat(ch): optionally disable migrations (#2715) 2025-11-28 19:39:29 +00:00
Eric Allam 61b338bea7 chore(repo): upgrade repo to pnpm@10 to prevent executing dep scripts on install (#2712)
* chore: migrate pnpm lockfile to v9 format via pnpm@9

* Upgrade to pnpm 10.23.0

* update the dockerfile and added a few deps to bundle in remix app
2025-11-27 16:26:19 +00:00
Kim Hallberg 83ddf721a4 docs: update Firecrawl example (#2652)
Update the example to use Firecrawl v2 API
2025-11-24 14:54:51 +00:00
Kim Hallberg a4dd2562d2 docs: format embedding example (#2656) 2025-11-24 14:51:53 +00:00
Felipe Martinez Albeche 5ff21a758c fix(docs): tooltip color contrast accessibility (#2662)
* fix(docs): color contrast accessibility on tooltip

* add changeset

* Delete .changeset/tough-feet-accept.md

---------

Co-authored-by: Eric Allam <eric@trigger.dev>
2025-11-24 14:39:14 +00:00
Lindsey 47a64c0335 docs: improve env var documentation with .env upload + ctx environment details (#2680) 2025-11-24 14:37:04 +00:00
Eric Allam fc351cb6c4 chore(repo): remove GITHUB_TOKEN requirement for publishing prerelease packages (#2679) 2025-11-24 14:33:40 +00:00
Eric Allam c4f2a9d065 chore(references): added prisma-generator-ts-enums to prisma ref project as an example (#2701) 2025-11-24 14:33:22 +00:00
Eric Allam 2762c542c2 chore(repo): remove format script to AI stops calling it (#2703) 2025-11-24 14:33:14 +00:00
Eric Allam 72e286af2f feat(otel): support for custom resource attributes via config#telemetry.resource and OTEL_RESOURCE_ATTRIBUTES env var (#2704) 2025-11-24 14:33:07 +00:00
Eric Allam f7240a99e7 fix(react): prevent infinite useEffect when passing an array of tags to useRealtimeRunsWithTag (#2705) 2025-11-24 14:32:57 +00:00
github-actions[bot] 6e47377766 chore: Update version for release (#2700)
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 0s
🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker-v4 (push) Has been skipped
🚀 Publish Trigger.dev Docker / units (push) Failing after 21s
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2025-11-24 10:30:01 +00:00
nicktrn a8563ca534 fix(docker): support the latest docker version (#2686) 2025-11-21 15:24:31 +00:00
Eric Allam 5e5c97ea4c fix(cli): stop dev runs stuck in dequeued status (fix #2639) (#2699)
* stop deleting the first dev version files on the first change, prevents system failures

* prevent dev runs getting stuck in dequeued status by deleting workers

* add changeset
2025-11-21 15:24:07 +00:00
Matt Aitken abd99aa27f Don't show (or count) archived branches/projects against allocated concurrency (#2698)
Archived preview branches were being counted. Also archived projects.
2025-11-20 11:41:12 +00:00
Eric Allam 4347499799 fix(api): Fix preview branch targeting in environment variable API routes (#2697)
This PR fixes the `x-trigger-branch` header support for targeting specific preview branches when managing environment variables. The header was documented but not actually being extracted or used in the environment variable API routes. Additionally, the query logic in `authenticatedEnvironmentForAuthentication` was fundamentally broken—it searched for environments with both `slug: "preview"` (parent environment property) AND a specific `branchName` (child environment property), which no environment could satisfy simultaneously. The fix extracts the branch name using `branchNameFromRequest()` and correctly queries for child branch environments using `type: "PREVIEW"` and the specific `branchName`. This ensures that environment variable operations (create, update, get, list) properly target individual preview branches instead of affecting all preview environments.
2025-11-19 18:01:12 +00:00
Eric Allam abee783d3f chore(logs): remove unnecessary debug logs (#2696) 2025-11-19 13:56:15 +00:00
Eric Allam 6464eeed53 fix(webapp): correctly generate JWT tokens for preview branches after triggering a run (fix #2678) (#2695) 2025-11-19 13:42:12 +00:00
Matt Aitken bee59de3a0 Concurrency self serve (#2681)
* Don't use the organization max concurrency anymore

* Early draft of the concurrency page

* WIP adding a new stepper input component

* Move stepper to be alphabetical

* When max value is reached, disabled the + button

* Show placeholder if you delete all numbers

* Make all the html input values available to the component

* Adds size variants

* Move stepper into its own component

* Work on showing the extra concurrency

* The purchase form styling and functionality (minus actually purchasing)

* New style for outline input fields

* Concurrency purchasing working

* Purchasing concurrency and quota emails working

* Improvements to the modal

* Show cost breakdown in the modal

* Fix for allocated concurrency including DEV

* Improved types

* Allocating concurrency is working

* Live updates total env concurrency

* Implemented reset

* Fix for concurrency allocation editing across multiple projects

* Tabular numbers

* Added an error from allocating concurrency

* Fixes for allocating concurrency where it didn't calculate correctly

* "Increase limit" link to concurrency page

* Indent environments

* Added Preview limit when updating concurrency for an org

* Show error when changing plan fails

* Added maximumProjectCount column to Org

* Limit project count and display a rich error toast (with title and button now)

* Added title and button to toasts. Use it for new project error

* @trigger.dev/platform 1.0.20

* Allow submitting zero concurrency so you can downgrade back to nothing

* Use the server as the truth for omitted environments

* Updated the pricing panels

---------

Co-authored-by: James Ritchie <james@trigger.dev>
2025-11-19 11:22:23 +00:00
Eric Allam 3af7303156 feat(docs): update prismaExtension for new mode functionality in 4.1.1 (#2690) 2025-11-19 11:16:35 +00:00
github-actions[bot] 6231ddc67a Release v4.1.1
* chore: Update version for release

* chore: Update pnpm-lock.yaml (#2694)

* Initial plan

* Update pnpm-lock.yaml

Co-authored-by: ericallam <534+ericallam@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: ericallam <534+ericallam@users.noreply.github.com>

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com>
Co-authored-by: ericallam <534+ericallam@users.noreply.github.com>
2025-11-19 11:09:26 +00:00
Eric Allam 15fef916f6 feat(build): update prisma extension to work with generated clients and rust-free clients (#2689)
* prisma extension fixes WIP

* More prisma stuff

* more prisma stuff

* remove changelog

* upgrade github workflows to use node 20.19 because installing prisma@7 breaks with lower versions

* Don't use generate for the prisma reference projects

* make sure it works if no mode is passed in
2025-11-19 10:48:15 +00:00
Eric Allam e2a703bfa8 fix(dashboard): continuously apply log filter during live run (#2692) 2025-11-18 17:00:29 +00:00
Saadi Myftija 1a7ee24b9e fix(ui): refresh deployments logs stream on status change (#2688)
The logs stream is created after the deployment moves from `PENDING` status to `INSTALLING`.
2025-11-17 14:50:05 +01:00
Saadi Myftija bb99af52cb feat(deployments): expose native build server option in build settings (#2685)
Adds a build setting about using our build server for remote builds.
2025-11-17 12:46:57 +01:00
Saadi Myftija 01797a1668 feat(deployments): ECR repo adaptations to enable external build cache (#2684)
* Update aws sdk ecr client to the latest version

* Exlude the cache tag from the immutability enforcement

* Attach a policy to ECR repos to expire untagged images

* Fix filterType
2025-11-17 12:34:54 +01:00
Eric Allam 19fa669318 Release v4.1.0 (#2683) 2025-11-14 09:55:01 +00:00
github-actions[bot] 78fcba518a chore: Update version for release (#2666)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2025-11-14 09:52:22 +00:00
Saadi Myftija 53047ab648 feat(deployments): external cache support for local builds (#2682)
* Add registry cache support for local builds

* Add changeset
2025-11-14 10:46:31 +01:00
Eric Allam bb5cefa92c docs: realtime streams v2 (#2673) 2025-11-14 09:42:20 +00:00
Eric Allam b8b198579c chore(rules): upgrade rules for 4.1.0 (#2676) 2025-11-14 09:42:15 +00:00
Eric Allam 892bed8c4c Upgrade to electricsql 1.2.4 (#2668) 2025-11-13 15:19:59 +00:00
Eric Allam a94a11f44d feat(sdk): replace onStart lifecycle hook with onStartAttempt (#2515)
* fix(sdk): prevent uncaught errors thrown onSuccess, onComplete, and onFailure hooks to fail attempts & in some cases runs

* Add onStartAttempt hook and deprecate onSuccess

* Add onStartAttempt hook and deprecate onStart hook

* Fix onStartAttempt overload types

* Update lifecycle functions diagram
2025-11-13 14:51:13 +00:00
Matt Aitken f116e93e01 Docs: tags can be up to 128 chars (#2678) 2025-11-13 14:43:31 +00:00
Eric Allam 6137338da9 feat(streams): make v2 streams the default when using 4.1.0+ if they are supported (#2677) 2025-11-13 13:53:42 +00:00
Lindsey 8cec3b763b docs: add section on using .env.production and dotenvx for environmen… (#2674)
* docs: add section on using .env.production and dotenvx for environment variables

* fix: correct dotenvx API usage in documentation example
2025-11-13 11:28:26 +00:00
Eric Allam 343ba54c69 fix(streams): restore realtime stream writing for v3 tasks (#2675) 2025-11-13 11:21:48 +00:00
Eric Allam a70ab10809 fix(streams): fixed broken wrapping in streams inspector (#2672) 2025-11-12 16:00:54 +00:00
Eric Allam f7cb637b32 fix(streams): scope s2 access token to environment and fix streams v1 appends (#2670)
* fix(streams): scope s2 access token to environment and fix streams v1 appends

* Less stale time
2025-11-12 12:34:15 +00:00
Eric Allam 668559ec1a fix(streams): buffer v1 streams on read to prevent split chunks (#2669) 2025-11-11 21:08:49 +00:00
Eric Allam d0ad38d684 chore(docker): remove unused seed copy from dockerfile (#2667) 2025-11-11 15:13:10 +00:00
Eric Allam 536d9fa217 feat(realtime): Realtime streams v2 (#2632) 2025-11-11 14:54:00 +00:00
github-actions[bot] d75c3aeadd chore: Update version for release (#2665)
* chore: Update version for release

* Release 4.0.7

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Matt Aitken <matt@mattaitken.com>
2025-11-11 14:29:24 +00:00
Matt Aitken a342332146 Fix for the MCP tool that gets logs for debugging runs (#2653)
* Fix for the MCP tool that gets logs for debugging runs

This was broken when we changed the data on the backend that returns 
log/span data from runs. We changed the data structured and the internal
API that the MCP client uses was failing to parse with the Zod schema

* add changeset

* Revert "add changeset"

This reverts commit 86eca836d5907fa0d0f8ac595d4d5ebade140514.

---------

Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
2025-11-11 13:26:07 +00:00
Saadi Myftija 9624465ee2 fix: error handling issue with s2 streams (#2664)
* Fix logger import

*old man yells at auto-import*

* Fix s2 error handling for non-existing streams
2025-11-11 09:44:54 +01:00
Saadi Myftija 42f53b12b7 feat(deployments): stream build server logs (#2663)
* Use read-only project-scoped s2 tokens for streaming deployment logs

* Add http2 to remix polyfills

Needed for using s2 client-side.

* Stream build-server logs in the deployment details page

* Disable 12-hour format in the DateTime component

* Enable collapsing the logs panel

* Auto-collapse logs for succesful/timedout/queued deployments

* Make S2 env vars optional

* Show the logs section only for gh-triggered deployments

* Cache s2 access tokens in redis

* Reset streaming state

* Expose 12h format as a param for the Datetime components
2025-11-10 12:42:40 +01:00
github-actions[bot] 9fdf91a1c4 Release v4.0.6 (#2647)
* chore: Update version for release

* Update lock file

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: myftija <saadi.myftija@gmail.com>
2025-11-03 14:42:02 +01:00
nicktrn 9593a46364 fix(extensions): prevent audiowaveform binary removal (#2643)
* fix(extensions): prevent audiowaveform binary removal

* add changeset
2025-11-03 14:17:38 +01:00
Saadi Myftija 27376df903 fix(cli): show a useful error message when config file is missing (#2650)
* fix(cli): show a useful error message when config file is missing

* Add changeset
2025-10-31 16:34:48 +01:00
Saadi Myftija e0cece72ae docs: deployments with local builds (#2645)
* docs: deployments with local builds

Adds a section to the deployment docs page about the option to deploy using a locally built image.

* Update installation link
2025-10-30 18:10:23 +01:00
Saadi Myftija 2f3f82f3a9 feat(deployments): show local build hint during depot outages (#2646)
* Add an API endpoint to query remote build provider status

* Show local build hint for failed deployments when Depot is down

* Show the local build flag in the help output

* Add changeset

* Fix import

* Fix docs link
2025-10-30 18:09:54 +01:00
Dan 4264bdf429 Added use cases section in the docs (#2641)
* Started adding use cases pages

* Updates

* More improvements

* Improved diagrams

* Added overview page

* More copy + diagram updates

* Improved diagram titles

* Further diagram improvements

* Corrected workflow

* Updated copy

* Typos

* Updated intro file

* Reverted aiRunFilterService.server.ts
2025-10-30 09:21:31 +00:00
nicktrn bf6735bd56 fix(webapp): concurrency override upper bound should be env not org (#2642) 2025-10-29 19:16:42 +00:00
James Ritchie dacd53b906 Fix for showing the incident UI in the side menu (#2638) 2025-10-28 21:13:23 +00:00
github-actions[bot] d4fe71df34 Release v4.0.5 (#2531)
* chore: Update version for release

* Release v4.0.5

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
2025-10-28 14:20:20 +00:00
nicktrn ae8e83b2d0 chore(runner): move max duration logic into parent process (#2637)
* chore(runner): move max duration logic into parent process

* chore(rsc): remove type-marker package.json

* add changeset

* chore(core): remove irrelevant test after our changes

* chore(core): clarify we don't care about the timeout promise
2025-10-28 13:49:59 +00:00
nicktrn 2283ca6ad3 fix(webapp): persist concurrency overrides on deploy (#2636)
* fix(webapp): display correct concurrency override base value

* fix(webapp): persist concurrency overrides on deploy

* fix(webapp): use correct override base value type

* fix(webapp): override input is bounded by env concurrency
2025-10-28 11:28:38 +00:00
nicktrn 8fdbbeb02f chore(helm): increase default clickhouse resources (#2635) 2025-10-27 13:45:09 +00:00
Marcus Nerløe 255ea0a4b3 fix(supervisor): prevent escalating duplicate reconnections in failedPodHandler (#2627)
⚒️ Publish Worker (v4) / build (supervisor) (push) Has been cancelled
* fix(supervisor): prevent escalating duplicate reconnections in failedPodHandler

* fix: added catch handler for informer.start() failures

* fix: removed 'errorStack' from error log
2025-10-23 11:13:45 +01:00
Saadi Myftija 7f25e82299 feat(deployments): support local builds in cloud (#2628)
* Enable skipping image push during deployment finalization step

* Add endpoint to generate registry credentials for a deployment

* Add a --force-local-build flag to the deployment command to skip remote build

* Do not show the new flag in the help output

* Add changeset

* Remove registry login logs from onLog, not useful

* Rename var

* Update platform package to the latest version
2025-10-23 10:57:36 +02:00
Eric Allam d90da7abf7 fix(replication): allow disabling of task run payload inserts via env var (#2626) 2025-10-22 11:22:14 +01:00
Matt Aitken 2affe541e8 Dev concurrency limit by env (with optional global limit) (#2625)
* Limit local dev concurrency using the dev environment concurrency limit

Previously it was limited to max of 25, no matter the environment limit

* Have global dev limit
2025-10-22 10:55:05 +01:00
Matt Aitken 3157b657c7 Test page recent payload: use ClickHouse to get the latest run ids (#2614)
The Postgres query to get the latest run ids for the test page was very 
slow when there were a lot of runs and/or versions.

This now uses the standard runs list we use everywhere else.
2025-10-17 17:17:36 +01:00
Matt Aitken 41bdab58d5 Document the dev --analyze command (#2613) 2025-10-17 13:00:30 +01:00
James Ritchie fe3fe01fe8 feat(queues): Override queue concurrency limits from the dashboard or API (#2609)
* feat(queues): add ability to override concurrency limit via API and dashboard

* Updates the modal layout and tweaks copy

* Improves the dropdown menu item

* Popover supports both Button and LinkButton

* Right align the columns and fix the dropdown menu item styles

* Organize imports,

* Fix spinner icon in dropdown menu

* Remove unused props

* Adds a tooltip to the Concurrency override badge

* Fixes console error with popover menu

* typo

* Fixes incorrect className

* Minimal buttons to view runs

---------

Co-authored-by: Eric Allam <eallam@icloud.com>
2025-10-17 12:58:25 +01:00
Matt Aitken 68d0037e60 fix(dev): dotenv issues when getting setup with the repo (#2612)
- The .env.example was missing required Clickhouse values.
- A symlink was needed for .env from the root to the apps/webapp folder
2025-10-17 11:49:40 +01:00
Matt Aitken 885d2d3560 Tags listing now uses ClickHouse (#2576)
* WIP using ClickHouse for the tags filter list

* WIP on tags listing

* Webapp: exclude test files when typechecking

* Tags filtering working with CH

* Remove unused import

* The AI filter should only look at the last past 30d of tags

* Do the text query in ClickHouse

* Deal with encoded characters better

* More encoding fixes

* Fix for wrong items being checked

* Put applied tags back

* Add the env.id to the dependencies array
2025-10-15 12:56:08 +01:00
Eric Allam a445af1b79 feat(otel): allow clickhouse task events to be inserted using async_insert via env vars (#2608) 2025-10-15 10:37:17 +01:00
nicktrn 5781783c74 fix(engine): default to paid placement on billing errors (#2604)
* chore(billing): improve logs to distinguish between failure modes

* fix(engine): default to paid placement on billing errors

* chore(engine): set plan type according to paying field when missing
2025-10-15 10:32:27 +01:00
nicktrn cca10c22d3 chore(supervisor): add machine label (#2603) 2025-10-15 10:31:37 +01:00
Leonardo Kaynan 63b6fc93fa fix(helm): align values-production-example.yaml with values.yaml (#2606)
- Move S3 credentials from secrets.objectStore to s3.auth
- Update external PostgreSQL config to use databaseUrl/directUrl approach
- Add existingSecret support for PostgreSQL with secretKeys
- Add TLS configuration for external Redis
- Add existingSecret support for Redis, ClickHouse and S3
- Add complete external S3 configuration example
- Improve secure credential management documentation

These changes align the production example file with the current values.yaml
structure, making it easier to configure external services with better
support for secret management.
2025-10-14 23:16:58 +01:00
Saadi Myftija aa66462971 docs(build-server): adjust build env vars section (#2605)
Adds a clarification about the env var prefix stripping.
2025-10-14 17:29:42 +02:00
James Ritchie f6461684ad chore(webapp): adds shortcut key for admin area (#2570)
* Allow shortcuts hook to work if undefined

* Conditionally show shortcut button if only 1 result

* LinkButton can accept conditionally shown shortcuts
2025-10-13 13:26:00 +01:00
James Ritchie a6896b411a Adds a link to edit the profile icon from the main menu (#2572) 2025-10-13 13:25:06 +01:00
Saadi Myftija 0cabbdd31f docs: deploying using the github integration (#2598)
* docs: deploying using the github integration

* Add hint in the gh actions docs page

* Remove extra space

* Add a couple of hints to the build config fields
2025-10-13 14:15:48 +02:00
Eric Allam f8977a7b70 chore(db): remove unnecessary FK constraints on TaskRunExecutionSnapshot (#2533) 2025-10-09 14:07:45 +01:00
Saadi Myftija f0643f76f5 feat(build-server): add option to specify pre-build command (#2596)
* feat(build-server): add option to specify pre-build command

Adds an option to specify a pre-build command in the build settings. Can
be useful for projects that need a step before the build, e.g., to
generate a prisma client.

Also, remove the install directory in favor of simplicity. Both
pre-build and install commands are run from the root of the repo. Users
that need to run the commands in a different dir can just prepend to the
command, e.g., `cd apps/web && pnpm run primsa:migrate`

* Fix spelling
2025-10-08 17:25:44 +02:00
Saadi Myftija 416dbcd536 fix(webapp): disable gh-triggered preview deployments if the preview env is disabled (#2595)
* Show hint if preview branches are disabled in the project

* Enable preview deployments only if the preview environemtn is enabled

* Fix prisma reference
2025-10-08 16:59:25 +02:00
Eric Allam 679b41dc7e chore(electric): upgrade server to 1.1.14 (#2590) 2025-10-08 14:33:10 +01:00
Eric Allam be98aecbfd fix(otel): prevent unpaired unicode surrogate pairs from causing insert errors (#2594)
* fix(otel): prevent unpaired unicode surrogate pairs from causing insert errors

* only check parts of the string that are not going to get truncated
remove unnecessary taks
2025-10-08 14:32:35 +01:00
nicktrn 129dc02f2a chore: add deepwiki badge to main readme (#2587)
This means DeepWiki will automatically re-index our repo and keep our page fresh
2025-10-08 13:15:06 +01:00
nicktrn 8917478d3c fix(runner): SIGTERM handling during warm start long poll (#2593) 2025-10-08 12:51:10 +01:00
Leonardo Kaynan f5caa66348 fix(helm): use bitnami legacy repo for minio console (#2592)
🧭 Helm Chart Release / lint-and-test (push) Has been cancelled
🧭 Helm Chart Release / release (push) Has been cancelled
* fix(charts): use bitnamilegacy repo for MinIO Object Browser

The Bitnami `minio-object-browser` image has been removed from Docker
Hub. This patch updates the Trigger Helm chart to reference the
`bitnamilegacy/minio-object-browser` repository under the
`s3.console.image` section, ensuring that the legacy MinIO browser pod
can still be deployed without image pull errors.

All other MinIO components continue using the `bitnamilegacy/minio`
image for consistency across Bitnami Legacy dependencies.

* chore(helm): bump Helm chart version to 4.0.4
2025-10-07 22:17:37 +01:00
Eric Allam 64fcc88fa7 fix(otel): spans with entities (like waitpoints) now correctly returned from clickhouse repo (#2591) 2025-10-07 16:55:00 +01:00
nicktrn 23dbe282ed security: upgrade transitive axios dep (CVE-2025-58754) (#2589) 2025-10-07 14:56:16 +01:00
Eric Allam 692316e82a fix(realtime): Upgrade to @electric-sql/client@1.0.14 to prevent cached 409 Conflict errors from breaking realtime updates (#2588) 2025-10-07 14:26:03 +01:00
Saadi Myftija 107f4dc87c fix(deployments): retry transient depot build init failures (#2586)
The Depot build init with `depot.build.v1.BuildService.createBuild` fails surprisingly often due to transient errors, causing the whole deployment to fail. This PR adds a simple retry mechanism with backoff using p-retry. This should improve the failure rate.
2025-10-06 11:33:31 +02:00
Eric Allam b90f3e2173 fix(otel): remove clickhouse event repo feature flag support from v3, now v4 only (#2585) 2025-10-06 09:54:23 +01:00
Eric Allam b3b2553651 fix(otel): propagate the task event store to run descendants (#2583) 2025-10-04 07:28:16 -07:00
Eric Allam cdd1a8838c fix(otel): prevent spans with negative durations (#2582) 2025-10-03 06:50:16 -07:00
Eric Allam 200b7354d0 fix(otel): clickhouse logs/span metrics now exclude partials and debug events (#2581) 2025-10-02 13:51:50 -07:00
Eric Allam eeed38d223 fix(clickhouse): correctly format datetime64(9) input format (#2580) 2025-10-02 11:32:32 -07:00
nicktrn 0ca092651b feat(supervisor): optional custom scheduler (#2579) 2025-10-02 16:22:53 +01:00
Eric Allam 53acdf8ef5 fix(otel): don't pass isDebug when creating postgresql task events (#2578) 2025-10-01 21:57:35 -07:00
Eric Allam 128bc437f6 feat(otel): Add support for storing run spans and log data in Clickhouse (#2567) 2025-10-01 12:41:18 -07:00
James Ritchie 0597691001 Adds 200 and 500 % billing alert options (#2571) 2025-09-30 13:59:44 -07:00
mintlify[bot] dae84a0d29 Update docs/idempotency.mdx (#2575)
Co-authored-by: mintlify[bot] <109931778+mintlify[bot]@users.noreply.github.com>
2025-09-30 13:50:02 -07:00
nicktrn f72d63aac2 chore(helm): migrate to bitnami legacy registry and add configurable utility images (#2574)
🧭 Helm Chart Release / lint-and-test (push) Has been cancelled
🧭 Helm Chart Release / release (push) Has been cancelled
* chore(docker): use bitnami legacy repo

* chore(helm): use bitnami legacy repo

* Make Helm webapp chart images configurable

Adds configurability for init and token syncer container images through
new values in the Helm chart configuration

* chore(helm): refactor utility image config

* chore(helm): bump chart version to 4.0.3

---------

Co-authored-by: LeoKaynan <leokaynan@hotmail.com>
2025-09-30 16:02:08 +01:00
nicktrn 12cceaa779 feat(helm): support topology spread constraints for webapp (#2560)
* feat(helm): support topology spread constraints

* chore(helm): update topology env var

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>

* chore(helm): limit spread constraints to webapp for now

---------

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
2025-09-30 15:31:04 +01:00
Saadi Myftija ddebe4dce0 feat(webapp): rate limit magic-link login attempts (#2568)
* feat(webapp): rate limit magic-link login attempts

Adds a simple rate limiter to the login with magic link flow. Similar implementation to the MFA rate limits.

* Fix error message

* Add an env var feature flags for login rate limiting

* Use BoolEnv instead of `0`/`1`

* Parse xff properly
2025-09-29 17:28:36 +02:00
Saadi Myftija 09d51c6d24 fix(webapp): add recommended security headers (#2569)
Sets `Referrer-Policy`, `X-Content-Type-Options` and `Permissions-Policy` headers.
Relevant against certain types of attacks.
2025-09-29 16:55:22 +02:00
Saadi Myftija 3ceea774a8 fix(run-engine): waitpoint update misleading error logs (#2566) 2025-09-26 21:16:12 +02:00
Eric Allam 05b6a26c4f fix(run-engine): pass through engine fair dequeue selection strategy options instead of using defaults (#2565) 2025-09-26 17:12:36 +01:00
Eric Allam 558fb11b89 feat(run-engine): ability to repair runs in QUEUED, SUSPENDED, and FINISHED execution status (#2564)
* feat(server): add two admin endpoints for queue and environment concurrency debugging and repairing
feat(run-engine): ability to repair runs in QUEUED, SUSPENDED, and FINISHED execution status

* Handle FINISHED snapshot in the repair
2025-09-26 15:16:10 +01:00
nicktrn 9aedda23a4 fix(run-engine): carryover batchId after PENDING_EXECUTING stalls (#2563) 2025-09-26 14:14:28 +01:00
Eric Allam 743b8dbe0c chore(run-engine): add additional logging around dequeueing and worker queues (#2562) 2025-09-26 11:50:21 +01:00
Eric Allam eb0263e942 feat(server): add two admin endpoints for queue and environment concurrency debugging and repairing (#2559) 2025-09-25 19:33:10 +01:00
Saadi Myftija 7bf579fa50 fix: use higher entropy invite tokens (#2558)
* fix: use higher entropy invite tokens

We currently use CUIDs for invite tokens, which are generated using
a pattern and are not cryptographically secure. This PR switches to
a higher entropy string generated with `nanoid`.

* Dedupe the invite emails in the application
2025-09-25 19:42:59 +02:00
Saadi Myftija 69d52db856 fix(webapp): org scoping issues in plan selection, alerts, pats and usage (#2549)
* fix: org scoping in the select plan flow

Adds proper org scoping in the loader and action in the plans page.

* Fix billing alerts scope

* Fix org usage page scope

* Fix token revoking flow scope check

* Throw error for failed PAT revokes instead of silent failure
2025-09-25 19:17:51 +02:00
Saadi Myftija a3cea1302e fix(webapp): org invite scoping (#2554)
* fix: org invite scoping

Fixes some scoping issues with team invites.

* Fix invite flow changes
2025-09-25 19:17:19 +02:00
Eric Allam a3bdd3c64b chore(run-engine): improve concurrency sweeper logging to get better visibility (#2557) 2025-09-25 16:43:43 +01:00
Saadi Myftija 6d6e98aa11 fix(webapp): project scoping for runs (#2553)
* fix: project scoping for runs

* Apply some 🐰 suggestions
2025-09-25 17:43:13 +02:00
Eric Allam e22c321dd1 fix(engine) truncate errors before storing them on a run and waitpoint output (#2552) 2025-09-25 13:39:51 +01:00
Eric Allam 59df4af1eb chore(engine): add additional logging when we fail to get snapshots since (#2551) 2025-09-25 12:59:02 +02:00
Eric Allam 8863ff05c9 fix(engine): limit the number of snapshots returned when getting latest snapshots since (#2550) 2025-09-25 11:53:35 +01:00
Matt Aitken d10281e655 Additional files docs with legacyDevProcessCwdBehaviour (#2543) 2025-09-24 14:32:02 +01:00
nicktrn 6798d57e72 chore(helm): bump image versions to 4.0.4 (#2537) 2025-09-24 14:31:30 +01:00
Saadi Myftija 480c0d34d3 fix(webapp): toast message issue after gh app installation (#2546)
* fix(webapp): toast message issue after gh app installation

Fixes an issue with displaying toasts messages in the project settings
page. The github callback cookie was interfering with the flash cookie used
for toast messages.

* Do not set a tracking branch in the staging env by default
2025-09-24 15:20:03 +02:00
Saadi Myftija 700a6ea598 feat: enable canceling deployments (#2545)
* Add canceledAt to the deployment db schema

* Expose an api endpoint to cancel deployments

* Show the canceled status description in the dashboard

* Enable canceling deployments from the dashboard

* Show cancelation reason in the deployment details

* Make verifyProjectMembership a function for consistency

* Apply some good 🐰 suggestions
2025-09-24 11:14:29 +02:00
Saadi Myftija cc94d121f2 feat: installing status for deployments (#2544)
* Add installing status to the deployment db schema

* Replace the deployments /start endpoint with /progress

* Show the installing status in the dashboard

* Add installing status to the api schema and cli

* Add changeset
2025-09-24 10:27:43 +02:00
Saadi Myftija 412e80fdde fix(webapp): hide outdated connected repos from deleted installations (#2538) 2025-09-23 13:24:58 +01:00
Saadi Myftija 49728b5a5f feat(api): defer remote build creation for pending deployments (#2536)
Depot builds have short-lived tokens and their TTL is not exposed in the SDK. As queued deployments can stay in the queue for an arbitrary amount of time, deferring the remote build creation helps avoid expired Depot token issues.
2025-09-22 13:01:56 +02:00
Saadi Myftija d45696c000 fix(api): 204 response issue in deployment start endpoint (#2534) 2025-09-19 19:05:18 +02:00
Eric Allam 8313800746 fix(webapp): don't override spans from ancestors unless the span is partial (#2532) 2025-09-19 14:58:53 +01:00
James Ritchie 28f8cee3a4 Fix for errors returned from searching time specific queries (#2525)
* Fix for errors returned from searching time specific queries

* Adds prompt patterns
2025-09-19 13:49:43 +01:00
Eric Allam 87b3603b23 feat(webapp): completing spans server-side no longer write-after-read, improving efficiency and perf (#2530)
* Cancel run events which then propogate cancellation status to span ancestors

* WIP

* convert closing cached run spans to new system

* converted expired complete span event to new method

* move v3 over to new methods

* Convert getDetailedTraceSummary to use the new ancestor override stuff

* remove debug logs

* Don't return UNSPECIFIED task events in getRunEvents

* fix the call site for cancelling run event in v3

* Add changeset

* remove methods
2025-09-19 13:39:48 +01:00
James Ritchie 365adc24a6 Chore(webapp): adds more copy buttons (#2529)
* Pass up asChild

* Include padding when asChild specified

* Adds copy buttons to useful data in Details tab

* Improve view batch tooltip message
2025-09-19 13:02:10 +01:00
Saadi Myftija 7d17730b52 feat(api): handle build server deployment init gracefully for older cli versionsi (#2526)
This PR adapts the deployment initialization endpoint to handle build server deployments with older CLI versions gracefully.

When we introduced automatic deployments via the build server, we slightly changed the deployment flow
mainly in the initialization and starting step: now deployments are first initialized in the `PENDING` status
and updated to `BUILDING` once the build server dequeues the build job.
Newer versions of the `deploy` command in the CLI will automatically attach to the existing deployment
and continue with the build process. For older versions, we can't change the command's client-side behavior,
so we need to handle this case here in the initialization endpoint. As we control the env variables which
the git meta is extracted from in the build server, we can use those to pass the existing deployment ID
to this endpoint. This doesn't affect the git meta on the deployment as it is set prior to this step using the
/start endpoint. It's a rather hacky solution, but it will do for now as it enables us to avoid degrading the
build server experience for users with older CLI versions. We'll eventually be able to remove this workaround
once we stop supporting 3.x CLI versions.
2025-09-19 10:40:59 +02:00
Saadi Myftija e4982bfd6d feat(webapp): deployments page live reloading (#2524)
* Fix `current` badge inconsistency in the deployment details page

* Add custom hook for auto revalidation based on an interval and/or focus change

* Use the autoRevalidate hook for live reloading of the deployments page

* Extract autoReloadPollIntervalMs to an env var

* Replace the sse-based autoreload in bulk actions and queues page with the simpler autoRevalidate hook
2025-09-19 10:40:07 +02:00
github-actions[bot] a03783d1a0 Release v4.0.4
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 19s
🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker-v4 (push) Has been skipped
🚀 Publish Trigger.dev Docker / units (push) Failing after 19s
* chore: Update version for release

* Release v4.0.4

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Eric Allam <eallam@icloud.com>
2025-09-18 23:18:59 +01:00
Eric Allam 0178bdbb00 fix(packages): remove effect from optional peer dependencies (#2527)
Fixes ERROR: Could not resolve "effect"
2025-09-18 23:09:37 +01:00
github-actions[bot] ad51168181 Release v4.0.3 (#2486)
* chore: Update version for release

* Release v4.0.3

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
2025-09-18 21:28:30 +01:00
nicktrn db87295049 fix(runner): reduce restore recovery time and deprecated runner false positives (#2523)
* fix(runner): improve restore detection

* chore(supervisor): skip schema parsing when debug logs disabled

* fix(runner): deprecation race condition

* add changeset
2025-09-18 16:59:44 +01:00
Saadi Myftija a3ef6ea236 feat: separate deployment initialize and start steps (#2522)
* Enable setting the initial status on deployment creation

* Expose endpoint to start deployments

* Extend build timeout on deployment start

* Use separate timeout value for queued deployments

* Add startedAt to the deployment schema

* Show the new startedAt instead of createdAt in the dashboard

* Show github user tag also in the deployment details page

* Show `pending` deployment status as `queued` in the dashboard

* Apply some good 🐰 suggestions

* Add missing return
2025-09-18 16:12:17 +02:00
James Ritchie ae22000409 chore(webapp): pricing page improvements (#2457)
* Improve styling of onboarding pricing plans

* type only import

* Improve the onboarding plan page so it scrolls on smaller screens

* Adds additional pricing for bolt-ons for the Pro plan

* fix text wrapping issue

* add bg color back in
2025-09-18 14:31:06 +01:00
James Ritchie 2bbf8ec6cd chore(webapp): a new cancelation reason (#2465)
* Adds a new cancelation option

* Better wording
2025-09-18 14:30:25 +01:00
James Ritchie c8858edf0a New jump to parent or root run buttons (#2067)
* Change the color to indigo

* Pro tier pricing information now matches the marketing site

* Update the button styles to secondary

* WIP adding separate links to Parent and Root runs

* TextLink now supports optional shortcuts

* Adds shortcut keys to the root and parent links + the shortcut help panel

* Adds new icons for root and parent

* root friendlyId works

* Updates icons for jump to root and parent

* Copy tweak

* Improve how the Free tier shows no preview branches

* Improve the wording in the tooltip

* Align the x icon better

* Show price for additional preview branches

* Change the shortcut key

* Fixes button alignment

* Adds nested dependencies task hello-world

* Fixes typo “Cancelled”

* Removes taskIdentifier, not needed

* Removes unused taskIdentifier
2025-09-18 14:29:47 +01:00
Eric Allam 9c087646bf fix(engine): prevent race condition that prevents triggerAndWait runs from resuming by atomically creating associated waitpoint records (#2519) 2025-09-17 13:58:17 +01:00
Saadi Myftija 691903cf58 feat(api): accept OATs in preview branch related endpoints (#2517)
Adjusts the authentication in a couple of endpoints to accept OATs too.
2025-09-17 14:10:25 +02:00
nicktrn 885ae5e06f fix(runner): immediate poll to decrease restore time (#2516)
* fix(runner): immediate poll to decrease restore time

* chore: bump default prerelease tag

* fix(cli): s is not a function

* add changeset
2025-09-16 16:17:04 +01:00
Saadi Myftija 501a383bcd feat: expose project build settings (#2507)
This PR enables setting project build settings in the settings page:
root directory, install command and trigger config file path.

For most cases there should be no need to set these explicitly.
2025-09-16 15:02:09 +02:00
nicktrn 3c199e6d9c chore(supervisor): remove deprecated route (#2513) 2025-09-16 13:47:07 +01:00
nicktrn a55294b7dd fix(run-engine): retry SIGSEGV errors (#2514) 2025-09-16 13:46:58 +01:00
Eric Allam c9d1aadfc3 core(docs): link to the v4 self-hosting in the introduction (#2512) 2025-09-16 13:26:55 +02:00
Eric Allam 08702cd710 chore(mcp): Add our MCP server to the official MCP registry (#2510)
See: 
https://blog.modelcontextprotocol.io/posts/2025-09-08-mcp-registry-preview/
2025-09-16 10:44:41 +01:00
Dan 04dcb81496 Added product image generator and replicate examples (#2511)
* Added nano banana task

* Renamed the file and updated docs.json

* Added product imgae generator demo project

* Removed old section

* Code tweak
2025-09-16 10:26:11 +01:00
Eric Allam 0f9b83db09 fix(core): prettyPrintingPacket will now do a structuredClone on non-circular references instead of outputting [Circular] (#2508)
* Mo-Stashed changes

* fix(core): prettyPrintingPacket will now do a structuredClone on non-circular references instead of outputting [Circular]

This also fixes an issue with replaying of runs that include 
non-circular references
2025-09-15 17:43:46 +01:00
Saadi Myftija 7d333e5b3c feat(cli): attach to existing deployments in the build server (#2501)
* Add external build data and image platform to the get deployment endpoint

* If provided, attach to an existing deployment in the deploy command

* Check status for existing deployments

* Add changeset
2025-09-15 17:16:26 +02:00
nicktrn 7c4ce6f76b feat(supervisor): add optional memory limit overhead (#2506) 2025-09-15 16:09:05 +01:00
Eric Allam dc42ae7aa4 chore(repo): update changeset snapshot instructions (#2505) 2025-09-15 15:27:54 +02:00
Saadi Myftija b0b88f1e05 fix(webapp): show deployments where triggeredById is missing (#2504)
Small fix for the sql query used in the deployments list page.
2025-09-15 14:12:48 +01:00
Eric Allam 6483a0f1c6 fix(core): Improves our schema to JSON Schema conversion (fix for zod 4) (#2483) 2025-09-15 14:11:40 +01:00
Eric Allam 6f1abe058b chore(docs): heartbeat timeout is now 5 minutes (#2500) 2025-09-15 15:11:32 +02:00
Eric Allam 83bd6f5f9e fix(engine): carry over completed waitpoints on PENDING_EXECUTING stalls (#2503) 2025-09-15 12:19:46 +01:00
Eric Allam e36d78e4fc fix: don't carry over the checkpoint ID when nack and requeuing (#2502) 2025-09-15 12:06:52 +01:00
Eric Allam 3188dc9b28 perf(webapp): Add BatchTaskRun index to speed up the batch list dashboard page (#2499) 2025-09-12 17:23:52 +01:00
Eric Allam 10e7985fbc Add index for waitpoint tokens dashboard query (#2498) 2025-09-12 16:48:42 +01:00
Eric Allam 2eddda1233 fix(webapp): worker actions now catch service validation errors and respond properly (#2481)
This also stops all the unnecessary error logging when throwing 
ServiceValidationErrors
2025-09-12 14:56:54 +01:00
HUORT Louis 49f2c54031 fix: webapp crash on clickhouse data corruption (#2491) 2025-09-12 14:55:29 +01:00
Eric Allam f077d49291 feat(engine): Improve execution stalls troubleshooting, align dev and prod behavior, adding heartbeats.yield utility (#2489)
* feat(engine): Improve execution stalls troubleshooting, align dev and prod behavior, adding heartbeats.yield utility

* A few improvements via the 🐇 review

* Allow treating EXECUTION stalls as OOM errors, improve the error message, add more information to the docs, improve resource monitor and add it to the docs

* Add changeset
2025-09-12 14:54:39 +01:00
Eric Allam 5db583b6cd docs: add machine option in triggering docs (#2487) 2025-09-11 15:28:26 +02:00
Matt Aitken 1227e5463e Fix for broken retrieve runs docs page (#2496)
The API path was accidentally edited as part of a /v3 purge
2025-09-10 16:59:37 +01:00
Eric Allam 8e66913e59 fix(run-engine): Preserve snapshot checkpoint ID when a PENDING_EXECUTING snapshot stalls (#2493) 2025-09-10 14:55:57 +01:00
Dan b9b17e24ac Added anchor browser example (#2488)
* Added anchor browser example project

* Updated node ver
2025-09-09 14:32:26 +01:00
Saadi Myftija 12bef0a938 feat: env command in the cli (#2485)
* Add a CLI command to list and view env vars

* Add changeset

* Restrict pemissions on env files created with `env pull`

* Escape env vars when exporting to file

* Switch changeset to patch
2025-09-09 13:13:24 +02:00
Saadi Myftija 5567f49846 feat(webapp): expose project git settings (#2464)
* Fix settigns page delete project width issue

* Apply a couple of touch-ups to the project settings page

* Add UI flow to connect gh repos

* Enabling adding another gh account in the ui

* Enable connecting a repo to a project

* Enable updating git settings

* Enable disconnecting gh repos from a project

* Remove prisma migration drifts

* Hide git settings when github app is disabled

* Fix migration order

* Avoid using `location` to avoid SSR issues

* Make branch tracking optional

* Disable save buttons when there are no field changes

* Disable delete project button unless the input matches the project slug

* Show connected repo connectedAt date

* Check that tracking branch exists when updating git settings

* Show tracking branch hint in the deployments page

* Fix positioning issue of the pagination pane in the deployments page

* Use mono font for branch names

* Add link to git settings

* Show tracking branch hint for the preview env too

* Add a confirmation prompt on repo disconnect

* Add link to configure repo access in gh

* Add rel prop to github links

* Automatically open repo connection modal after app installation

* Apply some fixes suggested by mr rabbit

* Fix flash cookie issue

* Extract project settings actions into a service

* Extract project settings loader into a presenter service

* Introduce neverthrow for error handling

* Try out neverthrow for error handling in the project setting flows

* Move env gh branch resolution to the presenter service
2025-09-09 13:03:43 +02:00
Dan 71060d93b1 Removed some remaining /v3’s from the docs (#2478)
* Removed /v3 from numerous files

* Fix AWS SDK v3 documentation link for S3 uploads
2025-09-08 10:01:09 +01:00
Eric Allam 89b1d8ba13 fix all lifecycle hooks (#2480) 2025-09-05 11:17:38 +01:00
Eric Allam 97015ba8c8 fix(docs): update sentry error tracking guide for v4 (#2479) 2025-09-05 11:13:13 +01:00
nicktrn 99660112bd feat(supervisor): add configurable resource requests (#2474) 2025-09-04 17:09:34 +01:00
Matt Aitken e6586d3c1a Remove deprecated releaseConcurrency from wait.forToken() docs (#2477) 2025-09-04 14:57:17 +01:00
Eric Allam 00d32ed4ee feat(webapp): add support for running web services (api, engine, webapp) in cluster mode for better perf (#2472)
* feat(webapp): add support for running web services (api, engine, webapp) in cluster mode for better perf

* cleaned up signal handling and resolved some valid 🐇 issues
2025-09-04 09:53:15 +01:00
nicktrn a1e9738faa fix(webapp): prevent duplicate preview env image tags (#2475)
* fix(webapp): prevent duplicate preview env image tags

* use deploy shortcode instead of new nanoid

* replace regexps in tests
2025-09-04 09:37:22 +01:00
nicktrn 59c17e04e9 feat(run-engine): worker queue resolver (#2476) 2025-09-04 09:27:06 +01:00
Eric Allam ed23615aa4 perf(webapp): add event loop utilization metric (#2471)
* perf(webapp): add event loop utilization metric

* add event loop utilization logging as well
2025-09-03 11:39:03 +01:00
Saadi Myftija 436d951b65 feat(webapp): github app installation flow (#2463)
* Add schemas for gh app installations

* Implement gh app installation flow

* Make the gh app configs optional

* Add additional org check on gh app installation callback

* Save account handle and repo default branch on install

* Do repo hard deletes in favor of simplicity

* Disable github app by default

* Fix gh env schema union issue

* Use octokit's iterator for paginating repos

* Parse gh app install callback with a discriminated union

* Remove duplicate env vars

* Use bigint for github integer IDs

* Sanitize redirect paths in the gh installation and auth flow

* Regenerate migration after rebase on main to fix ordering

* Handle gh install updates separately from new installs
2025-09-02 16:35:33 +02:00
nicktrn cf9398b56e fix(run-engine): retry SIGTERM errors (#2468) 2025-09-02 15:18:16 +01:00
Eric Allam 9b1877bef2 fix(run-engine): retry non-zero exit code errors (#2467)
We’re also now saving the retryConfig from the BackgroundWorkerTask on 
TaskRun.lockedRetryConfig when the run is first locked to the version
2025-09-02 14:12:14 +01:00
Eric Allam 0b2b73fc52 chore(docs): improve version pinning advise to use installed trigger.dev CLI (#2466) 2025-09-02 14:10:01 +01:00
nicktrn 0d1eac9406 feat(supervisor): dynamic queue consumer pool (#2461)
* feat(supervisor): dynamic queue consumer pool

* add changeset

* fix: correctly handle zero median and even samples

* feat(supervisor): consumer pool metrics

* fix tests

* more tests and fixes

* decrease default scaling cooldowns

* don't treat initial pool size as scale up

* handle scale down when queue length drops to zero

* remove changeset, supervisor changes only

* add damping factor env var
2025-09-02 13:52:42 +01:00
Willow (GHOST) ddbae6b6b4 fix(docker): clickhouse healthcheck (#2462) 2025-09-01 20:20:17 +01:00
nicktrn 2b095b1072 chore(webapp): upgrade otel packages and add more metrics (#2458)
* feat(webapp): upgrade otel packages and add more metrics

* add env var to disable additional detectors

* expose more prisma metrics

* chore(webapp): drop node 16 support
2025-09-01 16:24:12 +01:00
Dan 847ea866b6 Various docs improvements (#2456)
* Added MCP to the intro

* Added human-in-the-loop and new build extensions

* Waitpoint notes

* Moved openai guardrails example to Python

* Added a connection limit note

* Added Supabase + Prisma note to supabase auth

* Updates based on what the rabbit said

* Moved the supavisor section to the prismaExtension docs
2025-08-29 16:48:04 +01:00
James Ritchie fee31f2dc0 chore(webapp): update region message to say we are GDPR compliant (#2455)
* Updates the Region message to say we are GDPR compliant

* Hide the message if not on cloud
2025-08-29 14:33:46 +01:00
github-actions[bot] 10baa396d8 Release v4.0.2 (#2447)
* chore: Update version for release

* Release v4.0.2

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
2025-08-29 12:58:22 +01:00
James Ritchie 27bc2ad232 chore(webapp): show a regions info panel to explain about data residency (#2428) 2025-08-29 11:35:54 +01:00
James Ritchie 3a3e863daf chore(webapp): bulk action cancel as default (#2437)
* Switches “replay” and “cancel” options in the list

* Fixed incorrect cancel icon being used

* Fixes the alignment of the radiobuttons
2025-08-29 11:30:46 +01:00
Eric Allam f5f14c4206 chore(webapp): fix @trigger.dev/database imports so it doesn't end up in the frontend bundle (#2452) 2025-08-29 11:27:01 +01:00
Eric Allam 892adfe81c fix(mcp): don't require a trigger.config.ts file to be present when installing rules (#2453) 2025-08-29 11:20:45 +01:00
nicktrn 73e7378459 feat(webapp): add ecs otel resource attributes (#2454)
* add attributes to runlock dequeue span

* feat(webapp): add ecs otel resource attributes
2025-08-29 11:05:47 +01:00
Eric Allam b0b0df69be fix(run-engine): prevent stalled SUSPENDED snapshots from endlessly retrying (#2448) 2025-08-28 14:25:50 +01:00
Saadi Myftija e629810e96 fix: dev env connection issue (#2449)
This PR fixes an issue introduced in #2391 where the dev runtime env was not being resolved correctly.
2025-08-28 14:05:10 +01:00
Ognjen Simic e6f6d93e59 docs: update triggering.mdx fixing wrongly stated argument position for options (#2283)
* docs: update triggering.mdx fixing wrong argument numeration for options

Update the docs triggering.mdx: text indicating options argument position inside `tasks.trigger` function.

Options object is a third argument instead of stated second argument.

* update triggering.mdx to change options argument position in batchTrigger
2025-08-28 10:16:42 +01:00
Eric Allam 6f6ca01584 feat(mcp): add wait_for_run_to_complete tool so agents don't spam the get_run_details call after triggering (#2445)
* feat(mcp): add wait_for_run_to_complete tool so agents don't spam the get_run_details call after triggering

This also fixes the search docs MCP tool

* Install mcp using the latest tag, not the specific version
2025-08-28 10:13:46 +01:00
James Ritchie 11cbd1b5a0 Docs: Fixes incorrect retrieving runs from batchTrigger (#2427)
* Docs: fix for retrieving runs in batchTrigger

* Removes old v3-to-v4 migration guide (we have a redirect)

* Corrects tag limit from 5 to 10
2025-08-28 10:11:42 +01:00
Eric Allam 82401ec4fe fix(run-engine): dequeue performance improvements and improved telemetry (#2446)
* more eager dequeuing, queue cooloff periods, return workerQueueLength when dequeueing

* Cache worker group authentication and remove old self-hosted worker code (only managed is currently supported)

* add additional spans during dequeue

* Add env vars and additional spans
2025-08-27 21:22:10 +01:00
Matt Aitken 24a915133e Prisma 6.14.0 upgrade (#2444)
* Initial work on upgrading to 6.14.0

Set the output to node_modules still to make it easier

* Use ./generated Prisma folder, update types to fix issues

* Docker compose restart Clickhouse

* Prisma instrumentation update

* Docker

* Removed database dockerignore file, add generated prisma client to the top-level one

* Delete v3-catalog package.json

* Resolved pnpm lock file

* Log errors for very slow queries
2025-08-27 16:52:58 +01:00
Eric Allam 688b108ec3 chore(references): remove v3-catalog (#2443) 2025-08-27 14:35:55 +01:00
Saadi Myftija 1cc62230ab feat: introduce organization access tokens (#2391)
* Create schema and migration for organization access tokens

* Add helpers for creating and authenticating OATs

* Adapt the auth service to also accept OATs

* Accept OATs in the whoami v2 endpoint

* Enable deployments with the CLI using OATs

* Avoid reading env variables directly in the token utils

* Remove duplicate cli token utils

* Validate ENCRYPTION_KEY length when parsing env vars

* Make token utils a server-only module

* Disallow revoking already revoked OATs

* Simplify generics in authenticateRequest

* Use 32 bytes mock encryption key in the test setup

* Update dummy encryption key values in tests and templates

* Add a column in the OATs table to differentiate between user and system generated

* Simplify args for v3ProjectPath

Co-authored-by: Matt Aitken <matt@mattaitken.com>

* Add index on org id and createdAt

* Avoid storing the encrypted oat token and its obfuscated version in the DB at all

It is a safer approach. Also we do not need to ever read the decrypted token value after creation.

* Fix prisma update condition

* Add token type to the OAT table index

* Accept OATs in the mcp auth flow

* Simplify env auth flow around the /projects endpoints

---------

Co-authored-by: Matt Aitken <matt@mattaitken.com>
2025-08-27 13:23:45 +02:00
Eric Allam 96243efe16 docs: MCP Server 2.0 (#2385)
* Add MCP Server documentation to documentation site

This change introduces comprehensive documentation for the Trigger.dev MCP Server, covering its installation, usage, and available tools. The addition is necessary to provide users with detailed guidance on leveraging the MCP Server for managing Trigger.dev projects, tasks, and deployment processes.

- Updated `docs.json` to include a new section for the MCP Server documentation, improving navigation and accessibility.
- Added `mcp-introduction.mdx` as an introductory guide to the MCP Server, detailing its purpose and installation steps.
- Included `mcp-tools.mdx`, a document outlining each tool available in the MCP Server, complete with descriptions and example usages.
- The new content is aimed at improving user understanding and efficiency in using Trigger.dev's MCP Server capabilities.

* Refactor MCP installation documentation

The changes were necessary to streamline the MCP installation guidance and align our documentation with the latest client support capabilities. Removing the manual installation section and redundant information helps reduce confusion for users.

- Updated installation instructions to lead with auto-configuration for supported clients.
- Cleaned up the options and examples sections to enhance clarity.
- Reformatted client configuration details for better readability.
- Removed outdated links in the "Next Steps" section, focusing on more direct resources such as available MCP tools.

* mcp docs edits

* Add debugMode to get_run_details

* add cancel_run

* Add list deployments tools

* Add list preview branches tool

* Docs: adds agent rules page

* Adds redirect for the CLI path to work

* Adds subagent install image

---------

Co-authored-by: James Ritchie <james@trigger.dev>
2025-08-22 15:37:35 +01:00
Eric Allam ad26cded99 fix(v3): prevent saturated queues from dominating dequeue attempts by adding a cooloff period of successive failed dequeues (#2439)
* fix(v3): prevent saturated queues from dominating dequeue attempts by adding a cooloff period of successive failed dequeues

* Add env vars and add a marqs shutdown on SIGTERM/INT
2025-08-22 14:37:21 +01:00
Eric Allam 5e4756f8b7 fix(v3): eagerly dequeue messages from a queue when that queue is added to or removed from (v4 backport) (#2438)
* fix(v3): eagerly dequeue messages from a queue when that queue is added to or removed from (v4 backport)

* fixed configuration
2025-08-22 13:23:34 +01:00
Eric Allam c16cc577b0 fix(v3): adopt v4 dequeue strategy to increase perf (#2436) 2025-08-22 07:40:59 +01:00
github-actions[bot] 5b40c44ca6 Release v4.0.1
* chore: Update version for release

* Release v4.0.1

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Eric Allam <eallam@icloud.com>
2025-08-21 17:41:58 +01:00
Eric Allam c7e3811794 fix(webapp): correctly associate event-loop-blocked spans with parent span context (#2435) 2025-08-21 16:21:26 +01:00
Eric Allam bf2120d5a7 fix(runengine): remove dev worker queues from metrics (#2434) 2025-08-21 14:11:22 +01:00
Eric Allam 895e24a0d7 fix(cli): various mcp server fixes (#2430)
* fix: span events are optional, as logs don't have them

* remove CI: true env var to prevent massive tool responses

* fix github resolver relative path

* prevent the dev command from asking for user input when not interactive

* Remove lodash.get because it's deprecated

* span output can be object other than an object, like a string

* prevent large traces from causing get_run_details failures

* Decreased max log lines, made it a tool input property

* Some install mcp tweaks
2025-08-21 13:57:14 +01:00
Matt Aitken 58ff034325 Add attemptDequeue span to marqs (#2433)
* Add attemptDequeue span to marqs

More telemetry to figure out why dequeuing is slow

* More spans

* use "receive" for all operation names
2025-08-21 13:09:08 +01:00
Dan 84750f68a4 Updated main README (#2432)
* readme wip

* Added new commands

* Simplified readme

* Added docs links and improved about

* Added nice header and updated links

* Consistent headers

* Copy update

* Updated badges

* Added features / agents section

* Updated list

* Added links

* Updated links

* Minor readme tweaks

* Updated banner

* removed /v3/

---------

Co-authored-by: Matt Aitken <matt@mattaitken.com>
2025-08-21 12:28:52 +01:00
Matt Aitken 1087e5ede0 When replaying a v3 run on v4, don't pass the region (#2431) 2025-08-21 10:15:26 +01:00
Eric Allam f086626a41 feat(cli): MCP server 2.0 (#2384) 2025-08-20 16:04:50 +01:00
Matt Aitken 8483900832 Updated queue/concurrency docs for v4 (#2418) 2025-08-20 11:47:56 +01:00
Matt Aitken 4264fc066e chore(webapp): More false errors turned to logs (#2422)
* Remove variables from dequeue log message

* Continue snapshot throw json

* Waiting for deploy error removed

* Realtime ECONNRESET is expected

* Redis worker logErrors changes, removed ids

* Preview branch without a branch shouldn't log an error, it's a user provided issue

* "Task run is not in a cancellable state" isn't an error, it's expected

* "CreateCheckpointService: Child run already resumed" is expected

* "CreateCheckpointService: Batch already resumed" is expected

* "Failed to insert events, will attempt bisection" changed to info, we have errors for complete failures

* Ignore "PrismaClient error"

* Don't log Redis worker DLQ errors if we're ignoring

* "Failed to parse machine config" is fine, sometimes a config is null or undefined

* "Failed to parse machine config" for v3

* MetadataTooLargeError shouldn't log an error
2025-08-20 11:44:26 +01:00
nicktrn da79c2b185 chore(docker): add registry namespace var to .env.example (#2426) 2025-08-20 11:03:30 +01:00
nicktrn 677b7aed9c chore(docs): update bun guide (#2425)
* chore(docs): remove missing snippet references

* chore(docs): update bun guide
2025-08-20 11:03:23 +01:00
nicktrn 9092ca863f feat(supervisor): add ecr support to docker client (#2424) 2025-08-20 10:49:19 +01:00
James Ritchie b41129a24f Tabular-nums for the individually selected runs number (#2421) 2025-08-19 16:24:02 +01:00
James Ritchie 0261ccbc68 Fixes conflicting shortcut keys (#2420) 2025-08-19 15:38:53 +01:00
nicktrn 32e3fab5ea feat(helm): release 4.0.0 chart version (#2416)
* feat(helm): release 4.0.0 chart version

* chore(helm): remove last few beta references

* fix(docker): use latest tag for commands shown in webapp
2025-08-19 11:06:35 +02:00
Matt Aitken 0ee4234117 Reduce false error logging (round 1) (#2415)
* Don't log an error when the snapshot shouldn't be created, it's normal for this to happen

* Slack alerts, skip `account_inactive` errors

* v3 finalize run with no locked isn't an error

* Another false error

* Finalize run CRASHED runs were logging errors

* All slack alert errors are warnings except invalid blocks
2025-08-19 09:58:21 +01:00
Matt Aitken 471c96079a Some more queue docs changes for v4 (#2414) 2025-08-18 16:49:43 +01:00
Eric Allam 0ae59cf7b9 fix appEnv on undefined error and useShortcuts error on ErrorBoundary (#2412) 2025-08-18 15:42:16 +01:00
Matt Aitken 49e6470b6b Added missing trigger.config options to the docs (#2409) 2025-08-18 15:11:34 +01:00
Matt Aitken 9dd831f68a Fix for v4 pending versions not getting dequeued after a deploy (#2411)
We didn’t call `scheduleEnqueueRunsForBackgroundWorker` in the right 
place…
2025-08-18 15:09:12 +01:00
nicktrn 112f6f602e feat(supervisor): optionally strip digests from image refs (#2410) 2025-08-18 15:08:49 +01:00
github-actions[bot] 0b35cc35e0 Release v4.0.0 🚀
🚀 Publish Trigger.dev Docker / units (push) Failing after 0s
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 4s
🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker-v4 (push) Has been skipped
2025-08-18 12:35:03 +01:00
James Ritchie fade22015f Docs – v4 GA updates (#2298)
* Adds new features table to top of v4 upgrade guide

* Adds wait idempotency to wait-until, wait-for, and wait-for-token pages

* Adds new priority docs page and updates the v4 upgrade guide

* Adds new task lifecycle hooks

* Removes the message about requiring tasks to be exported

* Adds new global lifecycle hooks section

* Moves sections from upgrade guide into the table

* Adds hidden task page

* Improves the global lifecycle hooks section

* Updates middleware and locals section

* Adds new useWaitToken page to the react hooks section

* Adds a new ai.tool section

* Moves Docker (legacy) page into self-hosting section

* Removes known issues from v4 upgrade guide

* Replace “toolTask” with “ai.tool” in the Streams page example

* Renames guide to “Migrating from v3” and adds redirect

* Remove references to v4

* Removes changelog from migration guide

* The installation guide now references `@latest update`

* Changes all references from `/sdk/v3` to `/sdk`

* Updates @v4-beta to @latest

* Fixed broken link

* Fixes broken link

* Adds an upgrade to v4 using AI section

* Fixes 2 broken links

* Adds an entry for targetting preview branches

* Updates the run statuses

* Adds boolean helpers section to the runs and realtime pages

* Updates the concurrency page

* Updates the test page to include the new options

* Adds SDK and curl options for the preview branch targeting

* Updates new bulk actions page

* Remove the releasing concurrency section

* Got rid of some more @v4-beta mentions

* Improved rate limit docs

* Improved migrating docs

* Removed commented sections of the docs

* useWaitToken hook

* Fixed the description

* Fix for missing test image

---------

Co-authored-by: Matt Aitken <matt@mattaitken.com>
Co-authored-by: Dan <8297864+D-K-P@users.noreply.github.com>
2025-08-18 12:34:55 +01:00
Dan 78af477efc Updated readmes (#2408)
* readme wip

* Added new commands

* Simplified readme

* Added docs links and improved about

* Added nice header and updated links

* Consistent headers

* Copy update
2025-08-18 12:10:15 +01:00
Eric Allam e864973e01 chore(packages): leave changesets prerelease mode (#2406) 2025-08-18 11:06:31 +01:00
github-actions[bot] 132008ff64 Release 4.0.0-v4-beta.28 (#2386)
* chore: Update version for release (v4-beta)

* Release 4.0.0-v4-beta.28

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
2025-08-17 20:30:33 +01:00
nicktrn 26d4d08bdc chore: move all placement tag helpers to core (#2403)
* chore: move all placement tag helpers to core

* move placement tag utils into server-only
2025-08-15 17:19:39 +01:00
Eric Allam 1a8880971d fix(runner): prevent retry immediately race condition which can cause stuck runs that end up being system failures (#2402) 2025-08-15 17:05:02 +01:00
Eric Allam 3d17ce5559 feat: AI SDK 5.0 support (#2396)
* feat: AI SDK 5.0 support

* Add changeset
2025-08-15 17:03:39 +01:00
nicktrn a76905e272 chore(bun): upgrade to bun v1.2.20 (#2398) 2025-08-15 17:01:06 +01:00
Eric Allam 1c73ab930b fix(realtime): waitUntil now correctly waits for metadata.streams to finish (#2399) 2025-08-15 17:00:45 +01:00
Eric Allam 816c7ad56e fix(webapp): remove unnecessary log (#2401) 2025-08-15 12:47:55 +01:00
Eric Allam 4d975b19e7 fix: external traces now respect parent sampling, and prevent broken traces when there is no external trace context (#2395)
* fix: external traces now respect parent sampling, and prevent broken traces when there is no external trace context

* Add changeset

* improve trace flag handling and better internal host checking

* the traceFlags are now being properly passed through as a number
2025-08-15 10:12:38 +01:00
Eric Allam 4c4b774bab fix(webapp): Don't propogate region when replaying a run from or to development (#2400) 2025-08-15 10:07:59 +01:00
nicktrn fa7f4b1fed feat(k8s): add placement tags for flexible node selection (#2390)
* add tier scheduling support to supervisor

* add billing info to dequeued message w/o cache

* add cache with best effort invalidation

* fix invalidate circular dep

* add changeset

* use new plan type on runs as fallback during dequeue

* tidy up

* be more explicit with plan type fallback

* remove additional billing check from hot path

* switch to placement tags

* update changeset

* update platform package

* start using new entitlement response

* ensure skipChecks optimization validates at batch level

* add optional items to add to queue manager limits

* make the bool env helper only accept boolean defaults

* remove redundant private field

* update placement tag helper to prevent unsupported tags
2025-08-15 09:52:14 +01:00
Saadi Myftija f76765448b feat: show gh username and avatar for deployments triggerd by the github app integration (#2394)
Small change that shows the github user in the deployments page in cases
where it was triggered by the github app integration.

These deployments are not triggered using a PAT, so we cannot associate
them with a Trigger user ID.
2025-08-14 18:20:59 +02:00
Saadi Myftija 944b1875f1 feat: gather deployment git meta from the build server (#2392)
This PR adapts the deployment command to also evaluate build server env
variables when creating the git meta. The build server will initially
only support deployments triggered by the github app, but we might add
other git providers in the future.

Sticking to the GH actions naming convention for the env variables set
by the build server in favor of consistency.
2025-08-14 16:07:36 +02:00
Eric Allam a5dd6389b2 fix: sentry memory leak by patching @sentry/remix to stop cloning request (#2389)
* fix: sentry memory leak by disabling includeLocalVariables

* Enhance heap snapshot consistency and labeling

To facilitate more accurate and consistent heap memory snapshots, a new function forceConsistentGC was added before taking a snapshot. This ensures the garbage collector (GC) runs multiple times, stabilizing the heap state for more reliable analysis. This is particularly helpful when debugging memory-related issues.

Updates to the memory-leak-detector script now allow labeling of snapshot runs using the --label flag. This helps in distinguishing different runs for easier tracking and comparison of memory usage across test sessions. Additionally, the --expose-gc flag ensures that the GC can be manually triggered during test runs, leading to more consistent memory states and potentially uncovering hidden memory leaks.

* Refactor forceConsistentGC for improved readability

The function forceConsistentGC was refactored to enhance code readability and consistency. The main improvements include:

- Updated syntax for consistent string quotation and spacing.
- Simplified garbage collection by removing specific major/minor GC calls, as the distinction isn't necessary.
- Implemented minor changes to arrow function formatting for consistency.

These changes neither impact the program logic nor the function behavior but help maintain code quality standards and readability.

* Fix memory leak by removing request.clone() usage

Identified that the memory leak in the project was linked to the usage of request.clone() within the `@sentry/remix` package's callRouteAction handler. Although initially suspected as a Sentry issue, the problem appears to arise from the handling of request.clone() in Remix version 2.1.0. By removing the call to request.clone(), the memory leak has been resolved.

- Introduced garbage collection execution before snapshot to manage memory allocation effectively.
- Improved error handling and timeout mechanisms in the memory leak detector to enhance its resilience during runtime.
- Expanded testing for both GET and POST requests to monitor and validate potential memory leaks better. The POST requests involve sending large payloads to stress-test the system.
- The modification particularly focuses on enhancing robust memory tracking and providing detailed progress reporting during request phases.

* patch @sentry/remix to prevent memory leaks

* Fix pnpm lock

* undo some unrelated changes
2025-08-14 14:50:20 +01:00
Saadi Myftija 3ec95cd72d fix: code editor/block theme inconsistency (#2393)
Tiny fix to make the themes across code blocks and editor components across the app consistent.
2025-08-14 11:41:36 +02:00
Eric Allam 6f29b7dc34 fix: legacyDevProcessCwdBehaviour defaults to true (#2387)
This fixes an issue with prisma not being able to locate the native 
add-on binary
2025-08-13 14:30:10 +01:00
Eric Allam ed07bf9306 fix: prevent circular reference errors on task indexing when using schemaTask (#2383)
* fix: prevent circular reference errors on task indexing when using schemaTask

* add changeset
2025-08-13 14:10:57 +01:00
Matt Aitken cd6a0b3a3e fix: heartbeat race condition (acking) (#2380)
* The logger now supports metadata

* Added metadata to ServiceValidationError in some critical places

* Don't ack the heartbeat if there's a mismatch, it might prevent a brand new one

* Don't ack the heartbeat inside stalled. By returning it will be acked IF the deduplication key matches
2025-08-12 10:35:21 +02:00
Eric Allam e9cc7de8ea docs: Add beta.27 changelog (#2381) 2025-08-11 22:06:12 +01:00
github-actions[bot] 0448f3603a Release 4.0.0-v4-beta.27
* chore: Update version for release (v4-beta)

* Release 4.0.0-v4-beta.27

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Matt Aitken <matt@mattaitken.com>
2025-08-11 15:39:59 +01:00
Matt Aitken dd4515a2eb Merge pull request #2379 from triggerdotdev/remove-trigger-and-poll
Remove triggerAndPoll()
2025-08-11 15:11:58 +01:00
Matt Aitken 97e5c22051 Added a changeset 2025-08-11 13:42:14 +01:00
Matt Aitken b8d2fb215f Remove triggerAndPoll() 2025-08-11 13:40:37 +01:00
Takeoff 21c2c136be test coverage for utils (#2057) 2025-08-11 11:53:03 +01:00
Matt Aitken ae19d0f491 Only start an attempt if not finished. Send message to worker if pending executing (#2377)
* Only start an attempt if not finished. Send message to worker if pending executing

* Fix the exit process reason tet

* Fixed cancelling test since bug fix

The old behaviour was wrong for pending executing in the test
2025-08-11 11:46:10 +01:00
Matt Aitken c58b78f43f Exit the process if status is finished and not in the middle of completing (#2378)
This solves an issue where a run is system failed from the platform and 
the worker is never shut down in the cluster.
2025-08-11 11:22:26 +01:00
Matt Aitken a968c23678 Don't use a failed run with idempotency in v4 (#2376) 2025-08-08 14:01:18 +01:00
nicktrn 4b08597c77 feat: optional v4 deploy registry overrides (#2370)
* feat: optional v4 deploy registry overrides

* only require credentials for non-ecr registries
2025-08-07 17:02:01 +01:00
Matt Aitken 9ce4f57a6a Upgraded v4 experimental flags. keepNames and autoDetectExternal now default to true. (#2371)
* keepNames default true, not experimental now

* Added docs for the `environments` declarative schedules feature

* minify and autoDetectExternal (true now)

* processKeepAlive

* devProcessCwdInBuildDir

* Changeset

* Switch to legacyDevProcessCwdBehaviour
2025-08-07 16:47:45 +01:00
nicktrn 363b9b15b4 fix(engine): set correct run status after requeue and snapshot status after dequeue (#2367)
* set correct run status on snapshot after dequeue

* set run status back to PENDING when we requeue

* remove retrying after failure status from v4 and fix tests

* fix one last test
2025-08-07 16:46:27 +01:00
James Ritchie 51305b6023 Onboarding dashboard background (#2346)
* Adds a background image dashboard wrapper

* Dashboard background image

* Adds a background image to the welcome onboarding page

* Background is constructed of 3 images

* Adds the background to the create org page

* Adds a background to the choose plan page

* Change the default button spinner color to white

* Adds the background to the create new project page

* Updates the invite team member page

* Adds background image to received invite page
2025-08-07 16:13:42 +01:00
Dan 9e41e83818 Restructured the realtime docs (#2317)
* Reordered react hooks + frontend sections

* Updated the overview and nav

* Separated out SWR hooks

* Restructured metadata sections

* Improved backend docs

* Fixed broken link

* Fixed broken links

* Updated the structure

* Restructured overview

* Updated examples cards

* Improved overview

* Updated how it works

* Updated auth

* Added type safety to the run object page

* made the subscribe description clearer

* Fixed links in triggering

* Fixed link

* Removed examples footers

* Copy tweak

* Consolidated metadata and subscribe pages

* moved metadata task examples to metadata

* Fixed links

* Fixing links like zelda

* Removed dead import

* Clearer titles

* Cap R for Realtime

* Fixes broken link

---------

Co-authored-by: James Ritchie <james@trigger.dev>
2025-08-07 16:02:52 +01:00
Matt Aitken da7dc74fa4 Replaying keeps the original run region (#2368) 2025-08-07 14:43:28 +01:00
Matt Aitken af14621683 Specify a region when triggering (#2366)
* Map new allowedMasterQueues → allowedWorkerQueues

* ClickHouse worker_queue on task runs

* Added the Region to the run inspector

* Pass a region in when triggering

* Added a changeset

* Added triggering regions docs

* Added region to the ctx

* Fix for backfiller masterQueue/workerQueue
2025-08-07 12:41:39 +01:00
Matt Aitken 9787120cfb We need to set TRIGGER_OTEL_EXPORTER_OTLP_ENDPOINT for deployed runs (#2365) 2025-08-07 11:40:14 +01:00
Eric Allam e46e9bd30f docs: Manual setup guide (#2358)
* Add manual monorepo setup guide for Trigger.dev

This commit introduces a new "Manual setup" guide for setting up Trigger.dev in projects, specifically focused on monorepo configurations. The guide outlines two primary approaches for monorepos: creating a dedicated tasks package and integrating tasks directly within apps. This detailed documentation aims to help developers manually configure their projects, bypassing automated steps, and understanding the setup better.

- Provides step-by-step instructions for both 'Tasks as a package' and 'Tasks in apps' approaches.
- Includes example configurations for various package managers, environment setups, and runtime options.
- Enhances user understanding of Trigger.dev's configuration requirements in complex monorepo environments.

The purpose of adding this guide is to enable developers to seamlessly integrate Trigger.dev into their monorepos, whether they choose to abstract tasks into packages or embed them within individual applications. This flexibility supports diverse project structures while maintaining consistency with Trigger.dev's operational prerequisites.

* Correct tasks usage in documentation

The 'tasks as package' section in the documentation had an incorrect example under 'Use tasks in your apps' which needed correction to align with the actual package usage.

- Fixed incorrect import of tasks by updating to the correct import from '@repo/tasks/trigger'.
- Updated the syntax to use 'tasks.trigger' with type parameters, following the new pattern established for triggering tasks with TypeScript.
- Added error handling to catch and log errors during task execution, returning a meaningful error message instead of just failing silently.

This update ensures developers have an accurate reference when implementing tasks in their applications, especially given the breaking changes in TypeScript compatibility due to recent package updates.

* Update package configuration for Zod 3-4 compatibility

The recent Zod package updates from version 3 to 4 introduce breaking changes in TypeScript compatibility that require adjustments in our project configuration files. The primary updates involve adding type annotations and modifying import statements to support the shift without breaking existing functionality.

- Updated `package.json` and initialization files to align with new compatibility requirements.
- Modified server task examples to explicitly use type imports for improved error handling and consistency.
- Adjusted workspace and project settings to maintain compatibility and improve configuration clarity.
- Provided references to current examples for better implementation guidance.

* More manual setup guide steps

* Fix bun docs link
2025-08-07 10:22:37 +01:00
James Ritchie 3cd7dd86d4 Fixed code example in reduce spend docs page (#2350) 2025-08-06 17:18:59 +01:00
Matt Aitken a782813c6c Regions – dashboard page, switching default, allowedMasterQueues (#2354)
* Initial Regions page

* Fix for bad attribute name

* Switching regions is working. Some style improvements

* Use a dialog for confirmation, not great yet

* Added allowedMasterQueues

* Improves flag icons

* Improves the region switch modal with more info

* Adds “suggest a region” table row

* New icons for the buttons

* Improved the tooltip information

* New “small” badge style

* Make the default badge live in its column

* Better DO icon size

* Remove unused export of regions options

* Show upgrade message for free users to get static IPs

* Admins can view all regions and switch at will

---------

Co-authored-by: James Ritchie <james@trigger.dev>
2025-08-06 15:56:01 +01:00
Eric Allam d950a969bd Update zod package to version 3.25.76 across all modules (#2352)
* Update zod package to version 3.25.76 across all modules

Update the zod library from version 3.23.8 to 3.25.76 in multiple package files to ensure compatibility and take advantage of new features or bug fixes introduced in recent releases. Keeping all modules synchronized with the latest version of zod helps maintain consistency across the project and reduces potential compatibility issues.

- Modified zod version in apps/supervisor, webapp, and various internal packages.
- Updated zod references in pnpm-lock.yaml to reflect the new version.
- Ensure dependencies that rely on zod are using the updated version to avoid mismatches.

* Add changeset
2025-08-06 14:48:43 +01:00
Eric Allam 8eecc7e548 chore: json schema missing changeset (#2353) 2025-08-06 13:56:57 +01:00
Eric Allam 1294076484 feat: index json schemas on tasks and schemaTask (#2351)
* Add payload schema handling for task indexing

This change introduces support for handling payload schemas during task indexing. By incorporating the `payloadSchema` attribute into various components, we ensure that each task's payload structure is clearly defined and can be validated before processing.

- Updated the TaskManifest and task metadata structures to include an optional `payloadSchema` attribute. This addition allows for more robust validation and handling of task payloads.
- Enhanced several core modules to export and utilize the new `getSchemaToJsonSchema` function, providing easier conversion of schema types to JSON schemas.
- Modified the database schema to store the `payloadSchema` attribute, ensuring that the payload schema information is persisted.
- The change helps in maintaining consistency in data handling and improves the integrity of task data across the application.

* Refactor: Remove getSchemaToJsonSchema in favor of schemaToJsonSchema

The `getSchemaToJsonSchema` function was removed and replaced with `schemaToJsonSchema` across the codebase. This update introduces a new `@trigger.dev/schema-to-json` package to handle conversions of schema validation libraries to JSON Schema format, centralizing the functionality and improving maintainability.

- Removed `getSchemaToJsonSchema` exports and references.
- Added new schema conversion utility `@trigger.dev/schema-to-json`.
- Updated `trigger-sdk` package to utilize `schemaToJsonSchema` for payloads.
- Extensive testing coverage included to ensure conversion accuracy across various schema libraries including Zod, Yup, ArkType, Effect, and TypeBox.
- The update ensures consistent and reliable schema conversions, facilitating future enhancements and supporting additional schema libraries.

* Add support for Zod 4 in schema-to-json

This change enhances the schema-to-json package by adding support for Zod version 4, which introduces the native `toJsonSchema` method. This method facilitates a direct conversion of Zod schemas to JSON Schema format, improving performance and reducing reliance on the `zod-to-json-schema` library.

- Updated README to reflect Zod 4 support with native method and retained support for Zod 3 via existing library.
- Modified package.json to allow installation of both Zod 3 and 4 versions.
- Implemented handling for Zod 4 schemas in `src/index.ts` using their native method.
- Added a test case to verify the proper conversion of Zod 4 schemas to JSON Schema.
- Included a script for updating the package version based on the root package.json.
- Introduced a specific TypeScript config for source files.

* Revise schema-to-json for bundle safety and tests

The package @trigger.dev/schema-to-json has been revised to ensure bundle safety by removing direct dependencies on schema libraries such as Zod, Yup, and Effect. This change minimizes bundle size and enhances tree-shaking by allowing external conversion libraries to be utilized only at runtime if necessary. As a result, the README was updated to reflect this usage pattern.

- Introduced `initializeSchemaConverters` function to load necessary conversion libraries at runtime, keeping the base package slim.
- Adjusted test suite to initialize converters before tests, ensuring accurate testing of schema conversion capabilities.
- Updated `schemaToJsonSchema` function to dynamically check for availability of conversion libraries, improving flexibility without increasing the package size.
- Added configuration files for Vitest to support the new testing framework, reflecting the transition from previous test setups.

These enhancements ensure that only the schema libraries actively used in an application are bundled, optimizing performance and resource usage.

* Refine JSON Schema typing across packages

The changes introduce stricter typing for JSON Schema-related definitions, specifically replacing vague types with more precise ones, such as using `z.record(z.unknown())` instead of `z.any()` and `Record<string, unknown>` in place of `any`. This is part of an effort to better align with common practices and improve type safety in the packages.

- Updated the `payloadSchema` in several files to use `z.record(z.unknown())`, enhancing the type strictness and consistency with JSON Schema Draft 7 recommendations.
- Added `@types/json-schema` as a dependency, utilizing its definitions for improved type clarity and adherence to best practices in TypeScript.
- Modified various comments to explicitly mention JSON Schema Draft 7, ensuring developers are aware of the JSON Schema version being implemented.
- These adjustments are informed by research into how popular libraries and tools handle JSON Schema typing, aiming to integrate best practices for improved maintainability and interoperability.

* Add JSON Schema examples using various libraries

The change introduces extensive examples of using JSON Schemas in the 'references/hello-world' project within the 'trigger.dev' repository. These examples utilize libraries like Zod, Yup, and TypeBox for JSON Schema conversion and validation. The new examples demonstrate different use cases, including automatic conversion with schemaTask, manual schema provision, and schema conversion at build time. We also updated the dependencies in 'package.json' to include the necessary libraries for schema conversion and validation.

- Included examples of processing tasks with JSON Schema using libraries such as Zod, Yup, TypeBox, and ArkType.
- Showcased schema conversion techniques and type-safe JSON Schema creation.
- Updated 'package.json' to ensure all necessary dependencies for schema operations are available.
- Created illustrative scripts that cover task management from user processing to complex schema implementations.

* Refactor SDK to encapsulate schema-to-json package

The previous implementation required users to directly import and initialize functions from the `@trigger.dev/schema-to-json` package, which was not the intended user experience. This change refactors the SDK so that all necessary functions and types from `@trigger.dev/schema-to-json` are encapsulated within the `@trigger.dev/*` packages.

- The examples in `usage.ts` have been updated to clearly mark `@trigger.dev/schema-to-json` as an internal-only package.
- Re-export JSON Schema types and conversions in the SDK to improve developer experience (DX).
- Removed unnecessary direct dependencies on `@trigger.dev/schema-to-json` from user-facing code, ensuring initialization and conversion logic is handled internally.
- Replaced instances where users were required to manually perform schema conversions with automatic handling within the SDK for simplification and better maintainability.

* Add JSONSchema type for payloadSchema in tasks

The change was necessary to improve type safety by using a proper JSONSchema type definition instead of a generic Record<string, unknown>. This enhances the developer experience and ensures that task payloads conform to the JSON Schema Draft 7 specification. The JSONSchema type is now re-exported from the SDK for user convenience, hiding internal complexity and maintaining a seamless developer experience.

- Added JSONSchema type based on Draft 7 specification
- Updated task metadata and options to use JSONSchema type
- Hid internal schema conversion logic from users by re-exporting types from SDK
- Improved bundle safety and dependency management

* Add JSON schema testing and revert package dependencies

This commit introduces a comprehensive set of JSON schema testing within the monorepo, specifically adding a new test project in `references/json-schema-test`. This includes a variety of schema definitions and tasks utilizing multiple validation libraries to ensure robust type-checking and runtime validation.

Additionally, the dependency versions for `@effect/schema` have been adjusted from `^0.76.5` to `^0.75.5` to maintain compatibility across the project components. This ensures consistent behavior and compatibility with existing code bases without introducing breaking changes or unexpected behavior due to version discrepancies.

Key updates include:
- Added new test project with extensive schema validation tests.
- Ensured type safety across various task implementations.
- Reverted dependency versions to ensure compatibility.
- Created multiple schema tasks using libraries like Zod, Yup, and others for thorough testing.

* Refactor JSON Schema test files for clarity

Whitespace and formatting changes were applied across the `json-schema-test` reference project to enhance code readability and cohesion. This included removing unnecessary trailing spaces and ensuring consistent indentation patterns, which improves maintainability and readability by following the project's code style guidelines.

- Renamed JSONSchema type annotations to adhere to TypeScript conventions, ensuring that all schema definitions properly satisfy the JSONSchema interface.
- Restructured some object declarations for improved clarity, especially within complex schema definitions.
- These adjustments are crucial for better future maintainability, reducing potential developer errors when interacting with these test schemas.

* Fixed some stuff

* WIP

* we now convert schema to jsonSchema on the CLI side via the indexing

* Remove the json-schema-test reference project

* Improve schema-to-json peer deps and fix effect schema

* Explain the casting and match the version numbers

* Fixed a bunch more schema stuff

* Don't clean files that might be written to

* Don't use a custom version of vitest in the new package

* fix attw in schema-to-json
2025-08-06 13:44:56 +01:00
nicktrn f82a4e9651 fix(playwright): improve chrome installation and fix spinner duplication on narrow terminals (#2347)
* headless: false will now install binaries for headless mode too

* fix spinner message duplication on narrow terminals

* add changeset
2025-08-06 12:08:33 +02:00
Eric Allam 1f4434035e latest @opentelemetry packages and correlate external traces (#2334) 2025-08-06 08:10:46 +01:00
Dan 7bb88451be Sidebar fix (#2349)
* Commented out tooltip style

* Deleted stylesheet
2025-08-05 11:08:44 +01:00
James Ritchie be70f42bec Adds a link to the reduce spend docs page from the billing alerts page (#2345)
* Adds a link to the reduce spend docs page from the billing alerts page

* Copy tweak
2025-08-04 13:22:18 +01:00
James Ritchie bb54e97806 Show text wrapping on code blocks (#2338)
* Show text wrapping on deployment code blocks

* Show textWrapping on some more CodeBlocks

* Adds missing margins to the error block
2025-08-04 13:21:30 +01:00
James Ritchie edafc71142 Explain run status on run inspector panel (#2342) 2025-08-04 13:21:03 +01:00
James Ritchie 792c97339e Force-wrap long strings in the run error callout (#2340) 2025-08-04 13:19:57 +01:00
James Ritchie 0fc234b970 Fixes incorrect icon colors (#2339) 2025-08-04 13:18:32 +01:00
James Ritchie f8126871f3 Improves large output messaging (#2341)
* Adds more information to the large payloads/outputs

* Changes the Download logs button to secondary
2025-08-04 13:18:07 +01:00
James Ritchie df0dc07678 New Task page deploy blank state (#2344)
* Adds a step by step deploy blank state to Tasks page

* Adds links to Ask AI, docs and troubleshooting
2025-08-04 13:17:06 +01:00
James Ritchie 63857b0445 Makes it clear you only get preview branches if you’re on v4 (#2343) 2025-08-04 13:13:38 +01:00
James Ritchie b985843642 Prevents long project menu titles pushing the admin buttons out of bounds (#2332) 2025-08-04 13:13:03 +01:00
Boon Kai 9ecda816dc fix(cli): improve contrast for chalkWorker yellow in light mode (#2239)
* fix(cli-output): improve color contrast for chalkWorker on light mode

* add changeset

---------

Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
2025-08-04 12:09:38 +01:00
nicktrn 7b54c3527e fix(otel): prevent infinite retry loops on unicode hex escape errors (#2337)
* prevent infinite retry loop for unicode errors

* structured logs for prisma events

* preserve all prisma event fields

* either use structured logs or stdout, never both

* split runs repo tests

* decrease test shards to 8
2025-08-02 16:54:11 +01:00
nicktrn 60b8c76adc fix(hosting): pass registry namespace env var to webapp, bump helm (#2336)
* pass deploy registry namespace to webapp

* bump helm chart and images
2025-08-02 14:20:55 +01:00
Eric Allam 3fd8ea80da fix: backwards compatible snapshot runStatus to prevent DEQUEUED status from breaking older runners (#2335)
🚀 Publish Trigger.dev Docker / units (push) Failing after 0s
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 3s
🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker-v4 (push) Has been skipped
2025-08-01 17:19:58 +01:00
Matt Aitken c576e98dbc Fix v4 changelog latest date (#2333) 2025-08-01 15:23:55 +01:00
Matt Aitken b9e62139ff v4 docs upgrade changelog (#2331) 2025-08-01 14:17:40 +01:00
github-actions[bot] 44af80e055 Release v4.0.0-v4-beta.26
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 4s
🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker-v4 (push) Has been skipped
🚀 Publish Trigger.dev Docker / units (push) Failing after 4s
* chore: Update version for release (v4-beta)

* Release v4.0.0-v4-beta.26

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Matt Aitken <matt@mattaitken.com>
2025-08-01 12:48:08 +01:00
James Ritchie 1feabd732e Docs: Adds a Reduce your spend docs page (#2330) 2025-08-01 09:20:10 +01:00
Eric Allam 134942b2dd The schedule HTTP API now accepts deduplicationKeys as well as schedule IDs in the URL (#2329) 2025-07-31 14:12:31 +01:00
Eric Allam b153324cd9 fix: importing from runEngine/index.js breaks non-node runtimes (#2328) 2025-07-31 13:22:02 +01:00
James Ritchie 59bfce86db Docs: Fixes incorrect concurrency pricing on the limits page (#2324) 2025-07-31 12:07:42 +01:00
Eric Allam 748565c9db Fix backwards compatible execution by allowing additional properties to passthrough (#2327) 2025-07-31 11:14:46 +01:00
Matt Aitken 8b31871998 Usage billing alerts (#2323)
* First draft billing alerts page

* Budget alert form working

* Don't let free plan users change the billing alert amount

* Fix missing key in map in the form

* Disable queues/org from admin API endpoint

* Don't allow resuming if runsEnabled is false

* Refer to "Billing alerts" not "Plans"

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>

* Form missing dependencies fix

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>

* Deal with thrown errors, fix for duplicating email fields

* Added a RuntimeEnvironment organizationId index

---------

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
2025-07-30 13:48:02 +01:00
Eric Allam 14dcc76f93 feat: run.ctx tidying and additions (#2322)
* Cleanup context and execution creation, cache stuff, add parent and root task run ids

* more efficient by using friendly IDs instead of doing joins

* metadata.root/parent now reference current run when run has no root/parent

* Adding changeset

* try to make test less flaky

* Clean imports

* Another attempt to fix the flaky test

* Fix usage by still passing durationMs and costInCents to the execution, just not the run.ctx
2025-07-30 13:47:12 +01:00
Nicolas Rigaudière 5ea6605ec6 feat(extensions): add lightpanda (#2192)
* feat: add lightpanda structure

* chore: add lightpanda doc links

* fix: lightpanda extension instructions

* feat: add Lightpanda guide and examples

* feat: lightpanda - add 3rd example

* feat: add lightpandaTask

* fix: lightpanda 3rd example

* fix: lightpanda 1st example

* chore: add changeset

* add v4 tag to guide

* fix: merge lightpanda docker instructions

* fix: add failsafes

* add scrape warning

* lint

* successful login also switches to that profile

* update docs and links as this is v4 only

* extension tweaks

* simplify extension

* update examples

* update docs

* remove from extensions list as v4 only

* remove from catalog

* update changeset

---------

Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
2025-07-28 18:38:13 +01:00
nicktrn 3e1cc69db9 feat(ecr): make assume role optional (#2321) 2025-07-28 14:47:30 +01:00
nicktrn 2ea52da628 fix(runner): disable warm starts after SIGTERM (#2316)
* security: bump form-data to latest patch versions (CVE-2025-7783)

* disable warm starts on sigterm

* add changeset
2025-07-28 11:29:49 +01:00
Saadi Myftija 3493b9464f Cancel readonly ClickHouse queries on client close (#2319) 2025-07-28 11:00:35 +01:00
Eric Allam a7ff6de388 fix: otel logs better DynamicFlushScheduler (#2318)
* Improve dynamic flush scheduler for otel data

The changes introduce a more flexible and adaptive dynamic flush scheduler to address production issues where the system wasn't flushing data fast enough, causing memory growth and crashes. This issue arises from the existing scheduler handling only a single flush at a time, limiting concurrency and failing to cope with the influx of logs.

- Added configuration options for setting minimum and maximum concurrency levels, maximum batch size, and memory pressure threshold. These parameters ensure that flush operations adjust dynamically based on workload and pressure.
- Implemented `pLimit` to facilitate concurrent flush operations, with adjustments made according to batch queue length and memory pressure.
- Metrics reporting improvements were added to monitor the dynamic behavior of the flush scheduler, aiding in identifying performance issues and optimizing the operation accordingly.

* Implement load shedding for TaskEvent records

This change introduces load shedding mechanisms to manage TaskEvent
records, particularly those of kind LOG, when the system experiences
high volumes and is unable to flush to the database in a timely
manner. The addition aims to prevent overwhelming the system and
ensure critical tasks are prioritized.

- Added configuration options for `loadSheddingThreshold` and
  `loadSheddingEnabled` in multiple modules to activate load shedding.
- Introduced `isDroppableEvent` function to allow specific events to
  be dropped when load shedding is enabled.
- Ensured metrics are updated to reflect dropped events and load
  shedding status, providing visibility into system performance
during high load conditions.
- Updated loggers to inform about load shedding state changes,
  ensuring timely awareness of load management activities.

* Fix undefined 'queuePressure' variable in DynamicFlushScheduler

The 'queuePressure' variable was being used without being defined
in the DynamicFlushScheduler class, causing potential runtime
errors. This commit adds the missing definition and ensures that
the variable is correctly calculated based on the 'totalQueuedItems'
and 'memoryPressureThreshold'.

- Addressed code inconsistencies and improved formatting.
- Defined 'queuePressure' in the 'adjustConcurrency' method
  to prevent potential undefined errors.
- Enhanced readability by maintaining consistent spacing and
  format across the file, contributing to the stability and
  maintainability of the code.
- Adjusted batch size logic based on the newly defined 'queuePressure'
  variable.

* Refactor concurrency adjustment logic in scheduler

The concurrency adjustment logic in the dynamic flush scheduler has been refactored to improve clarity and maintainability. This change moves the calculation of pressure metrics outside of the conditional blocks to ensure they are always determined prior to decision-making.

- The queue pressure and time since last flush calculations were moved up in the code to be independent of the 'backOff' condition.
- This refactor sets up the groundwork for more reliable concurrency scaling and better performance monitoring capabilities. The overall logic of adjusting concurrency based on system pressure metrics remains unchanged.

This adjustment addresses ongoing issues with the scheduler that were not resolved by previous changes.

* Some tweaks
2025-07-26 11:06:56 +01:00
Eric Allam 0d136a3e0c fix runs.retrieve when the payload or output has unstringifiable JSON (#2315) 2025-07-25 12:00:05 +01:00
nicktrn c8165ccd44 security: bump form-data to latest patch versions (CVE-2025-7783) (#2314) 2025-07-24 17:44:19 +01:00
Eric Allam 00cc07dad3 Ignore queryRoute() call aborted errors (#2313) 2025-07-24 16:22:21 +01:00
Eric Allam 71693eb9b3 stop double reporting errors to sentry (#2312) 2025-07-24 16:02:26 +01:00
Eric Allam fb59bfd8f8 sentry: remove node native integration because it breaks (#2311) 2025-07-24 15:37:06 +01:00
Matt Aitken 038ee75c28 Queue "View runs" buttons turn off root only (#2310) 2025-07-24 15:26:56 +01:00
Eric Allam 09d0e80804 Add sentry error reporting (#2309)
* Sentry WIP

* Configure sentry for uploading and releasing during the publish webapp step

* Delete source maps after uploading

* Forward logger.error calls to sentry through Logger.onError

* Couple tweaks to the dockerfile
2025-07-24 15:09:50 +01:00
Saadi Myftija 74808d7400 Pre-merge and pre-sort run batches before sending to clickhouse (#2308)
* Premerge run batch before sending it to clickhouse

* Pre-order batch items before sending to clickhouse in favor of performance

* existing

* Emit event on batch flushes

* When merging batches, keep the last occurrence items with the same version

* Add a couple of tests
2025-07-24 14:17:26 +01:00
Matt Aitken 6791328a94 Queues dashboard update (burst concurrency) (#2293)
* Initial burst changes to queue page

* Added tooltip and changed wording around

* View runs from Queues page

* Fix for ugly Version filter "Current" badge
2025-07-24 10:17:54 +01:00
Eric Allam 3dce1f66f7 docs: beta.25 changelog entry (#2306) 2025-07-24 09:38:24 +01:00
github-actions[bot] 17bb0c0cef Release v4.0.0-v4-beta.25
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 3s
🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker-v4 (push) Has been skipped
🚀 Publish Trigger.dev Docker / units (push) Failing after 4s
* chore: Update version for release (v4-beta)

* Release v4.0.0-v4-beta.25

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Eric Allam <eallam@icloud.com>
2025-07-23 20:22:33 +01:00
Eric Allam ef59a62368 Ability to perform bulk run backfills (#2304)
* Ability to perform bulk run backfills

* Fix tests

* Allow controlling the delay interval when creating the batch
2025-07-23 20:12:35 +01:00
Eric Allam b447a8040f Use sync insert strategy since we are already batching client-side (#2303) 2025-07-23 17:17:13 +01:00
Matt Aitken ed86b4f5db Run repository test fix (#2302)
The normal database needs to be mocked because an import somewhere is 
using it which causes issues with the test container dbs and hanging at 
the end. 

Strategy copied from the trigger test file.
2025-07-23 17:10:20 +01:00
Matt Aitken 9b0eb64035 Fallback from ClickHouse to Postgres (#2300)
* Set the default replication concurrency to 2 for self-hosters

100 was a bit crazy

* Made the run repository an interface, deferring just to CH for now

* Added run repository feature flag, allowing passing a default when getting a flag

* Switch run repository using a feature flag

* Added spans

* Pass the default repository in, so we can try Postgres in the tests

* Fallback to Postgres if ClickHouse errors

* Update feature flags API endpoint
2025-07-23 16:30:30 +01:00
Eric Allam c927fbc8e4 Gracefully shutdown task run processes (#2299)
* Gracefully shutdown task run processes

* better log, thanks coderabbit
2025-07-23 16:11:26 +01:00
Matt Aitken bf4fff9cca The tasks page shows an error rather than breaking the entire page (#2297) 2025-07-23 12:28:06 +01:00
Saadi Myftija ad1672d95f Fail gracefully when runs cannot be loaded (#2296) 2025-07-23 11:21:54 +01:00
Eric Allam 6df7eb2bb5 docs: new release concurrency system and run statuses (#2287)
* WIP new docs for the new release concurrency and run statuses

* Fixed status description

* Add the beta.24 changelog entry
2025-07-22 11:30:28 +01:00
github-actions[bot] 36ced50c04 Release 4.0.0-v4-beta.24
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 0s
🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker-v4 (push) Has been skipped
🚀 Publish Trigger.dev Docker / units (push) Failing after 4s
* chore: Update version for release (v4-beta)

* Release 4.0.0-v4-beta.24

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Eric Allam <eallam@icloud.com>
2025-07-22 10:42:59 +01:00
Eric Allam 684e02c8c8 fix: prevent large root/parent metadata updates from endlessly retrying (#2290)
* fixing metadata WIP

* WIP

* fix: prevent large root/parent metadata updates from endlessly retrying

* Fixed other calls to handleMetadataPacket
2025-07-21 23:04:44 +01:00
Matt Aitken 4b9f33082e AI run filtering (#2285)
* Queue in run table and filtering

* Debounce the filter changes

* Remove console log

* Added machine filtering

* Added version filtering

* Filter by version in the db

* Removed duplicate classes

* Version filtering hasFilters consistency

* Added queues and machines to the bulk action summary

* runs.list filtering for queue and machine

* Fix for machine errors

* Input field now has accessory instead of shortcut

* First experiments with the UI

* Got the fake filtering working

* AI filtering is working pretty well 

* Started working on tool calling

* Tool calling is working

* Styling progress

* Working on the error

* Errors work, improved the styling

* Nice glow effect

* Tweak the darkness of the text field

* Re-ordered the UI, set AI settings to use system prompt and telemetry

* Refactored to make it testable

* Added basic evals

* Better time inputs and evals

* Removed some code comments

* Remove unused useSearchParam change

* Tidy imports

* If no OpenAI API key send json back

* Fix for merge conflict with duplicate query filters

* Another conflict resolved

* Another merge conflict resolved

* Pass the model in, allow changing it
2025-07-21 17:53:43 +01:00
Matt Aitken a90b73c7ca Filter runs by queue, machine, version (#2277)
* Queue in run table and filtering

* Debounce the filter changes

* Remove console log

* Added machine filtering

* Added version filtering

* Filter by version in the db

* Removed duplicate classes

* Version filtering hasFilters consistency

* Added queues and machines to the bulk action summary

* runs.list filtering for queue and machine

* Fix for machine errors
2025-07-21 16:21:46 +01:00
Eric Allam ff018718f7 Added experimental_devProcessCwdInBuildDir config option (#2269)
* Added experimental_devProcessCwdInBuildDir config option

Added experimental_devProcessCwdInBuildDir config option to opt-in to new process.cwd behavior when executing tasks in the dev CLI. Currently process.cwd maps to the "root" of your trigger.dev project (the directory that contains your trigger.config.ts file). Setting experimental_devProcessCwdInBuildDir to true changes process.cwd to instead be the temporary build directory inside of the .trigger directory.

This makes it so the files added via the additionalFiles extension can be read using the same path in dev and deployed tasks.

* Remove claude code reference because it breaks windows tests 😡
2025-07-21 16:06:25 +01:00
Eric Allam 07980f8ebd fix: allow updating run metadata up to 1 hour after completion (#2288) 2025-07-21 16:05:29 +01:00
Eric Allam 8d5c86fea0 v4: simplified release concurrency system and status changes (#2284)
* WIP

* Make release concurrency system extremely simple, everything just releases all the time

* update the deadlock detection to use the new lockedQueueReleaseConcurrencyOnWaitpoint column

* WIP new release concurrency system

* Remove releaseConcurrency and releaseConcurrencyOnWaitpoint

Also removed deadlock detection, and added environment burst concurrency

* Added new DEQUEUED status

Cleaned up the API run statuses, including now detecting new clients and not breaking older clients by adding an API version header to all requests

* Introduce the new "current dequeued concurrency set"

* Remove QUEUED_EXECUTING because we no longer "eagerly" release before checkpointing

* Remove waitpoint test for QUEUED_EXECUTING

* Add isWaiting

* Add changeset

* Use createdAt for ordering realtime runs instead of number

* Clarify the envCurrentDequeuedKey usage

* mock the db.server file to fix the tests

* Updated changset "EXECUTED" -> "EXECUTING"

---------

Co-authored-by: Matt Aitken <matt@mattaitken.com>
2025-07-21 15:45:27 +01:00
Eric Allam a9b0b4f7c9 Add beta.23 changelog entry (#2286) 2025-07-21 14:34:59 +01:00
nicktrn 4dee2c360e fix(helm): allow electric db url var to be set from secret (#2282)
* correctly use db url from secret for electric

* bump app and chart version

* bump electric version
2025-07-21 13:57:23 +01:00
James Ritchie 4565f19d3a Copyable admin area table fields and autofocus search (#2280) 2025-07-18 18:48:56 +01:00
github-actions[bot] 0796df2aa9 Release 4.0.0-v4-beta.23
🚀 Publish Trigger.dev Docker / units (push) Failing after 0s
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 3s
🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker-v4 (push) Has been skipped
* Flatten undefined properties correctly (#2276)

* chore: Update version for release (v4-beta)

* Release 4.0.0-v4-beta.23

---------

Co-authored-by: Eric Allam <eric@trigger.dev>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Matt Aitken <matt@mattaitken.com>
2025-07-18 11:53:43 +01:00
Eric Allam 6bf580ac21 Flatten undefined properties correctly (#2276) 2025-07-17 18:15:57 +01:00
James Ritchie 3ad4b8b32d Add machine to run list (#2275)
* Access machinePreset from the run list presenter

* New icons for machine presets

* New icon + name combo label for the machine preset

* Adds new “Machine” column to the runs list

* Make a separate component for the machine tooltip info

* add machinePreset to the span presenter

* Show the Machine in the Details tab in the Run inspector

* Show an admin only separator

* Fix docs icon in the button

* Small padding tweak

* Move the Machine nearer the costs

* Don't cast the machine preset

* Remove typecast, better to have a bad label if we add a new machine and don't update thos

---------

Co-authored-by: Matt Aitken <matt@mattaitken.com>
2025-07-17 14:26:52 +01:00
Dan f685c6637d Added Mastra + Trigger example (#2278)
* Added mastra agent docs

* Added links in the ai agents section

* Caught typos all thanks to coderabbit. Thanks coderabbit
2025-07-17 13:58:59 +01:00
Dan 1dbcf661f3 Added openai agents sdk for typescript playground docs (#2194)
* Added openai agents sdk typescript playground docs and updated guardrails

* Removed broken link and import

* Typo fix
2025-07-17 13:39:18 +01:00
Matt Aitken 6f6852621d Bundle p-map (#2274) 2025-07-16 16:36:36 +01:00
nicktrn aa3a56a85d fix(docker): apply max old space size env var to webapp (#2273) 2025-07-16 16:36:05 +01:00
Matt Aitken 5e999c8992 Use p-map to speed up bulk actions (#2272) 2025-07-16 15:52:42 +01:00
Matt Aitken 569cecb675 The email counts should come from the completed bulk action… not this iteration (#2271) 2025-07-16 13:46:48 +01:00
Saadi Myftija c0b6c41afc Disable payload editing for run replays with large payloads (#2268)
* Use sexy scrollbars for templates and recent runs popover content

@Nick this is for you!!1

* Fix TabButton component disabled state

* Disable payload tab when replaying runs with large payloads
2025-07-15 21:54:43 +01:00
Matt Aitken 35c7a1ecb0 Bulk action 2 fixes (#2267)
* Fix for filtering by run id

Simplified to have a single filter for runId which accepts an array. This unifies the filtering by multiple runs (used by the waitpoint page) and the individual run filtering. It fixes a bug with bulk actions

* Truncate long bulk action titles
2025-07-15 14:43:19 +01:00
Saadi Myftija a64875a297 fix: regression affecting replayed runs with superjson payloads (#2266)
* Fix regression that broke superjson behavior for the replay flow

* Ignore undefined type metadata when overriding superjson payloads

Addresses https://github.com/triggerdotdev/trigger.dev/issues/1968

* Add a few unit tests for replaceSuperJsonPayload
2025-07-15 12:30:33 +01:00
Matthieu Mordrel 8690878a86 Updated generatedCopy to generatedCopy.text (#2247)
I believe the generatedCopy is the object that the AI SDK returns and we want to pass the text of the object rather in our template literal.
2025-07-14 15:58:55 +01:00
Saadi Myftija b78b3dceb2 feat: link the original run from replayed runs (#2262)
* Add ID of the replayedFrom run to the TaskRun schema

* Propagate the replayedFrom run ID in the replay flow

* Link the replayed run in the run details pane
2025-07-14 15:57:54 +01:00
Matt Aitken 630e9556b0 Bulk actions 2.0 (and switch all run listing to ClickHouse) (#2264)
* useSearchParams has

* useSearchParams has

* useSearchParams has

* Consistent way to get the run filters

* Consistent way to get the run filters

* Consistent way to get the run filters

* Initial work on the new bulk actions

* Initial work on the new bulk actions

* Initial work on the new bulk actions

* WIP actions and filtering

* WIP actions and filtering

* WIP actions and filtering

* Empty filter arrays are set to undefined

* Empty filter arrays are set to undefined

* Empty filter arrays are set to undefined

* WIP prisma schema

Removed extra runtimeEnvironmentId

* WIP prisma schema

Removed extra runtimeEnvironmentId

* WIP prisma schema

Removed extra runtimeEnvironmentId

* Migrations

* Migrations

* Migrations

* BulkActionGroup changed some columns around

* BulkActionGroup changed some columns around

* BulkActionGroup changed some columns around

* New badge variant, removed unused ones

* New badge variant, removed unused ones

* New badge variant, removed unused ones

* Bulk action button

* Bulk action button

* Bulk action button

* Make the next runs page the default now

* Make the next runs page the default now

* Make the next runs page the default now

* Improved the RadioButton style

* Improved the RadioButton style

* Improved the RadioButton style

* Remove the old bulk action bar

* Remove the old bulk action bar

* Remove the old bulk action bar

* More UI progress

* More UI progress

* More UI progress

* Lots of UI changes to the Runs page

* Lots of UI changes to the Runs page

* Lots of UI changes to the Runs page

* Fixed period filter resetting everything

* Fixed period filter resetting everything

* Fixed period filter resetting everything

* Improved the Switch secondary style

* Improved the Switch secondary style

* Improved the Switch secondary style

* Buggy filter fixes

* Buggy filter fixes

* Buggy filter fixes

* Improved the filter display and fixed a bug with search param from object

* Improved the filter display and fixed a bug with search param from object

* Improved the filter display and fixed a bug with search param from object

* Clear button is minimal

* Clear button is minimal

* Clear button is minimal

* Using a presenter now

* Using a presenter now

* Using a presenter now

* Bulk actions are created, but not actually processed (yet)

* Bulk actions are created, but not actually processed (yet)

* Bulk actions are created, but not actually processed (yet)

* Bulk replay/cancel is working

* Bulk replay/cancel is working

* Bulk replay/cancel is working

* Multiple fixes, added bulk column to PG

* Multiple fixes, added bulk column to PG

* Multiple fixes, added bulk column to PG

* Bulk action run filtering working using CH

* Bulk action run filtering working using CH

* Bulk action run filtering working using CH

* Replay setting the bulk id on the runs

* Replay setting the bulk id on the runs

* Replay setting the bulk id on the runs

* Properly cap the time when doing a bulk action

* Properly cap the time when doing a bulk action

* Properly cap the time when doing a bulk action

* If the bulk action isn't recent, add it to the dropdown anyway

* If the bulk action isn't recent, add it to the dropdown anyway

* If the bulk action isn't recent, add it to the dropdown anyway

* Blank version of the bulk actions page

* Blank version of the bulk actions page

* Blank version of the bulk actions page

* Individually selected runs working

* Individually selected runs working

* Individually selected runs working

* Use selected mode if runs are checked

* Use selected mode if runs are checked

* Use selected mode if runs are checked

* Added the modal

* Added the modal

* Added the modal

* Marked the old bulk actions stuff as deprecated

* Marked the old bulk actions stuff as deprecated

* Marked the old bulk actions stuff as deprecated

* Renamed bulk action file

* Renamed bulk action file

* Renamed bulk action file

* Bulk run filter with the name and a default

* Bulk run filter with the name and a default

* Bulk run filter with the name and a default

* WIP on bulk actions page

* WIP on bulk actions page

* WIP on bulk actions page

* Updated panel, added new truncated id component

* Updated panel, added new truncated id component

* Updated panel, added new truncated id component

* Style improvements to the radio buttons

* Style improvements to the radio buttons

* Style improvements to the radio buttons

* Added an option action completion email

* Added an option action completion email

* Added an option action completion email

* Adds a blank state for the bulk actions page

* Adds a blank state for the bulk actions page

* Adds a blank state for the bulk actions page

* Nicer completed email

* Nicer completed email

* Nicer completed email

* Don't open the bulk action panel if there are no runs

* Don't open the bulk action panel if there are no runs

* Don't open the bulk action panel if there are no runs

* Runs blank state and bulk action accordion

* Runs blank state and bulk action accordion

* Runs blank state and bulk action accordion

* Updates secondary/small switch style

* Updates secondary/small switch style

* Updates secondary/small switch style

* Pagination buttons no longer split in twain (WIP)

* Pagination buttons no longer split in twain (WIP)

* Pagination buttons no longer split in twain (WIP)

* Aborting working

* Aborting working

* Aborting working

* Bulk action live reloading

* Bulk action live reloading

* Bulk action live reloading

* ListPagination works correctly in all states

* ListPagination works correctly in all states

* ListPagination works correctly in all states

* Run page, show friendlyId instead of number

* Run page, show friendlyId instead of number

* Run page, show friendlyId instead of number

* Bulk action help open by default if you have none

* Bulk action help open by default if you have none

* Bulk action help open by default if you have none

* Extra status filtering step because of replication delay

* Extra status filtering step because of replication delay

* Extra status filtering step because of replication delay

* Wider bulk action onboarding

* Wider bulk action onboarding

* Wider bulk action onboarding

* More sensible widths on the bulk action side panel

* More sensible widths on the bulk action side panel

* More sensible widths on the bulk action side panel

* Border color tweak to the RadioButton

* Border color tweak to the RadioButton

* Border color tweak to the RadioButton

* Improved the accordion component hover states

* Improved the accordion component hover states

* Improved the accordion component hover states

* Updates the bulk action blank state images to the latest UI

* Updates the bulk action blank state images to the latest UI

* Updates the bulk action blank state images to the latest UI

* Added R and C shortcuts back in

* Added R and C shortcuts back in

* Added R and C shortcuts back in

* Fix for selecting a single run

* Fix for selecting a single run

* Fix for selecting a single run

* Improved exit icon, added shortcut to modal

* Improved exit icon, added shortcut to modal

* Improved exit icon, added shortcut to modal

* Tidy imports

* Tidy imports

* Tidy imports

* Tidy imports

* Tidy imports

* Tidy imports

* Tidy imports

* Tidy imports

* Tidy imports

* Tidy imports

* Fix for grid layout when 1 page of bulk actions visible

* Fix for grid layout when 1 page of bulk actions visible

* Fix for grid layout when 1 page of bulk actions visible

* Removed the ... on the abort button

* Removed the ... on the abort button

* Removed the ... on the abort button

* Removed the ... on the abort button

* Animate the progress bar

* Set TZ="UTC" in the env example

* Filter summary in the bulk inspector

* Improves the pagination styling

* Improves the pagination styling

* Delete old bulk action routes

* Removed old Postgres RunListPresenter

* Retry any replication error where the message contains "timeout"

* Increase wait to make test less flaky

* The test was using run id instead of friendly id

* Safer array access

* Remove error log if there's a bad status

* Nicer frontend type safety with the bulk action and mode

* Switched a log to a debug log

* Retry replication unless the error is a known non-retry error

Flip the strategy to retry by default

* Make ClickHouse required

* Backfill run replication admin API endpoint

* Set a CLICKHOUSE_URL for unit tests

---------

Co-authored-by: James Ritchie <james@trigger.dev>
2025-07-14 15:50:01 +01:00
Eric Allam 6db1b71202 fix: prevent stuck batchTriggerAndWait by finalizing runs when they are cancelled (#2265) 2025-07-14 14:29:14 +01:00
Eric Allam 70b4b1257e fix: Logging large objects is now much more performant and uses less memory (#2263)
* Logging large objects performance fixes

* Add changeset

* docs: new otel attribute count defaults

* Scope the switch/case statement with braces

* Add tests for nested map and sets

---------

Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
2025-07-14 13:16:04 +01:00
Saadi Myftija 87c21ab50a feat: enable overriding run options when replaying (#2260)
* Rearrange the layout of the replay modal

* Add run options to the replay modal

* Handle payload and metadata correctly in the shared json editor

* Apply run options in replays

* Fix clear button issue in json editor

Memoization of the clear function caused some problems. Removing it should not cause performance issues.

* Update replay modal hint

* Move machine and version fields to the top for visibility

* Use the same field ordering in the test page

* Reload queues and versions on env override

* Adapt json editor to fill full height

* Clean up a few excessive ternaries

* Avoid ui jump on env selection

* Switch to sexy scrollbars for scheduled tasks in the test page
2025-07-11 16:54:20 +02:00
Matt Aitken 4c635f72a2 Fixed queues SSE live reloading (#2259) 2025-07-11 12:53:28 +01:00
Eric Allam 46dad7dc76 fix: prevent processKeepAlive OOM error process reuse (#2261)
* Improve TaskRunProcess health detection so we don't try and reuse an unhealthy process

This was happening after the process was killed internally, like by an OOM error

* Add changeset
2025-07-11 10:50:05 +01:00
nicktrn 4fdf23b38b fix(deploy): fail in CI with descriptive message and instructions if no access token set (#2258)
* don't try interactive login in ci, link to docs

* add ci note to cli deploy docs

* add ci note to github actions

* add changeset

* make cron test less strict
2025-07-10 17:28:12 +01:00
nicktrn 0c71dc74df feat: add node 22 and bun runtime support with version display (#2254)
* update node-22 image

* update bun image

* disable io_uring

* fix fallback bun path

* prevent duplicate warnings

* add runtime and version to deployments

* runtime icons

* fallback to nodejs

* prevent empty table cell menu

* log if local build on deploy

* pass io_uring env var to child

* denormalize runtime and version, display on run details

* add changesets

* disable pr checks for changeset commits..

* add runtime data to deployed bg workers
2025-07-10 16:43:10 +01:00
Eric Allam 6ff0b923e1 New internal idempotency implementation for trigger and batch trigger (#2256)
* Introduce request idempotency to prevent duplicate triggers

* Implement request idempotency on trigger

* Use x-trigger-request-idempotency-key header instead

Plus cleanup shared logic in the routes

* Add changeset

* Oops, lets not hardcode a 408

* A couple of improvements
2025-07-10 16:42:32 +01:00
nicktrn 1a64013825 fix(helm): external config and secret support (#2245)
* fix postgres and redis external config, incl redis tls

* external secret support for postgres, redis, clickhouse, plus fixes

* s3: full existing secret support, external fixes

* bump chart version

* add subchart links

* update external config docs

* tidy up validation and helper
2025-07-10 10:08:00 +02:00
nicktrn f456f68f88 fix(docs): force latest buildx version in sample gh action (#2252) 2025-07-09 19:41:02 +02:00
Saadi Myftija 28b6be2491 feat: introduce run templates for reusing test run configs (#2253)
* Add new prisma model for task run templates

* Create run templates in a new service

* Add modal to create run templates in the test page

* Show templates list and apply values when selected

* Hide template creation time in the dropdown list, only show date

* Enable deleting run templates

* Show success toast on template creation

* Validate template label length

* Improve the template creation success indicator

* Use formAction consistently to differentiate submissions

* Type formAction for better editor support

* Prettify run template payload and metadata

* Add triggerSource, concurrencyKey and ttl to run templates
2025-07-09 19:09:25 +02:00
Saadi Myftija dbcf632b4e fix: add missing field descriptions in the test page (#2251)
* Switch to sexy scrollbars in the options pane

* Add missing field descriptions

* Add a run options link to the docs

* Make the copy for the field descriptions more concise

* Fix tooltip issue in the Select component
2025-07-09 16:25:06 +02:00
Saadi Myftija 3cfde48bfd feat: expose all run options in the test run page (#2227)
* Implement a new primitive UI component for picking durations

* Implement a new component to input run tags

* Expose all run options in the test run page

* Add subtle animations when adding/removing run tags in the test page

* Add a new resource endpoint for fetching queues

* Fetch usable queues for the selected task

* Fix width display issue in the select component

* Enable locking a run to a version from the test page

* Disable entering max attemps <0

* Validate tags

* Add recent runs popover

* Only show latest version for development environments

* Update run options when selecting a recent run

* Rearrange the test page layout

* Add subtle animation to the duration picker segments on focus

* Improve queue selection dropdown styling

* Fix disabled state issue for the SelectTrigger component

* Disable version selection field for dev envs

* Add usage hints next to the run option fields

* Add machine preset to the run options list

* Allow arbitrary queue inputs for v1 engine runs

* Show truncated run ID instead of run numbers for recent runs

Run numbers will soon get deprecated due to contention issues

* Fix duplicate queue issue

* Extract common elements across the standard and scheduled test task forms

* Apply values from recent runs to scheduled tasks too

* Add additional run options for scheduled tasks

* Use a slightly smaller font size for run option labels

* Disallow commas in the run tag input field

* Switch to a custom icon for recent runs button

* Flatten the load function test task result object

* Avoid redefining machine presets, use zod schema instead

* Fix ClockRotateLeftIcon jsx issues

* Remove recent runs button tooltip as it causes nesting errors

* Adjust the page layout to make it clear which task is currently selected

* Inline the tab group with the copy/clear buttons
2025-07-09 10:40:10 +01:00
Eric Allam ba2e0cc1c6 Prevent circular references from breaking run logs in task outputs (#2249) 2025-07-09 10:28:12 +01:00
Eric Allam b113e5a99c Fix non-MFA login (#2246)
* Fix non-MFA login

* fixed typecheck errors
2025-07-08 14:59:31 +01:00
Eric Allam b119a52e08 Implement MFA (#2244)
* Adds a new route for logging in with mfa

* New path for security page

* Adds “Security” link to account side menu

* Update the Switch component to allow label positions left and right

* Optionally hide the Close button in the Dialog title bar

* Installs `qrcode` react package for generating QR codes.

* CopyButton component now takes children

* New Security route for setting up MFA

* Adds new OTP package for the chadcn InputOTP component

* Adds new InputOTP chadcn component

* Adds InputOTP chadcn component to the MFA login screen

* InputOTP component supports variant styles

* Improvements to form handling

* Show a confirmation modal before you can disable MFA

* Revert redirect back to the dashboard for now

* Implement MFA enabling and disabling

* Refactor and cleanup mfa management code

* More cleanup

* Handle errors in the management action

* Implement mfa login flow

* recovery code input should be password

* Implement rate limiting on the mfa validation endpoint

* Better error ux

* Implement mfa emails and apply James' updates

* Use latest @better-auth/utils

* Improvements via CodeRabbit review

---------

Co-authored-by: James Ritchie <james@trigger.dev>
2025-07-08 13:44:47 +01:00
nicktrn e02c90a4ea Update self hosting images to beta.22 (#2241)
* docker: set new cli tag env var

* helm: update app and chart versions
2025-07-08 09:33:27 +01:00
Eric Allam 493ca11c43 Add ability to disable the prisma query in the healthcheck endpoint (#2242) 2025-07-07 16:41:55 +01:00
Eric Allam b36d81a622 Add v4 changelog to v4 docs (#2240)
* Add v4-beta changelog docs

* Add beta.22 changelog entry
2025-07-06 11:38:10 +01:00
Eric Allam 275ce9f290 Release 4.0.0-v4-beta.22 (#2238)
🚀 Publish Trigger.dev Docker / units (push) Failing after 0s
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 4s
🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker-v4 (push) Has been skipped
2025-07-04 16:15:15 +01:00
github-actions[bot] 5cff5c45e9 chore: Update version for release (v4-beta) (#2159)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2025-07-04 16:13:20 +01:00
Eric Allam 47e583c19d Properly scope the batch list to the selected environment (#2236)
Prevents showing batch runs from environments that are not in the selected environment
2025-07-04 14:41:10 +01:00
nicktrn 1019c9cdea Switch trigger Helm chart RBAC from cluster to namespace scope and bump version (#2232)
* replace supervisor cluster role with ns-scoped role

* fix cross-ns binding

* bump chart version
2025-07-04 13:54:25 +01:00
Eric Allam 469a8df0cf Prevent redis worker from enqueuing pending jobs (#2235) 2025-07-04 13:53:32 +01:00
Eric Allam 922c7cb3a6 Increase batch trigger processing job visibility timeout (#2234) 2025-07-04 13:43:26 +01:00
nicktrn 0fcb9705ae Document new otel limits (#2196)
* update webapp env vars

* update limits
2025-07-04 13:41:54 +01:00
nicktrn 74fdfa93c5 Show project details when running the whoami command (#2231)
* add project details to whoami

* findFirst

* skip project check if no orgs found

* add changeset
2025-07-04 13:40:41 +01:00
Eric Allam bdaa2ed15a Move batch trigger processing jobs to a dedicated redis worker (#2233) 2025-07-04 12:16:05 +01:00
nicktrn afc9a6ade1 Add ECR support for remote builds (#2224)
* create repo if doesn't exist

* fresh auth token for each deploy

* optional assume role

* log when machine overrides enabled

* make test repo namespace configurable

* assume role fix and env var changes

* improve ecr check

* improve tag parsing

* tag parsing tests

* track if repo created and fix test

* missing tryCatch for sts call
2025-07-03 14:25:30 +01:00
Eric Allam 7bb7e7aedc When sharding, use the where clause in the shard key to distribute requests more evenly (#2229) 2025-07-03 14:21:58 +01:00
Eric Allam e90235b317 fix processKeepAlive disabled performance regression (#2228)
* processKeepAlive disabled performance fix

When processKeepAlive was disabled, the eagerly created TaskRunProcess being discarded and a whole new TaskRunProcess created at execution time.

* Don't use a TaskRunProcess if it's being killed, fixes retryImmediately
2025-07-03 14:21:49 +01:00
James Ritchie b7134dd769 Docs: Adds 2 references to the node.js versions we use for v3 and v4 (#2118)
* Adds 2 snippets to reference node.js version for v3 and v4

* Updated the node versions we support

* Remove specific node version from title
2025-07-02 13:08:49 +01:00
nicktrn 6fea802505 helm: set default webapp container and add extraVolumes (#2222)
* set default webapp container

* bump version

* add webapp extraVolumes and extraVolumeMounts

* bump version
2025-07-02 13:03:02 +01:00
Eric Allam 09ef7625ff Merge pull request #2219 from triggerdotdev/metadata-serialization
Serialize metadata to prevent invalid data from breaking run completions
2025-07-02 12:49:08 +01:00
Eric Allam 5cf37e9f8a Merge pull request #2189 from triggerdotdev/increased-and-configurable-attribute-limits
Increase otel attribute limits and make them configurable
2025-07-02 12:48:38 +01:00
Eric Allam e7b6c2fab6 Merge pull request #2183 from triggerdotdev/process-keep-alive
v4: Experimental process keep alive
2025-07-02 12:39:43 +01:00
nicktrn d619e79f74 helm: increase default postgres resource preset (#2221)
* switch to small preset for postgres

* lint

* bump version
2025-07-02 11:57:14 +01:00
Eric Allam 38a94b3b70 Add changeset 2025-07-02 11:30:13 +01:00
Eric Allam 0c2af6d87c Increase otel attribute limits and make them configurable 2025-07-02 11:30:13 +01:00
Eric Allam 0deb2c640d remove change from process-keep-alive 2025-07-02 11:30:13 +01:00
Eric Allam 21c70ad673 Serialize metadata to prevent invalid data from breaking run completions 2025-07-02 11:30:13 +01:00
Eric Allam 0793c2b0f4 address some nick comments in the review 2025-07-02 11:30:13 +01:00
Eric Allam 8dc90d61b4 increase run engine test timeouts 2025-07-02 11:30:13 +01:00
Eric Allam 57f3a89a86 Fix timeline span events tests 2025-07-02 11:30:13 +01:00
Eric Allam ceb737e9f1 Fix broken webapp test 2025-07-02 11:30:13 +01:00
Eric Allam 632d6b91e9 Fix taskSlug extraction 2025-07-02 11:30:13 +01:00
Eric Allam b548561220 Remvoe the sometimes incorrect import file name 2025-07-02 11:30:13 +01:00
Eric Allam aecee4d35f Don't reset task lifecycle hooks between executions 2025-07-02 11:30:13 +01:00
Eric Allam 9ab590bead Better handle snapshot changes with the new persistent task run process,
Also don’t re-import the task file if the task is already in the resourceCatalog
2025-07-02 11:30:13 +01:00
Eric Allam 2cc1453f25 Always set the cold/warm variant on attempt spans, even in dev 2025-07-02 11:30:13 +01:00
Eric Allam 82f3db6825 Only bootstrap on the cold start execution, and record how many executions in the attempt span 2025-07-02 11:30:13 +01:00
Eric Allam f0bc4ad2bd Don't delete env vars with falsy values 2025-07-02 11:30:13 +01:00
Eric Allam 4d15c8f7cf Better reset in the prod usage manager 2025-07-02 11:30:13 +01:00
Eric Allam 7f3b1c1735 Ensure the usage interval is cleared when resetting 2025-07-02 11:30:13 +01:00
Eric Allam 707bc09dac Make sure maxExecutionsPerProcess is a positive integer 2025-07-02 11:30:13 +01:00
Eric Allam a0f2c982f9 Make sure to clear existing metadata flush timer when resetting 2025-07-02 11:30:13 +01:00
Eric Allam bffd18a6b8 Properly reset listener arrays by setting length to 0 2025-07-02 11:30:13 +01:00
Eric Allam 2018b41d22 TRIGGER -> TRIGGERDOTDEV 2025-07-02 11:30:13 +01:00
Eric Allam c41d63497b Handle immediate retries better 2025-07-02 11:30:13 +01:00
Eric Allam 56146afcb6 TaskRunProcessProvider cleanup is async 2025-07-02 11:30:13 +01:00
Eric Allam 020565c009 Handle dev process pool shutdown errors 2025-07-02 11:30:13 +01:00
Eric Allam 58d56f8381 Use hostname instead of host 2025-07-02 11:30:13 +01:00
Eric Allam 5b49b420cb Add changeset 2025-07-02 11:30:13 +01:00
Eric Allam 3fb68b784a couple of tweaks for testing stuff 2025-07-02 11:30:13 +01:00
Eric Allam a414e65efe handle env vars that have been removed between executions 2025-07-02 11:30:13 +01:00
Eric Allam 38fe854559 Make refreshing a task run process work correctly 2025-07-02 11:30:13 +01:00
Eric Allam c24f3a56cc Reset deployed run worker and also get the warm start stuff to work properly 2025-07-02 11:30:13 +01:00
Eric Allam 8798352be3 Super-warm starts in deployed tasks proof 2025-07-02 11:30:13 +01:00
Eric Allam 1a1e70ae3d WIP deployed processKeepAlive 2025-07-02 11:30:13 +01:00
Eric Allam 19bdf3b6ea scope the pool by background version, allow configuring pool params in dev/prod 2025-07-02 11:30:13 +01:00
Eric Allam 427c450391 Fixed metrics, reusing a process counts as a warm start 2025-07-02 11:30:13 +01:00
Eric Allam 9374aea401 Split resource and span attributes 2025-07-02 11:30:13 +01:00
Eric Allam fd3db5f77b Reset execution environment between runs 2025-07-02 11:30:13 +01:00
Eric Allam e2049fcfbf Initial dev process keep alive. moved resource attributes to span attributes because resource attributes are global and immutable 2025-07-02 11:30:13 +01:00
Eric Allam 1be4fcbb82 Fixes the jitter option for the scanConcurrencySets cron job (#2220) 2025-07-02 11:29:54 +01:00
Eric Allam 124435e5e7 Eliminate race conditions in continueRunIfUnblocked to prevent stuck runs (#2217)
* Eliminate race conditions in continueRunIfUnblocked to prevent stuck runs

* Use assertNever for typecheck friendliness
2025-07-01 16:34:12 +01:00
Matt Aitken 4a79b3b35b Fixed retrying on run replication inserts on connection errors (#2216) 2025-07-01 13:35:13 +01:00
Boon Kai 0da6a19987 docs: Update triggering.mdx import for triggering from inside another task (#2215)
When triggering from inside another task, `runs` and `batch` should be imported from `@trigger.dev/sdk/v3`
2025-07-01 08:03:02 +01:00
Eric Allam 08d84eb596 Fix v4 concurrency sweeper flaky test (#2213) 2025-06-30 16:37:52 +01:00
Matt Aitken f38d35e9f6 ClickHouse replication improvements (retrying, strip bad unicode chars) (#2205)
* Add retry logic for insert operations

Add a generic retry mechanism for task run and payload inserts to handle
transient connection errors. The new #insertWithRetry method retries up to
three times with exponential backoff and jitter on retryable connection
errors such as connection resets or timeouts. Errors are logged and
recorded in tracing spans to improve observability and robustness of the
replication service.

* Replication settings are configurable

* Log out the runIds for failed batches

* Detecting bad JSON in run replication and ignoring it

* Reproduced split unicode error

* Move output file

* Massively improved the performance

* Minor performance improvements

* Unskip tests

* Remove unused test in CH package

* Fix for the ClickHouse UI explorer

* RunReplication keepAlive defaults to false

* Add concurrency_key and bulk_action_group_ids to ClickHouse task runs

* ClickHouse package doesn't need to be built anymore for the webapp

* Set the concurrency_key from the run replication service
2025-06-30 16:32:02 +01:00
Eric Allam 1b4dabb8ba v4: current concurrency sweeper (#2206)
* v4: current concurrency sweeper

* Fix webapp tests

* Ensure only a single instance performs concurrency sweeping by using redis-worker cron jobs

* Improved the mark phase

* Ensure cron jobs get rescheduled even if the handler throws an error

* Better property names
2025-06-30 16:22:19 +01:00
nicktrn aa24a9aae6 Fix external registry config and other helm tweaks (#2212)
* fix both internal and external registry host

* add next steps to self-hosting overview

* separate into webapp and registry ingress

* fix external registry config check

* fix localhost registry warning

* bump chart version
2025-06-30 14:21:00 +01:00
nicktrn dc568e7f3e Self-hosting: improved worker and bucket bootstrap (#2209) 2025-06-29 10:42:52 +01:00
nicktrn fa6675eb00 Helm chart improvements (#2207)
* add helm deps

* configurablecli tag for setup commands

* enabled -> deploy

* extraEnv -> extraEnvVars

* update docs

* minio -> s3

* improve persistence config

* set cli tag to v4-beta

* chart and image bump

* fix tests

* lock chart

* workflow hotfix

* quote for niels

* make probes configurable

* fix webapp and supervisor templates

* bump chart
2025-06-28 12:59:29 +01:00
Eric Allam 8a86e7b41d Remove deprecated schedule trigger jobs when registering with the schedule engine (#2204)
* Remove deprecated schedule trigger jobs when registering with the schedule engine

* Use a tryCatch just to be absolutely sure
2025-06-27 12:32:02 +01:00
nicktrn 119bfc0068 Fix update command version detection (#2199)
* fix root package.json resolution

* add changeset
2025-06-27 10:06:36 +01:00
nicktrn 7a6d9dc225 v4 self-hosting docs updates (#2202)
* quote version strings

* Update architecture diagram to include kubernetes

---------

Co-authored-by: James Ritchie <james@trigger.dev>
2025-06-26 16:53:10 +01:00
Eric Allam 0f3179f493 Provide realtime skipColumns option via untamperable public access tokens (#2201)
* Provide realtime skipColumns option via untamperable public access tokens

* Fixed realtime client tests

* Create red-rings-marry.md
2025-06-26 15:13:26 +01:00
Eric Allam b87e1c4630 Improve clickhouse client debug logging (#2197)
* Improve clickhouse client debug logging

* Stop hardcoding ScheduleEngineWorker log level

* Add DLQ debug log message
2025-06-26 15:12:54 +01:00
Eric Allam 14058d557f fix: queues no longer unpaused on deploy (#2198) 2025-06-26 15:08:52 +01:00
nicktrn 6535cf918b v4 helm chart and docs (#2195)
* v4 helm chart

* ignore packaged chart archives

* chart publish prep

* add helm chart release workflow

* enable draft gh release

* stateful services should be statefulsets

* expose internal otel config

* podAnnotations for all services

* postgresql.extraArgs

* postgresql -> postgres

* add clickhouse vars as extraEnv examples

* support external secrets

* improve clickhouse config

improve clickhouse config

* disable internal otel config by default

* allow nil internal config

* fix gh release permission

* fix validation step for ci

* secrets.enabled

* simplify secret keys

* bump chart version

* fix contents permission

* remove redundant print steps

* remove helm release heading

* optimise clickhouse config

* support wait-for-it.sh

* fix ch overrides

* add low-resource ch overrides to compose setup

* supervisor bootstrap config

* disable otel trace logging

* retain shared volume

* enable clickhouse for compose setup

* disable trace logging

* bump chart

* add extraManifests

* telemetry config

* add repo namespace config

* scope cluster role to namespace to support multiple installs

* external s3 config

* values tweaks

* helm docs

* bump chart

* set contents read permission

Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>

* add object store root credentials warning

* run init containers as uid 1000

* pass ch user creds with explicit flag

* improve ingress anotation handling

* remove duplicate colume claim block

* trim htpasswd auth

* lock curl images used for tests

* add v4-beta package version warning

* make schema and sslmode configurable

* package tag info -> warning

* improve secrets config example

* update docs

* bump version

* bump webapp reqs to account for ch

* make ch creds more easily configurable

* publish ch http port

* example point at latest beta chart

* small docs tweaks

---------

Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
2025-06-25 23:57:51 +01:00
Eric Allam aaf40c3be3 Fix "Duplicate path found during parsing JSON object" clickhouse error (#2193) 2025-06-24 14:34:32 +01:00
James Ritchie 35dedee1eb Update number of allowed tags (#2187) 2025-06-24 10:38:59 +01:00
Eric Allam 485f71c8d4 Improve schedule engine performance (#2191)
* Improve schedule engine performance

* Convert an info log to a debug log
2025-06-23 16:55:18 +01:00
nicktrn ec2175fec8 Some docs tweaks (#2190)
* add v4 tag to playwright extension

* add self-hosting architecture section

* add v4-beta tag warning
2025-06-23 10:25:07 +01:00
Eric Allam fb9abe5289 fix: Incorrect/missing logs on new schedule engine runs (#2185)
* Fix incorrect logs on new schedule engine triggered taskss

Also added the ability to recover schedules in the schedule engine via an Admin API endpoint in the new schedule engine

* Fixed schedule recovery failing test
2025-06-19 15:37:50 +01:00
Eric Allam f2db1b8aa5 Create dedicated alerts worker (#2184)
* Add a new alerts worker that just handle alerts

* fallback to graphile worker when a tx is required
2025-06-19 11:54:49 +01:00
nicktrn 3c93783e8b remove nodetype label requirement (#2182) 2025-06-18 11:34:35 +01:00
nicktrn 55f41a2168 Ensure webapp container does not fetch pnpm at runtime (#2181)
* install pnpm during build

* install pnpm for node user as well
2025-06-18 11:31:36 +01:00
Matt Aitken 4ca6506655 Delay reschedule fix (v3 & v4) (#2180)
* Fix for v3 rescheduling bug…

* v4 rescheduling working
2025-06-17 18:22:15 +01:00
Matt Aitken 9f9e74b0b6 Fix hydration error (#2179)
* Remove unused hook

* The Impersonation button was causing DOM hydration issues
2025-06-17 11:17:17 +01:00
Eric Allam ea6f66bdfc Improved schedule engine (#2174) 2025-06-17 06:48:05 +01:00
Eric Allam a42c9acf44 v4: Improved run locking (#2173) 2025-06-17 06:45:59 +01:00
nicktrn f7f8bc28c5 Docker self-hosting improvements (#2177)
* shallow clone

* fix image tag locking example

* electric should use db url env var

* add htpasswd note

* use local driver with log rotation by default

* make worker urls more easily configurable

* configure dev otel endpoint via .env

* increase min recommended worker specs

* move worker url section
2025-06-16 22:22:02 +01:00
Matt Aitken 5f6fd8e235 Tasks page didn't include Delayed runs (#2176)
* Delayed runs weren’t shown in the Tasks page mini graph

* The Tasks page Queued count didn’t include Delayed runs (since ClickHouse switchover)
2025-06-16 14:14:09 +01:00
James Ritchie 33e1008d83 Adds a docker error entry to common problems docs page (#2172) 2025-06-14 11:11:24 +01:00
Matt Aitken a0815c8e3e Clickhouse performance improvements (#2175)
* Runs filter by org id and add created at to ordering

* CopyableText can accept an alternative value for copying

* The runs table now shows the ID instead of run number

* Paginating back/forwards fix

* The task stats need org id and project id too
2025-06-14 11:10:28 +01:00
James Ritchie 8a5e7f34b1 Added more limits and information to the limits page (#2171) 2025-06-13 10:32:44 +01:00
Matt Aitken c3dca5835d Fix for the Kapa widget not working with SSR (#2170)
* Move all the logic into a single component, preparing for client only

* Use ClientOnly

* Fix for search param not working
2025-06-12 22:17:22 +01:00
James Ritchie 20a8e7c6c1 Updated kapa sdk to 0.1.3 (#2169) 2025-06-12 16:21:34 +01:00
Eric Allam cc5514e57b Add status-first TaskRun index to speed up uncommon status filters (#2168) 2025-06-12 15:02:40 +01:00
Eric Allam ab432a2d7d Fix realtime re-subscribing stale data issue (#2162)
* Fix realtime re-subscribing stale data issue

Fixes an issue with realtime when re-subscribing to a run, that would temporarily display stale data and the changes. Now when re-subscribing to a run only the latest changes will be vended

* removed logs
2025-06-12 14:58:01 +01:00
James Ritchie e12c82b444 Use the Kapa AI SDK instead of the Kapa AI widget (#2113)
* Install the kapa sdk

* WIP using the SDK for the Kapa Ask AI widget

* Removes old kapa from root

* Now rendering everything inside the dialog component

* Fixes min-height of dialog content

* Remove kapa from root

* prevents kapa using reCaptcha

* Adds more functionailty with temporary UI placement for now

* Reset conversation button

* Adds a new sparkle list icon

* Adds some example questions as a blank state

* Animate in the example questions

* use “marked” package to render markdown

* Improve some animations

* Submit a question from the URL param

* adds custom scroll bar styling

* fixes modal to correct height after re-opening it

* Add button to stop generating answer mid-stream

* Adds buttons states to show submitting, generating, submittable

* Adds a helpfull sentence in the blank state

* Show a message if the chat returns an error

* Adds reset chat and feedback buttons to the bottom of an answer

* Makes sure you can give feedback in the different states of chat

* Adds a suble background to the dialog

* Fix a button inside button error

* Improve the shortcut esc key on dialog and sheet component

* Fix classname error

* organize imports

* Use our custom focus-visible

* Move the Tooltip for the button into the AskAI component

* Improved error message

* Organize imports

* Animated the modal gradient

* Small layout improvements

* Adds most asked questions from Kapa

* border glow tweak

* AskAI component is now a hook that can take a question

* remove kapa script

* Add a delay before the modal opens when usign the URL params

* Remove old component

* Update to the latest Kapa version

* Rephrased error message

* Use correct types for conversation

* Fixed types for addFeedback

* Adds DOMPurify package

* removed unused const

* Removed unnecessary platform specification

* Reset the timeout when the ai panel pops up

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>

* Fix for coderabbit bad commit

* Clean up imports

---------

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
2025-06-12 14:09:03 +01:00
Dan 808c299008 Added deep research example (#2166)
* Added docs

* Improved docs and added video

* Added upgrade badge
2025-06-11 16:19:25 +01:00
Eric Allam 498b9a21af Improved TaskRun environment indexes (#2164)
### PR: Optimize **TaskRun** indexes for hot-path queries

**What changed**

| Object                          | Type                                                            | Purpose                                                                                                    |
| ------------------------------- | --------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- |
| `taskrun_runtime_id_desc_idx`   | **BTREE** `(runtimeEnvironmentId, id DESC) INCLUDE (createdAt)` | Eliminates explicit sort for the “latest task runs” query (`ORDER BY id DESC`) while remaining index-only. |
| `taskrun_runtime_createdat_idx` | **BTREE** `(runtimeEnvironmentId, createdAt DESC) INCLUDE (id)` | Accelerates the filter-only path that scans by `createdAt >= …` without any ordering requirement.          |
| `taskrun_createdat_brin`        | **BRIN** on `createdAt` (`pages_per_range = 128`)               | Lets the planner skip whole blocks older than the time window for both queries at < 100 MB cost.           |
| *(cleanup)*                     | **DROP** `TaskRun_runtimeEnvironmentId_createdAt_id_idx`        | Retires the 3-column index once the new ones are built.                                                    |

**Key details**

* All indexes created **CONCURRENTLY** to avoid write blocking.
* `fillfactor = 90` on b-trees for balanced space vs. future growth.
* Net disk usage drops **≈ 15–20 GB** while each query now gets a purpose-built access path.

**Why**

* Remove planner Sort nodes for the top-N “latest runs” view.
* Speed up environment-filtered range scans.
* Shrink index bloat and improve cache efficiency.
2025-06-11 09:24:11 +01:00
Eric Allam 48e1e7250b v4: fix stuck batch issue when processing happens before transaction commits (#2163)
* v4: fix stuck batch issue when processing happens before transaction commits

* Fix flaky test
2025-06-10 17:02:34 +01:00
Eric Allam 3f2b748a80 Delete the secure param before passing it to the ClickHouse client (#2161) 2025-06-10 13:55:19 +01:00
Eric Allam a060ceef0b Fix clickhouse migrations by adding the secure=true query param (#2160) 2025-06-10 13:45:24 +01:00
Eric Allam e6fb32194e Add environment filter to declarative schedules (#2148) 2025-06-10 12:11:11 +01:00
Eric Allam b38405cb88 Realtime and task run performance improvements (#2158)
* Add createdAt filter to realtime subscribing with tags

* Filter realtime colums and expose ability to skip some columns

* Add sharding support for electric

* Use unkey cache for the created at filter caching

* Remove 2 unused indexes on TaskRun

* Run list now filters by a single runtime environment

* Remove project ID indexes

* Use clickhouse in task list aggregation queries instead of pg (keep pg for self-hosters)

* WIP clickhouse powered runs list
stuff

* Improve the query to get the latest tasks for the task list presenter

* Update the usage task list to use clickhouse

* Implement next runs list powered by clickhouse

* Add new index for TaskRun for the runs list, by environment ID

* Add runTags gin index

* Handle possibly malicious inputs

* Ignore claude settings

* Better handling not finding an environment on the schedule page

* Use ms since epoch in test, not seconds

* Remove unused function

* Fix test

* Use an env var for the realtime maximum createdAt filter duration (defaults to 1 day)

* Fixed the query builder to correct the group by / order by order

* Make sure runs.list still works

* Create small-birds-arrive.md
2025-06-10 12:11:01 +01:00
James Ritchie e617c14456 Adds a troubleshooting entry for react.email not being bundled correctly (#2117) 2025-06-09 16:43:32 +01:00
James Ritchie 15fd731f3d Fixes embeded video width (#2121) 2025-06-09 16:40:59 +01:00
Eric Allam 4c7742c258 Enable docs AI contextual menu (#2157) 2025-06-09 10:04:56 +01:00
nicktrn e72820c080 Docs improvements for v4 self hosting (#2156)
* add switch command

* couple of changes
2025-06-07 19:51:47 +01:00
nicktrn 7a34c1102b Feat: v4 self-hosting (#2155)
* self-hosting stuff goes in /hosting

* add v4 tags

* add main compose file

* draft overview

* add webapp env vars

* overview tweaks

* add supervisor env vars

* move old docker guide

* new sidebar structure

* update github actions docs

* docker draft

* use env vars for s3 creds

* this might just work

* split into multiple files

* update guide

* document machine overrides

* split legacy docs into different section

* some fixes

* some tweaks

* add login and init instructions

* don't cursorignore .env.example
2025-06-07 00:57:47 +01:00
Eric Allam 2b3d54aff5 Make removing the httpEndpointEnvironmentId column migration correctly idempotent (#2154) 2025-06-06 11:52:14 +01:00
github-actions[bot] bb1f0a711a Release 4.0.0-v4-beta.21 (#2149)
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 4s
🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker-v4 (push) Has been skipped
🚀 Publish Trigger.dev Docker / units (push) Failing after 3s
* chore: Update version for release (v4-beta)

* Release 4.0.0-v4-beta.21

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
2025-06-06 10:50:28 +01:00
nicktrn 47b19a49d4 A few small v4 fixes (#2153)
* remove unused env vars

* actually use ALERT_REPLY_TO_EMAIL for alerts (non-breaking)

* increase fallback branch limit to 100M

* more unused env vars and example

* hide usage page when self-hosted

* fix for init flow

* set default or concurrency to 300 to match 100 on env

* fail fast when registry env vars are empty strings
2025-06-06 10:15:07 +01:00
Eric Allam 5301239e09 chore: remove v2 models from schema (#2107)
* Round 1 of v2 model removals

* Round 2

* Remove trigger http endpoint env

* Round 3

* Round 4

* Round 5

* Round 6

* Round 7

* Round 8

* Round 9

* Round 10

* Removed the remainder v2 code from the webapp

* GitButler WIP Commit

---------

Co-authored-by: GitButler <gitbutler@gitbutler.com>
2025-06-05 22:57:29 +01:00
Eric Allam 261dce102a v4: fix batchTriggerAndWait completion issues by processing batch chunks sequentially (#2152)
* v4: fix batchTriggerAndWait completion issues by processing batch chunks sequentially

* Remove unnecessary test
2025-06-05 17:56:56 +01:00
Eric Allam 575413c414 fix: moved dequeuing from a worker queue out of lua to be able to use blpop (#2151) 2025-06-04 22:04:30 +01:00
nicktrn e7795a06ad Fix: fixes and prerequisites for v4 self-hosting (#2150)
* remove pgadmin

* remove V3_ENABLED

* v3 is always enabled

* enfore docker machine presets by default

* rename autoremove env var

* prefix more k8s-specific env vars

* same prefix for all docker settings

* improve profile switcher copy

* supervisor can load token from file

* optional webapp worker group bootstrap

* fix error message

* fix app origin fallback for otlp endpoint

* use pnpm cache for webapp docker builds

* increase default org and env concurrency limit to 100

* optional machine preset overrides

* improve s3 pre-signing errors

* fix DOCKER_ENFORCE_MACHINE_PRESETS bool coercion

* shard unit tests

* fix for s3-compatible services

* optional object store region

* Update apps/supervisor/src/workerToken.ts

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>

* fix DEPLOY_REGISTRY_HOST example

* fix platform mock

* remove remaining v3Enabled refs

* fix error type.. bad bot

---------

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
2025-06-04 18:02:38 +01:00
Eric Allam 2b3ea692fe v4: dequeue performance improvements (split concurrency from dequeue) (#2127)
* WIP

* Run queue now works with the worker queue / master queue split

* Acking should also cause the master queue to be processed

* Convert run engine tests and run engine to use runQueue changes

* Include the util files in the test tsconfig

* coordinator target should be es2020 as well

* providers target 2020

* Fix the triggerTask tests in the webapp

* v4 now working with the new worker queues, and added the legacy master queue migration stuff

* report worker queue lengths via opentelemetry metrics

* Adding lock metrics

* Release concurrency bucket metrics

* • Updated RunQueue.removeEnvironmentQueuesFromMasterQueue() method signature to take runtimeEnvironmentId instead of masterQueue parameter
• Added automatic master queue shard calculation using this.keys.masterQueueKeyForEnvironment(runtimeEnvironmentId, this.shardCount) 
• Updated RunEngine wrapper method to use new runtimeEnvironmentId parameter
• Updated DeleteProjectService to call the method once per environment instead of once per master queue
• Simplified API by encapsulating master queue sharding logic within RunQueue class

* metrics now working, configure the run queue settings, additional metrics for run engine and redis-worker

* Fix CodeRabbit suggestions

* return undefined from dequeueFromWorkerQueue, not null

* Remove message from worker queue in certain circumstances when acking

* Update log

* Ensure master queue consumers cannot stop from a processing error, and make the consumer interval configurable via an env var

* Change how the run queue master queue consumers are disabled internally

* Fixed tests

* process the queue on nack

* Fix more tests

* Fix priority tests

* Fixed dequeueing test
2025-06-04 17:18:46 +01:00
nicktrn f603725393 Feat: unified deploys for self-hosted and cloud users incl. multi-platform support (#2138)
* remove registry proxy

* remove --self-hosted flag

* automatically set network build flag

* update syncEnvVars debug log

* improve switch command

* always display deploy errors if they exist

* fix stuck deploy command after finalize error

* webapp-driven deploys, multi-platform support, lots of fixes

* add worker deployment migration

* rename image platform env var

* only try to sync parent env vars for preview deployments

* add KEEP_TMP_DIRS

* supervisor: docker api version lock, auth, multi-platform

* set image ref on create, validate digest

* use metadata for digest, fix local multi-platform builds

* print git meta branch before commit

* improve push and load flag handling

* make runs after local builds compatible with load and push

* small improvement for platform overrides

* add image platform to dequeued message

* remove deprecated init request body fields

* fix fail deployment id param

* remove build debug logs

* pass report merge with no tests

* structured run debug logs

* add required env var for tests

* should not be an error log

* add changeset
2025-06-04 15:54:10 +01:00
nicktrn a61951049f Enhance webapp version with build info (#2146)
* improve app version output

* set build info

* fix typo

* always show additional build info when self-hosting

* Update apps/webapp/app/routes/_app.orgs.$organizationSlug.settings/route.tsx

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>

* Update apps/webapp/app/components/navigation/OrganizationSettingsSideMenu.tsx

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>

* fix build timestamp name

---------

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
2025-06-04 15:49:07 +01:00
nicktrn 7ee0992a2f Runtime agnostic SDK configuration via env vars (#2132)
* use std-env in getEnvVar

* add changeset
2025-06-04 15:48:52 +01:00
nicktrn eee0fb2198 Fix tar-fs dependabot alerts v3 (#2145)
* update testcontainers

* override tar-fs used by testcontainers
2025-06-03 14:29:54 +01:00
nicktrn 1b62b348a9 Fix tar-fs dependabot alerts (#2143)
* update dockerode

* override tar-fs for remix dev
2025-06-03 14:10:35 +01:00
github-actions[bot] a2a5204ac5 chore: Update version for release (v4-beta) (#2144)
* chore: Update version for release (v4-beta)

* Release 4.0.0-v4-beta.20

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Matt Aitken <matt@mattaitken.com>
2025-06-03 13:40:04 +01:00
Matt Aitken d4a9cefd78 syncVercelEnvVars fix for syncing the wrong preview branch env vars (#2141)
* syncVercelEnvVars fix for syncing the wrong preview branch env vars

* syncVercelEnvVars() fix for syncing the wrong preview branch env vars
2025-06-03 13:15:03 +01:00
nicktrn 9fca9e0508 Release 4.0.0-v4-beta.19 (#2140) 2025-06-02 15:32:25 +01:00
Matt Aitken d6dd2c9210 Preview branches docs (#2130)
* Draft preview branches docs

* More wip on docs

* Explain what the GH action does

* Added manual deploy/archive instructions

* Added dashboard section to the docs

* More wip on the preview branch docs

* Added preview/branch to the CLI deploy docs

* Add preview branch note to triggering docs

* Added preview branch notes to the API keys docs

* Added preview stuff to the Context docs

* CLI preview archive command

* Add a better explanation and details about env vars

* Fix for weird reference to the dashboard
2025-06-02 15:31:48 +01:00
github-actions[bot] 01e7ef3d04 chore: Update version for release (v4-beta) (#2110)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2025-06-02 15:21:16 +01:00
nicktrn e84d8cfa1d Fix: lock vitest version in CI (#2139) 2025-06-02 15:14:33 +01:00
Matt Aitken 26b0523cd0 Support passing “production” when deploying (#2137) 2025-06-02 14:18:36 +01:00
nicktrn 84d8fb575f Fix syncing parent env vars for non-preview deployments (#2131)
* only try to sync parent env vars for preview deployments

* add changeset
2025-06-02 10:40:06 +01:00
Matt Aitken 80d449bfdb Preview branch alerts (#2136)
* Preview branch alerts

* Show “Preview” not the branch for new alert modal
2025-06-02 10:37:37 +01:00
Matt Aitken ff157e51a1 If the run is already PENDING_EXECUTING, log an error and ack but don’t fail the run (#2129) 2025-05-30 11:44:40 +01:00
Matt Aitken a027e6a1bb Set the branch header for the SDK realtime functions (#2128) 2025-05-30 09:27:47 +01:00
Matt Aitken 5fdc7e66af Preview branch fixes (#2126)
* SDK: auto-detect the branch using the VERCEL_GIT_COMMIT_REF env var

* Waitpoint token completion with HTTP is now working
2025-05-29 21:23:11 +01:00
Matt Aitken ea853b243e Remove onsole.log… oops (#2125) 2025-05-29 14:39:36 +01:00
Matt Aitken 12f4653cd6 Fix long preview branch names (#2124)
* Truncate long branch names, don’t let the icon shrink

* Truncate and show tooltip for long branch names
2025-05-29 14:36:24 +01:00
Matt Aitken 47f4726515 Remove log that actually appears to users (#2123) 2025-05-29 13:47:19 +01:00
Matt Aitken 7ecaf2ce53 Sync env vars (parent env vars) (#2120)
* Prefer branch variables, but load preview envs too

* WIP with vercel preview env var syncing

* Return success if either child or parent env vars are synced

* Removed log of env vars

* Remove another log
2025-05-29 12:18:25 +01:00
nicktrn 7b463710f5 Fix metrics format (#2119)
* ensure eof marker is always at the end

* strip empty lines from prisma metrics
2025-05-29 09:52:42 +01:00
nicktrn fda9565eb0 Optionally disable run debug logs (#2116)
* disable run debug logs by default

* lightweight webapp health check

* disable debug logs for dev runs

* disable run debug logs for supervisor client

* add changeset
2025-05-29 09:27:08 +01:00
nicktrn 82e7484da8 Feat: import timings and bundle size analysis (#2114)
* cursor should ignore .env files

* fix for duplicate builds when starting dev

* output metafile in dev

* attach metafile to background worker

* attach import timings to worker manifest

* warn during dev if imports take more than 1s

* add analyze command

* update disable warnings flag message

* add changeset
2025-05-28 15:01:25 +01:00
Eric Allam 0f66023b54 v4: metadata collapsing now preserves operation order for correctness (#2115)
* v4: metadata collapsing now preserves operation order for correctness

* Add changeset
2025-05-28 14:52:06 +01:00
Eric Allam 021d6d8f20 Improve efficiency of cancelling in progress TaskEvent records in v3 and v4 (#2112) 2025-05-28 11:06:19 +01:00
Matt Aitken 068c024477 Preview branches (#2086)
* Initial preview migrations

* Modified the staging endpoint to create preview environments

* Added isBranchableEnvironment to RuntimeEnvironment

* Staging = yellow Preview = orange

* Changed the env sort order

* Set isBranchableEnvironment correctly. Create preview for new projects

* Very basic branch menu

* Creating branches from the dashboard

* Fix for string icons on project delete page

* Don’t show branch API keys

* WIP on the manage branches page

* RuntimeEnvironment added projectId index

* Only create the parentEnvironmentId column if it doesn’t exist already

* Improved the limit wording

* Add search to the branch list

* contains in both places

* Many style improvements

* Branch dropdown and v4 badge

* Arching/unarchive branches working in the dashboard

* Tidied imports

* Change preview slug from `prev` to `preview`

* Use correct color for side menu preview branch icon

* Upsert the branch and use the shortcode as a unique constraint

* Upserting working with nice messages in the dashboard

* Better errors when upserting branches

* Button shortcut, don’t allow event to propagate

* Better duplicate error message

* Filter out archived branches from the env selector

* Archiving/creating tweaked some more

* Add an archived banner to the app, fixes for archived branches and upsells

* Fixed pagination

* Disable editing schedules, pausing queues, testing tasks

* Don’t allow replaying if the env is archived

* When deploying detect the correct environment

* Get the projectClient when there’s a branch

* createGitMeta function, most code from the vercel CLI repo

* Deploy, getting the correct environment client

* Added git column to WorkerDeployment

* Add GitMeta to core schemas

* Create branch when deploying

* WIP on branch support in the API

* Delete old createTaskRunAttempt fn

* apiAuth remove export from internal functions

* Rename env var to “TRIGGER_PREVIEW_BRANCH”

* Add TRIGGER_PREVIEW_BRANCH to resolved env vars for runs

* First preview deploy and run working

* Set the preview branch in the main SDK

* Added git links to the preview branches table

* Better errors when replaying/testing archived branches

* Don’t dequeue archived environments

* Env var resolution with parent environment

* Hello world default machine small-2x to save my memory

* Fix for more env var functions

* Only return non-archived envs

* Switch to controlled state for the checkboxes

* Uncheck everything when PREVIEW is checked

* WIP on branch UI

* Show the preview branch label on the env vars list

* Fix for overriding env vars

* Adding preview branch env vars working

* Progress on new env vars

* Only allow selecting a single branch

* Layout fix when there are errors

* Set the defaultValue so there are some fields

* Conform fix for team invite page

* Archived environments don’t run scheduled tasks

* Added Git data to deployments

* Added git data to the deployment inspector

* Don’t allow upserting schedules when archived

* Deduplicate and blacklist some env vars

* Fix for wrong conform function being used

* Show a better error if all vars were blacklisted

* Added environment variable search (by key and value)

* Improved preview branch icon

* Replay now supports branches

* Schedule page render branches properly

* Show the env icon in bottom-left of the test page

* When editing older schedules (that have multi-env) show preview branches correctly

* Fix for incorrect disallowed branch name character

* Extract and improve the directory verification code

* WIP for CLI preview archive command

* Improved the preview branch action buttons

* Redirect to the project if we don’t find a matching env

* Archiving branch via the CLI working

* Fix for archiving branches

* Public access token test task

* JWTs working are with preview branches

* Add branch and git data to the Run ctx

* Updated GitMeta functions to work in CI

* Added pullRequestState

* Archive when deploying if the PR is closed/merged

* Fix for the changesets guide

* Fix for CLI dev bug introduced

* CLI promote now supports preview branches

* Add PR title. Reordered them and added tooltips

* syncEnvVars working with branches

* Added preview branch support to syncVercelEnvVars()

* Detect the branch from Vercel env var (set during build)

* Allow passing a branch in

* Use process.env.VERCEL_TOKEN as well… this used in Vercel CI

* Temp delete

* Improved regenerate api key modal

* Added Accordion component (with styles)

* Redesigned the API keys page

* Revert "Temp delete"

This reverts commit 177b92cd935a6161456bde65d01294e23ecfd47f.

* Changeset

* Fixed docs link

* The new branch panel closes when a branch is created

* Update apps/webapp/app/services/upsertBranch.server.ts

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>

* Removed findUniques from WorkerGroupTokenService

* Made the parentEnvironmentId migrations safe

* Latest lockfile

* Update packages/cli-v3/src/commands/workers/build.ts

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>

* Move isValidGitBranchName to a separate file

* Move the sanitize fn too

* removeBlacklistedVariables moved to a separate file

* Moved deduplicateVariableArray to a separate file…

* Fix broken sanitizeBranchName import

* Another import fix…

* Improved blacklisted error message

* SImplified migration to use `ADD COLUMN IF NOT EXISTS "parentEnvironmentId" TEXT`

---------

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
2025-05-28 11:02:56 +01:00
Eric Allam b74c000fb6 Improve metadata flushing efficiency by collapsing operations (fix #2104) (#2106)
* Improve metadata flushing efficiency by collapsing operations (fix #2104)

* Add changeset

* A couple of style tweaks
2025-05-27 14:08:15 +01:00
Eric Allam 0410316e14 Enable prisma metrics and add them to the /metrics endpoint (#2111)
* Enable prisma metrics and add them to the /metrics endpoint

* Add support for bearer token auth on metrics endpoint
2025-05-27 13:28:45 +01:00
Eric Allam c571c5ef32 v4: Can now set project ref using the TRIGGER_PROJECT_REF env var (#2109)
* v4: Can now set project ref using the TRIGGER_PROJECT_REF env var

* Add changeset

* Improve the name of the resolvEnvVars function and move setting NODE_ENV into the dev supervisor

* Removed moved function
2025-05-27 12:51:04 +01:00
Eric Allam e84ede9599 Upgrade to electric@1.0.13 (#2089)
* Upgrade to electric@1.0.13

* Add ELECTRIC_INSECURE env var to test container
2025-05-23 16:59:53 +01:00
Eric Allam 3fc5630be7 fix: calculating next schedule should always just be from now (#2102) 2025-05-23 16:55:52 +01:00
Eric Allam 13d378f6df v4: batches can only complete when all their runs have been created (#2101)
* v4: batches can only complete when all their runs have been created

* Fix flaky runs replication test
2025-05-23 16:53:50 +01:00
Eric Allam 1a6a9ca97a perf fix: calculateNextScheduledTimestamp speed improvement (#2099)
* Simulate "next steps" when calculating the next timestamp to prevent millions of iterations on dev schedues

* Add additional tests

* Fix typescript issue
2025-05-23 16:24:33 +01:00
Dan 6cb57b803c Added openai agent sdk guardrails docs (#2100) 2025-05-23 16:09:12 +01:00
nicktrn 7c791dd519 Improve unit test workflow performance (#2096)
* shard unit tests

* temp enable for all pushes

* fix test workflow

* update to latest vitest and only add to root package.json

* additionally use default reporter

* gather reports before uploading

* split up slow replication tests

* split up unit tests workflow

* move workflows to parent dir

* use new paths in parent workflow

* prevent artifact clashes

* we always need to create the reports dir

* speed up merge reports

* gather reports even when tests fail

* fix artifact patterns

* increase shards

* disable push trigger again

* improve dequeue snapshot test reliability
2025-05-23 12:22:14 +01:00
github-actions[bot] cf3735976e Release 4.0.0-v4-beta.18 (#2095)
* chore: Update version for release (v4-beta)

* Release 4.0.0-v4-beta.18

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
2025-05-22 21:59:11 +01:00
nicktrn a9a40b5184 Fix dev indexing (#2094)
* save build manifest for dev runs again

* add changeset
2025-05-22 21:54:00 +01:00
github-actions[bot] 0f928e224b Release 4.0.0-v4-beta.17 (#2071)
* chore: Update version for release (v4-beta)

* Release 4.0.0-v4-beta.17

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
2025-05-22 19:51:18 +01:00
Eric Allam a306545952 Always find debug logs when in the span sidebar (#2093) 2025-05-22 14:22:12 +01:00
Eric Allam 1bb1fc4c8f v4 run engine debug logs now filtered out in SQL by default (#2092) 2025-05-22 10:44:06 +01:00
nicktrn 1ca37f5745 Expose esbuild keepNames and minify options (experimental) (#2091)
* Expose esbuild `keepNames` option (experimental)

* Expose esbuild `minify` option (experimental)

* test new experimental options
2025-05-22 10:42:48 +01:00
nicktrn 8112c12092 Improve email whitelisting and extend to GitHub auth (#2090)
* make github auth respect email whitelist

* display github login errors

* fix smtp secure env vars

* prevent sending magic link if email not allowed

* fix MARQS_DISABLE_REBALANCING and RUN_ENGINE_DEBUG_WORKER_NOTIFICATIONS
2025-05-22 10:30:00 +01:00
nicktrn c8d252e300 Add experimental externals detection (#2083)
* add auto detect externals

* stop spinner on esbuild errors

* test problematic packages

* fix braces types

* improve detection 10x

* ignore sentry stub

* improve main package json detection

* rename to experimental_autoDetectExternal
2025-05-21 16:40:37 +01:00
nicktrn d89f740d18 Write esbuild metafile to output dir (#2087)
* write esbuild metafile to output dir

* add changeset
2025-05-21 16:17:10 +01:00
Eric Allam 41a3bdec68 Add id-token permission to the publish webapp workflow (#2088) 2025-05-21 14:30:03 +01:00
Eric Allam a9a937dd25 Implement example of streamText fallback using an AI data stream (#2068) 2025-05-21 13:22:06 +01:00
Eric Allam 14e081e17d release concurrency system reliability improvements (#2081)
* Refill release concurrency tokens when a run is cancelled

* Improved release concurrency accounting system + a sweeper to auto-refill tokens for snapshots that are no longer the latest snapshot on a run (e.g. the run has moved to a new snapshot state)

* Fix order of arguments to the releasings sweeper

* Add a heartbeat for SUSPENDED snapshots, where when stalled will attempt to continue the run if unblocked
2025-05-21 13:21:17 +01:00
Eric Allam 500a173d67 Set default values on the invite and new env var forms to fix an issue after updating conform (#2084) 2025-05-21 13:20:16 +01:00
Eric Allam cc562c58fa Remove unused publish job from the release github workflow to fix permissions error (#2085) 2025-05-21 13:19:49 +01:00
Eric Allam 0334df70ad Prevent socket hangup error by removing the aggressive clickhouse keep alive ttl env var default (#2082) 2025-05-21 13:15:15 +01:00
Dan e08e493b3c Potential fix for code scanning alert no. 2: Workflow does not contain permissions (#2078)
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
2025-05-19 21:17:35 +01:00
Dan f667d61d4c Potential fix for code scanning alert no. 3: Workflow does not contain permissions (#2077)
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
2025-05-19 21:16:58 +01:00
Dan 325ef04c7f Potential fix for code scanning alert no. 6: Workflow does not contain permissions (#2076)
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
2025-05-19 21:15:42 +01:00
Dan 79f0a6a662 Potential fix for code scanning alert no. 42: Workflow does not contain permissions (#2074)
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
2025-05-19 21:14:26 +01:00
Dan ad5027b5fd Potential fix for code scanning alert no. 1: Workflow does not contain permissions (#2079)
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
2025-05-19 20:59:50 +01:00
Dan f0809e6ad6 Potential fix for code scanning alert no. 38: Workflow does not contain permissions (#2075)
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
2025-05-19 20:48:58 +01:00
Eric Allam c63559f55d v4: fix race condition when continuing run when blocked at the same time (#2073)
* v4: fix race condition when continuing run when blocked at the same time

* Attempt to fix flaky e2e test

* Make waitpoint race condition test less flaky
2025-05-19 16:50:36 +01:00
Eric Allam 8471169b30 Fix various dependabot alerts (#2063)
* Upgrade posthog-node to clear axios vulns

* Upgrade @slack/web-api to use a secure version of axios

* Update parse-duration to fix security vulns

* Mitigate against the ws DoS vuln

* Upgrade body-parser to 1.20.3 in the webapp

* Upgrade react-use to 17.5.1 to remove the fast-loops transitive dep

* Remove unused babel dev deps and config file that's no longer used

* Upgrade prismjs to 1.30.0 and bundle parse-duration now

* upgrade express to 4.20.0 to fix issue with XSS when redirecting

* Upgrade @conform/zod to 0.9.2
2025-05-19 12:01:51 +01:00
nicktrn 558a39c644 Remove supervisor docker binary requirement and other tweaks (#2062)
* don't require docker binary

* more structured logs

* remove docker type dep

* add core changeset

* use implicit DOCKER_HOST instead

* disable resource monitor for now

* support attaching docker runners to multiple networks

* warn if dequeue interval > idle dequeue interval

* verbose logs

* add changeset
2025-05-17 12:33:41 +01:00
Eric Allam b4ab06600b Create AGENTS.md file (#2070) 2025-05-17 07:54:04 +01:00
Eric Allam c8e8bc9306 engine v2: Fix selectTopEnvs missing queues (#2069) 2025-05-17 07:53:48 +01:00
Eric Allam a69621bdcc Allow self-hosted deploys locally by pointing to docker.host.internal on macOS (#2064) 2025-05-16 15:20:48 +01:00
Eric Allam ad4daa3301 Fix run replication payloads/outputs trying to send invalid types like BigInt to clickhouse (#2065) 2025-05-16 11:02:30 +01:00
Eric Allam bb7173b4ab Improve run replication logging (#2060)
* Remove the tcp buffer monitors

* Improve run replication logging
2025-05-15 14:12:01 +01:00
Eric Allam 2aee3debc0 Docs: 3rd party log exporters (#2058) 2025-05-15 10:50:15 +01:00
Eric Allam d1e40643fc Allow creating and monitoring run replication services with different settings (#2055)
* Allow creating and monitoring run replication services with different settings

* Fix test
2025-05-14 20:40:47 +01:00
Eric Allam a8e38d9231 chore: Skip unit tests dependency when running publish image workflow (#2054) 2025-05-14 19:31:50 +01:00
github-actions[bot] 2ae0e22d81 Release 4.0.0-v4-beta.16 (#2053)
* chore: Update version for release (v4-beta)

* Release 4.0.0-v4-beta.16

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
2025-05-14 16:26:28 +01:00
nicktrn 206a2c3245 Fix init.ts detection for sentry plugin users (#2051)
* fix init.ts detection when using the sentry plugin

* add changeset
2025-05-14 16:20:05 +01:00
Eric Allam eb3929880f runs replication leader lock expiration fix (#2050)
* runs replication leader lock expiration fix

* Allow configuring the container image --max-old-space-size using NODE_MAX_OLD_SPACE_SIZE

* Ability to configure the clickhouse keep alive settings

* Add some logging because we might not be able to do telemetry
2025-05-14 15:49:48 +01:00
github-actions[bot] 0661ee5fdd Release 4.0.0-v4-beta.15 (#2040)
* chore: Update version for release (v4-beta)

* Release 4.0.0-v4-beta.15

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
2025-05-14 12:33:37 +01:00
Eric Allam 2deef00d48 Improvements to the runs replication service to prevent buffer exhaustion (#2047) 2025-05-14 11:35:51 +01:00
nicktrn fa5360a99d Feat: supervisor ARM builds (#2046)
* v4 worker multi-arch builds

* v4-beta tag for webapp

* add oidc permission

* deps fetcher needs python

* node-gyp needs complete toolchain

* fix headers

* bust pnpm cache

* add setuptools

* switch back to old cache id

* push worker images

* pass the correct tags
2025-05-14 11:23:41 +01:00
Eric Allam 7ace1c3e25 Fix run replication telemetry by passing in a tracer (#2045) 2025-05-14 09:52:09 +01:00
Eric Allam a815633712 Add external log exporters and fix missing external trace exporters in deployed tasks (#2038)
* Add external log exporters and fix missing external trace exporters in deployed tasks

* Generate the external traceID correctly and exporter 3rd party logs with the external traceID as well
2025-05-13 16:54:24 +01:00
Eric Allam 84a0336bdd Fix #1955: schemaTask parse payload failures handled gracefully (#2043) 2025-05-13 16:47:21 +01:00
Eric Allam 7be07b4d0e allow configuring the replication leader lock settings (#2042) 2025-05-13 14:17:36 +01:00
nicktrn bb96e848de Fix init.ts import for deployed workers (#2041)
* import init file in managed worker

* refactor for easier diff

* managed worker: attach configured incorrectly error details

* add changeset

* disable pre

* we also didn't rewrite the output path

* Revert "disable pre"

This reverts commit 89afe44f6c42d4bd7b5a51f7fda1caec6cab8a24.
2025-05-13 13:27:49 +01:00
Rudra Sankha Sinhamahapatra 39f03085b4 feat: ffmpeg v7 (#1777)
* adding ffmpeg v7

* Renaming id ffmpeg to ffmpeg7

* added space

* fixing formatting issue

* fix unit test workflow for forks

* merge ffmpeg v7 into existing extension

* verify md5 checksum

* fix dns resolution

* fix v7 install

* remove redundant code

* add changeset

* mark fluent-ffmpeg as external

* add docs

* Revert "mark fluent-ffmpeg as external"

This reverts commit 74cd317ae69c7453a22d3fa412ff00b402840b87.

---------

Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
2025-05-13 12:27:35 +01:00
Mendy Landa 21c0dc619e feat: playwright extension (#1764)
* feat: playwright build extension

* feat: add playwright task

* full dep list & added some comments

* fix v3-catalog

* log image size for local builds

* detect playwright version

* add changesets

* add docs

* additional checks

* fix unit test workflow for forks

* use middleware and update docs

* update jsdoc

---------

Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
2025-05-13 12:22:01 +01:00
James Ritchie da0ef73724 Docs: Removes an issue listed in ‘known issues’ in the v4 migration guide (#2039) 2025-05-13 10:14:27 +01:00
Eric Allam 65da20c225 feat: replicate task runs to clickhouse to power dashboard improvements (#2035)
* WIP clickhouse package with test containers setup

* More clickhouse client setup now with otel and real tests, and the v1 of raw run events

* Add some additional columns to raw_run_events_v1

* WIP runs dashboard service

* Create a new run engine event bus event for the runs dashboard to hook into

* Track run events in the run engine

* make sure engine v1 runs get synced to CH

* Update the attemptNumber of v3 task runs

* Restructure the run events to be more sparse

* emit more stuff

* Setup replication package

* scaffold the replication package

* replication wip

* resolve conflicts

* more replication stuff

* Add ability to drop the replication slot completely on teardown

* Use the new single replacingmergetree task events table for replication

* get it working

* insert payloads into their own table only on insert and then join

* prepare for using clickhouse cloud and now running ch migrations during boot in the entrypoint.sh

* Handover WIP and tests

* Testing the replication service

* Remove the runs dashboard stuff that we aren't using anymore

* Added a test for large payloads

* hacky typecheck fix

* Fix new internal package typecheck issues and start adding telemetry to the replication service

* tracing over spans, some other improvements

* Improvements to the runs replication service, now ready for testing

* Some fixes and cleanups

* Don't need this code anymore

* move transaction types into the runs replication service

* only send spans where there are transaction events

* A couple of suggested tweaks
2025-05-12 22:12:36 +01:00
James Ritchie ccc9764383 Active incident status panel in the side menu (#2033)
* WIP adding a side menu panel to display incident statuses

* Fixes re-rendering bug and copy tweak

* cache the betterstack response using unkey

* Style the button to fit the panel colors
2025-05-09 15:52:17 +01:00
Matt Aitken 584722d48b [v4] Complete a waitpoint and then get affected runs (#2034)
* Complete waitpoint before getting affected runs. Added more logging

* Added another error log
2025-05-09 15:46:41 +01:00
nicktrn 1b1ad16d94 Improve worker management admin route (#2032)
* improve route to add worker group, handles existing groups gracefully

* add option to remove default worker group from project

* separate project id field

* update supervisor readme with route changes
2025-05-09 10:43:31 +01:00
Matt Aitken 37aadfc80b We can use TaskRun error instead of attempt. Faster and works across v3/v4 (#2031) 2025-05-09 09:36:40 +01:00
github-actions[bot] 6736cb5ae9 Release 4.0.0-v4-beta.14 (#2029)
* chore: Update version for release (v4-beta)

* Release 4.0.0-v4-beta.14

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Matt Aitken <matt@mattaitken.com>
2025-05-08 11:12:25 +01:00
Matt Aitken 87630bc64f Fix for TS issue with old DB type (#2030) 2025-05-07 20:51:36 +01:00
Matt Aitken d23fa38a0a Waitpoint token callback URLs (#2025)
* Initial commit with a plan for what we’re going to do

* Some initial types and improved plan

* Add Waitpoint resolver

* Add resolver + status index

* Remove type + status index

* Only drop if exists

* Remove type index

* Update waitpoint list presenter to use resolver

* Added resolver to the engine

* Made the existing waitpoint list presenter more flexible

* Initial implentation ofr wait.forHttpCallback()

* Added the callback endpoint (no API rate limit)

* schema version

* Added jsdocs, removed schema version because of errors

* Show callback URL if it’s set

* Dashboard pages and panels

* Remove todos

* Added temporary icon

* Added a blank state

* Some tweaks and added a Replicate example

* Implement unwrap() for httpCallback

* Added unwrap to wait.forToken() as well

* Improved jsdocs

* Added docs

* Added unwrap to the token docs

* Show a dash if there are no tags

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>

* Make the timeout error safer

* Fixed migrations… should use id desc not createdAt desc

* Fixed page title

* Fixed migration so it only adds them if they don’t exist. This allows us to manuall run in cloud first

* Respect the max content length by getting the length of the body

* Added more docs details about the callback format

* Remove code comment

* Improved the error

* Added a hash to the HTTP callback URLs

* Add the apiKey to the API input type to fix TS error

* Return the error responses. They were being caught and not preserved

* The content-length header is required. Deal with an empty body

* Removed unused types

* Added some new span icons

* Reworked http callback to be a create call then just use wait.forToken()

* Added a changeset

* Updated the docs

* Updated the wait overview docs

* Simplify to just a call

* WIP stripping right back to waitpoints just having a URL associated with them…

* More deletions

* Remove missing icon

* Updated the changeset

* Add URL to the token return types

* Remove wait for http callback page

* Updated docs

* More tidying

* Type and import fix

* Remove unused import

* Some type fixes for the retrieve

---------

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
2025-05-07 19:55:29 +01:00
James Ritchie 4ec0395424 Fixes a console error introduced when updating the loading animation (#2019) 2025-05-07 18:40:04 +01:00
James Ritchie e94558a0d5 Stop indexing the test-cloud domain (#2026) 2025-05-07 18:39:38 +01:00
James Ritchie 6f4af89adb Docs: Adds Yarn ‘Plug’n’Play” conflicts fix to troubleshooting page (#2028) 2025-05-07 18:38:40 +01:00
James Ritchie 3e45f785a0 Dashboard: Org menu improvements for admins (#1969)
* Adds path for admin route

* Updates impersonation banner and show new org button if only 1 org

* Simpler UI for displaying admin access
2025-05-07 18:38:00 +01:00
James Ritchie 04b9973f20 Shows the tooltip date time on DateTImeAccurate component (#2021) 2025-05-07 18:35:48 +01:00
github-actions[bot] 8da3043111 Release 4.0.0-v4-beta.13 (#2027)
* chore: Update version for release (v4-beta)

* Release 4.0.0-v4-beta.13

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
2025-05-07 15:10:50 +01:00
nicktrn 2b586c87a0 Fix controller waitpoint resolution, suspendable state, and snapshot race conditions (#2006)
* remove dead code

* rename managed to shared runtime manager

* rename to resolve waitpoint for clarity

* add resolver id helper

* store and correctly resolve waipoints that come in early

* fix ipc message type change

* branded type for resolver ids

* add fixme comments

* remove more unused ipc schemas

* fix entitlement validation when client doesn't exist

* restore hello world reference workspace imports

* runtime manager debug logs

* prefix engine run logs

* managed run logger accepts nested props

* runtime suspendable state and improved logs

* require suspendable state for checkpoints, fix snapshot processing queue

* add terminal link as cli module so we can more easily patch it

* apply cursor patch

* add license info

* remove terminal-link package and add deprecation notice

* remove old patch

* remove terminal-link from sdk

* rename snapshot module

* add cli test tsconfig

* add run logger base type

* add snapshot manager tests

* fix cli builds

* improve QUEUED_EXECUTING test

* changeset

* make testcontainers wait until container has stopped

* require unit tests for publishing again

* avoid mutation during iteration when resolving pending waitpoints

* improve debug logs and make them less noisy

* always update poller snapshot id for accurate logs

* detach task run process handlers

* check for env overrides in a few more places and add verbose logs

* log when poller is still executing when we stop it

* add supervisor to publish workflow

* always print full deploy logs in CI

* Revert "avoid mutation during iteration when resolving pending waitpoints"

This reverts commit 87b0ce1e5b.

* disable pre

* print prerelease script errors

* Revert "disable pre"

This reverts commit 9403409637.

* misc fixes

* better debug logs

* add snapshots since methods and route

* prep for snapshots since

* improve deprecated execution detection

* update supervisor and schema

* properly log http server errors

* detect restore after failed snapshot fetch

* run and snapshot id can be overridden

* fix restore detection

* fix deprecation checks, move into snapshot manager

* less logs

* rename snapshot manager stop

* restore detection was moved into snapshot manager

* fix notifier logs

* make runtime manager status a debug log

* no need to attach runtime status twice

* findUnique -> findFirst

* sort snapshots by created at everywhere
2025-05-07 13:02:32 +01:00
nicktrn 053389d9b3 Fix testcontainers cleanup (#2023)
* update testcontainers packages

* increase cleanup timeout and add better logs

* small tweaks

* decrease docker network size so we can have more of them

* add a test flow to check this all works

* await all engine.quit calls

* add docker diagnostics

* reduce number of test runs

* improve network attachment output

* add setup logs

* log redis setup errors

* add cleanup log type

* stop redis container if setup fails

* disable ipv6

* tidy things up a bit

* fix unit tests workflow

* decrease container cleanup timeout

* fix types

* fix webapp typecheck
2025-05-06 08:56:34 +01:00
Eric Allam 43a4139424 docs: onCancel (#2024)
* docs: onCancel

* Add availability note
2025-05-03 21:40:03 +01:00
github-actions[bot] 7b509849e9 chore: Update version for release (v4-beta) (#1996)
* chore: Update version for release (v4-beta)

* Release 4.0.0-v4-beta.12

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Eric Allam <eallam@icloud.com>
2025-05-03 21:26:12 +01:00
Eric Allam 13f750a18b v4: implement onCancel callbacks (#2022)
* v4: implement onCancel callbacks

* chat example

* Add changeset

* A few improvements
2025-05-03 20:50:11 +01:00
nicktrn 15816e9955 Fix long retry delays (#2020)
* make testcontainers wait until container has stopped

* require unit tests for publishing again

* add failing test case

* make it pass

* add retry threshold ms env var and use it
2025-05-02 16:57:21 +01:00
nicktrn a10e2a8b16 Small fixes in prep for v4 self-hosting (#2018)
* fix entitlement validation when client doesn't exist

* add supervisor to publish workflow

* fix v4 publish workflow name
2025-05-02 14:33:20 +01:00
Matt Aitken 423209661b Give helpful hint if you pass a bad directory when deploying (#2013) 2025-05-02 10:54:26 +01:00
James Ritchie 73f363f9d5 Changes the spinner to white now we have purple primary buttons (#2014) 2025-05-02 10:50:22 +01:00
James Ritchie 4456184f37 Fix task and schedule icons squishing (#2017) 2025-05-02 10:47:57 +01:00
James Ritchie 2448242be3 Small improvement to the loading animation so it moves left to right only (#2012) 2025-05-01 19:24:03 +01:00
James Ritchie 9c3aeb3efb Adds friendly formatting for max duration values (#2011) 2025-05-01 19:22:56 +01:00
James Ritchie df0bce3a8f Code blocks have an optional text-wrap toggle (#2009)
* Code blocks have an optional text-wrap button

* Wrap “words”, not “all”

* wrapping is default false

* Change the wording in the tooltip
2025-05-01 17:44:17 +01:00
James Ritchie 0fe85eaec4 Highlight queues when concurrency limit reached and new table col (#2008)
* Combines the Running and Concurrency limit cols into 1

* Display a badge when a queue is at the concurrency limit

* Colors the Running/Limit column text amber if the concurrency limit is hit

* Turns the “Running” big number amber and shows “At concurrency limit” text

* BigNumber now handles big values using formatNumber and formatNumberCompact

Also includes some responsive improvements to make sure things wrap when it gets tight

* Adds a new col for showing how the queue is limited

* Reinstates a threshold for making very big numbers compact

* Added border to make the search bar not float
2025-05-01 17:40:54 +01:00
Matt Aitken d2dde0a844 formatDateTimeISO() fix (#2010)
* Fix for the DateTime ISO function

* Fix implementation and added unit tests

* Preserve milliseconds properly
2025-05-01 17:35:55 +01:00
Dan aacec569d0 Added Python MarkItDown docs (#1977)
* Added Python MarkItDown docs

* Simplified the task

* Update example to use fetch and improved sidebar titles

* Fixed link
2025-05-01 14:18:12 +01:00
nicktrn 0df7af4403 Add idle dequeue interval to supervisor (#2007)
* supervisor uses new idle interval after empty dequeues and errors

* make supervisor heartbeat interval configurable
2025-05-01 10:53:19 +01:00
James Ritchie 553a03580f Show a tooltip when hovering over any DateTime (#2003)
* Adds a tooltip to display the UTC, timezone and local time for DateTime

* Made it safer for SSR

* Makes the copy function into a hook and separate button + adds copy dateTime button to the date tooltip

* Adds an extra-small size copy button

* Show the offset (UTC +1) next to your local time

* Optionally don’t show the date tooltip. Defaults to true (always show)

* Tidy imports

* Refactor the tooltip content to remove duplicated markup

* Coderabbit suggestion to include support for minutes for some timezones like Nepal
2025-05-01 10:00:42 +01:00
Matt Aitken af019473cd Get data for the last day of the month for usage graphs (#1999) 2025-05-01 09:53:06 +01:00
Matt Aitken cb47200d34 Added search to the Queues page (#2000)
* Added search to the queues page

* Fix for empty state copy
2025-05-01 09:33:03 +01:00
James Ritchie 98fd8045e9 Tooltip to explain dev schedules not counting towards your limit (#2004) 2025-05-01 09:32:13 +01:00
James Ritchie 44184a61c7 Updates the secret key text to be more explicit what you have to do to start triggering tasks (#2005) 2025-05-01 09:31:37 +01:00
nicktrn c0807ad0d0 Display terminal links in cursor (#1998)
* add terminal link as cli module so we can more easily patch it

* apply cursor patch

* add license info

* remove terminal-link package and add deprecation notice

* remove old patch

* remove terminal-link from sdk

* changeset
2025-05-01 08:53:11 +01:00
James Ritchie cf9460afee Post new sign up reasons to slack, not plain (#1995)
* Post new sign up reasons to slack, not plain

* Makes the Slack env vars optional

---------

Co-authored-by: Matt Aitken <matt@mattaitken.com>
2025-04-30 14:02:02 +01:00
Matt Aitken 7fe111abe9 AI dashboard widget + CLI AI generate fix suggestions (#1925)
* Made a better AI icon and improved how it looks on the create new schedules inspector

* WIP adding kapa ai to the app

* WIP adding a new Ask AI button to the side menu

* WIP using the react example from the docs

* Align the AI button in the bottom bar

* Kapa widget now works

* Trigger the Kapa modal from the custom button

* Fix imports

* Adds Ask AI shortcut to Shortcuts panel

* Adds a new enter shortcut key

* Adds a prop so you can optionally hide the shortcut key

* Latest

* Moved the Kapa/Help stuff into a component, out of root

* WIP using onModalClose

* Creates a wrapper provider to block shortcuts while kapa modal is open (has bugs)

* Fixes button alignment

* Hide the shortcut key at the button layer

* Fix for enable/disable shortcut keys globally

* Kapa is working

* You can bring up the shortcut keys without opening the help panel

* TODO remove listeners

* remove imports and fix invalid tailwind class

* style kapa widget as best i can

* Remove Kapa event listeners

* Allow passing in a query

* Open the AI widget if there’s a URL param

* Much cleaner implementation for Kapa

* Trying to auto-open the Kapa widget when the page loads

* Delay opening the widget because it was causing issues

* Improved Kapa widget colors

* Added an AI help link to the CLI

* We don’t need this anymore

* Exit with 1, indicating an error. This is important for CI

* Removed old auto-opening code

* Added some code comments to explain some of the annoying stuff

---------

Co-authored-by: James Ritchie <james@trigger.dev>
2025-04-30 11:11:42 +01:00
github-actions[bot] 525285da7f chore: Update version for release (v4-beta) (#1994)
* chore: Update version for release (v4-beta)

* Release 4.0.0-v4-beta.11

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Matt Aitken <matt@mattaitken.com>
2025-04-29 16:48:57 +01:00
Eric Allam 07365cd405 realtime: fix streams missing chunks when streams are longer than 5 minutes and receive 408 Request Timeout errors. Also now support multiple client streams being sent to a single stream key (#1993)
* realtime: fix streams missing chunks when streams are longer than 5 minutes and receive 408 Request Timeout errors. Also now support multiple client streams being sent to a single stream key

* Safely release the reader lock
2025-04-29 12:42:38 +01:00
Matt Aitken 83d0e8710d Fix for Schedules list page slow loading (#1992)
* Fix for Schedules list page slow loading

Getting BackgroundWorkerTask was very slow (Prisma was getting every single one…)

* Same fix for the upserting of schedules in the dashboard
2025-04-29 12:40:56 +01:00
Matt Aitken 280a6d0297 V4 dequeue test fixes (#1991)
* Fix priority tests by dequeuing in sequence

* Expect this test to fail until we support parallel dequeuing of an env with streaming

* Fix one and expect fail from the other

* Fix for batch trigger test dequeuing

* Fixed pending version test
2025-04-29 12:40:22 +01:00
Matt Aitken f579afb4cc V4 dequeue performance (return faster) (#1989)
* Loop through tenants only once, return as soon as we have enough.

* Hello world deadlock task set back to triggerAndWait
2025-04-28 14:49:38 +01:00
James Ritchie 3d744c6ecc Mergent migration guide (#1988)
* Adds a migration guide for converting mergent tasks to trigger.dev

* Adds link to migration guide in the docs side bar
2025-04-28 14:18:09 +01:00
github-actions[bot] 5691480dac Release 4.0.0-v4-beta.10 (#1984)
* chore: Update version for release (v4-beta)

* Release 4.0.0-v4-beta.10

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
2025-04-28 13:34:34 +01:00
nicktrn 09a859d056 Fix managed run controller edge cases (#1987)
* parse disconnect description

* decrease snapshot poller logs

* remove another useless log

* attempt number change detection

* ensure no executions or child processes are being reused

* prevent snapshot change handler from running after execution stopped

* prevent creating zombie intervals on reset

* add changeset

* add enabled poller check before getting latest snapshot

* make attempt number update more explicit, improve logs

* remove deprecated heartbeat service

* ..also remove the import
2025-04-28 13:22:30 +01:00
Saadi Myftija 7bdbbdc433 Handle rate limit errors from Slack on new alert creation (#1986) 2025-04-28 11:07:59 +01:00
Matt Aitken 8b0e4f2367 Fix for completing batches (when there are idempotent runs) (#1985) 2025-04-25 18:41:20 +01:00
Eric Allam e837500486 feat: v4 deadlock detection (#1970)
* Locked task runs will now require queues and tasks to be in the locked version

* Client errors caught in a run function now will skip retrying

* Extracted out the trigger queues logic

* extract validation, idempotency keys, payloads to concerns

* Extracted out a bunch of more stuff and getting trigger tests to work

* Add queue and locked version tests

* Deadlock detection WIP

* more deadlock detection

* Only detect deadlocks when the parent run is waiting on the child run

* Improve the error experience around deadlocks

* A couple tweaks to make CodeRabbit happy and fixing the tests in CI

* Fixed failing test

* Changeset

* wip

* Make sure to scope queries to the runtime env
2025-04-25 12:56:54 +01:00
github-actions[bot] 60fcf5fdac Release 4.0.0-v4-beta.9 (#1980)
* chore: Update version for release (v4-beta)

* Release 4.0.0-v4-beta.9

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
2025-04-25 12:43:32 +01:00
Eric Allam 3a62ee36d9 v3: prevent envs with excess queues from degrading dequeue performance (#1982)
* v3: WIP maximum queues per env implementation

* Add a queue distribution test

* Add edge case tests

* Set the marqs maximumQueuePerEnvCount
2025-04-24 18:02:49 +01:00
Matt Aitken bb606a1fcd Don’t allowed canceled delayed runs to be put into the queue (#1981) 2025-04-24 15:10:47 +01:00
Matt Aitken e9c6cc71f0 New batch id length is longer (#1944) 2025-04-24 13:38:15 +01:00
Eric Allam 30f93521c8 fix: default machine config indexing now works (#1979) 2025-04-24 13:38:03 +01:00
Matt Aitken 18f77660e0 Runlock telemetry (#1974)
* Added error throwing tests to runlock

* Added some more run lock tests

* Added customConditions so the test can use the Logger

* Fix for JSON error

* Added a span to the run lock with a name
2025-04-24 13:37:33 +01:00
Matt Aitken aa1027969b Increase v4 visibility timeouts & don't try continue finished runs (#1978)
* Increase v4 visibility timeouts

* Added some logging for #continueRunIfUnblocked: run has no checkpoint

* If a run is finished or pending cancel, don’t try and continue it
2025-04-24 13:36:18 +01:00
James Ritchie 32ff569844 Fixes bug on Safari preventing Queues and Schedules tables from scrolling (#1973) 2025-04-24 11:40:47 +01:00
James Ritchie dfca04aaea Docs: Updates the run function params syntax + adds new run.list() example (#1976) 2025-04-24 11:31:22 +01:00
Matt Aitken 672a6b8be7 RedisWorker clean up orphaned items when dequeuing (#1975)
* Remove setting the invisibility timeout because it’s already done in the dequeue Lua script

* Remove orphaned queue items when dequeuing

* Added an ack to the visibility timeout test
2025-04-24 11:15:42 +01:00
github-actions[bot] 56ec0523b5 Release 4.0.0-v4-beta.8 (#1972)
* chore: Update version for release (v4-beta)

* Release 4.0.0-v4-beta.8

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
2025-04-23 19:30:49 +01:00
nicktrn 73214d760e Prevent large outputs from overwriting each other (#1971)
* large outputs now get unique keys again

* replace other attempt id occurences

* add changeset
2025-04-23 19:21:21 +01:00
James Ritchie 853a7ef7e4 Updates button copy and layout tweaks (#1941) 2025-04-23 16:33:46 +01:00
HUORT Louis 0726620525 Switch to docker compose v2 (#1692)
* refactor: docker compose migration

* fix compose download link

* set static name for electric container

---------

Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
2025-04-23 14:29:26 +01:00
github-actions[bot] cfb77159d7 Release 4.0.0-v4-beta.7 (#1954)
* chore: Update version for release (v4-beta)

* Release 4.0.0-v4-beta.7

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
2025-04-23 13:34:26 +01:00
Saadi Myftija 2dbc085bb3 Fix alert creation bug related to the Slack integration (#1967)
* Fix alert creation bug related to the Slack integration

* Handle token_expired error from slack gracefully
2025-04-23 11:11:17 +01:00
Saadi Myftija 26aad58abd Add fuzzy matching to the tasks filter (#1960)
* Add a custom react hook to enable fuzzy searching in item lists

* Use fuzzy filtering in the tasks view list

* Use fuzzy filtering in the test tasks list

* Remove the old tasks filtering react hook
2025-04-22 16:34:56 +01:00
Saadi Myftija b62c17bec1 Show more recent payloads in the test task page (#1961) 2025-04-22 16:24:15 +01:00
Saadi Myftija fab565256e Allow empty payloads in test runs (#1962)
* Allow empty payloads in test runs

* Expose a basic linter configs in the code editor component

* Touch up the linting error elements in the code editor component
2025-04-22 16:23:52 +01:00
nicktrn cedd932f99 Fix queued snapshot status handler (#1963)
* handle queued status change gracefully

* add resource monitor to references

* add resource monitor example

* improve runtime manager debug logs

* fix resource monitor example

* add changeset
2025-04-22 15:32:16 +01:00
Matt Aitken e379fc15be Fix redis worker debounce (#1966)
* Added some Redis worker debounce tests (one failing that reproduces a bug)

* Added some tests for acking

* Added a deduplicationKey to prevent acking when items are queued

* The worker passes the deduplicationKey back in for acking

* Improved logs and removed events from test
2025-04-22 15:24:39 +01:00
Dan c0172d428d added cursor rules docs page (#1965) 2025-04-22 15:23:47 +01:00
Dmitry Sabanin 2c19693cfc Fixes runLimiter check on #dequeueRuns (#1953)
* Fixes runLimiter check on #dequeueRuns

This was causing issues in the dev environment with logs being spammed with `[DevSupervisor] dequeueRuns. Run limit reached, trying again later` and making the worker unresponsive and lose connection.

* Create polite-impalas-care.md

---------

Co-authored-by: Eric Allam <eallam@icloud.com>
2025-04-19 21:45:27 +01:00
github-actions[bot] 64ca9dc9a8 Release 4.0.0-v4-beta.6 (#1951)
* chore: Update version for release (v4-beta)

* Release 4.0.0-v4-beta.6

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
2025-04-18 16:50:22 +01:00
Eric Allam 39e535772f fix: realtime streams from tasks now retry if they receive a 408 timeout error from the realtime/trigger.dev server (#1952) 2025-04-18 16:20:13 +01:00
Eric Allam 006951aa53 Set a maximum number of times the dequeue loop can run (#1950) 2025-04-18 13:36:52 +01:00
nicktrn 0ebecbe574 Multiple queue consumers and dev engine URL changesets (#1949)
* add engine url changeset

* add queue consumer changeset
2025-04-18 11:36:29 +01:00
nicktrn aa74260646 Dev command engine URL now configurable via the webapp (#1948)
* return dev engine url from config call

* set and use engine url from dev config reply
2025-04-18 11:11:11 +01:00
nicktrn f683fe8d2d Multiple queue consumers per supervisor (#1947)
* multiple queue consumer in the same supervisor instance

* disable pre

* Revert "disable pre"

This reverts commit 4b154398982758d09667531e3a64761da83664eb.
2025-04-18 11:07:49 +01:00
Eric Allam 863ecf4830 fix: prevent unbounded looping while dequeueing by exiting early when no queues have messages to dequeue (#1946) 2025-04-18 09:41:09 +01:00
Matt Aitken bbf397e661 Completed batch waitpoints when we completed the BatchTaskRun (#1945)
* Completed batch waitpoints when we completed the BatchTaskRun

* Try complete the batch faster now it’s being used operationally

* Fix for tests that were using the old engine.unblockRunForCreatedBatch() function
2025-04-17 21:01:58 +01:00
github-actions[bot] 4a8303212c Release 4.0.0-v4-beta.5 (#1943)
* chore: Update version for release (v4-beta)

* Release 4.0.0-v4-beta.5

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Matt Aitken <matt@mattaitken.com>
2025-04-17 17:25:07 +01:00
github-actions[bot] 59eea654ac Release 4.0.0-v4-beta.4 (#1939)
* chore: Update version for release (v4-beta)

* Release 4.0.0-v4-beta.4

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Matt Aitken <matt@mattaitken.com>
2025-04-17 17:05:52 +01:00
Matt Aitken 0513bfadfb Changeset: The envvars.list() and retrieve() functions receive isSecret for each value. Secret values are always redacted. (#1942) 2025-04-17 16:55:51 +01:00
Matt Aitken a7e326c3f3 Remove concurrently, can’t have two in a single file and they’ve already been applied (#1940) 2025-04-17 15:30:37 +01:00
James Ritchie ad088456d4 Changes button spinner to white and Replay Run button changes to secondary variant (#1912)
* Uses the white variant of the spinner instead of dark on the replay run button

* Changes the replay run button on the run page to secondary
2025-04-17 15:09:29 +01:00
Dan e61466c3a5 Added meme generator example (#1929) 2025-04-17 15:08:47 +01:00
Eric Allam 4b42f778ce fix: release concurrency token exhaustion no longer happens on batchTriggerAndWait (#1937) 2025-04-17 15:07:06 +01:00
James Ritchie 2aacf76444 Adds a new entry in the help and feedback modal to request more feedback (#1936) 2025-04-17 15:06:41 +01:00
Matt Aitken 5597291380 Schedules list performance (#1938)
* Order the schedules: most recent first

* TaskSchedule dashboard speed indexes
2025-04-17 15:05:50 +01:00
Matt Aitken 5c9610b807 The mini task page graphs were missing Timed out and Expired (#1915) 2025-04-17 15:05:37 +01:00
Himanshu Sharma e0fd06490a maintain proper context in metadata.root and parent getters (#1917)
* fix(TRI-5140): maintain proper context in metadata.root and parent getters

* Create two-tigers-dream.md

---------

Co-authored-by: Eric Allam <eallam@icloud.com>
2025-04-17 15:04:01 +01:00
Matt Aitken 33e7b6865a Set env vars to be "secret" (#1923)
* WIP on secret env vars

* Editing individual env var values is working

* Sort the env vars by the key

* Deleting values

* Allowing setting secret env vars

* Added medium switch style

* Many style changes to the env var form

* “Copy text” -> “Copy”

* Draw a divider between hidden buttons

* Env var tweaks

* Don’t show Dev:you anymore

* Grouping the same env var keys together

* Styles improved

* Improved styling of edit panel

* Fix bun detection, dev flushing, and init command (#1914)

* update nypm to support text-based bun lockfiles

* add nypm changeset

* handle dev flushing failures gracefully

* fix path normalization for init.ts

* add changesets

* chore: remove pre.json after exiting pre mode

* init command to install v4-beta packages

* Revert "chore: remove pre.json after exiting pre mode"

This reverts commit f5694fde9314114c74a220c2213d19667bca1a6c.

* make init default to cli version for all packages

* Release 4.0.0-v4-beta.1 (#1916)

* chore: Update version for release (v4-beta)

* Release 4.0.0-v4-beta.1

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>

* Both run engines will only lock to versions they can handle (#1922)

* run engine v1 will only lock to v1 deployments

* run engine v2 will only lock to managed v2 deployments

* test: create background worker and deployment with correct engine version

* Add links to and from deployments (#1921)

* link from deployments tasks to filtered runs view

* jump to deployment

* don't add version links for dev (yet)

* Fix current worker deployment getter (#1924)

* only return last v1 deployment in the shared queue consumer

* be explicit about only returning managed deployments

* Add a docs page for the human-in-the-loop example project (#1919)

* Add a docs page for the human-in-the-loop example project

* Order guides, example projects and example tasks alphabetically in the docs list

* Managed run controller revamp (#1927)

* update nypm to support text-based bun lockfiles

* fix retry spans

* only download debug logs if admin

* add nypm changeset

* pull out env override logic

* use runner env gather helper

* handle dev flushing failures gracefully

* fix path normalization for init.ts

* add logger

* add execution heartbeat service

* add snapshot poller service

* fix poller

* add changesets

* create socket in constructor

* enable strictPropertyInitialization

* deprecate dequeue from version

* start is not async

* dependency injection in prep for tests

* add warm start count to all controller logs

* add restore count

* pull out run execution logic

* temp disable pre

* add a controller log when starting an execution

* refactor execution and squash some bugs

* cleanup completed docker containers by default

* execution fixes and logging improvements

* don't throw afet abort cleanup

* poller should use private interval

* rename heartbeat service file

* rename HeartbeatService to IntervalService

* restore old heartbeat service but deprecate it

* use the new interval service everywhere

* Revert "temp disable pre"

This reverts commit e03f4179de6a731c17253b68a6e00bcb7ac1736b.

* add changeset

* replace all run engine find uniques with find first

* Release 4.0.0-v4-beta.2 (#1928)

* chore: Update version for release (v4-beta)

* Release 4.0.0-v4-beta.2

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>

* Remove batch ID carryover for non-batch waits (#1930)

* add failing test case

* do not carry over previous batch id when blocking with waitpoint

* delete irrelevant test

* Delete project (#1913)

* Delete project

- Don’t schedule tasks if the project is deleted
- Delete queues from the master queues
- Add the old delete project UI back in

* Mark the project as deleted last

* Fix for overriding local variable

* Added a todo for deleting env queues

* Remove todo

* Improve usage flushing (#1931)

* add flush to global usage api

* enable controller debug logs

* initialize usage manager after env overrides

* add previous run id to more debug logs

* add changeset

* For secret env vars, don’t return the value

* Added a new env var repository function for getting secrets with redactions

* Test task for env vars

* Delete heartbeat file, merge mess up

---------

Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Saadi Myftija <saadi.myftija@gmail.com>
2025-04-17 14:39:09 +01:00
Matt Aitken 1b758bd894 Only prevent triggering runs if the parent is completed AND it’s a triggerAndWait (#1935) 2025-04-17 14:25:20 +01:00
Eric Allam e501113a73 fix: refill release concurrency token bucket queue when runs resume before checkpoints are created (#1933)
* fix: refill release concurrency token bucket queue when runs resume before checkpoints are created

* Fix the engine package.json tests

* Remove log
2025-04-16 20:18:38 +01:00
2072 changed files with 292148 additions and 62689 deletions
+16
View File
@@ -0,0 +1,16 @@
---
"@trigger.dev/core": patch
"@trigger.dev/sdk": patch
---
Add `maxDelay` option to debounce feature. This allows setting a maximum time limit for how long a debounced run can be delayed, ensuring execution happens within a specified window even with continuous triggers.
```typescript
await myTask.trigger(payload, {
debounce: {
key: "my-key",
delay: "5s",
maxDelay: "30m", // Execute within 30 minutes regardless of continuous triggers
},
});
```
-10
View File
@@ -1,10 +0,0 @@
---
"@trigger.dev/react-hooks": patch
"@trigger.dev/sdk": patch
"trigger.dev": patch
"@trigger.dev/build": patch
"@trigger.dev/core": patch
"@trigger.dev/rsc": patch
---
Run Engine 2.0 (alpha)
+5
View File
@@ -0,0 +1,5 @@
---
"@trigger.dev/react-hooks": patch
---
Fix `onComplete` callback firing prematurely when the realtime stream disconnects before the run finishes.
+5
View File
@@ -0,0 +1,5 @@
---
"@trigger.dev/sdk": patch
---
Aligned the SDK's `getRunIdForOptions` logic with the Core package to handle semantic targets (`root`, `parent`) in root tasks.
@@ -0,0 +1,5 @@
---
"@trigger.dev/sdk": patch
---
Export `AnyOnStartAttemptHookFunction` type to allow defining `onStartAttempt` hooks for individual tasks.
@@ -0,0 +1,5 @@
---
"trigger.dev": patch
---
Fix runner getting stuck indefinitely when `execute()` is called on a dead child process.
-12
View File
@@ -1,12 +0,0 @@
---
"@trigger.dev/redis-worker": major
"@trigger.dev/react-hooks": major
"@trigger.dev/sdk": major
"trigger.dev": major
"@trigger.dev/python": major
"@trigger.dev/build": major
"@trigger.dev/core": major
"@trigger.dev/rsc": major
---
Trigger.dev v4 release. Please see our upgrade to v4 docs to view the full changelog: https://trigger.dev/docs/upgrade-to-v4
-5
View File
@@ -1,5 +0,0 @@
---
"@trigger.dev/sdk": patch
---
Deprecate toolTask and replace with `ai.tool(mySchemaTask)`
-5
View File
@@ -1,5 +0,0 @@
---
"trigger.dev": patch
---
Fix init.ts in custom trigger dirs
-5
View File
@@ -1,5 +0,0 @@
---
"trigger.dev": patch
---
Init command will now correctly install v4-beta packages
-5
View File
@@ -1,5 +0,0 @@
---
"trigger.dev": patch
---
Improve warm start times by eagerly creating the child TaskRunProcess when a previous run as completed
-5
View File
@@ -1,5 +0,0 @@
---
"trigger.dev": patch
---
Update nypm package to support test-based bun.lock files
-35
View File
@@ -1,35 +0,0 @@
{
"mode": "pre",
"tag": "v4-beta",
"initialVersions": {
"coordinator": "0.0.1",
"docker-provider": "0.0.1",
"kubernetes-provider": "0.0.1",
"supervisor": "0.0.1",
"webapp": "1.0.0",
"@trigger.dev/build": "3.3.17",
"trigger.dev": "3.3.17",
"@trigger.dev/core": "3.3.17",
"@trigger.dev/python": "3.3.17",
"@trigger.dev/react-hooks": "3.3.17",
"@trigger.dev/redis-worker": "3.3.17",
"@trigger.dev/rsc": "3.3.17",
"@trigger.dev/sdk": "3.3.17"
},
"changesets": [
"breezy-turtles-talk",
"four-needles-add",
"honest-files-decide",
"late-chairs-ring",
"moody-squids-count",
"nice-colts-boil",
"polite-lies-fix",
"red-wasps-cover",
"shiny-kiwis-beam",
"smart-coins-hammer",
"sour-mirrors-accept",
"tiny-buckets-teach",
"tricky-houses-invite",
"weak-jobs-hide"
]
}
-5
View File
@@ -1,5 +0,0 @@
---
"@trigger.dev/core": patch
---
Suppress external instrumentation for fetch calls from ApiClient
-5
View File
@@ -1,5 +0,0 @@
---
"trigger.dev": patch
---
Handle flush errors gracefully in dev
-5
View File
@@ -1,5 +0,0 @@
---
"@trigger.dev/core": patch
---
fix: Realtime streams: prevent enqueuing into closed ReadableStream
-6
View File
@@ -1,6 +0,0 @@
---
"trigger.dev": patch
"@trigger.dev/core": patch
---
Improve usage flushing
-5
View File
@@ -1,5 +0,0 @@
---
"trigger.dev": patch
---
Fix stalled run detection
-6
View File
@@ -1,6 +0,0 @@
---
"trigger.dev": patch
"@trigger.dev/core": patch
---
Managed run controller performance and reliability improvements
+7
View File
@@ -0,0 +1,7 @@
---
"@trigger.dev/core": patch
---
fix: vendor superjson to fix ESM/CJS compatibility
Bundle superjson during build to avoid `ERR_REQUIRE_ESM` errors on Node.js versions that don't support `require(ESM)` by default (< 22.12.0) and AWS Lambda which intentionally disables it.
-7
View File
@@ -1,7 +0,0 @@
---
"@trigger.dev/sdk": patch
"trigger.dev": patch
"@trigger.dev/core": patch
---
v4: New lifecycle hooks
+200
View File
@@ -0,0 +1,200 @@
---
name: trigger-dev-tasks
description: Use this skill when writing, designing, or optimizing Trigger.dev background tasks and workflows. This includes creating reliable async tasks, implementing AI workflows, setting up scheduled jobs, structuring complex task hierarchies with subtasks, configuring build extensions for tools like ffmpeg or Puppeteer/Playwright, and handling task schemas with Zod validation.
allowed-tools: Read, Write, Edit, Glob, Grep, Bash
---
# Trigger.dev Task Expert
You are an expert Trigger.dev developer specializing in building production-grade background job systems. Tasks deployed to Trigger.dev run in Node.js 21+ and use the `@trigger.dev/sdk` package.
## Critical Rules
1. **Always use `@trigger.dev/sdk`** - Never use `@trigger.dev/sdk/v3` or deprecated `client.defineJob` pattern
2. **Never use `node-fetch`** - Use the built-in `fetch` function
3. **Export all tasks** - Every task must be exported, including subtasks
4. **Never wrap wait/trigger calls in Promise.all** - `triggerAndWait`, `batchTriggerAndWait`, and `wait.*` calls cannot be wrapped in `Promise.all` or `Promise.allSettled`
## Basic Task Pattern
```ts
import { task } from "@trigger.dev/sdk";
export const processData = task({
id: "process-data",
retry: {
maxAttempts: 10,
factor: 1.8,
minTimeoutInMs: 500,
maxTimeoutInMs: 30_000,
},
run: async (payload: { userId: string; data: any[] }) => {
console.log(`Processing ${payload.data.length} items`);
return { processed: payload.data.length };
},
});
```
## Schema Task (with validation)
```ts
import { schemaTask } from "@trigger.dev/sdk";
import { z } from "zod";
export const validatedTask = schemaTask({
id: "validated-task",
schema: z.object({
name: z.string(),
email: z.string().email(),
}),
run: async (payload) => {
// Payload is automatically validated and typed
return { message: `Hello ${payload.name}` };
},
});
```
## Triggering Tasks
### From Backend Code (type-only import to prevent dependency leakage)
```ts
import { tasks } from "@trigger.dev/sdk";
import type { processData } from "./trigger/tasks";
const handle = await tasks.trigger<typeof processData>("process-data", {
userId: "123",
data: [{ id: 1 }],
});
```
### From Inside Tasks
```ts
export const parentTask = task({
id: "parent-task",
run: async (payload) => {
// Trigger and wait - returns Result object, NOT direct output
const result = await childTask.triggerAndWait({ data: "value" });
if (result.ok) {
console.log("Output:", result.output);
} else {
console.error("Failed:", result.error);
}
// Or unwrap directly (throws on error)
const output = await childTask.triggerAndWait({ data: "value" }).unwrap();
},
});
```
## Idempotency (Critical for Retries)
Always use idempotency keys when triggering tasks from inside other tasks:
```ts
import { idempotencyKeys } from "@trigger.dev/sdk";
export const paymentTask = task({
id: "process-payment",
run: async (payload: { orderId: string }) => {
// Scoped to current run - survives retries
const key = await idempotencyKeys.create(`payment-${payload.orderId}`);
await chargeCustomer.trigger(payload, {
idempotencyKey: key,
idempotencyKeyTTL: "24h",
});
},
});
```
## Trigger Options
```ts
await myTask.trigger(payload, {
delay: "1h", // Delay execution
ttl: "10m", // Cancel if not started within TTL
idempotencyKey: key,
queue: "my-queue",
machine: "large-1x", // micro, small-1x, small-2x, medium-1x, medium-2x, large-1x, large-2x
maxAttempts: 3,
tags: ["user_123"], // Max 10 tags
debounce: { // Consolidate rapid triggers
key: "unique-key",
delay: "5s",
mode: "trailing", // "leading" (default) or "trailing"
},
});
```
## Debouncing
Consolidate multiple triggers into a single execution:
```ts
// Rapid triggers with same key = single execution
await myTask.trigger({ userId: "123" }, {
debounce: {
key: "user-123-update",
delay: "5s",
},
});
// Trailing mode: use payload from LAST trigger
await myTask.trigger({ data: "latest" }, {
debounce: {
key: "my-key",
delay: "10s",
mode: "trailing",
},
});
```
Use cases: user activity updates, webhook deduplication, search indexing, notification batching.
## Batch Triggering
Up to 1,000 items per batch, 3MB per payload:
```ts
const results = await myTask.batchTriggerAndWait([
{ payload: { userId: "1" } },
{ payload: { userId: "2" } },
]);
for (const result of results) {
if (result.ok) console.log(result.output);
}
```
## Machine Presets
| Preset | vCPU | Memory |
|-------------|------|--------|
| micro | 0.25 | 0.25GB |
| small-1x | 0.5 | 0.5GB |
| small-2x | 1 | 1GB |
| medium-1x | 1 | 2GB |
| medium-2x | 2 | 4GB |
| large-1x | 4 | 8GB |
| large-2x | 8 | 16GB |
## Design Principles
1. **Break complex workflows into subtasks** that can be independently retried and made idempotent
2. **Don't over-complicate** - Sometimes `Promise.allSettled` inside a single task is better than many subtasks (each task has dedicated process and is charged by millisecond)
3. **Always configure retries** - Set appropriate `maxAttempts` based on the operation
4. **Use idempotency keys** - Especially for payment/critical operations
5. **Group related subtasks** - Keep subtasks only used by one parent in the same file, don't export them
6. **Use logger** - Log at key execution points with `logger.info()`, `logger.error()`, etc.
## Reference Documentation
For detailed documentation on specific topics, read these files:
- `basic-tasks.md` - Task basics, triggering, waits
- `advanced-tasks.md` - Tags, queues, concurrency, metadata, error handling
- `scheduled-tasks.md` - Cron schedules, declarative and imperative
- `realtime.md` - Real-time subscriptions, streams, React hooks
- `config.md` - trigger.config.ts, build extensions (Prisma, Playwright, FFmpeg, etc.)
@@ -0,0 +1,485 @@
# Trigger.dev Advanced Tasks (v4)
**Advanced patterns and features for writing tasks**
## Tags & Organization
```ts
import { task, tags } from "@trigger.dev/sdk";
export const processUser = task({
id: "process-user",
run: async (payload: { userId: string; orgId: string }, { ctx }) => {
// Add tags during execution
await tags.add(`user_${payload.userId}`);
await tags.add(`org_${payload.orgId}`);
return { processed: true };
},
});
// Trigger with tags
await processUser.trigger(
{ userId: "123", orgId: "abc" },
{ tags: ["priority", "user_123", "org_abc"] } // Max 10 tags per run
);
// Subscribe to tagged runs
for await (const run of runs.subscribeToRunsWithTag("user_123")) {
console.log(`User task ${run.id}: ${run.status}`);
}
```
**Tag Best Practices:**
- Use prefixes: `user_123`, `org_abc`, `video:456`
- Max 10 tags per run, 1-64 characters each
- Tags don't propagate to child tasks automatically
## Batch Triggering v2
Enhanced batch triggering with larger payloads and streaming ingestion.
### Limits
- **Maximum batch size**: 1,000 items (increased from 500)
- **Payload per item**: 3MB each (increased from 1MB combined)
- Payloads > 512KB automatically offload to object storage
### Rate Limiting (per environment)
| Tier | Bucket Size | Refill Rate |
|------|-------------|-------------|
| Free | 1,200 runs | 100 runs/10 sec |
| Hobby | 5,000 runs | 500 runs/5 sec |
| Pro | 5,000 runs | 500 runs/5 sec |
### Concurrent Batch Processing
| Tier | Concurrent Batches |
|------|-------------------|
| Free | 1 |
| Hobby | 10 |
| Pro | 10 |
### Usage
```ts
import { myTask } from "./trigger/myTask";
// Basic batch trigger (up to 1,000 items)
const runs = await myTask.batchTrigger([
{ payload: { userId: "user-1" } },
{ payload: { userId: "user-2" } },
{ payload: { userId: "user-3" } },
]);
// Batch trigger with wait
const results = await myTask.batchTriggerAndWait([
{ payload: { userId: "user-1" } },
{ payload: { userId: "user-2" } },
]);
for (const result of results) {
if (result.ok) {
console.log("Result:", result.output);
}
}
// With per-item options
const batchHandle = await myTask.batchTrigger([
{
payload: { userId: "123" },
options: {
idempotencyKey: "user-123-batch",
tags: ["priority"],
},
},
{
payload: { userId: "456" },
options: {
idempotencyKey: "user-456-batch",
},
},
]);
```
## Debouncing
Consolidate multiple triggers into a single execution by debouncing task runs with a unique key and delay window.
### Use Cases
- **User activity updates**: Batch rapid user actions into a single run
- **Webhook deduplication**: Handle webhook bursts without redundant processing
- **Search indexing**: Combine document updates instead of processing individually
- **Notification batching**: Group notifications to prevent user spam
### Basic Usage
```ts
await myTask.trigger(
{ userId: "123" },
{
debounce: {
key: "user-123-update", // Unique identifier for debounce group
delay: "5s", // Wait duration ("5s", "1m", or milliseconds)
},
}
);
```
### Execution Modes
**Leading Mode** (default): Uses payload/options from the first trigger; subsequent triggers only reschedule execution time.
```ts
// First trigger sets the payload
await myTask.trigger({ action: "first" }, {
debounce: { key: "my-key", delay: "10s" }
});
// Second trigger only reschedules - payload remains "first"
await myTask.trigger({ action: "second" }, {
debounce: { key: "my-key", delay: "10s" }
});
// Task executes with { action: "first" }
```
**Trailing Mode**: Uses payload/options from the most recent trigger.
```ts
await myTask.trigger(
{ data: "latest-value" },
{
debounce: {
key: "trailing-example",
delay: "10s",
mode: "trailing",
},
}
);
```
In trailing mode, these options update with each trigger:
- `payload` — task input data
- `metadata` — run metadata
- `tags` — run tags (replaces existing)
- `maxAttempts` — retry attempts
- `maxDuration` — maximum compute time
- `machine` — machine preset
### Important Notes
- Idempotency keys take precedence over debounce settings
- Compatible with `triggerAndWait()` — parent runs block correctly on debounced execution
- Debounce key is scoped to the task
## Concurrency & Queues
```ts
import { task, queue } from "@trigger.dev/sdk";
// Shared queue for related tasks
const emailQueue = queue({
name: "email-processing",
concurrencyLimit: 5, // Max 5 emails processing simultaneously
});
// Task-level concurrency
export const oneAtATime = task({
id: "sequential-task",
queue: { concurrencyLimit: 1 }, // Process one at a time
run: async (payload) => {
// Critical section - only one instance runs
},
});
// Per-user concurrency
export const processUserData = task({
id: "process-user-data",
run: async (payload: { userId: string }) => {
// Override queue with user-specific concurrency
await childTask.trigger(payload, {
queue: {
name: `user-${payload.userId}`,
concurrencyLimit: 2,
},
});
},
});
export const emailTask = task({
id: "send-email",
queue: emailQueue, // Use shared queue
run: async (payload: { to: string }) => {
// Send email logic
},
});
```
## Error Handling & Retries
```ts
import { task, retry, AbortTaskRunError } from "@trigger.dev/sdk";
export const resilientTask = task({
id: "resilient-task",
retry: {
maxAttempts: 10,
factor: 1.8, // Exponential backoff multiplier
minTimeoutInMs: 500,
maxTimeoutInMs: 30_000,
randomize: false,
},
catchError: async ({ error, ctx }) => {
// Custom error handling
if (error.code === "FATAL_ERROR") {
throw new AbortTaskRunError("Cannot retry this error");
}
// Log error details
console.error(`Task ${ctx.task.id} failed:`, error);
// Allow retry by returning nothing
return { retryAt: new Date(Date.now() + 60000) }; // Retry in 1 minute
},
run: async (payload) => {
// Retry specific operations
const result = await retry.onThrow(
async () => {
return await unstableApiCall(payload);
},
{ maxAttempts: 3 }
);
// Conditional HTTP retries
const response = await retry.fetch("https://api.example.com", {
retry: {
maxAttempts: 5,
condition: (response, error) => {
return response?.status === 429 || response?.status >= 500;
},
},
});
return result;
},
});
```
## Machines & Performance
```ts
export const heavyTask = task({
id: "heavy-computation",
machine: { preset: "large-2x" }, // 8 vCPU, 16 GB RAM
maxDuration: 1800, // 30 minutes timeout
run: async (payload, { ctx }) => {
// Resource-intensive computation
if (ctx.machine.preset === "large-2x") {
// Use all available cores
return await parallelProcessing(payload);
}
return await standardProcessing(payload);
},
});
// Override machine when triggering
await heavyTask.trigger(payload, {
machine: { preset: "medium-1x" }, // Override for this run
});
```
**Machine Presets:**
- `micro`: 0.25 vCPU, 0.25 GB RAM
- `small-1x`: 0.5 vCPU, 0.5 GB RAM (default)
- `small-2x`: 1 vCPU, 1 GB RAM
- `medium-1x`: 1 vCPU, 2 GB RAM
- `medium-2x`: 2 vCPU, 4 GB RAM
- `large-1x`: 4 vCPU, 8 GB RAM
- `large-2x`: 8 vCPU, 16 GB RAM
## Idempotency
```ts
import { task, idempotencyKeys } from "@trigger.dev/sdk";
export const paymentTask = task({
id: "process-payment",
retry: {
maxAttempts: 3,
},
run: async (payload: { orderId: string; amount: number }) => {
// Automatically scoped to this task run, so if the task is retried, the idempotency key will be the same
const idempotencyKey = await idempotencyKeys.create(`payment-${payload.orderId}`);
// Ensure payment is processed only once
await chargeCustomer.trigger(payload, {
idempotencyKey,
idempotencyKeyTTL: "24h", // Key expires in 24 hours
});
},
});
// Payload-based idempotency
import { createHash } from "node:crypto";
function createPayloadHash(payload: any): string {
const hash = createHash("sha256");
hash.update(JSON.stringify(payload));
return hash.digest("hex");
}
export const deduplicatedTask = task({
id: "deduplicated-task",
run: async (payload) => {
const payloadHash = createPayloadHash(payload);
const idempotencyKey = await idempotencyKeys.create(payloadHash);
await processData.trigger(payload, { idempotencyKey });
},
});
```
## Metadata & Progress Tracking
```ts
import { task, metadata } from "@trigger.dev/sdk";
export const batchProcessor = task({
id: "batch-processor",
run: async (payload: { items: any[] }, { ctx }) => {
const totalItems = payload.items.length;
// Initialize progress metadata
metadata
.set("progress", 0)
.set("totalItems", totalItems)
.set("processedItems", 0)
.set("status", "starting");
const results = [];
for (let i = 0; i < payload.items.length; i++) {
const item = payload.items[i];
// Process item
const result = await processItem(item);
results.push(result);
// Update progress
const progress = ((i + 1) / totalItems) * 100;
metadata
.set("progress", progress)
.increment("processedItems", 1)
.append("logs", `Processed item ${i + 1}/${totalItems}`)
.set("currentItem", item.id);
}
// Final status
metadata.set("status", "completed");
return { results, totalProcessed: results.length };
},
});
// Update parent metadata from child task
export const childTask = task({
id: "child-task",
run: async (payload, { ctx }) => {
// Update parent task metadata
metadata.parent.set("childStatus", "processing");
metadata.root.increment("childrenCompleted", 1);
return { processed: true };
},
});
```
## Logging & Tracing
```ts
import { task, logger } from "@trigger.dev/sdk";
export const tracedTask = task({
id: "traced-task",
run: async (payload, { ctx }) => {
logger.info("Task started", { userId: payload.userId });
// Custom trace with attributes
const user = await logger.trace(
"fetch-user",
async (span) => {
span.setAttribute("user.id", payload.userId);
span.setAttribute("operation", "database-fetch");
const userData = await database.findUser(payload.userId);
span.setAttribute("user.found", !!userData);
return userData;
},
{ userId: payload.userId }
);
logger.debug("User fetched", { user: user.id });
try {
const result = await processUser(user);
logger.info("Processing completed", { result });
return result;
} catch (error) {
logger.error("Processing failed", {
error: error.message,
userId: payload.userId,
});
throw error;
}
},
});
```
## Hidden Tasks
```ts
// Hidden task - not exported, only used internally
const internalProcessor = task({
id: "internal-processor",
run: async (payload: { data: string }) => {
return { processed: payload.data.toUpperCase() };
},
});
// Public task that uses hidden task
export const publicWorkflow = task({
id: "public-workflow",
run: async (payload: { input: string }) => {
// Use hidden task internally
const result = await internalProcessor.triggerAndWait({
data: payload.input,
});
if (result.ok) {
return { output: result.output.processed };
}
throw new Error("Internal processing failed");
},
});
```
## Best Practices
- **Concurrency**: Use queues to prevent overwhelming external services
- **Retries**: Configure exponential backoff for transient failures
- **Idempotency**: Always use for payment/critical operations
- **Metadata**: Track progress for long-running tasks
- **Machines**: Match machine size to computational requirements
- **Tags**: Use consistent naming patterns for filtering
- **Debouncing**: Use for user activity, webhooks, and notification batching
- **Batch triggering**: Use for bulk operations up to 1,000 items
- **Error Handling**: Distinguish between retryable and fatal errors
Design tasks to be stateless, idempotent, and resilient to failures. Use metadata for state tracking and queues for resource management.
@@ -0,0 +1,199 @@
# Trigger.dev Basic Tasks (v4)
**MUST use `@trigger.dev/sdk`, NEVER `client.defineJob`**
## Basic Task
```ts
import { task } from "@trigger.dev/sdk";
export const processData = task({
id: "process-data",
retry: {
maxAttempts: 10,
factor: 1.8,
minTimeoutInMs: 500,
maxTimeoutInMs: 30_000,
randomize: false,
},
run: async (payload: { userId: string; data: any[] }) => {
// Task logic - runs for long time, no timeouts
console.log(`Processing ${payload.data.length} items for user ${payload.userId}`);
return { processed: payload.data.length };
},
});
```
## Schema Task (with validation)
```ts
import { schemaTask } from "@trigger.dev/sdk";
import { z } from "zod";
export const validatedTask = schemaTask({
id: "validated-task",
schema: z.object({
name: z.string(),
age: z.number(),
email: z.string().email(),
}),
run: async (payload) => {
// Payload is automatically validated and typed
return { message: `Hello ${payload.name}, age ${payload.age}` };
},
});
```
## Triggering Tasks
### From Backend Code
```ts
import { tasks } from "@trigger.dev/sdk";
import type { processData } from "./trigger/tasks";
// Single trigger
const handle = await tasks.trigger<typeof processData>("process-data", {
userId: "123",
data: [{ id: 1 }, { id: 2 }],
});
// Batch trigger (up to 1,000 items, 3MB per payload)
const batchHandle = await tasks.batchTrigger<typeof processData>("process-data", [
{ payload: { userId: "123", data: [{ id: 1 }] } },
{ payload: { userId: "456", data: [{ id: 2 }] } },
]);
```
### Debounced Triggering
Consolidate multiple triggers into a single execution:
```ts
// Multiple rapid triggers with same key = single execution
await myTask.trigger(
{ userId: "123" },
{
debounce: {
key: "user-123-update", // Unique key for debounce group
delay: "5s", // Wait before executing
},
}
);
// Trailing mode: use payload from LAST trigger
await myTask.trigger(
{ data: "latest-value" },
{
debounce: {
key: "trailing-example",
delay: "10s",
mode: "trailing", // Default is "leading" (first payload)
},
}
);
```
**Debounce modes:**
- `leading` (default): Uses payload from first trigger, subsequent triggers only reschedule
- `trailing`: Uses payload from most recent trigger
### From Inside Tasks (with Result handling)
```ts
export const parentTask = task({
id: "parent-task",
run: async (payload) => {
// Trigger and continue
const handle = await childTask.trigger({ data: "value" });
// Trigger and wait - returns Result object, NOT task output
const result = await childTask.triggerAndWait({ data: "value" });
if (result.ok) {
console.log("Task output:", result.output); // Actual task return value
} else {
console.error("Task failed:", result.error);
}
// Quick unwrap (throws on error)
const output = await childTask.triggerAndWait({ data: "value" }).unwrap();
// Batch trigger and wait
const results = await childTask.batchTriggerAndWait([
{ payload: { data: "item1" } },
{ payload: { data: "item2" } },
]);
for (const run of results) {
if (run.ok) {
console.log("Success:", run.output);
} else {
console.log("Failed:", run.error);
}
}
},
});
export const childTask = task({
id: "child-task",
run: async (payload: { data: string }) => {
return { processed: payload.data };
},
});
```
> Never wrap triggerAndWait or batchTriggerAndWait calls in a Promise.all or Promise.allSettled as this is not supported in Trigger.dev tasks.
## Waits
```ts
import { task, wait } from "@trigger.dev/sdk";
export const taskWithWaits = task({
id: "task-with-waits",
run: async (payload) => {
console.log("Starting task");
// Wait for specific duration
await wait.for({ seconds: 30 });
await wait.for({ minutes: 5 });
await wait.for({ hours: 1 });
await wait.for({ days: 1 });
// Wait until specific date
await wait.until({ date: new Date("2024-12-25") });
// Wait for token (from external system)
await wait.forToken({
token: "user-approval-token",
timeoutInSeconds: 3600, // 1 hour timeout
});
console.log("All waits completed");
return { status: "completed" };
},
});
```
> Never wrap wait calls in a Promise.all or Promise.allSettled as this is not supported in Trigger.dev tasks.
## Key Points
- **Result vs Output**: `triggerAndWait()` returns a `Result` object with `ok`, `output`, `error` properties - NOT the direct task output
- **Type safety**: Use `import type` for task references when triggering from backend
- **Waits > 5 seconds**: Automatically checkpointed, don't count toward compute usage
- **Debounce + idempotency**: Idempotency keys take precedence over debounce settings
## NEVER Use (v2 deprecated)
```ts
// BREAKS APPLICATION
client.defineJob({
id: "job-id",
run: async (payload, io) => {
/* ... */
},
});
```
Use SDK (`@trigger.dev/sdk`), check `result.ok` before accessing `result.output`
+346
View File
@@ -0,0 +1,346 @@
# Trigger.dev Configuration
**Complete guide to configuring `trigger.config.ts` with build extensions**
## Basic Configuration
```ts
import { defineConfig } from "@trigger.dev/sdk";
export default defineConfig({
project: "<project-ref>", // Required: Your project reference
dirs: ["./trigger"], // Task directories
runtime: "node", // "node", "node-22", or "bun"
logLevel: "info", // "debug", "info", "warn", "error"
// Default retry settings
retries: {
enabledInDev: false,
default: {
maxAttempts: 3,
minTimeoutInMs: 1000,
maxTimeoutInMs: 10000,
factor: 2,
randomize: true,
},
},
// Build configuration
build: {
autoDetectExternal: true,
keepNames: true,
minify: false,
extensions: [], // Build extensions go here
},
// Global lifecycle hooks
onStartAttempt: async ({ payload, ctx }) => {
console.log("Global task start");
},
onSuccess: async ({ payload, output, ctx }) => {
console.log("Global task success");
},
onFailure: async ({ payload, error, ctx }) => {
console.log("Global task failure");
},
});
```
## Build Extensions
### Database & ORM
#### Prisma
```ts
import { prismaExtension } from "@trigger.dev/build/extensions/prisma";
extensions: [
prismaExtension({
schema: "prisma/schema.prisma",
version: "5.19.0", // Optional: specify version
migrate: true, // Run migrations during build
directUrlEnvVarName: "DIRECT_DATABASE_URL",
typedSql: true, // Enable TypedSQL support
}),
];
```
#### TypeScript Decorators (for TypeORM)
```ts
import { emitDecoratorMetadata } from "@trigger.dev/build/extensions/typescript";
extensions: [
emitDecoratorMetadata(), // Enables decorator metadata
];
```
### Scripting Languages
#### Python
```ts
import { pythonExtension } from "@trigger.dev/build/extensions/python";
extensions: [
pythonExtension({
scripts: ["./python/**/*.py"], // Copy Python files
requirementsFile: "./requirements.txt", // Install packages
devPythonBinaryPath: ".venv/bin/python", // Dev mode binary
}),
];
// Usage in tasks
const result = await python.runInline(`print("Hello, world!")`);
const output = await python.runScript("./python/script.py", ["arg1"]);
```
### Browser Automation
#### Playwright
```ts
import { playwright } from "@trigger.dev/build/extensions/playwright";
extensions: [
playwright({
browsers: ["chromium", "firefox", "webkit"], // Default: ["chromium"]
headless: true, // Default: true
}),
];
```
#### Puppeteer
```ts
import { puppeteer } from "@trigger.dev/build/extensions/puppeteer";
extensions: [puppeteer()];
// Environment variable needed:
// PUPPETEER_EXECUTABLE_PATH: "/usr/bin/google-chrome-stable"
```
#### Lightpanda
```ts
import { lightpanda } from "@trigger.dev/build/extensions/lightpanda";
extensions: [
lightpanda({
version: "latest", // or "nightly"
disableTelemetry: false,
}),
];
```
### Media Processing
#### FFmpeg
```ts
import { ffmpeg } from "@trigger.dev/build/extensions/core";
extensions: [
ffmpeg({ version: "7" }), // Static build, or omit for Debian version
];
// Automatically sets FFMPEG_PATH and FFPROBE_PATH
// Add fluent-ffmpeg to external packages if using
```
#### Audio Waveform
```ts
import { audioWaveform } from "@trigger.dev/build/extensions/audioWaveform";
extensions: [
audioWaveform(), // Installs Audio Waveform 1.1.0
];
```
### System & Package Management
#### System Packages (apt-get)
```ts
import { aptGet } from "@trigger.dev/build/extensions/core";
extensions: [
aptGet({
packages: ["ffmpeg", "imagemagick", "curl=7.68.0-1"], // Can specify versions
}),
];
```
#### Additional NPM Packages
Only use this for installing CLI tools, NOT packages you import in your code.
```ts
import { additionalPackages } from "@trigger.dev/build/extensions/core";
extensions: [
additionalPackages({
packages: ["wrangler"], // CLI tools and specific versions
}),
];
```
#### Additional Files
```ts
import { additionalFiles } from "@trigger.dev/build/extensions/core";
extensions: [
additionalFiles({
files: ["wrangler.toml", "./assets/**", "./fonts/**"], // Glob patterns supported
}),
];
```
### Environment & Build Tools
#### Environment Variable Sync
```ts
import { syncEnvVars } from "@trigger.dev/build/extensions/core";
extensions: [
syncEnvVars(async (ctx) => {
// ctx contains: environment, projectRef, env
return [
{ name: "SECRET_KEY", value: await getSecret(ctx.environment) },
{ name: "API_URL", value: ctx.environment === "prod" ? "api.prod.com" : "api.dev.com" },
];
}),
];
```
#### ESBuild Plugins
```ts
import { esbuildPlugin } from "@trigger.dev/build/extensions";
import { sentryEsbuildPlugin } from "@sentry/esbuild-plugin";
extensions: [
esbuildPlugin(
sentryEsbuildPlugin({
org: process.env.SENTRY_ORG,
project: process.env.SENTRY_PROJECT,
authToken: process.env.SENTRY_AUTH_TOKEN,
}),
{ placement: "last", target: "deploy" } // Optional config
),
];
```
## Custom Build Extensions
```ts
import { defineConfig } from "@trigger.dev/sdk";
const customExtension = {
name: "my-custom-extension",
externalsForTarget: (target) => {
return ["some-native-module"]; // Add external dependencies
},
onBuildStart: async (context) => {
console.log(`Build starting for ${context.target}`);
// Register esbuild plugins, modify build context
},
onBuildComplete: async (context, manifest) => {
console.log("Build complete, adding layers");
// Add build layers, modify deployment
context.addLayer({
id: "my-layer",
files: [{ source: "./custom-file", destination: "/app/custom" }],
commands: ["chmod +x /app/custom"],
});
},
};
export default defineConfig({
project: "my-project",
build: {
extensions: [customExtension],
},
});
```
## Advanced Configuration
### Telemetry
```ts
import { PrismaInstrumentation } from "@prisma/instrumentation";
import { OpenAIInstrumentation } from "@langfuse/openai";
export default defineConfig({
// ... other config
telemetry: {
instrumentations: [new PrismaInstrumentation(), new OpenAIInstrumentation()],
exporters: [customExporter], // Optional custom exporters
},
});
```
### Machine & Performance
```ts
export default defineConfig({
// ... other config
defaultMachine: "large-1x", // Default machine for all tasks
maxDuration: 300, // Default max duration (seconds)
enableConsoleLogging: true, // Console logging in development
});
```
## Common Extension Combinations
### Full-Stack Web App
```ts
extensions: [
prismaExtension({ schema: "prisma/schema.prisma", migrate: true }),
additionalFiles({ files: ["./public/**", "./assets/**"] }),
syncEnvVars(async (ctx) => [...envVars]),
];
```
### AI/ML Processing
```ts
extensions: [
pythonExtension({
scripts: ["./ai/**/*.py"],
requirementsFile: "./requirements.txt",
}),
ffmpeg({ version: "7" }),
additionalPackages({ packages: ["wrangler"] }),
];
```
### Web Scraping
```ts
extensions: [
playwright({ browsers: ["chromium"] }),
puppeteer(),
additionalFiles({ files: ["./selectors.json", "./proxies.txt"] }),
];
```
## Best Practices
- **Use specific versions**: Pin extension versions for reproducible builds
- **External packages**: Add modules with native addons to the `build.external` array
- **Environment sync**: Use `syncEnvVars` for dynamic secrets
- **File paths**: Use glob patterns for flexible file inclusion
- **Debug builds**: Use `--log-level debug --dry-run` for troubleshooting
Extensions only affect deployment, not local development. Use `external` array for packages that shouldn't be bundled.
@@ -0,0 +1,244 @@
# Trigger.dev Realtime
**Real-time monitoring and updates for runs**
## Core Concepts
Realtime allows you to:
- Subscribe to run status changes, metadata updates, and streams
- Build real-time dashboards and UI updates
- Monitor task progress from frontend and backend
## Authentication
### Public Access Tokens
```ts
import { auth } from "@trigger.dev/sdk";
// Read-only token for specific runs
const publicToken = await auth.createPublicToken({
scopes: {
read: {
runs: ["run_123", "run_456"],
tasks: ["my-task-1", "my-task-2"],
},
},
expirationTime: "1h", // Default: 15 minutes
});
```
### Trigger Tokens (Frontend only)
```ts
// Single-use token for triggering tasks
const triggerToken = await auth.createTriggerPublicToken("my-task", {
expirationTime: "30m",
});
```
## Backend Usage
### Subscribe to Runs
```ts
import { runs, tasks } from "@trigger.dev/sdk";
// Trigger and subscribe
const handle = await tasks.trigger("my-task", { data: "value" });
// Subscribe to specific run
for await (const run of runs.subscribeToRun<typeof myTask>(handle.id)) {
console.log(`Status: ${run.status}, Progress: ${run.metadata?.progress}`);
if (run.status === "COMPLETED") break;
}
// Subscribe to runs with tag
for await (const run of runs.subscribeToRunsWithTag("user-123")) {
console.log(`Tagged run ${run.id}: ${run.status}`);
}
// Subscribe to batch
for await (const run of runs.subscribeToBatch(batchId)) {
console.log(`Batch run ${run.id}: ${run.status}`);
}
```
### Realtime Streams v2
```ts
import { streams, InferStreamType } from "@trigger.dev/sdk";
// 1. Define streams (shared location)
export const aiStream = streams.define<string>({
id: "ai-output",
});
export type AIStreamPart = InferStreamType<typeof aiStream>;
// 2. Pipe from task
export const streamingTask = task({
id: "streaming-task",
run: async (payload) => {
const completion = await openai.chat.completions.create({
model: "gpt-4",
messages: [{ role: "user", content: payload.prompt }],
stream: true,
});
const { waitUntilComplete } = aiStream.pipe(completion);
await waitUntilComplete();
},
});
// 3. Read from backend
const stream = await aiStream.read(runId, {
timeoutInSeconds: 300,
startIndex: 0, // Resume from specific chunk
});
for await (const chunk of stream) {
console.log("Chunk:", chunk); // Fully typed
}
```
## React Frontend Usage
### Installation
```bash
npm add @trigger.dev/react-hooks
```
### Triggering Tasks
```tsx
"use client";
import { useTaskTrigger, useRealtimeTaskTrigger } from "@trigger.dev/react-hooks";
import type { myTask } from "../trigger/tasks";
function TriggerComponent({ accessToken }: { accessToken: string }) {
// Basic trigger
const { submit, handle, isLoading } = useTaskTrigger<typeof myTask>("my-task", {
accessToken,
});
// Trigger with realtime updates
const {
submit: realtimeSubmit,
run,
isLoading: isRealtimeLoading,
} = useRealtimeTaskTrigger<typeof myTask>("my-task", { accessToken });
return (
<div>
<button onClick={() => submit({ data: "value" })} disabled={isLoading}>
Trigger Task
</button>
<button onClick={() => realtimeSubmit({ data: "realtime" })} disabled={isRealtimeLoading}>
Trigger with Realtime
</button>
{run && <div>Status: {run.status}</div>}
</div>
);
}
```
### Subscribing to Runs
```tsx
"use client";
import { useRealtimeRun, useRealtimeRunsWithTag } from "@trigger.dev/react-hooks";
import type { myTask } from "../trigger/tasks";
function SubscribeComponent({ runId, accessToken }: { runId: string; accessToken: string }) {
// Subscribe to specific run
const { run, error } = useRealtimeRun<typeof myTask>(runId, {
accessToken,
onComplete: (run) => {
console.log("Task completed:", run.output);
},
});
// Subscribe to tagged runs
const { runs } = useRealtimeRunsWithTag("user-123", { accessToken });
if (error) return <div>Error: {error.message}</div>;
if (!run) return <div>Loading...</div>;
return (
<div>
<div>Status: {run.status}</div>
<div>Progress: {run.metadata?.progress || 0}%</div>
{run.output && <div>Result: {JSON.stringify(run.output)}</div>}
<h3>Tagged Runs:</h3>
{runs.map((r) => (
<div key={r.id}>
{r.id}: {r.status}
</div>
))}
</div>
);
}
```
### Realtime Streams with React
```tsx
"use client";
import { useRealtimeStream } from "@trigger.dev/react-hooks";
import { aiStream } from "../trigger/streams";
function StreamComponent({ runId, accessToken }: { runId: string; accessToken: string }) {
// Pass defined stream directly for type safety
const { parts, error } = useRealtimeStream(aiStream, runId, {
accessToken,
timeoutInSeconds: 300,
throttleInMs: 50, // Control re-render frequency
});
if (error) return <div>Error: {error.message}</div>;
if (!parts) return <div>Loading...</div>;
const text = parts.join(""); // parts is typed as AIStreamPart[]
return <div>Streamed Text: {text}</div>;
}
```
### Wait Tokens
```tsx
"use client";
import { useWaitToken } from "@trigger.dev/react-hooks";
function WaitTokenComponent({ tokenId, accessToken }: { tokenId: string; accessToken: string }) {
const { complete } = useWaitToken(tokenId, { accessToken });
return <button onClick={() => complete({ approved: true })}>Approve Task</button>;
}
```
## Run Object Properties
Key properties available in run subscriptions:
- `id`: Unique run identifier
- `status`: `QUEUED`, `EXECUTING`, `COMPLETED`, `FAILED`, `CANCELED`, etc.
- `payload`: Task input data (typed)
- `output`: Task result (typed, when completed)
- `metadata`: Real-time updatable data
- `createdAt`, `updatedAt`: Timestamps
- `costInCents`: Execution cost
## Best Practices
- **Use Realtime over SWR**: Recommended for most use cases due to rate limits
- **Scope tokens properly**: Only grant necessary read/trigger permissions
- **Handle errors**: Always check for errors in hooks and subscriptions
- **Type safety**: Use task types for proper payload/output typing
- **Cleanup subscriptions**: Backend subscriptions auto-complete, frontend hooks auto-cleanup
@@ -0,0 +1,113 @@
# Scheduled Tasks (Cron)
Recurring tasks using cron. For one-off future runs, use the **delay** option.
## Define a Scheduled Task
```ts
import { schedules } from "@trigger.dev/sdk";
export const task = schedules.task({
id: "first-scheduled-task",
run: async (payload) => {
payload.timestamp; // Date (scheduled time, UTC)
payload.lastTimestamp; // Date | undefined
payload.timezone; // IANA, e.g. "America/New_York" (default "UTC")
payload.scheduleId; // string
payload.externalId; // string | undefined
payload.upcoming; // Date[]
payload.timestamp.toLocaleString("en-US", { timeZone: payload.timezone });
},
});
```
> Scheduled tasks need at least one schedule attached to run.
## Attach Schedules
**Declarative (sync on dev/deploy):**
```ts
schedules.task({
id: "every-2h",
cron: "0 */2 * * *", // UTC
run: async () => {},
});
schedules.task({
id: "tokyo-5am",
cron: { pattern: "0 5 * * *", timezone: "Asia/Tokyo", environments: ["PRODUCTION", "STAGING"] },
run: async () => {},
});
```
**Imperative (SDK or dashboard):**
```ts
await schedules.create({
task: task.id,
cron: "0 0 * * *",
timezone: "America/New_York", // DST-aware
externalId: "user_123",
deduplicationKey: "user_123-daily", // updates if reused
});
```
### Dynamic / Multi-tenant Example
```ts
// /trigger/reminder.ts
export const reminderTask = schedules.task({
id: "todo-reminder",
run: async (p) => {
if (!p.externalId) throw new Error("externalId is required");
const user = await db.getUser(p.externalId);
await sendReminderEmail(user);
},
});
```
```ts
// app/reminders/route.ts
export async function POST(req: Request) {
const data = await req.json();
return Response.json(
await schedules.create({
task: reminderTask.id,
cron: "0 8 * * *",
timezone: data.timezone,
externalId: data.userId,
deduplicationKey: `${data.userId}-reminder`,
})
);
}
```
## Cron Syntax (no seconds)
```
* * * * *
| | | | └ day of week (07 or 1L7L; 0/7=Sun; L=last)
| | | └── month (112)
| | └──── day of month (131 or L)
| └────── hour (023)
└──────── minute (059)
```
## When Schedules Won't Trigger
- **Dev:** only when the dev CLI is running.
- **Staging/Production:** only for tasks in the **latest deployment**.
## SDK Management
```ts
await schedules.retrieve(id);
await schedules.list();
await schedules.update(id, { cron: "0 0 1 * *", externalId: "ext", deduplicationKey: "key" });
await schedules.deactivate(id);
await schedules.activate(id);
await schedules.del(id);
await schedules.timezones(); // list of IANA timezones
```
+11
View File
@@ -0,0 +1,11 @@
# Remove AI code slop
Check the diff against main, and remove all AI generated slop introduced in this branch.
This includes:
- Extra comments that a human wouldn't add or is inconsistent with the rest of the file
- Extra defensive checks or try/catch blocks that are abnormal for that area of the codebase (especially if called by trusted / validated codepaths)
- Casts to any to get around type issues
- Any other style that is inconsistent with the file
Report at the end with only a 1-3 sentence summary of what you changed
+2 -6
View File
@@ -1,7 +1,3 @@
{
"mcpServers": {
"trigger.dev": {
"url": "http://localhost:3333/sse"
}
}
}
"mcpServers": {}
}
+6
View File
@@ -0,0 +1,6 @@
---
description: how to create and apply database migrations
alwaysApply: false
---
Follow our [migrations.md](mdc:ai/references/migrations.md) guide for how to create and apply database migrations.
+66
View File
@@ -0,0 +1,66 @@
---
description: Guidelines for creating OpenTelemetry metrics to avoid cardinality issues
globs:
- "**/*.ts"
---
# OpenTelemetry Metrics Guidelines
When creating or editing OTEL metrics (counters, histograms, gauges), always ensure metric attributes have **low cardinality**.
## What is Cardinality?
Cardinality refers to the number of unique values an attribute can have. Each unique combination of attribute values creates a new time series, which consumes memory and storage in your metrics backend.
## Rules
### DO use low-cardinality attributes:
- **Enums**: `environment_type` (PRODUCTION, STAGING, DEVELOPMENT, PREVIEW)
- **Booleans**: `hasFailures`, `streaming`, `success`
- **Bounded error codes**: A finite, controlled set of error types
- **Shard IDs**: When sharding is bounded (e.g., 0-15)
### DO NOT use high-cardinality attributes:
- **UUIDs/IDs**: `envId`, `userId`, `runId`, `projectId`, `organizationId`
- **Unbounded integers**: `itemCount`, `batchSize`, `retryCount`
- **Timestamps**: `createdAt`, `startTime`
- **Free-form strings**: `errorMessage`, `taskName`, `queueName`
## Example
```typescript
// BAD - High cardinality
this.counter.add(1, {
envId: options.environmentId, // UUID - unbounded
itemCount: options.runCount, // Integer - unbounded
});
// GOOD - Low cardinality
this.counter.add(1, {
environment_type: options.environmentType, // Enum - 4 values
streaming: true, // Boolean - 2 values
});
```
## Prometheus Metric Naming
When metrics are exported via OTLP to Prometheus, the exporter automatically adds unit suffixes to metric names:
| OTel Metric Name | Unit | Prometheus Name |
|------------------|------|-----------------|
| `my_duration_ms` | `ms` | `my_duration_ms_milliseconds` |
| `my_counter` | counter | `my_counter_total` |
| `items_inserted` | counter | `items_inserted_inserts_total` |
| `batch_size` | histogram | `batch_size_items_bucket` |
Keep this in mind when writing Grafana dashboards or Prometheus queries—the metric names in Prometheus will differ from the names defined in code.
## Reference
See the schedule engine (`internal-packages/schedule-engine/src/engine/index.ts`) for a good example of low-cardinality metric attributes.
High cardinality metrics can cause:
- Memory bloat in metrics backends (Axiom, Prometheus, etc.)
- Slow queries and dashboard timeouts
- Increased costs (many backends charge per time series)
- Potential data loss or crashes at scale
+6 -3
View File
@@ -6,7 +6,7 @@ alwaysApply: false
The main trigger.dev webapp, which powers it's API and dashboard and makes up the docker image that is produced as an OSS image, is a Remix 2.1.0 app that uses an express server, written in TypeScript. The following subsystems are either included in the webapp or are used by the webapp in another part of the monorepo:
- `@trigger.dev/database` exports a Prisma 5.4.1 client that is used extensively in the webapp to access a PostgreSQL instance. The schema file is [schema.prisma](mdc:internal-packages/database/prisma/schema.prisma)
- `@trigger.dev/database` exports a Prisma 6.14.0 client that is used extensively in the webapp to access a PostgreSQL instance. The schema file is [schema.prisma](mdc:internal-packages/database/prisma/schema.prisma)
- `@trigger.dev/core` is a published package and is used to share code between the `@trigger.dev/sdk` and the webapp. It includes functionality but also a load of Zod schemas for data validation. When importing from `@trigger.dev/core` in the webapp, we never import the root `@trigger.dev/core` path, instead we favor one of the subpath exports that you can find in [package.json](mdc:packages/core/package.json)
- `@internal/run-engine` has all the code needed to trigger a run and take it through it's lifecycle to completion.
- `@trigger.dev/redis-worker` is a custom redis based background job/worker system that's used in the webapp and also used inside the run engine.
@@ -31,7 +31,10 @@ We originally the Trigger.dev "Run Engine" not as a single system, but just spre
- The batch trigger API endpoint is [api.v1.tasks.batch.ts](mdc:apps/webapp/app/routes/api.v1.tasks.batch.ts)
- Setup code for the prisma client is in [db.server.ts](mdc:apps/webapp/app/db.server.ts)
- The run engine is configured in [runEngine.server.ts](mdc:apps/webapp/app/v3/runEngine.server.ts)
- All the "services" that are found in app/v3/services/**/*.server.ts
- All the "services" that are found in app/v3/services/\*_/_.server.ts
- The code for the TaskEvent data, which is the otel data sent from tasks to our servers, is in both the [eventRepository.server.ts](mdc:apps/webapp/app/v3/eventRepository.server.ts) and also the [otlpExporter.server.ts](mdc:apps/webapp/app/v3/otlpExporter.server.ts). The otel endpoints which are hit from production and development otel exporters is [otel.v1.logs.ts](mdc:apps/webapp/app/routes/otel.v1.logs.ts) and [otel.v1.traces.ts](mdc:apps/webapp/app/routes/otel.v1.traces.ts)
- We use "presenters" to move more complex loader code into a class, and you can find those are app/v3/presenters/**/*.server.ts
- We use "presenters" to move more complex loader code into a class, and you can find those are app/v3/presenters/\*_/_.server.ts
- All the "services" that are found in app/v3/services/\*_/_.server.ts
- The code for the TaskEvent data, which is the otel data sent from tasks to our servers, is in both the [eventRepository.server.ts](mdc:apps/webapp/app/v3/eventRepository.server.ts) and also the [otlpExporter.server.ts](mdc:apps/webapp/app/v3/otlpExporter.server.ts). The otel endpoints which are hit from production and development otel exporters is [otel.v1.logs.ts](mdc:apps/webapp/app/routes/otel.v1.logs.ts) and [otel.v1.traces.ts](mdc:apps/webapp/app/routes/otel.v1.traces.ts)
- We use "presenters" to move more complex loader code into a class, and you can find those are app/v3/presenters/\*_/_.server.ts
-22
View File
@@ -431,28 +431,6 @@ export async function POST(request: Request) {
}
```
### tasks.triggerAndPoll()
Triggers a task and polls until completion. Not recommended for web requests as it blocks until the run completes. Consider using Realtime docs for better alternatives.
```ts
import { tasks } from "@trigger.dev/sdk/v3";
import type { emailSequence } from "~/trigger/emails";
export async function POST(request: Request) {
const data = await request.json();
const result = await tasks.triggerAndPoll<typeof emailSequence>(
"email-sequence",
{
to: data.email,
name: data.name,
},
{ pollIntervalMs: 5000 }
);
return Response.json(result);
}
```
### batch.trigger()
Triggers multiple runs of different tasks at once, useful when you need to execute multiple tasks simultaneously.
+3 -1
View File
@@ -4,4 +4,6 @@ apps/proxy/
apps/coordinator/
packages/rsc/
.changeset
.zed
.zed
.env
!.env.example
+2
View File
@@ -16,6 +16,8 @@
**/dist
**/node_modules
**/generated/prisma
apps/webapp/build
apps/webapp/public/build
+21 -31
View File
@@ -12,7 +12,14 @@ APP_ENV=development
APP_ORIGIN=http://localhost:3030
ELECTRIC_ORIGIN=http://localhost:3060
NODE_ENV=development
V3_ENABLED=true
# Clickhouse
CLICKHOUSE_URL=http://default:password@localhost:8123
RUN_REPLICATION_CLICKHOUSE_URL=http://default:password@localhost:8123
RUN_REPLICATION_ENABLED=1
# Set this to UTC because Node.js uses the system timezone
TZ="UTC"
# Redis is used for the v3 queuing and v2 concurrency control
REDIS_HOST="localhost"
@@ -22,11 +29,14 @@ REDIS_TLS_DISABLED="true"
DEV_OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:3030/otel"
DEV_OTEL_BATCH_PROCESSING_ENABLED="0"
# When the domain is set to `localhost` the CLI deploy command will only --load the image by default and not --push it
DEPLOY_REGISTRY_HOST=localhost:5000
# OPTIONAL VARIABLES
# This is used for validating emails that are allowed to log in. Every email that do not match this regex will be rejected.
# WHITELISTED_EMAILS="authorized@yahoo\.com|authorized@gmail\.com"
# WHITELISTED_EMAILS="^(authorized@yahoo\.com|authorized@gmail\.com)$"
# Accounts with these emails will get global admin rights. This grants access to the admin UI.
# ADMIN_EMAILS="admin@example\.com|another-admin@example\.com"
# ADMIN_EMAILS="^(admin@example\.com|another-admin@example\.com)$"
# This is used for logging in via GitHub. You can leave these commented out if you don't want to use GitHub for authentication.
# AUTH_GITHUB_CLIENT_ID=
# AUTH_GITHUB_CLIENT_SECRET=
@@ -60,34 +70,11 @@ DEV_OTEL_BATCH_PROCESSING_ENABLED="0"
# FROM_EMAIL=
# REPLY_TO_EMAIL=
# Remove the following line to enable logging telemetry traces to the console
LOG_TELEMETRY="false"
# CLOUD VARIABLES
POSTHOG_PROJECT_KEY=
PLAIN_API_KEY=
CLOUD_AIRTABLE_CLIENT_ID=
CLOUD_AIRTABLE_CLIENT_SECRET=
CLOUD_GITHUB_CLIENT_ID=
CLOUD_GITHUB_CLIENT_SECRET=
CLOUD_LINEAR_CLIENT_ID=
CLOUD_LINEAR_CLIENT_SECRET=
CLOUD_SLACK_APP_HOST=
CLOUD_SLACK_CLIENT_ID=
CLOUD_SLACK_CLIENT_SECRET=
# v3 variables
PROVIDER_SECRET=provider-secret # generate the actual secret with `openssl rand -hex 32`
COORDINATOR_SECRET=coordinator-secret # generate the actual secret with `openssl rand -hex 32`
# Uncomment the following line to enable the registry proxy
# ENABLE_REGISTRY_PROXY=true
# DEPOT_ORG_ID=<Depot org id>
# DEPOT_TOKEN=<Depot org token>
# DEPOT_PROJECT_ID=<Depot project id>
# DEPLOY_REGISTRY_HOST=${APP_ORIGIN} # This is the host that the deploy CLI will use to push images to the registry
# CONTAINER_REGISTRY_ORIGIN=<Container registry origin e.g. https://registry.digitalocean.com>
# CONTAINER_REGISTRY_USERNAME=<Container registry username e.g. Digital ocean email address>
# CONTAINER_REGISTRY_PASSWORD=<Container registry password e.g. Digital ocean PAT>
# DEV_OTEL_EXPORTER_OTLP_ENDPOINT="http://0.0.0.0:4318"
# These are needed for the object store (for handling large payloads/outputs)
# OBJECT_STORE_BASE_URL="https://{bucket}.{accountId}.r2.cloudflarestorage.com"
@@ -97,8 +84,11 @@ COORDINATOR_SECRET=coordinator-secret # generate the actual secret with `openssl
# These control the server-side internal telemetry
# INTERNAL_OTEL_TRACE_EXPORTER_URL=<URL to send traces to>
# INTERNAL_OTEL_TRACE_EXPORTER_AUTH_HEADER_NAME=<Header name for the auth token>
# INTERNAL_OTEL_TRACE_EXPORTER_AUTH_HEADER_VALUE=<Auth token value>
# INTERNAL_OTEL_TRACE_LOGGING_ENABLED=1
# INTERNAL_OTEL_TRACE_SAMPING_RATE=20 # this means 1/20 traces or 5% of traces will be sampled (sampled = recorded)
# INTERNAL_OTEL_TRACE_INSTRUMENT_PRISMA_ENABLED=0,
# INTERNAL_OTEL_TRACE_INSTRUMENT_PRISMA_ENABLED=0
# Enable local observability stack (requires `pnpm run docker` to start otel-collector)
# Uncomment these to send metrics to the local Prometheus via OTEL Collector:
# INTERNAL_OTEL_METRIC_EXPORTER_ENABLED=1
# INTERNAL_OTEL_METRIC_EXPORTER_URL=http://localhost:4318/v1/metrics
# INTERNAL_OTEL_METRIC_EXPORTER_INTERVAL_MS=15000
+102
View File
@@ -0,0 +1,102 @@
name: 🦋 Changesets PR
on:
push:
branches:
- main
paths:
- "packages/**"
- ".changeset/**"
- "package.json"
- "pnpm-lock.yaml"
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
jobs:
release-pr:
name: Create Release PR
runs-on: ubuntu-latest
permissions:
contents: write
pull-requests: write
if: github.repository == 'triggerdotdev/trigger.dev'
steps:
- name: Checkout
uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Setup pnpm
uses: pnpm/action-setup@v4
- name: Setup node
uses: buildjet/setup-node@v4
with:
node-version: 20.20.0
cache: "pnpm"
- name: Install dependencies
run: pnpm install --frozen-lockfile
- name: Create release PR
id: changesets
uses: changesets/action@v1
with:
version: pnpm run changeset:version
commit: "chore: release"
title: "chore: release"
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- name: Update PR title with version
if: steps.changesets.outputs.published != 'true'
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
PR_NUMBER=$(gh pr list --head changeset-release/main --json number --jq '.[0].number')
if [ -n "$PR_NUMBER" ]; then
git fetch origin changeset-release/main
# we arbitrarily reference the version of the cli package here; it is the same for all package releases
VERSION=$(git show origin/changeset-release/main:packages/cli-v3/package.json | jq -r '.version')
gh pr edit "$PR_NUMBER" --title "chore: release v$VERSION"
fi
update-lockfile:
name: Update lockfile on release PR
runs-on: ubuntu-latest
needs: release-pr
permissions:
contents: write
steps:
- name: Checkout release branch
uses: actions/checkout@v4
with:
ref: changeset-release/main
- name: Setup pnpm
uses: pnpm/action-setup@v4
with:
version: 10.23.0
- name: Setup node
uses: buildjet/setup-node@v4
with:
node-version: 20.20.0
- name: Install and update lockfile
run: pnpm install --no-frozen-lockfile
- name: Commit and push lockfile
run: |
set -e
if git diff --quiet pnpm-lock.yaml; then
echo "No lockfile changes"
else
git config user.name "github-actions[bot]"
git config user.email "github-actions[bot]@users.noreply.github.com"
git add pnpm-lock.yaml
git commit -m "chore: update lockfile for release"
git push origin changeset-release/main
fi
+70
View File
@@ -0,0 +1,70 @@
name: Claude Code
on:
issue_comment:
types: [created]
pull_request_review_comment:
types: [created]
issues:
types: [opened, assigned]
pull_request_review:
types: [submitted]
jobs:
claude:
if: |
(github.event_name == 'issue_comment' && contains(github.event.comment.body, '@claude')) ||
(github.event_name == 'pull_request_review_comment' && contains(github.event.comment.body, '@claude')) ||
(github.event_name == 'pull_request_review' && contains(github.event.review.body, '@claude')) ||
(github.event_name == 'issues' && (contains(github.event.issue.body, '@claude') || contains(github.event.issue.title, '@claude')))
runs-on: ubuntu-latest
permissions:
contents: read
pull-requests: read
issues: read
id-token: write
actions: read # Required for Claude to read CI results on PRs
steps:
- name: Checkout repository
uses: actions/checkout@v4
with:
fetch-depth: 1
- name: ⎔ Setup pnpm
uses: pnpm/action-setup@v4
with:
version: 10.23.0
- name: ⎔ Setup node
uses: buildjet/setup-node@v4
with:
node-version: 20.20.0
cache: "pnpm"
- name: 📥 Download deps
run: pnpm install --frozen-lockfile
- name: 📀 Generate Prisma Client
run: pnpm run generate
- name: Run Claude Code
id: claude
uses: anthropics/claude-code-action@v1
with:
claude_code_oauth_token: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }}
# This is an optional setting that allows Claude to read CI results on PRs
additional_permissions: |
actions: read
claude_args: |
--model claude-opus-4-5-20251101
--allowedTools "Bash(pnpm:*),Bash(turbo:*),Bash(git:*),Bash(gh:*),Bash(npx:*),Bash(docker:*),Edit,MultiEdit,Read,Write,Glob,Grep,LS,Task"
# Optional: Give a custom prompt to Claude. If this is not specified, Claude will perform the instructions specified in the comment that tagged it.
# prompt: 'Update the pull request description to include a summary of changes.'
# Optional: Add claude_args to customize behavior and configuration
# See https://github.com/anthropics/claude-code-action/blob/main/docs/usage.md
# or https://code.claude.com/docs/en/cli-reference for available options
# claude_args: '--allowed-tools Bash(gh pr:*)'
+5 -2
View File
@@ -1,5 +1,8 @@
name: "E2E"
permissions:
contents: read
on:
workflow_call:
inputs:
@@ -28,12 +31,12 @@ jobs:
- name: ⎔ Setup pnpm
uses: pnpm/action-setup@v4
with:
version: 8.15.5
version: 10.23.0
- name: ⎔ Setup node
uses: buildjet/setup-node@v4
with:
node-version: 20.11.1
node-version: 20.20.0
- name: 📥 Download deps
run: pnpm install --frozen-lockfile --filter trigger.dev...
+138
View File
@@ -0,0 +1,138 @@
name: 🧭 Helm Chart PR Prerelease
on:
pull_request:
types: [opened, synchronize, reopened]
paths:
- "hosting/k8s/helm/**"
concurrency:
group: helm-prerelease-${{ github.event.pull_request.number }}
cancel-in-progress: true
env:
REGISTRY: ghcr.io
CHART_NAME: trigger
jobs:
lint-and-test:
runs-on: ubuntu-latest
permissions:
contents: read
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Set up Helm
uses: azure/setup-helm@v4
with:
version: "3.18.3"
- name: Build dependencies
run: helm dependency build ./hosting/k8s/helm/
- name: Extract dependency charts
run: |
cd ./hosting/k8s/helm/
for file in ./charts/*.tgz; do echo "Extracting $file"; tar -xzf "$file" -C ./charts; done
- name: Lint Helm Chart
run: |
helm lint ./hosting/k8s/helm/
- name: Render templates
run: |
helm template test-release ./hosting/k8s/helm/ \
--values ./hosting/k8s/helm/values.yaml \
--output-dir ./helm-output
- name: Validate manifests
uses: docker://ghcr.io/yannh/kubeconform:v0.7.0
with:
entrypoint: "/kubeconform"
args: "-summary -output json ./helm-output"
prerelease:
needs: lint-and-test
runs-on: ubuntu-latest
permissions:
contents: read
packages: write
pull-requests: write
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Set up Helm
uses: azure/setup-helm@v4
with:
version: "3.18.3"
- name: Build dependencies
run: helm dependency build ./hosting/k8s/helm/
- name: Extract dependency charts
run: |
cd ./hosting/k8s/helm/
for file in ./charts/*.tgz; do echo "Extracting $file"; tar -xzf "$file" -C ./charts; done
- name: Log in to Container Registry
uses: docker/login-action@v3
with:
registry: ${{ env.REGISTRY }}
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Generate prerelease version
id: version
run: |
BASE_VERSION=$(grep '^version:' ./hosting/k8s/helm/Chart.yaml | awk '{print $2}')
PR_NUMBER=${{ github.event.pull_request.number }}
SHORT_SHA=$(echo "${{ github.event.pull_request.head.sha }}" | cut -c1-7)
PRERELEASE_VERSION="${BASE_VERSION}-pr${PR_NUMBER}.${SHORT_SHA}"
echo "version=$PRERELEASE_VERSION" >> $GITHUB_OUTPUT
echo "Prerelease version: $PRERELEASE_VERSION"
- name: Update Chart.yaml with prerelease version
run: |
sed -i "s/^version:.*/version: ${{ steps.version.outputs.version }}/" ./hosting/k8s/helm/Chart.yaml
- name: Package Helm Chart
run: |
helm package ./hosting/k8s/helm/ --destination /tmp/
- name: Push Helm Chart to GHCR
run: |
VERSION="${{ steps.version.outputs.version }}"
CHART_PACKAGE="/tmp/${{ env.CHART_NAME }}-${VERSION}.tgz"
# Push to GHCR OCI registry
helm push "$CHART_PACKAGE" "oci://${{ env.REGISTRY }}/${{ github.repository_owner }}/charts"
- name: Find existing comment
uses: peter-evans/find-comment@v3
id: find-comment
with:
issue-number: ${{ github.event.pull_request.number }}
comment-author: "github-actions[bot]"
body-includes: "Helm Chart Prerelease Published"
- name: Create or update PR comment
uses: peter-evans/create-or-update-comment@v4
with:
comment-id: ${{ steps.find-comment.outputs.comment-id }}
issue-number: ${{ github.event.pull_request.number }}
body: |
### 🧭 Helm Chart Prerelease Published
**Version:** `${{ steps.version.outputs.version }}`
**Install:**
```bash
helm upgrade --install trigger \
oci://ghcr.io/${{ github.repository_owner }}/charts/trigger \
--version "${{ steps.version.outputs.version }}"
```
> ⚠️ This is a prerelease for testing. Do not use in production.
edit-mode: replace
+5
View File
@@ -5,6 +5,7 @@ on:
types: [opened, synchronize, reopened]
paths-ignore:
- "docs/**"
- ".changeset/**"
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
@@ -28,3 +29,7 @@ jobs:
with:
package: cli-v3
secrets: inherit
sdk-compat:
uses: ./.github/workflows/sdk-compat.yml
secrets: inherit
+29 -2
View File
@@ -1,5 +1,10 @@
name: "🐳 Publish Webapp"
permissions:
contents: read
packages: write
id-token: write
on:
workflow_call:
inputs:
@@ -43,13 +48,25 @@ jobs:
ref_without_tag=ghcr.io/triggerdotdev/trigger.dev
image_tags=$ref_without_tag:${{ steps.get_tag.outputs.tag }}
# if tag is a semver, also tag it as v3
# if tag is a semver, also tag it as v4
if [[ "${{ steps.get_tag.outputs.is_semver }}" == true ]]; then
image_tags=$image_tags,$ref_without_tag:v3
# TODO: switch to v4 tag on GA
image_tags=$image_tags,$ref_without_tag:v4-beta
fi
echo "image_tags=${image_tags}" >> "$GITHUB_OUTPUT"
- name: 📝 Set the build info
id: set_build_info
run: |
tag=${{ steps.get_tag.outputs.tag }}
if [[ "${{ steps.get_tag.outputs.is_semver }}" == true ]]; then
echo "BUILD_APP_VERSION=${tag}" >> "$GITHUB_OUTPUT"
fi
echo "BUILD_GIT_SHA=${{ github.sha }}" >> "$GITHUB_OUTPUT"
echo "BUILD_GIT_REF_NAME=${{ github.ref_name }}" >> "$GITHUB_OUTPUT"
echo "BUILD_TIMESTAMP_SECONDS=$(date +%s)" >> "$GITHUB_OUTPUT"
- name: 🐙 Login to GitHub Container Registry
uses: docker/login-action@v3
with:
@@ -64,3 +81,13 @@ jobs:
platforms: linux/amd64,linux/arm64
tags: ${{ steps.set_tags.outputs.image_tags }}
push: true
build-args: |
BUILD_APP_VERSION=${{ steps.set_build_info.outputs.BUILD_APP_VERSION }}
BUILD_GIT_SHA=${{ steps.set_build_info.outputs.BUILD_GIT_SHA }}
BUILD_GIT_REF_NAME=${{ steps.set_build_info.outputs.BUILD_GIT_REF_NAME }}
BUILD_TIMESTAMP_SECONDS=${{ steps.set_build_info.outputs.BUILD_TIMESTAMP_SECONDS }}
SENTRY_RELEASE=${{ steps.set_build_info.outputs.BUILD_GIT_SHA }}
SENTRY_ORG=triggerdev
SENTRY_PROJECT=trigger-cloud
secrets: |
sentry_auth_token=${{ secrets.SENTRY_AUTH_TOKEN }}
@@ -1,4 +1,4 @@
name: "⚒️ Publish Worker RE2"
name: "⚒️ Publish Worker (v4)"
on:
workflow_call:
@@ -14,6 +14,7 @@ on:
- "re2-prod-*"
permissions:
id-token: write
packages: write
contents: read
@@ -35,6 +36,9 @@ jobs:
env:
DOCKER_BUILDKIT: "1"
steps:
- name: 🏭 Setup Depot CLI
uses: depot/setup-action@v1
- name: ⬇️ Checkout git repo
uses: actions/checkout@v4
@@ -49,26 +53,26 @@ jobs:
fi
echo "repo=${repo}" >> "$GITHUB_OUTPUT"
- id: get_tag
- name: "#️⃣ Get image tag"
id: get_tag
uses: ./.github/actions/get-image-tag
with:
tag: ${{ inputs.image_tag }}
- name: 🐋 Set up Docker Buildx
uses: docker/setup-buildx-action@v3
# ..to avoid rate limits when pulling images
- name: 🐳 Login to DockerHub
uses: docker/login-action@v3
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: 🚢 Build Container Image
- name: 📛 Set tags to push
id: set_tags
run: |
docker build -t infra_image -f ./apps/${{ matrix.package }}/Containerfile .
ref_without_tag=ghcr.io/triggerdotdev/${{ steps.get_repository.outputs.repo }}
image_tags=$ref_without_tag:${{ steps.get_tag.outputs.tag }}
# if tag is a semver, also tag it as v4
if [[ "${{ steps.get_tag.outputs.is_semver }}" == true ]]; then
# TODO: switch to v4 tag on GA
image_tags=$image_tags,$ref_without_tag:v4-beta
fi
echo "image_tags=${image_tags}" >> "$GITHUB_OUTPUT"
# ..to push image
- name: 🐙 Login to GitHub Container Registry
uses: docker/login-action@v3
with:
@@ -76,20 +80,10 @@ jobs:
username: ${{ github.repository_owner }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: 🐙 Push to GitHub Container Registry
run: |
docker tag infra_image "$REGISTRY/$REPOSITORY:$IMAGE_TAG"
docker push "$REGISTRY/$REPOSITORY:$IMAGE_TAG"
env:
REGISTRY: ghcr.io/triggerdotdev
REPOSITORY: ${{ steps.get_repository.outputs.repo }}
IMAGE_TAG: ${{ steps.get_tag.outputs.tag }}
- name: 🐙 Push 'v3' tag to GitHub Container Registry
if: steps.get_tag.outputs.is_semver == 'true'
run: |
docker tag infra_image "$REGISTRY/$REPOSITORY:v3"
docker push "$REGISTRY/$REPOSITORY:v3"
env:
REGISTRY: ghcr.io/triggerdotdev
REPOSITORY: ${{ steps.get_repository.outputs.repo }}
- name: 🐳 Build image and push to GitHub Container Registry
uses: depot/build-push-action@v1
with:
file: ./apps/${{ matrix.package }}/Containerfile
platforms: linux/amd64,linux/arm64
tags: ${{ steps.set_tags.outputs.image_tags }}
push: true
+8 -8
View File
@@ -77,11 +77,11 @@ jobs:
REPOSITORY: ${{ steps.get_repository.outputs.repo }}
IMAGE_TAG: ${{ steps.get_tag.outputs.tag }}
- name: 🐙 Push 'v3' tag to GitHub Container Registry
if: steps.get_tag.outputs.is_semver == 'true'
run: |
docker tag infra_image "$REGISTRY/$REPOSITORY:v3"
docker push "$REGISTRY/$REPOSITORY:v3"
env:
REGISTRY: ghcr.io/triggerdotdev
REPOSITORY: ${{ steps.get_repository.outputs.repo }}
# - name: 🐙 Push 'v3' tag to GitHub Container Registry
# if: steps.get_tag.outputs.is_semver == 'true'
# run: |
# docker tag infra_image "$REGISTRY/$REPOSITORY:v3"
# docker push "$REGISTRY/$REPOSITORY:v3"
# env:
# REGISTRY: ghcr.io/triggerdotdev
# REPOSITORY: ${{ steps.get_repository.outputs.repo }}
+8
View File
@@ -1,6 +1,7 @@
name: 🚀 Publish Trigger.dev Docker
on:
workflow_dispatch:
workflow_call:
inputs:
image_tag:
@@ -68,3 +69,10 @@ jobs:
secrets: inherit
with:
image_tag: ${{ inputs.image_tag }}
publish-worker-v4:
needs: [typecheck]
uses: ./.github/workflows/publish-worker-v4.yml
secrets: inherit
with:
image_tag: ${{ inputs.image_tag }}
+143
View File
@@ -0,0 +1,143 @@
name: 🧭 Helm Chart Release
on:
push:
tags:
- 'helm-v*'
workflow_dispatch:
inputs:
chart_version:
description: 'Chart version to release'
required: true
type: string
env:
REGISTRY: ghcr.io
CHART_NAME: trigger
jobs:
lint-and-test:
runs-on: ubuntu-latest
permissions:
contents: read
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Set up Helm
uses: azure/setup-helm@v4
with:
version: "3.18.3"
- name: Build dependencies
run: helm dependency build ./hosting/k8s/helm/
- name: Extract dependency charts
run: |
cd ./hosting/k8s/helm/
for file in ./charts/*.tgz; do echo "Extracting $file"; tar -xzf "$file" -C ./charts; done
- name: Lint Helm Chart
run: |
helm lint ./hosting/k8s/helm/
- name: Render templates
run: |
helm template test-release ./hosting/k8s/helm/ \
--values ./hosting/k8s/helm/values.yaml \
--output-dir ./helm-output
- name: Validate manifests
uses: docker://ghcr.io/yannh/kubeconform:v0.7.0
with:
entrypoint: '/kubeconform'
args: "-summary -output json ./helm-output"
release:
needs: lint-and-test
runs-on: ubuntu-latest
permissions:
contents: write # for gh-release
packages: write
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Set up Helm
uses: azure/setup-helm@v4
with:
version: "3.18.3"
- name: Build dependencies
run: helm dependency build ./hosting/k8s/helm/
- name: Extract dependency charts
run: |
cd ./hosting/k8s/helm/
for file in ./charts/*.tgz; do echo "Extracting $file"; tar -xzf "$file" -C ./charts; done
- name: Log in to Container Registry
uses: docker/login-action@v3
with:
registry: ${{ env.REGISTRY }}
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Extract version from tag or input
id: version
run: |
if [ "${{ github.event_name }}" = "workflow_dispatch" ]; then
VERSION="${{ github.event.inputs.chart_version }}"
else
VERSION="${{ github.ref_name }}"
VERSION="${VERSION#helm-v}"
fi
echo "version=$VERSION" >> $GITHUB_OUTPUT
echo "Releasing version: $VERSION"
- name: Check Chart.yaml version matches release version
run: |
VERSION="${{ steps.version.outputs.version }}"
CHART_VERSION=$(grep '^version:' ./hosting/k8s/helm/Chart.yaml | awk '{print $2}')
echo "Chart.yaml version: $CHART_VERSION"
echo "Release version: $VERSION"
if [ "$CHART_VERSION" != "$VERSION" ]; then
echo "❌ Chart.yaml version does not match release version!"
exit 1
fi
echo "✅ Chart.yaml version matches release version."
- name: Package Helm Chart
run: |
helm package ./hosting/k8s/helm/ --destination /tmp/
- name: Push Helm Chart to GHCR
run: |
VERSION="${{ steps.version.outputs.version }}"
CHART_PACKAGE="/tmp/${{ env.CHART_NAME }}-${VERSION}.tgz"
# Push to GHCR OCI registry
helm push "$CHART_PACKAGE" "oci://${{ env.REGISTRY }}/${{ github.repository_owner }}/charts"
- name: Create GitHub Release
id: release
uses: softprops/action-gh-release@v1
if: github.event_name == 'push'
with:
tag_name: ${{ github.ref_name }}
name: "Helm Chart ${{ steps.version.outputs.version }}"
body: |
### Installation
```bash
helm upgrade --install trigger \
oci://${{ env.REGISTRY }}/${{ github.repository_owner }}/charts/${{ env.CHART_NAME }} \
--version "${{ steps.version.outputs.version }}"
```
### Changes
See commit history for detailed changes in this release.
files: |
/tmp/${{ env.CHART_NAME }}-${{ steps.version.outputs.version }}.tgz
token: ${{ secrets.GITHUB_TOKEN }}
draft: true
prerelease: true
+142 -48
View File
@@ -1,101 +1,195 @@
name: 🦋 Changesets Release
on:
push:
pull_request:
types: [closed]
branches:
- main
paths-ignore:
- "docs/**"
- "**.md"
- ".github/CODEOWNERS"
- ".github/ISSUE_TEMPLATE/**"
workflow_dispatch:
inputs:
type:
description: "Select release type"
required: true
type: choice
options:
- release
- prerelease
default: "prerelease"
ref:
description: "The ref (branch, tag, or SHA) to checkout and release from"
required: true
type: string
prerelease_tag:
description: "The npm dist-tag for the prerelease (e.g., 'v4-prerelease')"
required: false
type: string
default: "prerelease"
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
group: ${{ github.workflow }}
cancel-in-progress: false
jobs:
release:
name: 🦋 Changesets Release
show-release-summary:
name: 📋 Release Summary
runs-on: ubuntu-latest
if: github.repository == 'triggerdotdev/trigger.dev'
if: |
github.repository == 'triggerdotdev/trigger.dev' &&
github.event_name == 'pull_request' &&
github.event.pull_request.merged == true &&
github.event.pull_request.head.ref == 'changeset-release/main'
steps:
- name: Show release summary
env:
PR_BODY: ${{ github.event.pull_request.body }}
run: |
echo "$PR_BODY" | sed -n '/^# Releases/,$p' >> $GITHUB_STEP_SUMMARY
release:
name: 🚀 Release npm packages
runs-on: ubuntu-latest
environment: npm-publish
permissions:
contents: write
packages: write
id-token: write
if: |
github.repository == 'triggerdotdev/trigger.dev' &&
(
(github.event_name == 'workflow_dispatch' && github.event.inputs.type == 'release') ||
(github.event_name == 'pull_request' && github.event.pull_request.merged == true && github.event.pull_request.head.ref == 'changeset-release/main')
)
outputs:
published: ${{ steps.changesets.outputs.published }}
published_packages: ${{ steps.changesets.outputs.publishedPackages }}
published_package_version: ${{ steps.get_version.outputs.package_version }}
steps:
- name: ⬇️ Checkout repo
- name: Checkout repo
uses: actions/checkout@v4
with:
fetch-depth: 0
ref: ${{ github.event_name == 'workflow_dispatch' && github.event.inputs.ref || github.sha }}
- name: ⎔ Setup pnpm
- name: Verify ref is on main
if: github.event_name == 'workflow_dispatch'
run: |
if ! git merge-base --is-ancestor ${{ github.event.inputs.ref }} origin/main; then
echo "Error: ref must be an ancestor of main (i.e., already merged)"
exit 1
fi
- name: Setup pnpm
uses: pnpm/action-setup@v4
with:
version: 8.15.5
version: 10.23.0
- name: Setup node
- name: Setup node
uses: buildjet/setup-node@v4
with:
node-version: 20.11.1
node-version: 20.20.0
cache: "pnpm"
- name: 📥 Download deps
# npm v11.5.1 or newer is required for OIDC support
# https://github.blog/changelog/2025-07-31-npm-trusted-publishing-with-oidc-is-generally-available/#whats-new
- name: Setup npm 11.x for OIDC
run: npm install -g npm@11.6.4
- name: Install dependencies
run: pnpm install --frozen-lockfile
- name: 📀 Generate Prisma Client
- name: Generate Prisma client
run: pnpm run generate
- name: 🏗️ Build
- name: Build
run: pnpm run build --filter "@trigger.dev/*" --filter "trigger.dev"
- name: 🔎 Type check
- name: Type check
run: pnpm run typecheck --filter "@trigger.dev/*" --filter "trigger.dev"
- name: 🔐 Setup npm auth
run: |
echo "registry=https://registry.npmjs.org" >> ~/.npmrc
echo "//registry.npmjs.org/:_authToken=${{ secrets.NPM_TOKEN }}" >> ~/.npmrc
# This action has two responsibilities. The first time the workflow runs
# (initial push to the `main` branch) it will create a new branch and
# then open a PR with the related changes for the new version. After the
# PR is merged, the workflow will run again and this action will build +
# publish to npm.
- name: 🚀 PR / Publish
if: ${{ !env.ACT }}
- name: Publish
id: changesets
uses: changesets/action@v1
with:
version: pnpm run changeset:version
commit: "chore: Update version for release"
title: "chore: Update version for release"
publish: pnpm run changeset:release
createGithubReleases: true
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
NPM_TOKEN: ${{ secrets.NPM_TOKEN }}
# - name: 🚀 PR / Publish (mock)
# if: ${{ env.ACT }}
# id: changesets
# run: |
# echo "published=true" >> "$GITHUB_OUTPUT"
# echo "publishedPackages=[{\"name\": \"@xx/xx\", \"version\": \"1.2.0\"}, {\"name\": \"@xx/xy\", \"version\": \"0.8.9\"}]" >> "$GITHUB_OUTPUT"
- name: 📦 Get package version
- name: Show package version
if: steps.changesets.outputs.published == 'true'
id: get_version
run: |
package_version=$(echo '${{ steps.changesets.outputs.publishedPackages }}' | jq -r '.[0].version')
echo "package_version=${package_version}" >> "$GITHUB_OUTPUT"
publish:
- name: Create and push Docker tag
if: steps.changesets.outputs.published == 'true'
run: |
set -e
git tag "v.docker.${{ steps.get_version.outputs.package_version }}"
git push origin "v.docker.${{ steps.get_version.outputs.package_version }}"
# Trigger Docker builds directly via workflow_call since tags pushed with
# GITHUB_TOKEN don't trigger other workflows (GitHub Actions limitation).
publish-docker:
name: 🐳 Publish Docker images
needs: release
if: needs.release.outputs.published == 'true'
uses: ./.github/workflows/publish.yml
secrets: inherit
# if: needs.release.outputs.published == 'true'
# disable automatic publishing for now
if: false
with:
image_tag: v${{ needs.release.outputs.published_package_version }}
# The prerelease job needs to be on the same workflow file due to a limitation related to how npm verifies OIDC claims.
prerelease:
name: 🧪 Prerelease
runs-on: ubuntu-latest
environment: npm-publish
permissions:
contents: read
id-token: write
if: github.repository == 'triggerdotdev/trigger.dev' && github.event_name == 'workflow_dispatch' && github.event.inputs.type == 'prerelease'
steps:
- name: Checkout repo
uses: actions/checkout@v4
with:
fetch-depth: 0
ref: ${{ github.event.inputs.ref }}
- name: Setup pnpm
uses: pnpm/action-setup@v4
with:
version: 10.23.0
- name: Setup node
uses: buildjet/setup-node@v4
with:
node-version: 20.20.0
cache: "pnpm"
# npm v11.5.1 or newer is required for OIDC support
# https://github.blog/changelog/2025-07-31-npm-trusted-publishing-with-oidc-is-generally-available/#whats-new
- name: Setup npm 11.x for OIDC
run: npm install -g npm@11.6.4
- name: Download deps
run: pnpm install --frozen-lockfile
- name: Generate Prisma Client
run: pnpm run generate
- name: Snapshot version
run: pnpm exec changeset version --snapshot ${{ github.event.inputs.prerelease_tag }}
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- name: Clean
run: pnpm run clean --filter "@trigger.dev/*" --filter "trigger.dev"
- name: Build
run: pnpm run build --filter "@trigger.dev/*" --filter "trigger.dev"
- name: Publish prerelease
run: pnpm exec changeset publish --no-git-tag --snapshot --tag ${{ github.event.inputs.prerelease_tag }}
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+178
View File
@@ -0,0 +1,178 @@
name: "🔌 SDK Compatibility Tests"
permissions:
contents: read
on:
workflow_call:
jobs:
node-compat:
name: "Node.js ${{ matrix.node }} (${{ matrix.os }})"
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
matrix:
os: [ubuntu-latest]
node: ["20.20", "22.12"]
steps:
- name: ⬇️ Checkout repo
uses: actions/checkout@v4
with:
fetch-depth: 0
- name: ⎔ Setup pnpm
uses: pnpm/action-setup@v4
with:
version: 10.23.0
- name: ⎔ Setup node
uses: buildjet/setup-node@v4
with:
node-version: ${{ matrix.node }}
cache: "pnpm"
- name: 📥 Download deps
run: pnpm install --frozen-lockfile
- name: 📀 Generate Prisma Client
run: pnpm run generate
- name: 🔨 Build SDK dependencies
shell: bash
run: pnpm run build --filter '@trigger.dev/sdk^...'
- name: 🔨 Build SDK
shell: bash
run: pnpm run build --filter '@trigger.dev/sdk'
- name: 🧪 Run SDK Compatibility Tests
shell: bash
run: pnpm --filter @internal/sdk-compat-tests test
bun-compat:
name: "Bun Runtime"
runs-on: ubuntu-latest
steps:
- name: ⬇️ Checkout repo
uses: actions/checkout@v4
with:
fetch-depth: 0
- name: ⎔ Setup pnpm
uses: pnpm/action-setup@v4
with:
version: 10.23.0
- name: ⎔ Setup node
uses: buildjet/setup-node@v4
with:
node-version: 20.20.0
cache: "pnpm"
- name: 🥟 Setup Bun
uses: oven-sh/setup-bun@v2
with:
bun-version: latest
- name: 📥 Download deps
run: pnpm install --frozen-lockfile
- name: 📀 Generate Prisma Client
run: pnpm run generate
- name: 🔨 Build SDK dependencies
run: pnpm run build --filter @trigger.dev/sdk^...
- name: 🔨 Build SDK
run: pnpm run build --filter @trigger.dev/sdk
- name: 🧪 Run Bun Compatibility Test
working-directory: internal-packages/sdk-compat-tests/src/fixtures/bun
run: bun run test.ts
deno-compat:
name: "Deno Runtime"
runs-on: ubuntu-latest
steps:
- name: ⬇️ Checkout repo
uses: actions/checkout@v4
with:
fetch-depth: 0
- name: ⎔ Setup pnpm
uses: pnpm/action-setup@v4
with:
version: 10.23.0
- name: ⎔ Setup node
uses: buildjet/setup-node@v4
with:
node-version: 20.20.0
cache: "pnpm"
- name: 🦕 Setup Deno
uses: denoland/setup-deno@v2
with:
deno-version: v2.x
- name: 📥 Download deps
run: pnpm install --frozen-lockfile
- name: 📀 Generate Prisma Client
run: pnpm run generate
- name: 🔨 Build SDK dependencies
run: pnpm run build --filter @trigger.dev/sdk^...
- name: 🔨 Build SDK
run: pnpm run build --filter @trigger.dev/sdk
- name: 🔗 Link node_modules for Deno fixture
working-directory: internal-packages/sdk-compat-tests/src/fixtures/deno
run: ln -s ../../../../../node_modules node_modules
- name: 🧪 Run Deno Compatibility Test
working-directory: internal-packages/sdk-compat-tests/src/fixtures/deno
run: deno run --allow-read --allow-env --allow-sys test.ts
cloudflare-compat:
name: "Cloudflare Workers"
runs-on: ubuntu-latest
steps:
- name: ⬇️ Checkout repo
uses: actions/checkout@v4
with:
fetch-depth: 0
- name: ⎔ Setup pnpm
uses: pnpm/action-setup@v4
with:
version: 10.23.0
- name: ⎔ Setup node
uses: buildjet/setup-node@v4
with:
node-version: 20.20.0
cache: "pnpm"
- name: 📥 Download deps
run: pnpm install --frozen-lockfile
- name: 📀 Generate Prisma Client
run: pnpm run generate
- name: 🔨 Build SDK dependencies
run: pnpm run build --filter @trigger.dev/sdk^...
- name: 🔨 Build SDK
run: pnpm run build --filter @trigger.dev/sdk
- name: 📥 Install Cloudflare fixture deps
working-directory: internal-packages/sdk-compat-tests/src/fixtures/cloudflare-worker
run: pnpm install
- name: 🧪 Run Cloudflare Workers Compatibility Test (dry-run)
working-directory: internal-packages/sdk-compat-tests/src/fixtures/cloudflare-worker
run: npx wrangler deploy --dry-run --outdir dist
+7 -2
View File
@@ -3,6 +3,9 @@ name: "ʦ TypeScript"
on:
workflow_call:
permissions:
contents: read
jobs:
typecheck:
runs-on: ubuntu-latest
@@ -16,12 +19,12 @@ jobs:
- name: ⎔ Setup pnpm
uses: pnpm/action-setup@v4
with:
version: 8.15.5
version: 10.23.0
- name: ⎔ Setup node
uses: buildjet/setup-node@v4
with:
node-version: 20.11.1
node-version: 20.20.0
cache: "pnpm"
- name: 📥 Download deps
@@ -32,6 +35,8 @@ jobs:
- name: 🔎 Type check
run: pnpm run typecheck
env:
NODE_OPTIONS: --max-old-space-size=8192
- name: 🔎 Check exports
run: pnpm run check-exports
+141
View File
@@ -0,0 +1,141 @@
name: "🧪 Unit Tests: Internal"
permissions:
contents: read
on:
workflow_call:
jobs:
unitTests:
name: "🧪 Unit Tests: Internal"
runs-on: ubuntu-latest
strategy:
matrix:
shardIndex: [1, 2, 3, 4, 5, 6, 7, 8]
shardTotal: [8]
env:
DOCKERHUB_USERNAME: ${{ secrets.DOCKERHUB_USERNAME }}
SHARD_INDEX: ${{ matrix.shardIndex }}
SHARD_TOTAL: ${{ matrix.shardTotal }}
steps:
- name: 🔧 Disable IPv6
run: |
sudo sysctl -w net.ipv6.conf.all.disable_ipv6=1
sudo sysctl -w net.ipv6.conf.default.disable_ipv6=1
sudo sysctl -w net.ipv6.conf.lo.disable_ipv6=1
- name: 🔧 Configure docker address pool
run: |
CONFIG='{
"default-address-pools" : [
{
"base" : "172.17.0.0/12",
"size" : 20
},
{
"base" : "192.168.0.0/16",
"size" : 24
}
]
}'
mkdir -p /etc/docker
echo "$CONFIG" | sudo tee /etc/docker/daemon.json
- name: 🔧 Restart docker daemon
run: sudo systemctl restart docker
- name: ⬇️ Checkout repo
uses: actions/checkout@v4
with:
fetch-depth: 0
- name: ⎔ Setup pnpm
uses: pnpm/action-setup@v4
with:
version: 10.23.0
- name: ⎔ Setup node
uses: buildjet/setup-node@v4
with:
node-version: 20.20.0
cache: "pnpm"
# ..to avoid rate limits when pulling images
- name: 🐳 Login to DockerHub
if: ${{ env.DOCKERHUB_USERNAME }}
uses: docker/login-action@v3
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: 🐳 Skipping DockerHub login (no secrets available)
if: ${{ !env.DOCKERHUB_USERNAME }}
run: echo "DockerHub login skipped because secrets are not available."
- name: 🐳 Pre-pull testcontainer images
if: ${{ env.DOCKERHUB_USERNAME }}
run: |
echo "Pre-pulling Docker images with authenticated session..."
docker pull postgres:14
docker pull clickhouse/clickhouse-server:25.4-alpine
docker pull redis:7-alpine
docker pull testcontainers/ryuk:0.11.0
docker pull electricsql/electric:1.2.4
echo "Image pre-pull complete"
- name: 📥 Download deps
run: pnpm install --frozen-lockfile
- name: 📀 Generate Prisma Client
run: pnpm run generate
- name: 🧪 Run Internal Unit Tests
run: pnpm run test:internal --reporter=default --reporter=blob --shard=${{ matrix.shardIndex }}/${{ matrix.shardTotal }}
- name: Gather all reports
if: ${{ !cancelled() }}
run: |
mkdir -p .vitest-reports
find . -type f -path '*/.vitest-reports/blob-*.json' \
-exec bash -c 'src="$1"; basename=$(basename "$src"); pkg=$(dirname "$src" | sed "s|^\./||;s|/\.vitest-reports$||;s|/|_|g"); cp "$src" ".vitest-reports/${pkg}-${basename}"' _ {} \;
- name: Upload blob reports to GitHub Actions Artifacts
if: ${{ !cancelled() }}
uses: actions/upload-artifact@v4
with:
name: internal-blob-report-${{ matrix.shardIndex }}
path: .vitest-reports/*
include-hidden-files: true
retention-days: 1
merge-reports:
name: "📊 Merge Reports"
if: ${{ !cancelled() }}
needs: [unitTests]
runs-on: ubuntu-latest
steps:
- name: ⬇️ Checkout repo
uses: actions/checkout@v4
with:
fetch-depth: 0
- name: ⎔ Setup pnpm
uses: pnpm/action-setup@v4
with:
version: 10.23.0
- name: ⎔ Setup node
uses: buildjet/setup-node@v4
with:
node-version: 20.20.0
# no cache enabled, we're not installing deps
- name: Download blob reports from GitHub Actions Artifacts
uses: actions/download-artifact@v4
with:
path: .vitest-reports
pattern: internal-blob-report-*
merge-multiple: true
- name: Merge reports
run: pnpm dlx vitest@3.1.4 run --merge-reports --pass-with-no-tests
+141
View File
@@ -0,0 +1,141 @@
name: "🧪 Unit Tests: Packages"
permissions:
contents: read
on:
workflow_call:
jobs:
unitTests:
name: "🧪 Unit Tests: Packages"
runs-on: ubuntu-latest
strategy:
matrix:
shardIndex: [1]
shardTotal: [1]
env:
DOCKERHUB_USERNAME: ${{ secrets.DOCKERHUB_USERNAME }}
SHARD_INDEX: ${{ matrix.shardIndex }}
SHARD_TOTAL: ${{ matrix.shardTotal }}
steps:
- name: 🔧 Disable IPv6
run: |
sudo sysctl -w net.ipv6.conf.all.disable_ipv6=1
sudo sysctl -w net.ipv6.conf.default.disable_ipv6=1
sudo sysctl -w net.ipv6.conf.lo.disable_ipv6=1
- name: 🔧 Configure docker address pool
run: |
CONFIG='{
"default-address-pools" : [
{
"base" : "172.17.0.0/12",
"size" : 20
},
{
"base" : "192.168.0.0/16",
"size" : 24
}
]
}'
mkdir -p /etc/docker
echo "$CONFIG" | sudo tee /etc/docker/daemon.json
- name: 🔧 Restart docker daemon
run: sudo systemctl restart docker
- name: ⬇️ Checkout repo
uses: actions/checkout@v4
with:
fetch-depth: 0
- name: ⎔ Setup pnpm
uses: pnpm/action-setup@v4
with:
version: 10.23.0
- name: ⎔ Setup node
uses: buildjet/setup-node@v4
with:
node-version: 20.20.0
cache: "pnpm"
# ..to avoid rate limits when pulling images
- name: 🐳 Login to DockerHub
if: ${{ env.DOCKERHUB_USERNAME }}
uses: docker/login-action@v3
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: 🐳 Skipping DockerHub login (no secrets available)
if: ${{ !env.DOCKERHUB_USERNAME }}
run: echo "DockerHub login skipped because secrets are not available."
- name: 🐳 Pre-pull testcontainer images
if: ${{ env.DOCKERHUB_USERNAME }}
run: |
echo "Pre-pulling Docker images with authenticated session..."
docker pull postgres:14
docker pull clickhouse/clickhouse-server:25.4-alpine
docker pull redis:7-alpine
docker pull testcontainers/ryuk:0.11.0
docker pull electricsql/electric:1.2.4
echo "Image pre-pull complete"
- name: 📥 Download deps
run: pnpm install --frozen-lockfile
- name: 📀 Generate Prisma Client
run: pnpm run generate
- name: 🧪 Run Package Unit Tests
run: pnpm run test:packages --reporter=default --reporter=blob --shard=${{ matrix.shardIndex }}/${{ matrix.shardTotal }}
- name: Gather all reports
if: ${{ !cancelled() }}
run: |
mkdir -p .vitest-reports
find . -type f -path '*/.vitest-reports/blob-*.json' \
-exec bash -c 'src="$1"; basename=$(basename "$src"); pkg=$(dirname "$src" | sed "s|^\./||;s|/\.vitest-reports$||;s|/|_|g"); cp "$src" ".vitest-reports/${pkg}-${basename}"' _ {} \;
- name: Upload blob reports to GitHub Actions Artifacts
if: ${{ !cancelled() }}
uses: actions/upload-artifact@v4
with:
name: packages-blob-report-${{ matrix.shardIndex }}
path: .vitest-reports/*
include-hidden-files: true
retention-days: 1
merge-reports:
name: "📊 Merge Reports"
if: ${{ !cancelled() }}
needs: [unitTests]
runs-on: ubuntu-latest
steps:
- name: ⬇️ Checkout repo
uses: actions/checkout@v4
with:
fetch-depth: 0
- name: ⎔ Setup pnpm
uses: pnpm/action-setup@v4
with:
version: 10.23.0
- name: ⎔ Setup node
uses: buildjet/setup-node@v4
with:
node-version: 20.20.0
# no cache enabled, we're not installing deps
- name: Download blob reports from GitHub Actions Artifacts
uses: actions/download-artifact@v4
with:
path: .vitest-reports
pattern: packages-blob-report-*
merge-multiple: true
- name: Merge reports
run: pnpm dlx vitest@3.1.4 run --merge-reports --pass-with-no-tests
+149
View File
@@ -0,0 +1,149 @@
name: "🧪 Unit Tests: Webapp"
permissions:
contents: read
on:
workflow_call:
jobs:
unitTests:
name: "🧪 Unit Tests: Webapp"
runs-on: ubuntu-latest
strategy:
matrix:
shardIndex: [1, 2, 3, 4, 5, 6, 7, 8]
shardTotal: [8]
env:
DOCKERHUB_USERNAME: ${{ secrets.DOCKERHUB_USERNAME }}
SHARD_INDEX: ${{ matrix.shardIndex }}
SHARD_TOTAL: ${{ matrix.shardTotal }}
steps:
- name: 🔧 Disable IPv6
run: |
sudo sysctl -w net.ipv6.conf.all.disable_ipv6=1
sudo sysctl -w net.ipv6.conf.default.disable_ipv6=1
sudo sysctl -w net.ipv6.conf.lo.disable_ipv6=1
- name: 🔧 Configure docker address pool
run: |
CONFIG='{
"default-address-pools" : [
{
"base" : "172.17.0.0/12",
"size" : 20
},
{
"base" : "192.168.0.0/16",
"size" : 24
}
]
}'
mkdir -p /etc/docker
echo "$CONFIG" | sudo tee /etc/docker/daemon.json
- name: 🔧 Restart docker daemon
run: sudo systemctl restart docker
- name: ⬇️ Checkout repo
uses: actions/checkout@v4
with:
fetch-depth: 0
- name: ⎔ Setup pnpm
uses: pnpm/action-setup@v4
with:
version: 10.23.0
- name: ⎔ Setup node
uses: buildjet/setup-node@v4
with:
node-version: 20.20.0
cache: "pnpm"
# ..to avoid rate limits when pulling images
- name: 🐳 Login to DockerHub
if: ${{ env.DOCKERHUB_USERNAME }}
uses: docker/login-action@v3
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: 🐳 Skipping DockerHub login (no secrets available)
if: ${{ !env.DOCKERHUB_USERNAME }}
run: echo "DockerHub login skipped because secrets are not available."
- name: 🐳 Pre-pull testcontainer images
if: ${{ env.DOCKERHUB_USERNAME }}
run: |
echo "Pre-pulling Docker images with authenticated session..."
docker pull postgres:14
docker pull clickhouse/clickhouse-server:25.4-alpine
docker pull redis:7-alpine
docker pull testcontainers/ryuk:0.11.0
docker pull electricsql/electric:1.2.4
echo "Image pre-pull complete"
- name: 📥 Download deps
run: pnpm install --frozen-lockfile
- name: 📀 Generate Prisma Client
run: pnpm run generate
- name: 🧪 Run Webapp Unit Tests
run: pnpm run test:webapp --reporter=default --reporter=blob --shard=${{ matrix.shardIndex }}/${{ matrix.shardTotal }}
env:
DATABASE_URL: postgresql://postgres:postgres@localhost:5432/postgres
DIRECT_URL: postgresql://postgres:postgres@localhost:5432/postgres
SESSION_SECRET: "secret"
MAGIC_LINK_SECRET: "secret"
ENCRYPTION_KEY: "dummy-encryption-keeeey-32-bytes"
DEPLOY_REGISTRY_HOST: "docker.io"
CLICKHOUSE_URL: "http://default:password@localhost:8123"
- name: Gather all reports
if: ${{ !cancelled() }}
run: |
mkdir -p .vitest-reports
find . -type f -path '*/.vitest-reports/blob-*.json' \
-exec bash -c 'src="$1"; basename=$(basename "$src"); pkg=$(dirname "$src" | sed "s|^\./||;s|/\.vitest-reports$||;s|/|_|g"); cp "$src" ".vitest-reports/${pkg}-${basename}"' _ {} \;
- name: Upload blob reports to GitHub Actions Artifacts
if: ${{ !cancelled() }}
uses: actions/upload-artifact@v4
with:
name: webapp-blob-report-${{ matrix.shardIndex }}
path: .vitest-reports/*
include-hidden-files: true
retention-days: 1
merge-reports:
name: "📊 Merge Reports"
if: ${{ !cancelled() }}
needs: [unitTests]
runs-on: ubuntu-latest
steps:
- name: ⬇️ Checkout repo
uses: actions/checkout@v4
with:
fetch-depth: 0
- name: ⎔ Setup pnpm
uses: pnpm/action-setup@v4
with:
version: 10.23.0
- name: ⎔ Setup node
uses: buildjet/setup-node@v4
with:
node-version: 20.20.0
# no cache enabled, we're not installing deps
- name: Download blob reports from GitHub Actions Artifacts
uses: actions/download-artifact@v4
with:
path: .vitest-reports
pattern: webapp-blob-report-*
merge-multiple: true
- name: Merge reports
run: pnpm dlx vitest@3.1.4 run --merge-reports --pass-with-no-tests
+12 -47
View File
@@ -1,53 +1,18 @@
name: "🧪 Unit Tests"
permissions:
contents: read
on:
workflow_call:
jobs:
unitTests:
name: "🧪 Unit Tests"
runs-on: ubuntu-latest
steps:
- name: ⬇️ Checkout repo
uses: actions/checkout@v4
with:
fetch-depth: 0
- name: ⎔ Setup pnpm
uses: pnpm/action-setup@v4
with:
version: 8.15.5
- name: ⎔ Setup node
uses: buildjet/setup-node@v4
with:
node-version: 20.11.1
cache: "pnpm"
# ..to avoid rate limits when pulling images
- name: 🐳 Login to DockerHub
uses: docker/login-action@v3
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: 📥 Download deps
run: pnpm install --frozen-lockfile
- name: 📀 Generate Prisma Client
run: pnpm run generate
- name: 🧪 Run Webapp Unit Tests
run: pnpm run test:webapp
env:
DATABASE_URL: postgresql://postgres:postgres@localhost:5432/postgres
DIRECT_URL: postgresql://postgres:postgres@localhost:5432/postgres
SESSION_SECRET: "secret"
MAGIC_LINK_SECRET: "secret"
ENCRYPTION_KEY: "secret"
- name: 🧪 Run Package Unit Tests
run: pnpm run test:packages
- name: 🧪 Run Internal Unit Tests
run: pnpm run test:internal
webapp:
uses: ./.github/workflows/unit-tests-webapp.yml
secrets: inherit
packages:
uses: ./.github/workflows/unit-tests-packages.yml
secrets: inherit
internal:
uses: ./.github/workflows/unit-tests-internal.yml
secrets: inherit
+10 -5
View File
@@ -15,6 +15,9 @@ out/
dist
packages/**/dist
# vendored bundles (generated during build)
packages/**/src/**/vendor
# Tailwind
apps/**/styles/tailwind.css
packages/**/styles/tailwind.css
@@ -29,12 +32,10 @@ yarn-debug.log*
yarn-error.log*
# local env files
.env.docker
.env
.env.*
.docker/*.env
.env.local
.env.development.local
.env.test.local
.env.production.local
!.env.example
# turbo
.turbo
@@ -63,3 +64,7 @@ apps/**/public/build
/packages/core/src/package.json
/packages/trigger-sdk/src/package.json
/packages/python/src/package.json
**/.claude/settings.local.json
.mcp.log
.mcp.json
.cursor/debug.log
-5
View File
@@ -1,5 +0,0 @@
link-workspace-packages=false
public-hoist-pattern[]=*prisma*
prefer-workspace-packages=true
update-notifier=false
side-effects-cache=false
+1 -1
View File
@@ -1 +1 @@
v20.11.1
v20.20.0
+8 -8
View File
@@ -25,8 +25,8 @@
{
"type": "node-terminal",
"request": "launch",
"name": "Debug fairDequeuingStrategy.test.ts",
"command": "pnpm run test -t FairDequeuingStrategy",
"name": "Debug triggerTask.test.ts",
"command": "pnpm run test --run ./test/engine/triggerTask.test.ts",
"envFile": "${workspaceFolder}/.env",
"cwd": "${workspaceFolder}/apps/webapp",
"sourceMaps": true
@@ -59,7 +59,7 @@
"request": "launch",
"name": "Debug V3 Dev CLI",
"command": "pnpm exec trigger dev",
"cwd": "${workspaceFolder}/references/v3-catalog",
"cwd": "${workspaceFolder}/references/hello-world",
"sourceMaps": true
},
{
@@ -83,7 +83,7 @@
"request": "launch",
"name": "Debug V3 Deploy CLI",
"command": "pnpm exec trigger deploy --self-hosted --load-image",
"cwd": "${workspaceFolder}/references/v3-catalog",
"cwd": "${workspaceFolder}/references/hello-world",
"sourceMaps": true
},
{
@@ -91,7 +91,7 @@
"request": "launch",
"name": "Debug V3 list-profiles CLI",
"command": "pnpm exec trigger list-profiles --log-level debug",
"cwd": "${workspaceFolder}/references/v3-catalog",
"cwd": "${workspaceFolder}/references/hello-world",
"sourceMaps": true
},
{
@@ -99,7 +99,7 @@
"request": "launch",
"name": "Debug V3 update CLI",
"command": "pnpm exec trigger update",
"cwd": "${workspaceFolder}/references/v3-catalog",
"cwd": "${workspaceFolder}/references/hello-world",
"sourceMaps": true
},
{
@@ -107,7 +107,7 @@
"request": "launch",
"name": "Debug V3 Management",
"command": "pnpm run management",
"cwd": "${workspaceFolder}/references/v3-catalog",
"cwd": "${workspaceFolder}/references/hello-world",
"sourceMaps": true
},
{
@@ -146,7 +146,7 @@
"type": "node-terminal",
"request": "launch",
"name": "Debug RunQueue tests",
"command": "pnpm run test ./src/run-queue/index.test.ts",
"command": "pnpm run test ./src/run-queue/index.test.ts --run",
"cwd": "${workspaceFolder}/internal-packages/run-engine",
"sourceMaps": true
},
+2 -1
View File
@@ -6,5 +6,6 @@
"**/node_modules/**": true,
"packages/cli-v3/e2e": true
},
"vitest.disableWorkspaceWarning": true
"vitest.disableWorkspaceWarning": true,
"typescript.experimental.useTsgo": false
}
+68
View File
@@ -0,0 +1,68 @@
# Guidance for Coding Agents
This repository is a pnpm monorepo managed with Turbo. It contains multiple apps and packages that make up the Trigger.dev platform and SDK.
## Repository layout
- `apps/webapp` Remix application that serves as the main API and dashboard.
- `apps/supervisor` Node application for executing built tasks.
- `packages/*` Published packages such as `@trigger.dev/sdk`, the CLI (`trigger.dev`), and shared libraries.
- `internal-packages/*` Internal-only packages used by the webapp and other apps.
- `references/*` Example projects for manual testing and development of new features.
- `ai/references` Contains additional documentation including an overview (`repo.md`) and testing guidelines (`tests.md`).
See `ai/references/repo.md` for a more complete explanation of the workspaces.
## Development setup
1. Install dependencies with `pnpm i` (pnpm `10.23.0` and Node.js `20.20.0` are required).
2. Copy `.env.example` to `.env` and generate a random 16 byte hex string for `ENCRYPTION_KEY` (`openssl rand -hex 16`). Update other secrets if needed.
3. Start the local services with Docker:
```bash
pnpm run docker
```
4. Run database migrations:
```bash
pnpm run db:migrate
```
5. Build the webapp, CLI and SDK packages:
```bash
pnpm run build --filter webapp && pnpm run build --filter trigger.dev && pnpm run build --filter @trigger.dev/sdk
```
6. Launch the development server:
```bash
pnpm run dev --filter webapp
```
The webapp runs on <http://localhost:3030>.
For full setup instructions see `CONTRIBUTING.md`.
## Running tests
- Unit tests use **vitest**. Run all tests:
```bash
pnpm run test
```
- Run tests for a specific workspace (example for `webapp`):
```bash
pnpm run test --filter webapp
```
- Prefer running a single test file from within its directory:
```bash
cd apps/webapp
pnpm run test ./src/components/Button.test.ts
```
If packages in that workspace need to be built first, run `pnpm run build --filter webapp`.
Refer to `ai/references/tests.md` for details on writing tests. Tests should avoid mocks or stubs and use the helpers from `@internal/testcontainers` when Redis or Postgres are needed.
## Coding style
- Formatting is enforced using Prettier. Run `pnpm run format` before committing.
- Follow the existing project conventions. Test files live beside the files under test and use descriptive `describe` and `it` blocks.
- Do not commit directly to the `main` branch. All changes should be made in a separate branch and go through a pull request.
## Additional docs
- The root `README.md` describes Trigger.dev and links to documentation.
- The `docs` workspace contains our documentation site, which can be run locally with:
```bash
pnpm run dev --filter docs
```
- `references/README.md` explains how to create new reference projects for manual testing.
+7 -19
View File
@@ -30,28 +30,16 @@ Please follow the best-practice of adding changesets in the same commit as the c
## Snapshot instructions
!MAKE SURE TO UPDATE THE TAG IN THE INSTRUCTIONS BELOW!
1. Update the `.changeset/config.json` file to set the `"changelog"` field to this:
1. Add changesets as usual
```sh
pnpm run changeset:add
```json
"changelog": "@changesets/cli/changelog",
```
2. Create a snapshot version (replace "prerelease" with your tag)
2. Do a temporary commit (do NOT push this, you should undo it after)
```sh
pnpm exec changeset version --snapshot prerelease
```
3. Run `./scripts/publish-prerelease.sh prerelease`
3. Build the packages:
You can choose a different tag if you want, but usually `prerelease` is fine.
```sh
pnpm run build --filter "@trigger.dev/*" --filter "trigger.dev"
```
4. Publish the snapshot (replace "dev" with your tag)
```sh
pnpm exec changeset publish --no-git-tag --snapshot --tag prerelease
```
5. Undo the commit where you updated the config.json file.
+303
View File
@@ -0,0 +1,303 @@
# CLAUDE.md
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
## Build and Development Commands
This is a pnpm 10.23.0 monorepo using Turborepo. Run commands from root with `pnpm run`.
### Essential Commands
```bash
# Start Docker services (PostgreSQL, Redis, Electric)
pnpm run docker
# Run database migrations
pnpm run db:migrate
# Seed the database (required for reference projects)
pnpm run db:seed
# Build packages (required before running)
pnpm run build --filter webapp && pnpm run build --filter trigger.dev && pnpm run build --filter @trigger.dev/sdk
# Run webapp in development mode (http://localhost:3030)
pnpm run dev --filter webapp
# Build and watch for changes (CLI and packages)
pnpm run dev --filter trigger.dev --filter "@trigger.dev/*"
```
### Testing
We use vitest exclusively. **Never mock anything** - use testcontainers instead.
```bash
# Run all tests for a package
pnpm run test --filter webapp
# Run a single test file (preferred - cd into directory first)
cd internal-packages/run-engine
pnpm run test ./src/engine/tests/ttl.test.ts --run
# May need to build dependencies first
pnpm run build --filter @internal/run-engine
```
Test files go next to source files (e.g., `MyService.ts``MyService.test.ts`).
#### Testcontainers for Redis/PostgreSQL
```typescript
import { redisTest, postgresTest, containerTest } from "@internal/testcontainers";
// Redis only
redisTest("should use redis", async ({ redisOptions }) => {
/* ... */
});
// PostgreSQL only
postgresTest("should use postgres", async ({ prisma }) => {
/* ... */
});
// Both Redis and PostgreSQL
containerTest("should use both", async ({ prisma, redisOptions }) => {
/* ... */
});
```
### Changesets
When modifying any public package (`packages/*` or `integrations/*`), add a changeset:
```bash
pnpm run changeset:add
```
- Default to **patch** for bug fixes and minor changes
- Confirm with maintainers before selecting **minor** (new features)
- **Never** select major (breaking changes) without explicit approval
## Architecture Overview
### Apps
- **apps/webapp**: Remix 2.1.0 app - main API, dashboard, and Docker image. Uses Express server.
- **apps/supervisor**: Node.js app handling task execution, interfacing with Docker/Kubernetes.
### Public Packages
- **packages/trigger-sdk** (`@trigger.dev/sdk`): Main SDK
- **packages/cli-v3** (`trigger.dev`): CLI package
- **packages/core** (`@trigger.dev/core`): Shared code between SDK and webapp. Import subpaths only (never root).
- **packages/build**: Build extensions and types
- **packages/react-hooks**: React hooks for realtime and triggering
- **packages/redis-worker** (`@trigger.dev/redis-worker`): Custom Redis-based background job system
### Internal Packages
- **internal-packages/database** (`@trigger.dev/database`): Prisma 6.14.0 client and schema
- **internal-packages/clickhouse** (`@internal/clickhouse`): ClickHouse client and schema migrations
- **internal-packages/run-engine** (`@internal/run-engine`): "Run Engine 2.0" - run lifecycle management
- **internal-packages/redis** (`@internal/redis`): Redis client creation utilities
- **internal-packages/testcontainers** (`@internal/testcontainers`): Test helpers for Redis/PostgreSQL containers
- **internal-packages/zodworker** (`@internal/zodworker`): Graphile-worker wrapper (being replaced by redis-worker)
### Reference Projects
The `references/` directory contains test workspaces for developing and testing new SDK and platform features. Use these projects (e.g., `references/hello-world`) to manually test changes to the CLI, SDK, core packages, and webapp before submitting PRs.
## Webapp Development
### Key Locations
- Trigger API: `apps/webapp/app/routes/api.v1.tasks.$taskId.trigger.ts`
- Batch trigger: `apps/webapp/app/routes/api.v1.tasks.batch.ts`
- Prisma setup: `apps/webapp/app/db.server.ts`
- Run engine config: `apps/webapp/app/v3/runEngine.server.ts`
- Services: `apps/webapp/app/v3/services/**/*.server.ts`
- Presenters: `apps/webapp/app/v3/presenters/**/*.server.ts`
- OTEL endpoints: `apps/webapp/app/routes/otel.v1.logs.ts`, `otel.v1.traces.ts`
### Environment Variables
Access via `env` export from `apps/webapp/app/env.server.ts`, never `process.env` directly.
For testable code, **never import env.server.ts** in test files. Pass configuration as options instead. Example pattern:
- `realtimeClient.server.ts` (testable service)
- `realtimeClientGlobal.server.ts` (configuration)
### Legacy vs Run Engine 2.0
The codebase is transitioning from the "legacy run engine" (spread across codebase) to "Run Engine 2.0" (`@internal/run-engine`). Focus on Run Engine 2.0 for new work.
## Docker Image Guidelines
When updating Docker image references in `docker/Dockerfile` or other container files:
- **Always use multiplatform/index digests**, not architecture-specific digests
- Architecture-specific digests (e.g., for `linux/amd64` only) will cause CI failures on different build environments
- On Docker Hub, the multiplatform digest is shown on the main image page, while architecture-specific digests are listed under "OS/ARCH"
- Example: Use `node:20.20-bullseye-slim@sha256:abc123...` where the digest is from the multiplatform index, not from a specific OS/ARCH variant
## Database Migrations (PostgreSQL)
1. Edit `internal-packages/database/prisma/schema.prisma`
2. Create migration:
```bash
cd internal-packages/database
pnpm run db:migrate:dev:create --name "add_new_column"
```
3. **Important**: Generated migration includes extraneous changes. Remove lines related to:
- `_BackgroundWorkerToBackgroundWorkerFile`
- `_BackgroundWorkerToTaskQueue`
- `_TaskRunToTaskRunTag`
- `_WaitpointRunConnections`
- `_completedWaitpoints`
- `SecretStore_key_idx`
- Various `TaskRun` indexes unless you added them
4. Apply migration:
```bash
pnpm run db:migrate:deploy && pnpm run generate
```
### Index Migration Rules
- Indexes **must use CONCURRENTLY** to avoid table locks
- **CONCURRENTLY indexes must be in their own separate migration file** - they cannot be combined with other schema changes
## ClickHouse Migrations
ClickHouse migrations use Goose format and live in `internal-packages/clickhouse/schema/`.
1. Create a new numbered SQL file (e.g., `010_add_new_column.sql`)
2. Use Goose markers:
```sql
-- +goose Up
ALTER TABLE trigger_dev.your_table
ADD COLUMN new_column String DEFAULT '';
-- +goose Down
ALTER TABLE trigger_dev.your_table
DROP COLUMN new_column;
```
Follow naming conventions in `internal-packages/clickhouse/README.md`:
- `raw_` prefix for input tables
- `_v1`, `_v2` suffixes for versioning
- `_mv_v1` suffix for materialized views
## Writing Trigger.dev Tasks
Always import from `@trigger.dev/sdk`. Never use `@trigger.dev/sdk/v3` or deprecated `client.defineJob` pattern.
```typescript
import { task } from "@trigger.dev/sdk";
// Every task must be exported
export const myTask = task({
id: "my-task", // Unique ID
run: async (payload: { message: string }) => {
// Task logic - no timeouts
},
});
```
### SDK Documentation Rules
The `rules/` directory contains versioned documentation for writing Trigger.dev tasks, distributed to users via the SDK installer. Current version is defined in `rules/manifest.json`.
- `rules/4.3.0/` - Latest: batch trigger v2 (1,000 items, 3MB payloads), debouncing
- `rules/4.1.0/` - Realtime streams v2, updated config
- `rules/4.0.0/` - Base v4 SDK documentation
When adding new SDK features, create a new version directory with only the files that changed from the previous version. Update `manifest.json` to point unchanged files to previous versions.
### Claude Code Skill
The `.claude/skills/trigger-dev-tasks/` skill provides Claude Code with Trigger.dev task expertise. It includes:
- `SKILL.md` - Core instructions and patterns
- Reference files for basic tasks, advanced tasks, scheduled tasks, realtime, and config
Keep the skill in sync with the latest rules version when SDK features change.
## Testing with hello-world Reference Project
First-time setup:
1. Run `pnpm run db:seed` to seed the database (creates the hello-world project)
2. Build CLI: `pnpm run build --filter trigger.dev && pnpm i`
3. Authorize CLI: `cd references/hello-world && pnpm exec trigger login -a http://localhost:3030`
Running:
```bash
cd references/hello-world
pnpm exec trigger dev # or with --log-level debug
```
## Local Task Testing Workflow
This workflow enables Claude Code to run the webapp and trigger dev simultaneously, trigger tasks, and inspect results for testing code changes.
### Step 1: Start Webapp in Background
```bash
# Run from repo root with run_in_background: true
pnpm run dev --filter webapp
```
Verify webapp is running:
```bash
curl -s http://localhost:3030/healthcheck # Should return 200
```
### Step 2: Start Trigger Dev in Background
```bash
# Run from hello-world directory with run_in_background: true
cd references/hello-world && pnpm exec trigger dev
```
The worker will build and register tasks. Check output for "Local worker ready [node]" message.
### Step 3: Trigger and Monitor Tasks via MCP
Use the Trigger.dev MCP tools to interact with tasks:
```
# Get current worker and registered tasks
mcp__trigger__get_current_worker(projectRef: "proj_rrkpdguyagvsoktglnod", environment: "dev")
# Trigger a task
mcp__trigger__trigger_task(
projectRef: "proj_rrkpdguyagvsoktglnod",
environment: "dev",
taskId: "hello-world",
payload: {"message": "Hello from Claude"}
)
# List runs to see status
mcp__trigger__list_runs(
projectRef: "proj_rrkpdguyagvsoktglnod",
environment: "dev",
taskIdentifier: "hello-world",
limit: 5
)
```
### Step 4: Monitor Execution
- Check trigger dev output file for real-time execution logs
- Successful runs show: `Task | Run ID | Success (Xms)`
- Dashboard available at: http://localhost:3030/orgs/references-9dfd/projects/hello-world-97DT/env/dev/runs
### Key Project Refs
- hello-world: `proj_rrkpdguyagvsoktglnod`
+23 -82
View File
@@ -14,8 +14,8 @@ branch are tagged into a release periodically.
### Prerequisites
- [Node.js](https://nodejs.org/en) version 20.11.1
- [pnpm package manager](https://pnpm.io/installation) version 8.15.5
- [Node.js](https://nodejs.org/en) version 20.20.0
- [pnpm package manager](https://pnpm.io/installation) version 10.23.0
- [Docker](https://www.docker.com/get-started/)
- [protobuf](https://github.com/protocolbuffers/protobuf)
@@ -34,9 +34,9 @@ branch are tagged into a release periodically.
```
cd trigger.dev
```
3. Ensure you are on the correct version of Node.js (20.11.1). If you are using `nvm`, there is an `.nvmrc` file that will automatically select the correct version of Node.js when you navigate to the repository.
3. Ensure you are on the correct version of Node.js (20.20.0). If you are using `nvm`, there is an `.nvmrc` file that will automatically select the correct version of Node.js when you navigate to the repository.
4. Run `corepack enable` to use the correct version of pnpm (`8.15.5`) as specified in the root `package.json` file.
4. Run `corepack enable` to use the correct version of pnpm (`10.23.0`) as specified in the root `package.json` file.
5. Install the required packages using pnpm.
```
@@ -62,15 +62,13 @@ branch are tagged into a release periodically.
pnpm run docker
```
This will also start and run a local instance of [pgAdmin](https://www.pgadmin.org/) on [localhost:5480](http://localhost:5480), preconfigured with email `admin@example.com` and pwd `admin`. Then use `postgres` as the password to the Trigger.dev server.
9. Migrate the database
```
pnpm run db:migrate
```
10. Build the server app
10. Build everything
```
pnpm run build --filter webapp
pnpm run build --filter webapp && pnpm run build --filter trigger.dev && pnpm run build --filter @trigger.dev/sdk
```
11. Run the app. See the section below.
@@ -86,21 +84,19 @@ branch are tagged into a release periodically.
2. Once the app is running click the magic link button and enter your email. You will automatically be logged in, since you are running locally. Create an Org and your first project in the dashboard.
## Manual testing using v3-catalog
## Manual testing using hello-world
We use the `<root>/references/v3-catalog` subdirectory as a staging ground for testing changes to the SDK (`@trigger.dev/sdk` at `<root>/packages/trigger-sdk`), the Core package (`@trigger.dev/core` at `<root>packages/core`), the CLI (`trigger.dev` at `<root>/packages/cli-v3`) and the platform (The remix app at `<root>/apps/webapp`). The instructions below will get you started on using the `v3-catalog` for local development of Trigger.dev (v3).
We use the `<root>/references/hello-world` subdirectory as a staging ground for testing changes to the SDK (`@trigger.dev/sdk` at `<root>/packages/trigger-sdk`), the Core package (`@trigger.dev/core` at `<root>packages/core`), the CLI (`trigger.dev` at `<root>/packages/cli-v3`) and the platform (The remix app at `<root>/apps/webapp`). The instructions below will get you started on using the `hello-world` for local development of Trigger.dev.
### First-time setup
First, make sure you are running the webapp according to the instructions above. Then:
1. In Postgres go to the "Organizations" table and on your org set the `v3Enabled` column to `true`.
1. Visit http://localhost:3030 in your browser and create a new project called "hello-world".
2. Visit http://localhost:3030 in your browser and create a new V3 project called "v3-catalog". If you don't see an option for V3, you haven't set the `v3Enabled` flag to true.
2. In Postgres go to the "Projects" table and for the project you create change the `externalRef` to `proj_rrkpdguyagvsoktglnod`.
3. In Postgres go to the "Projects" table and for the project you create change the `externalRef` to `yubjwjsfkxnylobaqvqz`.
4. Build the CLI
3. Build the CLI
```sh
# Build the CLI
@@ -109,10 +105,10 @@ pnpm run build --filter trigger.dev
pnpm i
```
5. Change into the `<root>/references/v3-catalog` directory and authorize the CLI to the local server:
4. Change into the `<root>/references/hello-world` directory and authorize the CLI to the local server:
```sh
cd references/v3-catalog
cd references/hello-world
cp .env.example .env
pnpm exec trigger login -a http://localhost:3030
```
@@ -122,7 +118,7 @@ This will open a new browser window and authorize the CLI against your local use
You can optionally pass a `--profile` flag to the `login` command, which will allow you to use the CLI with separate accounts/servers. We suggest using a profile called `local` for your local development:
```sh
cd references/v3-catalog
cd references/hello-world
pnpm exec trigger login -a http://localhost:3030 --profile local
# later when you run the dev or deploy command:
pnpm exec trigger dev --profile local
@@ -131,7 +127,7 @@ pnpm exec trigger deploy --profile local
### Running
The following steps should be followed any time you start working on a new feature you want to test in v3:
The following steps should be followed any time you start working on a new feature you want to test:
1. Make sure the webapp is running on localhost:3030
@@ -141,84 +137,29 @@ The following steps should be followed any time you start working on a new featu
pnpm run dev --filter trigger.dev --filter "@trigger.dev/*"
```
3. Open another terminal window, and change into the `<root>/references/v3-catalog` directory.
3. Open another terminal window, and change into the `<root>/references/hello-world` directory.
4. You'll need to run the following commands to setup prisma and migrate the database:
4. Run the `dev` command, which will register all the local tasks with the platform and allow you to start testing task execution:
```sh
pnpm exec prisma migrate deploy
pnpm run generate:prisma
```
5. Run the `dev` command, which will register all the local tasks with the platform and allow you to start testing task execution:
```sh
# in <root>/references/v3-catalog
# in <root>/references/hello-world
pnpm exec trigger dev
```
If you want additional debug logging, you can use the `--log-level debug` flag:
```sh
# in <root>/references/v3-catalog
# in <root>/references/hello-world
pnpm exec trigger dev --log-level debug
```
6. If you make any changes in the CLI/Core/SDK, you'll need to `CTRL+C` to exit the `dev` command and restart it to pickup changes. Any changes to the files inside of the `v3-catalog/src/trigger` dir will automatically be rebuilt by the `dev` command.
6. If you make any changes in the CLI/Core/SDK, you'll need to `CTRL+C` to exit the `dev` command and restart it to pickup changes. Any changes to the files inside of the `hello-world/src/trigger` dir will automatically be rebuilt by the `dev` command.
7. Navigate to the `v3-catalog` project in your local dashboard at localhost:3030 and you should see the list of tasks.
7. Navigate to the `hello-world` project in your local dashboard at localhost:3030 and you should see the list of tasks.
8. Go to the "Test" page in the sidebar and select a task. Then enter a payload and click "Run test". You can tell what the payloads should be by looking at the relevant task file inside the `/references/v3-catalog/src/trigger` folder. Many of them accept an empty payload.
8. Go to the "Test" page in the sidebar and select a task. Then enter a payload and click "Run test". You can tell what the payloads should be by looking at the relevant task file inside the `/references/hello-world/src/trigger` folder. Many of them accept an empty payload.
9. Feel free to add additional files in `v3-catalog/src/trigger` to test out specific aspects of the system, or add in edge cases.
## Running end-to-end webapp tests (deprecated)
To run the end-to-end tests, follow the steps below:
1. Set up environment variables (copy example envs into the correct place)
```sh
cp ./.env.example ./.env
cp ./references/nextjs-test/.env.example ./references/nextjs-test/.env.local
```
2. Set up dependencies
```sh
# Build packages
pnpm run build --filter @references/nextjs-test^...
pnpm --filter @trigger.dev/database generate
# Move trigger-cli bin to correct place
pnpm install --frozen-lockfile
# Install playwrite browsers (ONE TIME ONLY)
npx playwright install
```
3. Set up the database
```sh
pnpm run docker
pnpm run db:migrate
pnpm run db:seed
```
4. Run the end-to-end tests
```sh
pnpm run test:e2e
```
### Cleanup
The end-to-end tests use a `setup` and `teardown` script to seed the database with test data. If the test runner doesn't exit cleanly, then the database can be left in a state where the tests can't run because the `setup` script will try to create data that already exists. If this happens, you can manually delete the `users` and `organizations` from the database using prisma studio:
```sh
# With the database running (i.e. pnpm run docker)
pnpm run db:studio
```
9. Feel free to add additional files in `hello-world/src/trigger` to test out specific aspects of the system, or add in edge cases.
## Adding and running migrations
+24 -15
View File
@@ -8,47 +8,56 @@ If you don't have Docker installed on your machine, you'll run into some complic
Below are the steps on how you can avoid that.
First you need to setup docker-compose as it is an underlying tool that this command: `pnpm run docker` fires behind the scene.
First you need to setup docker compose as it is an underlying tool that this command: `pnpm run docker` fires behind the scene.
## Linux
To install Docker Compose on Linux Ubuntu via the terminal, you can follow these steps:
To install Docker Compose on Linux Ubuntu, you can follow these steps:
1. Update the package index on your system by running the following command:
1. Create the Docker config directory and cli-plugins subdirectory:
```shell
sudo apt update
DOCKER_CONFIG=${DOCKER_CONFIG:-$HOME/.docker}
mkdir -p $DOCKER_CONFIG/cli-plugins
```
2. Install the required dependencies by running the following command:
2. Download the Docker Compose plugin:
```shell
sudo apt install curl
curl -SL "https://github.com/docker/compose/releases/latest/download/docker-compose-$(uname -s)-$(uname -m)" -o $DOCKER_CONFIG/cli-plugins/docker-compose
```
3. Download the Docker Compose binary into the `/usr/local/bin` directory using the `curl` command:
Note:
- To install for all users, replace `$DOCKER_CONFIG/cli-plugins` with `/usr/local/lib/docker/cli-plugins`
3. Set the appropriate permissions to make the Docker Compose plugin executable:
```shell
sudo curl -L "https://github.com/docker/compose/releases/latest/download/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
chmod +x $DOCKER_CONFIG/cli-plugins/docker-compose
```
4. Set the appropriate permissions to make the `docker-compose` binary executable:
If you installed for all users:
```shell
sudo chmod +x /usr/local/bin/docker-compose
sudo chmod +x /usr/local/lib/docker/cli-plugins/docker-compose
```
5. Verify that Docker Compose has been successfully installed by running the following command:
4. Verify that Docker Compose has been successfully installed:
```shell
docker-compose --version
docker compose version
```
This command should display the version information of Docker Compose without any errors.
You should see output similar to:
After following these steps, you should have Docker Compose installed on your Ubuntu system, and you can use it by running `docker-compose` commands in the terminal.
```
Docker Compose version vX.Y.Z
```
When you've verified that the `docker-compose` package is installed and you proceed to start Docker with `pnpm run docker`.
After following these steps, you should have Docker Compose installed on your Ubuntu system, and you can use it by running `docker compose` commands in the terminal.
When you've verified that the `docker compose` package is installed and you proceed to start Docker with `pnpm run docker`.
You'll probably get an error similar to the one below:
+68 -31
View File
@@ -1,44 +1,76 @@
<div align="center">
<picture>
<source media="(prefers-color-scheme: dark)" srcset="https://imagedelivery.net/3TbraffuDZ4aEf8KWOmI_w/a45d1fa2-0ae8-4a39-4409-f4f934bfae00/public">
<source media="(prefers-color-scheme: light)" srcset="https://imagedelivery.net/3TbraffuDZ4aEf8KWOmI_w/3f5ad4c1-c4c8-4277-b622-290e7f37bd00/public">
<img alt="Trigger.dev logo" src="https://imagedelivery.net/3TbraffuDZ4aEf8KWOmI_w/a45d1fa2-0ae8-4a39-4409-f4f934bfae00/public">
</picture>
### Open source background jobs and AI infrastructure
[Discord](https://trigger.dev/discord) | [Website](https://trigger.dev) | [Issues](https://github.com/triggerdotdev/trigger.dev/issues) | [Docs](https://trigger.dev/docs)
![Trigger.dev logo](https://content.trigger.dev/github-header-banner.jpg)
[![Twitter](https://img.shields.io/twitter/url/https/twitter.com/triggerdotdev.svg?style=social&label=Follow%20%40trigger.dev)](https://twitter.com/triggerdotdev)
### Build and deploy fullymanaged AI agents and workflows
[Website](https://trigger.dev) | [Docs](https://trigger.dev/docs) | [Issues](https://github.com/triggerdotdev/trigger.dev/issues) | [Example projects](https://github.com/triggerdotdev/examples) | [Feature requests](https://triggerdev.featurebase.app/) | [Public roadmap](https://triggerdev.featurebase.app/roadmap) | [Self-hosting](https://trigger.dev/docs/self-hosting/overview)
[![Open Source](https://img.shields.io/badge/Open%20Source-%E2%9D%A4-red.svg)](https://github.com/triggerdotdev/trigger.dev)
[![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/triggerdotdev/trigger.dev/blob/main/LICENSE)
[![npm](https://img.shields.io/npm/v/@trigger.dev/sdk.svg?label=npm)](https://www.npmjs.com/package/@trigger.dev/sdk)
[![SDK downloads](https://img.shields.io/npm/dm/@trigger.dev/sdk.svg?label=SDK%20downloads)](https://www.npmjs.com/package/@trigger.dev/sdk)
[![Twitter Follow](https://img.shields.io/twitter/follow/triggerdotdev?style=social)](https://twitter.com/triggerdotdev)
[![Discord](https://img.shields.io/discord/1066956501299777596?logo=discord&logoColor=white&color=7289da)](https://discord.gg/nkqV9xBYWy)
[![Ask DeepWiki](https://deepwiki.com/badge.svg)](https://deepwiki.com/triggerdotdev/trigger.dev)
[![GitHub stars](https://img.shields.io/github/stars/triggerdotdev/trigger.dev?style=social)](https://github.com/triggerdotdev/trigger.dev)
</div>
## About Trigger.dev
Trigger.dev is an open source platform and SDK which allows you to create long-running background jobs. Write normal async code, deploy, and never hit a timeout.
Trigger.dev is the open-source platform for building AI workflows in TypeScript. Long-running tasks with retries, queues, observability, and elastic scaling.
### Key features:
## The platform designed for building AI agents
- JavaScript and TypeScript SDK
- No timeouts
- Retries (with exponential backoff)
- Queues and concurrency controls
- Schedules and crons
- Full Observability; logs, live trace views, advanced filtering
- React hooks to interact with the Trigger API from your React app
- Pipe LLM streams straight to your users through the Realtime API
- Trigger tasks and display the run status and metadata anywhere in your app
- Custom alerts, get notified by email, Slack or webhooks
- No infrastructure to manage
- Elastic (scaling)
- Works with your existing tech stack
Build [AI agents](https://trigger.dev/product/ai-agents) using all the frameworks, services and LLMs you're used to, deploy them to Trigger.dev and get durable, long-running tasks with retries, queues, observability, and elastic scaling out of the box.
## In your codebase
- **Long-running without timeouts**: Execute your tasks with absolutely no timeouts, unlike AWS Lambda, Vercel, and other serverless platforms.
- **Durability, retries & queues**: Build rock solid agents and AI applications using our durable tasks, retries, queues and idempotency.
- **True runtime freedom**: Customize your deployed tasks with system packages run browsers, Python scripts, FFmpeg and more.
- **Human-in-the-loop**: Programmatically pause your tasks until a human can approve, reject or give feedback.
- **Realtime apps & streaming**: Move your background jobs to the foreground by subscribing to runs or streaming AI responses to your app.
- **Observability & monitoring**: Each run has full tracing and logs. Configure error alerts to catch bugs fast.
## Key features:
- **[JavaScript and TypeScript SDK](https://trigger.dev/docs/tasks/overview)** - Build background tasks using familiar programming models
- **[Long-running tasks](https://trigger.dev/docs/runs/max-duration)** - Handle resource-heavy tasks without timeouts
- **[Durable cron schedules](https://trigger.dev/docs/tasks/scheduled#scheduled-tasks-cron)** - Create and attach recurring schedules of up to a year
- **[Trigger.dev Realtime](https://trigger.dev/docs/realtime/overview)** - Trigger, subscribe to, and get real-time updates for runs, with LLM streaming support
- **[Build extensions](https://trigger.dev/docs/config/extensions/overview#build-extensions)** - Hook directly into the build system and customize the build process. Run Python scripts, FFmpeg, browsers, and more.
- **[React hooks](https://trigger.dev/docs/frontend/react-hooks#react-hooks)** - Interact with the Trigger.dev API on your frontend using our React hooks package
- **[Batch triggering](https://trigger.dev/docs/triggering#tasks-batchtrigger)** - Use batchTrigger() to initiate multiple runs of a task with custom payloads and options
- **[Structured inputs / outputs](https://trigger.dev/docs/tasks/schemaTask#schematask)** - Define precise data schemas for your tasks with runtime payload validation
- **[Waits](https://trigger.dev/docs/wait)** - Add waits to your tasks to pause execution for a specified duration
- **[Preview branches](https://trigger.dev/docs/deployment/preview-branches)** - Create isolated environments for testing and development. Integrates with Vercel and git workflows
- **[Waitpoints](https://trigger.dev/docs/wait-for-token#wait-for-token)** - Add human-in-the-loop judgment at critical decision points without disrupting workflow
- **[Concurrency & queues](https://trigger.dev/docs/queue-concurrency#concurrency-and-queues)** - Set concurrency rules to manage how multiple tasks execute
- **[Multiple environments](https://trigger.dev/docs/how-it-works#dev-mode)** - Support for DEV, PREVIEW, STAGING, and PROD environments
- **[No infrastructure to manage](https://trigger.dev/docs/how-it-works#trigger-dev-architecture)** - Auto-scaling infrastructure that eliminates timeouts and server management
- **[Automatic retries](https://trigger.dev/docs/errors-retrying)** - If your task encounters an uncaught error, we automatically attempt to run it again
- **[Checkpointing](https://trigger.dev/docs/how-it-works#the-checkpoint-resume-system)** - Tasks are inherently durable, thanks to our checkpointing feature
- **[Versioning](https://trigger.dev/docs/versioning)** - Atomic versioning allows you to deploy new versions without affecting running tasks
- **[Machines](https://trigger.dev/docs/machines)** - Configure the number of vCPUs and GBs of RAM you want the task to use
- **[Observability & monitoring](https://trigger.dev/product/observability-and-monitoring)** - Monitor every aspect of your tasks' performance with comprehensive logging and visualization tools
- **[Logging & tracing](https://trigger.dev/docs/logging)** - Comprehensive logging and tracing for all your tasks
- **[Tags](https://trigger.dev/docs/tags#tags)** - Attach up to ten tags to each run, allowing you to filter via the dashboard, realtime, and the SDK
- **[Run metadata](https://trigger.dev/docs/runs/metadata#run-metadata)** - Attach metadata to runs which updates as the run progresses and is available to use in your frontend for live updates
- **[Bulk actions](https://trigger.dev/docs/bulk-actions)** - Perform actions on multiple runs simultaneously, including replaying and cancelling
- **[Real-time alerts](https://trigger.dev/docs/troubleshooting-alerts#alerts)** - Choose your preferred notification method for run failures and deployments
## Write tasks in your codebase
Create tasks where they belong: in your codebase. Version control, localhost, test and review like you're already used to.
```ts
import { task } from "@trigger.dev/sdk/v3";
import { task } from "@trigger.dev/sdk";
//1. You need to export each task
export const helloWorld = task({
@@ -58,13 +90,13 @@ Use our SDK to write tasks in your codebase. There's no infrastructure to manage
## Environments
We support `Development`, `Staging`, and `Production` environments, allowing you to test your tasks before deploying them to production.
We support `Development`, `Staging`, `Preview`, and `Production` environments, allowing you to test your tasks before deploying them to production.
## Full visibility of every job run
View every task in every run so you can tell exactly what happened. We provide a full trace view of every task run so you can see what happened at every step.
![Trace view image](https://imagedelivery.net/3TbraffuDZ4aEf8KWOmI_w/7c1b347f-004c-4482-38a7-3f6fa9c00d00/public)
![Trace view image](https://content.trigger.dev/trace-view.png)
# Getting started
@@ -73,14 +105,19 @@ The quickest way to get started is to create an account and project in our [web
### Useful links:
- [Quick start](https://trigger.dev/docs/quick-start) - get up and running in minutes
- [How it works](https://trigger.dev/docs/v3/how-it-works) - understand how Trigger.dev works under the hood
- [How it works](https://trigger.dev/docs/how-it-works) - understand how Trigger.dev works under the hood
- [Guides and examples](https://trigger.dev/docs/guides/introduction) - walk-through guides and code examples for popular frameworks and use cases
## Self-hosting
If you prefer to self-host Trigger.dev, you can follow our [self-hosting guide](https://trigger.dev/docs/v3/open-source-self-hosting#overview).
If you prefer to self-host Trigger.dev, you can follow our [self-hosting guides](https://trigger.dev/docs/self-hosting/overview):
We also have a dedicated self-hosting channel in our [Discord server](https://trigger.dev/discord) for support.
- [Docker self-hosting guide](https://trigger.dev/docs/self-hosting/docker) - use Docker Compose to spin up a Trigger.dev instance
- [Kubernetes self-hosting guide](https://trigger.dev/docs/self-hosting/kubernetes) - use our official Helm chart to deploy Trigger.dev to your Kubernetes cluster
## Support and community
We have a large active community in our official [Discord server](https://trigger.dev/discord) for support, including a dedicated channel for self-hosting.
## Development
+121
View File
@@ -0,0 +1,121 @@
## Creating and applying migrations
We use prisma migrations to manage the database schema. Please follow the following steps when editing the `internal-packages/database/prisma/schema.prisma` file:
Edit the `schema.prisma` file to add or modify the schema.
Create a new migration file but don't apply it yet:
```bash
cd internal-packages/database
pnpm run db:migrate:dev:create --name "add_new_column_to_table"
```
The migration file will be created in the `prisma/migrations` directory, but it will have a bunch of edits to the schema that are not needed and will need to be removed before we can apply the migration. Here's an example of what the migration file might look like:
```sql
-- AlterEnum
ALTER TYPE "public"."TaskRunExecutionStatus" ADD VALUE 'DELAYED';
-- AlterTable
ALTER TABLE "public"."TaskRun" ADD COLUMN "debounce" JSONB;
-- AlterTable
ALTER TABLE "public"."_BackgroundWorkerToBackgroundWorkerFile" ADD CONSTRAINT "_BackgroundWorkerToBackgroundWorkerFile_AB_pkey" PRIMARY KEY ("A", "B");
-- DropIndex
DROP INDEX "public"."_BackgroundWorkerToBackgroundWorkerFile_AB_unique";
-- AlterTable
ALTER TABLE "public"."_BackgroundWorkerToTaskQueue" ADD CONSTRAINT "_BackgroundWorkerToTaskQueue_AB_pkey" PRIMARY KEY ("A", "B");
-- DropIndex
DROP INDEX "public"."_BackgroundWorkerToTaskQueue_AB_unique";
-- AlterTable
ALTER TABLE "public"."_TaskRunToTaskRunTag" ADD CONSTRAINT "_TaskRunToTaskRunTag_AB_pkey" PRIMARY KEY ("A", "B");
-- DropIndex
DROP INDEX "public"."_TaskRunToTaskRunTag_AB_unique";
-- AlterTable
ALTER TABLE "public"."_WaitpointRunConnections" ADD CONSTRAINT "_WaitpointRunConnections_AB_pkey" PRIMARY KEY ("A", "B");
-- DropIndex
DROP INDEX "public"."_WaitpointRunConnections_AB_unique";
-- AlterTable
ALTER TABLE "public"."_completedWaitpoints" ADD CONSTRAINT "_completedWaitpoints_AB_pkey" PRIMARY KEY ("A", "B");
-- DropIndex
DROP INDEX "public"."_completedWaitpoints_AB_unique";
-- CreateIndex
CREATE INDEX "SecretStore_key_idx" ON "public"."SecretStore"("key" text_pattern_ops);
-- CreateIndex
CREATE INDEX "TaskRun_runtimeEnvironmentId_id_idx" ON "public"."TaskRun"("runtimeEnvironmentId", "id" DESC);
-- CreateIndex
CREATE INDEX "TaskRun_runtimeEnvironmentId_createdAt_idx" ON "public"."TaskRun"("runtimeEnvironmentId", "createdAt" DESC);
```
All the following lines should be removed:
```sql
-- AlterTable
ALTER TABLE "public"."_BackgroundWorkerToBackgroundWorkerFile" ADD CONSTRAINT "_BackgroundWorkerToBackgroundWorkerFile_AB_pkey" PRIMARY KEY ("A", "B");
-- DropIndex
DROP INDEX "public"."_BackgroundWorkerToBackgroundWorkerFile_AB_unique";
-- AlterTable
ALTER TABLE "public"."_BackgroundWorkerToTaskQueue" ADD CONSTRAINT "_BackgroundWorkerToTaskQueue_AB_pkey" PRIMARY KEY ("A", "B");
-- DropIndex
DROP INDEX "public"."_BackgroundWorkerToTaskQueue_AB_unique";
-- AlterTable
ALTER TABLE "public"."_TaskRunToTaskRunTag" ADD CONSTRAINT "_TaskRunToTaskRunTag_AB_pkey" PRIMARY KEY ("A", "B");
-- DropIndex
DROP INDEX "public"."_TaskRunToTaskRunTag_AB_unique";
-- AlterTable
ALTER TABLE "public"."_WaitpointRunConnections" ADD CONSTRAINT "_WaitpointRunConnections_AB_pkey" PRIMARY KEY ("A", "B");
-- DropIndex
DROP INDEX "public"."_WaitpointRunConnections_AB_unique";
-- AlterTable
ALTER TABLE "public"."_completedWaitpoints" ADD CONSTRAINT "_completedWaitpoints_AB_pkey" PRIMARY KEY ("A", "B");
-- DropIndex
DROP INDEX "public"."_completedWaitpoints_AB_unique";
-- CreateIndex
CREATE INDEX "SecretStore_key_idx" ON "public"."SecretStore"("key" text_pattern_ops);
-- CreateIndex
CREATE INDEX "TaskRun_runtimeEnvironmentId_id_idx" ON "public"."TaskRun"("runtimeEnvironmentId", "id" DESC);
-- CreateIndex
CREATE INDEX "TaskRun_runtimeEnvironmentId_createdAt_idx" ON "public"."TaskRun"("runtimeEnvironmentId", "createdAt" DESC);
```
Leaving only this:
```sql
-- AlterEnum
ALTER TYPE "public"."TaskRunExecutionStatus" ADD VALUE 'DELAYED';
-- AlterTable
ALTER TABLE "public"."TaskRun" ADD COLUMN "debounce" JSONB;
```
After editing the migration file, apply the migration:
```bash
cd internal-packages/database
pnpm run db:migrate:deploy && pnpm run generate
```
+1 -1
View File
@@ -1,6 +1,6 @@
## Repo Overview
This is a pnpm 8.15.5 monorepo that uses turborepo @turbo.json. The following workspaces are relevant
This is a pnpm 10.23.0 monorepo that uses turborepo @turbo.json. The following workspaces are relevant
## Apps
+1 -1
View File
@@ -35,7 +35,7 @@ COPY --from=pruner --chown=node:node /app/out/full/ .
COPY --from=dev-deps --chown=node:node /app/ .
COPY --chown=node:node turbo.json turbo.json
RUN pnpm run -r --filter coordinator build:bundle
RUN pnpm run -r --filter @trigger.dev/core bundle-vendor && pnpm run -r --filter coordinator build:bundle
FROM alpine AS cri-tools
+1 -1
View File
@@ -1,6 +1,6 @@
{
"compilerOptions": {
"target": "es2018",
"target": "es2020",
"module": "commonjs",
"esModuleInterop": true,
"resolveJsonModule": true,
+1 -1
View File
@@ -31,7 +31,7 @@ COPY --from=pruner --chown=node:node /app/out/full/ .
COPY --from=dev-deps --chown=node:node /app/ .
COPY --chown=node:node turbo.json turbo.json
RUN pnpm run -r --filter docker-provider build:bundle
RUN pnpm run -r --filter @trigger.dev/core bundle-vendor && pnpm run -r --filter docker-provider build:bundle
FROM base AS runner
+1 -1
View File
@@ -1,6 +1,6 @@
{
"compilerOptions": {
"target": "es2018",
"target": "es2020",
"module": "commonjs",
"esModuleInterop": true,
"forceConsistentCasingInFileNames": true,
+1 -1
View File
@@ -31,7 +31,7 @@ COPY --from=pruner --chown=node:node /app/out/full/ .
COPY --from=dev-deps --chown=node:node /app/ .
COPY --chown=node:node turbo.json turbo.json
RUN pnpm run -r --filter kubernetes-provider build:bundle
RUN pnpm run -r --filter @trigger.dev/core bundle-vendor && pnpm run -r --filter kubernetes-provider build:bundle
FROM base AS runner
+1 -1
View File
@@ -1,6 +1,6 @@
{
"compilerOptions": {
"target": "es2018",
"target": "es2020",
"module": "commonjs",
"esModuleInterop": true,
"forceConsistentCasingInFileNames": true,
+1 -2
View File
@@ -7,12 +7,11 @@ MANAGED_WORKER_SECRET=managed-secret
# Point this at the webapp in prod
TRIGGER_API_URL=http://localhost:3030
# Point this at the OTel collector in prod
# Point this at the webapp or an OTel collector in prod
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:3030/otel
# Use this on macOS
# OTEL_EXPORTER_OTLP_ENDPOINT=http://host.docker.internal:3030/otel
# Optional settings
DEBUG=1
ENFORCE_MACHINE_PRESETS=1
TRIGGER_DEQUEUE_INTERVAL_MS=1000
+4 -3
View File
@@ -5,7 +5,7 @@ WORKDIR /app
FROM node-22-alpine AS pruner
COPY --chown=node:node . .
RUN npx -q turbo@1.10.9 prune --scope=supervisor --docker
RUN npx -q turbo@2.5.4 prune --scope=supervisor --docker
FROM node-22-alpine AS base
@@ -16,9 +16,10 @@ COPY --from=pruner --chown=node:node /app/out/json/ .
COPY --from=pruner --chown=node:node /app/out/pnpm-lock.yaml ./pnpm-lock.yaml
COPY --from=pruner --chown=node:node /app/out/pnpm-workspace.yaml ./pnpm-workspace.yaml
RUN corepack enable && corepack prepare --activate
RUN corepack enable && corepack prepare pnpm@10.23.0 --activate
FROM base AS deps-fetcher
RUN apk add --no-cache python3-dev py3-setuptools make g++ gcc linux-headers
RUN --mount=type=cache,id=pnpm,target=/root/.local/share/pnpm/store pnpm fetch --frozen-lockfile
FROM deps-fetcher AS dev-deps
@@ -36,7 +37,7 @@ COPY --chown=node:node scripts/updateVersion.ts scripts/updateVersion.ts
RUN pnpm run generate && \
pnpm run --filter supervisor... build&& \
pnpm deploy --filter=supervisor --prod /prod/supervisor
pnpm deploy --legacy --filter=supervisor --prod /prod/supervisor
FROM base AS runner
+47 -9
View File
@@ -19,6 +19,8 @@ curl -sS \
-d "{\"name\": \"$wg_name\"}"
```
If the worker group is newly created, the response will include a `token` field. If the group already exists, no token is returned.
2. Create `.env` and set the worker token
```sh
@@ -34,7 +36,7 @@ TRIGGER_WORKER_TOKEN=tr_wgt_...
pnpm dev
```
4. Build CLI, then deploy a reference project
4. Build CLI, then deploy a test project
```sh
pnpm exec trigger deploy --self-hosted
@@ -43,16 +45,36 @@ pnpm exec trigger deploy --self-hosted
pnpm exec trigger deploy --self-hosted --network host
```
## Additional worker groups
## Worker group management
When adding more worker groups you might also want to make them the default for a specific project. This will allow you to test it without having to change the global default:
### Shared variables
```sh
api_url=http://localhost:3030
admin_pat=tr_pat_... # edit this
```
- These are used by all commands
### Create a worker group
```sh
wg_name=my-worker
# edit these
admin_pat=tr_pat_...
curl -sS \
-X POST \
"$api_url/admin/api/v1/workers" \
-H "Authorization: Bearer $admin_pat" \
-H "Content-Type: application/json" \
-d "{\"name\": \"$wg_name\"}"
```
- If the worker group already exists, no token will be returned
### Set a worker group as default for a project
```sh
wg_name=my-worker
project_id=clsw6q8wz...
curl -sS \
@@ -60,8 +82,24 @@ curl -sS \
"$api_url/admin/api/v1/workers" \
-H "Authorization: Bearer $admin_pat" \
-H "Content-Type: application/json" \
-d "{
\"name\": \"$wg_name\",
\"makeDefaultForProjectId\": \"$project_id\"
}"
-d "{\"name\": \"$wg_name\", \"projectId\": \"$project_id\", \"makeDefaultForProject\": true}"
```
- If the worker group doesn't exist, yet it will be created
- If the worker group already exists, it will be attached to the project as default. No token will be returned.
### Remove the default worker group from a project
```sh
project_id=clsw6q8wz...
curl -sS \
-X POST \
"$api_url/admin/api/v1/workers" \
-H "Authorization: Bearer $admin_pat" \
-H "Content-Type: application/json" \
-d "{\"projectId\": \"$project_id\", \"removeDefaultFromProject\": true}"
```
- The project will then use the global default again
- When `removeDefaultFromProject: true` no other actions will be performed
+4 -7
View File
@@ -13,19 +13,16 @@
"typecheck": "tsc --noEmit"
},
"dependencies": {
"@aws-sdk/client-ecr": "^3.839.0",
"@kubernetes/client-node": "^1.0.0",
"@trigger.dev/core": "workspace:*",
"dockerode": "^4.0.3",
"nanoid": "^5.0.9",
"dockerode": "^4.0.6",
"prom-client": "^15.1.0",
"socket.io": "4.7.4",
"std-env": "^3.8.0",
"tinyexec": "^0.3.1",
"zod": "3.23.8"
"zod": "3.25.76"
},
"devDependencies": {
"@types/dockerode": "^3.3.33",
"docker-api-ts": "^0.2.2",
"vitest": "^1.4.0"
"@types/dockerode": "^3.3.33"
}
}
+4 -1
View File
@@ -3,9 +3,12 @@ import { Informer } from "@kubernetes/client-node";
import { ListPromise } from "@kubernetes/client-node";
import { KubernetesObject } from "@kubernetes/client-node";
import { assertExhaustive } from "@trigger.dev/core/utils";
import { SimpleStructuredLogger } from "@trigger.dev/core/v3/utils/structuredLogger";
export const RUNTIME_ENV = process.env.KUBERNETES_PORT ? "kubernetes" : "local";
const logger = new SimpleStructuredLogger("kubernetes-client");
export function createK8sApi() {
const kubeConfig = getKubeConfig();
@@ -31,7 +34,7 @@ export function createK8sApi() {
export type K8sApi = ReturnType<typeof createK8sApi>;
function getKubeConfig() {
console.log("getKubeConfig()", { RUNTIME_ENV });
logger.debug("getKubeConfig()", { RUNTIME_ENV });
const kubeConfig = new k8s.KubeConfig();
+83 -17
View File
@@ -6,14 +6,16 @@ import { AdditionalEnvVars, BoolEnv } from "./envUtil.js";
const Env = z.object({
// This will come from `spec.nodeName` in k8s
TRIGGER_WORKER_INSTANCE_NAME: z.string().default(randomUUID()),
TRIGGER_WORKER_HEARTBEAT_INTERVAL_SECONDS: z.coerce.number().default(30),
// Required settings
TRIGGER_API_URL: z.string().url(),
TRIGGER_WORKER_TOKEN: z.string(),
TRIGGER_WORKER_TOKEN: z.string(), // accepts file:// path to read from a file
MANAGED_WORKER_SECRET: z.string(),
OTEL_EXPORTER_OTLP_ENDPOINT: z.string().url(), // set on the runners
// Workload API settings (coordinator mode) - the workload API is what the run controller connects to
TRIGGER_WORKLOAD_API_ENABLED: BoolEnv.default("true"),
TRIGGER_WORKLOAD_API_ENABLED: BoolEnv.default(true),
TRIGGER_WORKLOAD_API_PROTOCOL: z
.string()
.transform((s) => z.enum(["http", "https"]).parse(s.toLowerCase()))
@@ -27,34 +29,97 @@ const Env = z.object({
RUNNER_HEARTBEAT_INTERVAL_SECONDS: z.coerce.number().optional(),
RUNNER_SNAPSHOT_POLL_INTERVAL_SECONDS: z.coerce.number().optional(),
RUNNER_ADDITIONAL_ENV_VARS: AdditionalEnvVars, // optional (csv)
RUNNER_DOCKER_AUTOREMOVE: BoolEnv.default(true),
RUNNER_PRETTY_LOGS: BoolEnv.default(false),
// Dequeue settings (provider mode)
TRIGGER_DEQUEUE_ENABLED: BoolEnv.default("true"),
TRIGGER_DEQUEUE_INTERVAL_MS: z.coerce.number().int().default(1000),
TRIGGER_DEQUEUE_MAX_RUN_COUNT: z.coerce.number().int().default(10),
TRIGGER_DEQUEUE_ENABLED: BoolEnv.default(true),
TRIGGER_DEQUEUE_INTERVAL_MS: z.coerce.number().int().default(250),
TRIGGER_DEQUEUE_IDLE_INTERVAL_MS: z.coerce.number().int().default(1000),
TRIGGER_DEQUEUE_MAX_RUN_COUNT: z.coerce.number().int().default(1),
TRIGGER_DEQUEUE_MIN_CONSUMER_COUNT: z.coerce.number().int().default(1),
TRIGGER_DEQUEUE_MAX_CONSUMER_COUNT: z.coerce.number().int().default(10),
TRIGGER_DEQUEUE_SCALING_STRATEGY: z.enum(["none", "smooth", "aggressive"]).default("none"),
TRIGGER_DEQUEUE_SCALING_UP_COOLDOWN_MS: z.coerce.number().int().default(5000), // 5 seconds
TRIGGER_DEQUEUE_SCALING_DOWN_COOLDOWN_MS: z.coerce.number().int().default(30000), // 30 seconds
TRIGGER_DEQUEUE_SCALING_TARGET_RATIO: z.coerce.number().default(1.0), // Target ratio of queue items to consumers (1.0 = 1 item per consumer)
TRIGGER_DEQUEUE_SCALING_EWMA_ALPHA: z.coerce.number().min(0).max(1).default(0.3), // Smooths queue length measurements (0=historical, 1=current)
TRIGGER_DEQUEUE_SCALING_BATCH_WINDOW_MS: z.coerce.number().int().positive().default(1000), // Batch window for metrics processing (ms)
TRIGGER_DEQUEUE_SCALING_DAMPING_FACTOR: z.coerce.number().min(0).max(1).default(0.7), // Smooths consumer count changes after EWMA (0=no scaling, 1=immediate)
// Optional services
TRIGGER_WARM_START_URL: z.string().optional(),
TRIGGER_CHECKPOINT_URL: z.string().optional(),
TRIGGER_METADATA_URL: z.string().optional(),
// Used by the workload manager, e.g docker/k8s
DOCKER_NETWORK: z.string().default("host"),
OTEL_EXPORTER_OTLP_ENDPOINT: z.string().url(),
ENFORCE_MACHINE_PRESETS: z.coerce.boolean().default(false),
KUBERNETES_IMAGE_PULL_SECRETS: z.string().optional(), // csv
// Used by the resource monitor
OVERRIDE_CPU_TOTAL: z.coerce.number().optional(),
OVERRIDE_MEMORY_TOTAL_GB: z.coerce.number().optional(),
RESOURCE_MONITOR_ENABLED: BoolEnv.default(false),
RESOURCE_MONITOR_OVERRIDE_CPU_TOTAL: z.coerce.number().optional(),
RESOURCE_MONITOR_OVERRIDE_MEMORY_TOTAL_GB: z.coerce.number().optional(),
// Kubernetes specific settings
// Docker settings
DOCKER_API_VERSION: z.string().optional(),
DOCKER_PLATFORM: z.string().optional(), // e.g. linux/amd64, linux/arm64
DOCKER_STRIP_IMAGE_DIGEST: BoolEnv.default(true),
DOCKER_REGISTRY_USERNAME: z.string().optional(),
DOCKER_REGISTRY_PASSWORD: z.string().optional(),
DOCKER_REGISTRY_URL: z.string().optional(), // e.g. https://index.docker.io/v1
DOCKER_ENFORCE_MACHINE_PRESETS: BoolEnv.default(true),
DOCKER_AUTOREMOVE_EXITED_CONTAINERS: BoolEnv.default(true),
/**
* Network mode to use for all runners. Supported standard values are: `bridge`, `host`, `none`, and `container:<name|id>`.
* Any other value is taken as a custom network's name to which all runners should connect to.
*
* Accepts a list of comma-separated values to attach to multiple networks. Additional networks are interpreted as network names and will be attached after container creation.
*
* **WARNING**: Specifying multiple networks will slightly increase startup times.
*
* @default "host"
*/
DOCKER_RUNNER_NETWORKS: z.string().default("host"),
// Kubernetes settings
KUBERNETES_FORCE_ENABLED: BoolEnv.default(false),
KUBERNETES_NAMESPACE: z.string().default("default"),
KUBERNETES_WORKER_NODETYPE_LABEL: z.string().default("v4-worker"),
EPHEMERAL_STORAGE_SIZE_LIMIT: z.string().default("10Gi"),
EPHEMERAL_STORAGE_SIZE_REQUEST: z.string().default("2Gi"),
KUBERNETES_IMAGE_PULL_SECRETS: z.string().optional(), // csv
KUBERNETES_EPHEMERAL_STORAGE_SIZE_LIMIT: z.string().default("10Gi"),
KUBERNETES_EPHEMERAL_STORAGE_SIZE_REQUEST: z.string().default("2Gi"),
KUBERNETES_STRIP_IMAGE_DIGEST: BoolEnv.default(false),
KUBERNETES_CPU_REQUEST_MIN_CORES: z.coerce.number().min(0).default(0),
KUBERNETES_CPU_REQUEST_RATIO: z.coerce.number().min(0).max(1).default(0.75), // Ratio of CPU limit, so 0.75 = 75% of CPU limit
KUBERNETES_MEMORY_REQUEST_MIN_GB: z.coerce.number().min(0).default(0),
KUBERNETES_MEMORY_REQUEST_RATIO: z.coerce.number().min(0).max(1).default(1), // Ratio of memory limit, so 1 = 100% of memory limit
// Per-preset overrides of the global KUBERNETES_CPU_REQUEST_RATIO
KUBERNETES_CPU_REQUEST_RATIO_MICRO: z.coerce.number().min(0).max(1).optional(),
KUBERNETES_CPU_REQUEST_RATIO_SMALL_1X: z.coerce.number().min(0).max(1).optional(),
KUBERNETES_CPU_REQUEST_RATIO_SMALL_2X: z.coerce.number().min(0).max(1).optional(),
KUBERNETES_CPU_REQUEST_RATIO_MEDIUM_1X: z.coerce.number().min(0).max(1).optional(),
KUBERNETES_CPU_REQUEST_RATIO_MEDIUM_2X: z.coerce.number().min(0).max(1).optional(),
KUBERNETES_CPU_REQUEST_RATIO_LARGE_1X: z.coerce.number().min(0).max(1).optional(),
KUBERNETES_CPU_REQUEST_RATIO_LARGE_2X: z.coerce.number().min(0).max(1).optional(),
// Per-preset overrides of the global KUBERNETES_MEMORY_REQUEST_RATIO
KUBERNETES_MEMORY_REQUEST_RATIO_MICRO: z.coerce.number().min(0).max(1).optional(),
KUBERNETES_MEMORY_REQUEST_RATIO_SMALL_1X: z.coerce.number().min(0).max(1).optional(),
KUBERNETES_MEMORY_REQUEST_RATIO_SMALL_2X: z.coerce.number().min(0).max(1).optional(),
KUBERNETES_MEMORY_REQUEST_RATIO_MEDIUM_1X: z.coerce.number().min(0).max(1).optional(),
KUBERNETES_MEMORY_REQUEST_RATIO_MEDIUM_2X: z.coerce.number().min(0).max(1).optional(),
KUBERNETES_MEMORY_REQUEST_RATIO_LARGE_1X: z.coerce.number().min(0).max(1).optional(),
KUBERNETES_MEMORY_REQUEST_RATIO_LARGE_2X: z.coerce.number().min(0).max(1).optional(),
KUBERNETES_MEMORY_OVERHEAD_GB: z.coerce.number().min(0).optional(), // Optional memory overhead to add to the limit in GB
KUBERNETES_SCHEDULER_NAME: z.string().optional(), // Custom scheduler name for pods
KUBERNETES_LARGE_MACHINE_POOL_LABEL: z.string().optional(), // if set, large-* presets affinity for machinepool=<value>
// Project affinity settings - pods from the same project prefer the same node
KUBERNETES_PROJECT_AFFINITY_ENABLED: BoolEnv.default(false),
KUBERNETES_PROJECT_AFFINITY_WEIGHT: z.coerce.number().int().min(1).max(100).default(50),
KUBERNETES_PROJECT_AFFINITY_TOPOLOGY_KEY: z.string().trim().min(1).default("kubernetes.io/hostname"),
// Placement tags settings
PLACEMENT_TAGS_ENABLED: BoolEnv.default(false),
PLACEMENT_TAGS_PREFIX: z.string().default("node.cluster.x-k8s.io"),
// Metrics
METRICS_ENABLED: BoolEnv.default(true),
@@ -73,6 +138,7 @@ const Env = z.object({
// Debug
DEBUG: BoolEnv.default(false),
SEND_RUN_DEBUG_LOGS: BoolEnv.default(false),
});
export const env = Env.parse(stdEnv);
+10 -2
View File
@@ -1,6 +1,9 @@
import { z } from "zod";
import { SimpleStructuredLogger } from "@trigger.dev/core/v3/utils/structuredLogger";
export const BoolEnv = z.preprocess((val) => {
const logger = new SimpleStructuredLogger("env-util");
const baseBoolEnv = z.preprocess((val) => {
if (typeof val !== "string") {
return val;
}
@@ -8,6 +11,11 @@ export const BoolEnv = z.preprocess((val) => {
return ["true", "1"].includes(val.toLowerCase().trim());
}, z.boolean());
// Create a type-safe version that only accepts boolean defaults
export const BoolEnv = baseBoolEnv as Omit<typeof baseBoolEnv, "default"> & {
default: (value: boolean) => z.ZodDefault<typeof baseBoolEnv>;
};
export const AdditionalEnvVars = z.preprocess((val) => {
if (typeof val !== "string") {
return val;
@@ -33,7 +41,7 @@ export const AdditionalEnvVars = z.preprocess((val) => {
// Return undefined if no valid key-value pairs were found
return Object.keys(result).length === 0 ? undefined : result;
} catch (error) {
console.warn("Failed to parse additional env vars", { error, val });
logger.warn("Failed to parse additional env vars", { error, val });
return undefined;
}
}, z.record(z.string(), z.string()).optional());
+76 -47
View File
@@ -9,6 +9,7 @@ import { type DequeuedMessage } from "@trigger.dev/core/v3";
import {
DockerResourceMonitor,
KubernetesResourceMonitor,
NoopResourceMonitor,
type ResourceMonitor,
} from "./resourceMonitor.js";
import { KubernetesWorkloadManager } from "./workloadManager/kubernetes.js";
@@ -23,6 +24,7 @@ import { collectDefaultMetrics } from "prom-client";
import { register } from "./metrics.js";
import { PodCleaner } from "./services/podCleaner.js";
import { FailedPodHandler } from "./services/failedPodHandler.js";
import { getWorkerToken } from "./workerToken.js";
if (env.METRICS_COLLECT_DEFAULTS) {
collectDefaultMetrics({ register });
@@ -33,7 +35,7 @@ class ManagedSupervisor {
private readonly metricsServer?: HttpServer;
private readonly workloadServer: WorkloadServer;
private readonly workloadManager: WorkloadManager;
private readonly logger = new SimpleStructuredLogger("managed-worker");
private readonly logger = new SimpleStructuredLogger("managed-supervisor");
private readonly resourceMonitor: ResourceMonitor;
private readonly checkpointClient?: CheckpointClient;
@@ -47,11 +49,11 @@ class ManagedSupervisor {
const { TRIGGER_WORKER_TOKEN, MANAGED_WORKER_SECRET, ...envWithoutSecrets } = env;
if (env.DEBUG) {
console.debug("[ManagedSupervisor] Starting up", { envWithoutSecrets });
this.logger.debug("Starting up", { envWithoutSecrets });
}
if (this.warmStartUrl) {
this.logger.log("[ManagedWorker] 🔥 Warm starts enabled", {
this.logger.log("🔥 Warm starts enabled", {
warmStartUrl: this.warmStartUrl,
});
}
@@ -66,12 +68,22 @@ class ManagedSupervisor {
heartbeatIntervalSeconds: env.RUNNER_HEARTBEAT_INTERVAL_SECONDS,
snapshotPollIntervalSeconds: env.RUNNER_SNAPSHOT_POLL_INTERVAL_SECONDS,
additionalEnvVars: env.RUNNER_ADDITIONAL_ENV_VARS,
dockerAutoremove: env.RUNNER_DOCKER_AUTOREMOVE,
dockerAutoremove: env.DOCKER_AUTOREMOVE_EXITED_CONTAINERS,
} satisfies WorkloadManagerOptions;
this.resourceMonitor = env.RESOURCE_MONITOR_ENABLED
? this.isKubernetes
? new KubernetesResourceMonitor(createK8sApi(), env.TRIGGER_WORKER_INSTANCE_NAME)
: new DockerResourceMonitor(new Docker())
: new NoopResourceMonitor();
this.workloadManager = this.isKubernetes
? new KubernetesWorkloadManager(workloadManagerOptions)
: new DockerWorkloadManager(workloadManagerOptions);
if (this.isKubernetes) {
if (env.POD_CLEANER_ENABLED) {
this.logger.log("[ManagedWorker] 🧹 Pod cleaner enabled", {
this.logger.log("🧹 Pod cleaner enabled", {
namespace: env.KUBERNETES_NAMESPACE,
batchSize: env.POD_CLEANER_BATCH_SIZE,
intervalMs: env.POD_CLEANER_INTERVAL_MS,
@@ -83,11 +95,11 @@ class ManagedSupervisor {
intervalMs: env.POD_CLEANER_INTERVAL_MS,
});
} else {
this.logger.warn("[ManagedWorker] Pod cleaner disabled");
this.logger.warn("Pod cleaner disabled");
}
if (env.FAILED_POD_HANDLER_ENABLED) {
this.logger.log("[ManagedWorker] 🔁 Failed pod handler enabled", {
this.logger.log("🔁 Failed pod handler enabled", {
namespace: env.KUBERNETES_NAMESPACE,
reconnectIntervalMs: env.FAILED_POD_HANDLER_RECONNECT_INTERVAL_MS,
});
@@ -97,35 +109,52 @@ class ManagedSupervisor {
reconnectIntervalMs: env.FAILED_POD_HANDLER_RECONNECT_INTERVAL_MS,
});
} else {
this.logger.warn("[ManagedWorker] Failed pod handler disabled");
this.logger.warn("Failed pod handler disabled");
}
}
this.resourceMonitor = new KubernetesResourceMonitor(
createK8sApi(),
env.TRIGGER_WORKER_INSTANCE_NAME
if (env.TRIGGER_DEQUEUE_INTERVAL_MS > env.TRIGGER_DEQUEUE_IDLE_INTERVAL_MS) {
this.logger.warn(
`⚠️ TRIGGER_DEQUEUE_INTERVAL_MS (${env.TRIGGER_DEQUEUE_INTERVAL_MS}) is greater than TRIGGER_DEQUEUE_IDLE_INTERVAL_MS (${env.TRIGGER_DEQUEUE_IDLE_INTERVAL_MS}) - did you mix them up?`
);
this.workloadManager = new KubernetesWorkloadManager(workloadManagerOptions);
} else {
this.resourceMonitor = new DockerResourceMonitor(new Docker());
this.workloadManager = new DockerWorkloadManager(workloadManagerOptions);
}
this.workerSession = new SupervisorSession({
workerToken: env.TRIGGER_WORKER_TOKEN,
workerToken: getWorkerToken(),
apiUrl: env.TRIGGER_API_URL,
instanceName: env.TRIGGER_WORKER_INSTANCE_NAME,
managedWorkerSecret: env.MANAGED_WORKER_SECRET,
dequeueIntervalMs: env.TRIGGER_DEQUEUE_INTERVAL_MS,
dequeueIdleIntervalMs: env.TRIGGER_DEQUEUE_IDLE_INTERVAL_MS,
queueConsumerEnabled: env.TRIGGER_DEQUEUE_ENABLED,
maxRunCount: env.TRIGGER_DEQUEUE_MAX_RUN_COUNT,
metricsRegistry: register,
scaling: {
strategy: env.TRIGGER_DEQUEUE_SCALING_STRATEGY,
minConsumerCount: env.TRIGGER_DEQUEUE_MIN_CONSUMER_COUNT,
maxConsumerCount: env.TRIGGER_DEQUEUE_MAX_CONSUMER_COUNT,
scaleUpCooldownMs: env.TRIGGER_DEQUEUE_SCALING_UP_COOLDOWN_MS,
scaleDownCooldownMs: env.TRIGGER_DEQUEUE_SCALING_DOWN_COOLDOWN_MS,
targetRatio: env.TRIGGER_DEQUEUE_SCALING_TARGET_RATIO,
ewmaAlpha: env.TRIGGER_DEQUEUE_SCALING_EWMA_ALPHA,
batchWindowMs: env.TRIGGER_DEQUEUE_SCALING_BATCH_WINDOW_MS,
dampingFactor: env.TRIGGER_DEQUEUE_SCALING_DAMPING_FACTOR,
},
runNotificationsEnabled: env.TRIGGER_WORKLOAD_API_ENABLED,
heartbeatIntervalSeconds: env.TRIGGER_WORKER_HEARTBEAT_INTERVAL_SECONDS,
sendRunDebugLogs: env.SEND_RUN_DEBUG_LOGS,
preDequeue: async () => {
if (!env.RESOURCE_MONITOR_ENABLED) {
return {};
}
if (this.isKubernetes) {
// Not used in k8s for now
return {};
}
const resources = await this.resourceMonitor.getNodeResources();
return {
maxResources: {
cpu: resources.cpuAvailable,
@@ -141,7 +170,7 @@ class ManagedSupervisor {
});
if (env.TRIGGER_CHECKPOINT_URL) {
this.logger.log("[ManagedWorker] 🥶 Checkpoints enabled", {
this.logger.log("🥶 Checkpoints enabled", {
checkpointUrl: env.TRIGGER_CHECKPOINT_URL,
});
@@ -152,43 +181,34 @@ class ManagedSupervisor {
});
}
// setInterval(async () => {
// const resources = await this.resourceMonitor.getNodeResources(true);
// this.logger.debug("[ManagedWorker] Current resources", { resources });
// }, 1000);
this.workerSession.on("runNotification", async ({ time, run }) => {
this.logger.log("[ManagedWorker] runNotification", { time, run });
this.logger.log("runNotification", { time, run });
this.workloadServer.notifyRun({ run });
});
this.workerSession.on("runQueueMessage", async ({ time, message }) => {
this.logger.log(
`[ManagedWorker] Received message with timestamp ${time.toLocaleString()}`,
message
);
this.logger.log(`Received message with timestamp ${time.toLocaleString()}`, message);
if (message.completedWaitpoints.length > 0) {
this.logger.debug("[ManagedWorker] Run has completed waitpoints", {
this.logger.debug("Run has completed waitpoints", {
runId: message.run.id,
completedWaitpoints: message.completedWaitpoints.length,
});
// TODO: Do something with them or if we don't need the data here, maybe we shouldn't even send it
}
if (!message.image) {
this.logger.error("[ManagedWorker] Run has no image", { runId: message.run.id });
this.logger.error("Run has no image", { runId: message.run.id });
return;
}
const { checkpoint, ...rest } = message;
if (checkpoint) {
this.logger.log("[ManagedWorker] Restoring run", { runId: message.run.id });
this.logger.log("Restoring run", { runId: message.run.id });
if (!this.checkpointClient) {
this.logger.error("[ManagedWorker] No checkpoint client", { runId: message.run.id });
this.logger.error("No checkpoint client", { runId: message.run.id });
return;
}
@@ -203,27 +223,33 @@ class ManagedSupervisor {
});
if (didRestore) {
this.logger.log("[ManagedWorker] Restore successful", { runId: message.run.id });
this.logger.log("Restore successful", { runId: message.run.id });
} else {
this.logger.error("[ManagedWorker] Restore failed", { runId: message.run.id });
this.logger.error("Restore failed", { runId: message.run.id });
}
} catch (error) {
this.logger.error("[ManagedWorker] Failed to restore run", { error });
this.logger.error("Failed to restore run", { error });
}
return;
}
this.logger.log("[ManagedWorker] Scheduling run", { runId: message.run.id });
this.logger.log("Scheduling run", { runId: message.run.id });
const didWarmStart = await this.tryWarmStart(message);
if (didWarmStart) {
this.logger.log("[ManagedWorker] Warm start successful", { runId: message.run.id });
this.logger.log("Warm start successful", { runId: message.run.id });
return;
}
try {
if (!message.deployment.friendlyId) {
// mostly a type guard, deployments always exists for deployed environments
// a proper fix would be to use a discriminated union schema to differentiate between dequeued runs in dev and in deployed environments.
throw new Error("Deployment is missing");
}
await this.workloadManager.create({
dequeuedAt: message.dequeuedAt,
envId: message.environment.id,
@@ -232,12 +258,15 @@ class ManagedSupervisor {
machine: message.run.machine,
orgId: message.organization.id,
projectId: message.project.id,
deploymentFriendlyId: message.deployment.friendlyId,
deploymentVersion: message.backgroundWorker.version,
runId: message.run.id,
runFriendlyId: message.run.friendlyId,
version: message.version,
nextAttemptNumber: message.run.attemptNumber,
snapshotId: message.snapshot.id,
snapshotFriendlyId: message.snapshot.friendlyId,
placementTags: message.placementTags,
});
// Disabled for now
@@ -246,7 +275,7 @@ class ManagedSupervisor {
// memory: message.run.machine.memory,
// });
} catch (error) {
this.logger.error("[ManagedWorker] Failed to create workload", { error });
this.logger.error("Failed to create workload", { error });
}
});
@@ -274,12 +303,12 @@ class ManagedSupervisor {
}
async onRunConnected({ run }: { run: { friendlyId: string } }) {
this.logger.debug("[ManagedWorker] Run connected", { run });
this.logger.debug("Run connected", { run });
this.workerSession.subscribeToRunNotifications([run.friendlyId]);
}
async onRunDisconnected({ run }: { run: { friendlyId: string } }) {
this.logger.debug("[ManagedWorker] Run disconnected", { run });
this.logger.debug("Run disconnected", { run });
this.workerSession.unsubscribeFromRunNotifications([run.friendlyId]);
}
@@ -300,7 +329,7 @@ class ManagedSupervisor {
});
if (!res.ok) {
this.logger.error("[ManagedWorker] Warm start failed", {
this.logger.error("Warm start failed", {
runId: dequeuedMessage.run.id,
});
return false;
@@ -310,7 +339,7 @@ class ManagedSupervisor {
const parsedData = z.object({ didWarmStart: z.boolean() }).safeParse(data);
if (!parsedData.success) {
this.logger.error("[ManagedWorker] Warm start response invalid", {
this.logger.error("Warm start response invalid", {
runId: dequeuedMessage.run.id,
data,
});
@@ -319,7 +348,7 @@ class ManagedSupervisor {
return parsedData.data.didWarmStart;
} catch (error) {
this.logger.error("[ManagedWorker] Warm start error", {
this.logger.error("Warm start error", {
runId: dequeuedMessage.run.id,
error,
});
@@ -328,7 +357,7 @@ class ManagedSupervisor {
}
async start() {
this.logger.log("[ManagedWorker] Starting up");
this.logger.log("Starting up");
// Optional services
await this.podCleaner?.start();
@@ -336,21 +365,21 @@ class ManagedSupervisor {
await this.metricsServer?.start();
if (env.TRIGGER_WORKLOAD_API_ENABLED) {
this.logger.log("[ManagedWorker] Workload API enabled", {
this.logger.log("Workload API enabled", {
protocol: env.TRIGGER_WORKLOAD_API_PROTOCOL,
domain: env.TRIGGER_WORKLOAD_API_DOMAIN,
port: env.TRIGGER_WORKLOAD_API_PORT_INTERNAL,
});
await this.workloadServer.start();
} else {
this.logger.warn("[ManagedWorker] Workload API disabled");
this.logger.warn("Workload API disabled");
}
await this.workerSession.start();
}
async stop() {
this.logger.log("[ManagedWorker] Shutting down");
this.logger.log("Shutting down");
await this.workerSession.stop();
// Optional services
+40 -8
View File
@@ -1,5 +1,4 @@
import type Docker from "dockerode";
import type * as TDocker from "docker-api-ts";
import type { MachineResources } from "@trigger.dev/core/v3";
import { SimpleStructuredLogger } from "@trigger.dev/core/v3/utils/structuredLogger";
import { env } from "./env.js";
@@ -71,18 +70,21 @@ export abstract class ResourceMonitor {
}
protected applyOverrides(resources: NodeResources): NodeResources {
if (!env.OVERRIDE_CPU_TOTAL && !env.OVERRIDE_MEMORY_TOTAL_GB) {
if (
!env.RESOURCE_MONITOR_OVERRIDE_CPU_TOTAL &&
!env.RESOURCE_MONITOR_OVERRIDE_MEMORY_TOTAL_GB
) {
return resources;
}
logger.debug("[ResourceMonitor] 🛡️ Applying resource overrides", {
cpuTotal: env.OVERRIDE_CPU_TOTAL,
memoryTotalGb: env.OVERRIDE_MEMORY_TOTAL_GB,
cpuTotal: env.RESOURCE_MONITOR_OVERRIDE_CPU_TOTAL,
memoryTotalGb: env.RESOURCE_MONITOR_OVERRIDE_MEMORY_TOTAL_GB,
});
const cpuTotal = env.OVERRIDE_CPU_TOTAL ?? resources.cpuTotal;
const memoryTotal = env.OVERRIDE_MEMORY_TOTAL_GB
? this.gbToBytes(env.OVERRIDE_MEMORY_TOTAL_GB)
const cpuTotal = env.RESOURCE_MONITOR_OVERRIDE_CPU_TOTAL ?? resources.cpuTotal;
const memoryTotal = env.RESOURCE_MONITOR_OVERRIDE_MEMORY_TOTAL_GB
? this.gbToBytes(env.RESOURCE_MONITOR_OVERRIDE_MEMORY_TOTAL_GB)
: resources.memoryTotal;
const cpuDiff = cpuTotal - resources.cpuTotal;
@@ -100,6 +102,11 @@ export abstract class ResourceMonitor {
}
}
type SystemInfo = {
NCPU: number | undefined;
MemTotal: number | undefined;
};
export class DockerResourceMonitor extends ResourceMonitor {
private docker: Docker;
@@ -114,7 +121,7 @@ export class DockerResourceMonitor extends ResourceMonitor {
return this.cachedResources;
}
const info: TDocker.SystemInfo = await this.docker.info();
const info: SystemInfo = await this.docker.info();
const stats = await this.docker.listContainers({ all: true });
// Get system-wide resources
@@ -208,6 +215,21 @@ export class KubernetesResourceMonitor extends ResourceMonitor {
}
}
export class NoopResourceMonitor extends ResourceMonitor {
constructor() {
super(NoopResourceParser);
}
async getNodeResources(): Promise<NodeResources> {
return {
cpuTotal: 0,
cpuAvailable: Infinity,
memoryTotal: 0,
memoryAvailable: Infinity,
};
}
}
abstract class ResourceParser {
abstract cpu(cpu: number | string): number;
abstract memory(memory: number | string): number;
@@ -244,3 +266,13 @@ class KubernetesResourceParser extends ResourceParser {
return parseInt(memory);
}
}
class NoopResourceParser extends ResourceParser {
cpu(cpu: number): number {
return cpu;
}
memory(memory: number): number {
return memory;
}
}
@@ -25,6 +25,7 @@ export class FailedPodHandler {
private readonly informer: Informer<V1Pod>;
private readonly reconnectIntervalMs: number;
private reconnecting = false;
// Metrics
private readonly register: Registry;
@@ -250,21 +251,48 @@ export class FailedPodHandler {
}
private makeOnError(informerName: string) {
return () => this.onError(informerName);
return (err?: unknown) => this.onError(informerName, err);
}
private async onError(informerName: string) {
private async onError(informerName: string, err?: unknown) {
if (!this.isRunning) {
this.logger.warn("onError: informer not running");
return;
}
this.logger.error("error event fired", { informerName });
this.informerEventsTotal.inc({ namespace: this.namespace, verb: "error" });
// Guard against multiple simultaneous reconnections
if (this.reconnecting) {
this.logger.debug("onError: reconnection already in progress, skipping", {
informerName,
});
return;
}
// Reconnect on errors
await setTimeout(this.reconnectIntervalMs);
await this.informer.start();
this.reconnecting = true;
try {
const error = err instanceof Error ? err : undefined;
this.logger.error("error event fired", {
informerName,
error: error?.message,
errorType: error?.name,
});
this.informerEventsTotal.inc({ namespace: this.namespace, verb: "error" });
// Reconnect on errors
await setTimeout(this.reconnectIntervalMs);
await this.informer.start();
} catch (handlerError) {
const error = handlerError instanceof Error ? handlerError : undefined;
this.logger.error("onError: reconnection attempt failed", {
informerName,
error: error?.message,
errorType: error?.name,
errorStack: error?.stack,
});
} finally {
this.reconnecting = false;
}
}
private makeOnConnect(informerName: string) {
+13 -4
View File
@@ -1,8 +1,17 @@
export function getDockerHostDomain() {
const isMacOs = process.platform === "darwin";
const isWindows = process.platform === "win32";
import { isMacOS, isWindows } from "std-env";
return isMacOs || isWindows ? "host.docker.internal" : "localhost";
export function normalizeDockerHostUrl(url: string) {
const $url = new URL(url);
if ($url.hostname === "localhost") {
$url.hostname = getDockerHostDomain();
}
return $url.toString();
}
export function getDockerHostDomain() {
return isMacOS || isWindows ? "host.docker.internal" : "localhost";
}
export function getRunnerId(runId: string, attemptNumber?: number) {
+29
View File
@@ -0,0 +1,29 @@
import { readFileSync } from "fs";
import { env } from "./env.js";
export function getWorkerToken() {
if (!env.TRIGGER_WORKER_TOKEN.startsWith("file://")) {
return env.TRIGGER_WORKER_TOKEN;
}
const tokenPath = env.TRIGGER_WORKER_TOKEN.replace("file://", "");
console.debug(
JSON.stringify({
message: "🔑 Reading worker token from file",
tokenPath,
})
);
try {
const token = readFileSync(tokenPath, "utf8").trim();
return token;
} catch (error) {
console.error(`Failed to read worker token from file: ${tokenPath}`, error);
throw new Error(
`Unable to read worker token from file: ${
error instanceof Error ? error.message : "Unknown error"
}`
);
}
}
+257 -44
View File
@@ -4,88 +4,301 @@ import {
type WorkloadManagerCreateOptions,
type WorkloadManagerOptions,
} from "./types.js";
import { x } from "tinyexec";
import { env } from "../env.js";
import { getDockerHostDomain, getRunnerId } from "../util.js";
import { getDockerHostDomain, getRunnerId, normalizeDockerHostUrl } from "../util.js";
import Docker from "dockerode";
import { tryCatch } from "@trigger.dev/core";
import { ECRAuthService } from "./ecrAuth.js";
export class DockerWorkloadManager implements WorkloadManager {
private readonly logger = new SimpleStructuredLogger("docker-workload-provider");
private readonly logger = new SimpleStructuredLogger("docker-workload-manager");
private readonly docker: Docker;
private readonly runnerNetworks: string[];
private readonly staticAuth?: Docker.AuthConfig;
private readonly platformOverride?: string;
private readonly ecrAuthService?: ECRAuthService;
constructor(private opts: WorkloadManagerOptions) {
this.docker = new Docker({
version: env.DOCKER_API_VERSION,
});
if (opts.workloadApiDomain) {
this.logger.warn("[DockerWorkloadProvider] ⚠️ Custom workload API domain", {
this.logger.warn("⚠️ Custom workload API domain", {
domain: opts.workloadApiDomain,
});
}
this.runnerNetworks = env.DOCKER_RUNNER_NETWORKS.split(",");
this.platformOverride = env.DOCKER_PLATFORM;
if (this.platformOverride) {
this.logger.info("🖥️ Platform override", {
targetPlatform: this.platformOverride,
hostPlatform: process.arch,
});
}
if (env.DOCKER_REGISTRY_USERNAME && env.DOCKER_REGISTRY_PASSWORD && env.DOCKER_REGISTRY_URL) {
this.logger.info("🐋 Using Docker registry credentials", {
username: env.DOCKER_REGISTRY_USERNAME,
url: env.DOCKER_REGISTRY_URL,
});
this.staticAuth = {
username: env.DOCKER_REGISTRY_USERNAME,
password: env.DOCKER_REGISTRY_PASSWORD,
serveraddress: env.DOCKER_REGISTRY_URL,
};
} else if (ECRAuthService.hasAWSCredentials()) {
this.logger.info("🐋 AWS credentials found, initializing ECR auth service");
this.ecrAuthService = new ECRAuthService();
} else {
this.logger.warn(
"🐋 No Docker registry credentials or AWS credentials provided, skipping auth"
);
}
}
async create(opts: WorkloadManagerCreateOptions) {
this.logger.log("[DockerWorkloadProvider] Creating container", { opts });
this.logger.log("create()", { opts });
const runnerId = getRunnerId(opts.runFriendlyId, opts.nextAttemptNumber);
const runArgs = [
"run",
"--detach",
`--network=${env.DOCKER_NETWORK}`,
`--env=TRIGGER_DEQUEUED_AT_MS=${opts.dequeuedAt.getTime()}`,
`--env=TRIGGER_POD_SCHEDULED_AT_MS=${Date.now()}`,
`--env=TRIGGER_ENV_ID=${opts.envId}`,
`--env=TRIGGER_RUN_ID=${opts.runFriendlyId}`,
`--env=TRIGGER_SNAPSHOT_ID=${opts.snapshotFriendlyId}`,
`--env=TRIGGER_SUPERVISOR_API_PROTOCOL=${this.opts.workloadApiProtocol}`,
`--env=TRIGGER_SUPERVISOR_API_PORT=${this.opts.workloadApiPort}`,
`--env=TRIGGER_SUPERVISOR_API_DOMAIN=${this.opts.workloadApiDomain ?? getDockerHostDomain()}`,
`--env=TRIGGER_WORKER_INSTANCE_NAME=${env.TRIGGER_WORKER_INSTANCE_NAME}`,
`--env=OTEL_EXPORTER_OTLP_ENDPOINT=${env.OTEL_EXPORTER_OTLP_ENDPOINT}`,
`--env=TRIGGER_RUNNER_ID=${runnerId}`,
`--hostname=${runnerId}`,
`--name=${runnerId}`,
// Build environment variables
const envVars: string[] = [
`OTEL_EXPORTER_OTLP_ENDPOINT=${env.OTEL_EXPORTER_OTLP_ENDPOINT}`,
`TRIGGER_DEQUEUED_AT_MS=${opts.dequeuedAt.getTime()}`,
`TRIGGER_POD_SCHEDULED_AT_MS=${Date.now()}`,
`TRIGGER_ENV_ID=${opts.envId}`,
`TRIGGER_DEPLOYMENT_ID=${opts.deploymentFriendlyId}`,
`TRIGGER_DEPLOYMENT_VERSION=${opts.deploymentVersion}`,
`TRIGGER_RUN_ID=${opts.runFriendlyId}`,
`TRIGGER_SNAPSHOT_ID=${opts.snapshotFriendlyId}`,
`TRIGGER_SUPERVISOR_API_PROTOCOL=${this.opts.workloadApiProtocol}`,
`TRIGGER_SUPERVISOR_API_PORT=${this.opts.workloadApiPort}`,
`TRIGGER_SUPERVISOR_API_DOMAIN=${this.opts.workloadApiDomain ?? getDockerHostDomain()}`,
`TRIGGER_WORKER_INSTANCE_NAME=${env.TRIGGER_WORKER_INSTANCE_NAME}`,
`TRIGGER_RUNNER_ID=${runnerId}`,
`TRIGGER_MACHINE_CPU=${opts.machine.cpu}`,
`TRIGGER_MACHINE_MEMORY=${opts.machine.memory}`,
`PRETTY_LOGS=${env.RUNNER_PRETTY_LOGS}`,
];
if (this.opts.dockerAutoremove) {
runArgs.push("--rm");
}
if (this.opts.warmStartUrl) {
runArgs.push(`--env=TRIGGER_WARM_START_URL=${this.opts.warmStartUrl}`);
envVars.push(`TRIGGER_WARM_START_URL=${normalizeDockerHostUrl(this.opts.warmStartUrl)}`);
}
if (this.opts.metadataUrl) {
runArgs.push(`--env=TRIGGER_METADATA_URL=${this.opts.metadataUrl}`);
envVars.push(`TRIGGER_METADATA_URL=${this.opts.metadataUrl}`);
}
if (this.opts.heartbeatIntervalSeconds) {
runArgs.push(
`--env=TRIGGER_HEARTBEAT_INTERVAL_SECONDS=${this.opts.heartbeatIntervalSeconds}`
);
envVars.push(`TRIGGER_HEARTBEAT_INTERVAL_SECONDS=${this.opts.heartbeatIntervalSeconds}`);
}
if (this.opts.snapshotPollIntervalSeconds) {
runArgs.push(
`--env=TRIGGER_SNAPSHOT_POLL_INTERVAL_SECONDS=${this.opts.snapshotPollIntervalSeconds}`
envVars.push(
`TRIGGER_SNAPSHOT_POLL_INTERVAL_SECONDS=${this.opts.snapshotPollIntervalSeconds}`
);
}
if (this.opts.additionalEnvVars) {
Object.entries(this.opts.additionalEnvVars).forEach(([key, value]) => {
runArgs.push(`--env=${key}=${value}`);
envVars.push(`${key}=${value}`);
});
}
if (env.ENFORCE_MACHINE_PRESETS) {
runArgs.push(`--cpus=${opts.machine.cpu}`, `--memory=${opts.machine.memory}G`);
runArgs.push(`--env=TRIGGER_MACHINE_CPU=${opts.machine.cpu}`);
runArgs.push(`--env=TRIGGER_MACHINE_MEMORY=${opts.machine.memory}`);
const hostConfig: Docker.HostConfig = {
AutoRemove: !!this.opts.dockerAutoremove,
};
const [firstNetwork, ...remainingNetworks] = this.runnerNetworks;
// Always attach the first network at container creation time. This has the following benefits:
// - If there is only a single network to attach, this will prevent having to make a separate request.
// - If there are multiple networks to attach, this will ensure the runner won't also be connected to the bridge network
hostConfig.NetworkMode = firstNetwork;
if (env.DOCKER_ENFORCE_MACHINE_PRESETS) {
hostConfig.NanoCpus = opts.machine.cpu * 1e9;
hostConfig.Memory = opts.machine.memory * 1024 * 1024 * 1024;
}
runArgs.push(`${opts.image}`);
let imageRef = opts.image;
try {
const { stdout, stderr } = await x("docker", runArgs);
this.logger.debug("[DockerWorkloadProvider] Create succeeded", { stdout, stderr });
} catch (error) {
this.logger.error("[DockerWorkloadProvider] Create failed:", { opts, error });
if (env.DOCKER_STRIP_IMAGE_DIGEST) {
imageRef = opts.image.split("@")[0]!;
}
const containerCreateOpts: Docker.ContainerCreateOptions = {
name: runnerId,
Hostname: runnerId,
HostConfig: hostConfig,
Image: imageRef,
AttachStdout: false,
AttachStderr: false,
AttachStdin: false,
};
if (this.platformOverride) {
containerCreateOpts.platform = this.platformOverride;
}
const logger = this.logger.child({ opts, containerCreateOpts });
const [inspectError, inspectResult] = await tryCatch(this.docker.getImage(imageRef).inspect());
let shouldPull = !!inspectError;
if (this.platformOverride) {
const imageArchitecture = inspectResult?.Architecture;
// When the image architecture doesn't match the platform, we need to pull the image
if (imageArchitecture && !this.platformOverride.includes(imageArchitecture)) {
shouldPull = true;
}
}
// If the image is not present, try to pull it
if (shouldPull) {
logger.info("Pulling image", {
error: inspectError,
image: opts.image,
targetPlatform: this.platformOverride,
imageArchitecture: inspectResult?.Architecture,
});
// Get auth config (static or ECR)
const authConfig = await this.getAuthConfig();
// Ensure the image is present
const [createImageError, imageResponseReader] = await tryCatch(
this.docker.createImage(authConfig, {
fromImage: imageRef,
...(this.platformOverride ? { platform: this.platformOverride } : {}),
})
);
if (createImageError) {
logger.error("Failed to pull image", { error: createImageError });
return;
}
const [imageReadError, imageResponse] = await tryCatch(readAllChunks(imageResponseReader));
if (imageReadError) {
logger.error("failed to read image response", { error: imageReadError });
return;
}
logger.debug("pulled image", { image: opts.image, imageResponse });
} else {
// Image is present, so we can use it to create the container
}
// Create container
const [createContainerError, container] = await tryCatch(
this.docker.createContainer({
...containerCreateOpts,
// Add env vars here so they're not logged
Env: envVars,
})
);
if (createContainerError) {
logger.error("Failed to create container", { error: createContainerError });
return;
}
// If there are multiple networks to attach to we need to attach the remaining ones after creation
if (remainingNetworks.length > 0) {
await this.attachContainerToNetworks({
containerId: container.id,
networkNames: remainingNetworks,
});
}
// Start container
const [startError, startResult] = await tryCatch(container.start());
if (startError) {
logger.error("Failed to start container", { error: startError, containerId: container.id });
return;
}
logger.debug("create succeeded", { startResult, containerId: container.id });
}
/**
* Get authentication config for Docker operations
* Uses static credentials if available, otherwise attempts ECR auth
*/
private async getAuthConfig(): Promise<Docker.AuthConfig | undefined> {
// Use static credentials if available
if (this.staticAuth) {
return this.staticAuth;
}
// Use ECR auth if service is available
if (this.ecrAuthService) {
const ecrAuth = await this.ecrAuthService.getAuthConfig();
return ecrAuth || undefined;
}
// No auth available
return undefined;
}
private async attachContainerToNetworks({
containerId,
networkNames,
}: {
containerId: string;
networkNames: string[];
}) {
this.logger.debug("Attaching container to networks", { containerId, networkNames });
const [error, networkResults] = await tryCatch(
this.docker.listNetworks({
filters: {
// Full name matches only to prevent unexpected results
name: networkNames.map((name) => `^${name}$`),
},
})
);
if (error) {
this.logger.error("Failed to list networks", { networkNames });
return;
}
const results = await Promise.allSettled(
networkResults.map((networkInfo) => {
const network = this.docker.getNetwork(networkInfo.Id);
return network.connect({ Container: containerId });
})
);
if (results.some((r) => r.status === "rejected")) {
this.logger.error("Failed to attach container to some networks", {
containerId,
networkNames,
results,
});
return;
}
this.logger.debug("Attached container to networks", {
containerId,
networkNames,
results,
});
}
}
async function readAllChunks(reader: NodeJS.ReadableStream) {
const chunks = [];
for await (const chunk of reader) {
chunks.push(chunk.toString());
}
return chunks;
}
@@ -0,0 +1,144 @@
import { ECRClient, GetAuthorizationTokenCommand } from "@aws-sdk/client-ecr";
import { SimpleStructuredLogger } from "@trigger.dev/core/v3/utils/structuredLogger";
import { tryCatch } from "@trigger.dev/core";
import Docker from "dockerode";
interface ECRTokenCache {
token: string;
username: string;
serverAddress: string;
expiresAt: Date;
}
export class ECRAuthService {
private readonly logger = new SimpleStructuredLogger("ecr-auth-service");
private readonly ecrClient: ECRClient;
private tokenCache: ECRTokenCache | null = null;
constructor() {
this.ecrClient = new ECRClient();
this.logger.info("🔐 ECR Auth Service initialized", {
region: this.ecrClient.config.region,
});
}
/**
* Check if we have AWS credentials configured
*/
static hasAWSCredentials(): boolean {
if (process.env.AWS_ACCESS_KEY_ID && process.env.AWS_SECRET_ACCESS_KEY) {
return true;
}
if (
process.env.AWS_PROFILE ||
process.env.AWS_ROLE_ARN ||
process.env.AWS_WEB_IDENTITY_TOKEN_FILE
) {
return true;
}
return false;
}
/**
* Check if the current token is still valid with a 10-minute buffer
*/
private isTokenValid(): boolean {
if (!this.tokenCache) {
return false;
}
const now = new Date();
const bufferMs = 10 * 60 * 1000; // 10 minute buffer before expiration
return now < new Date(this.tokenCache.expiresAt.getTime() - bufferMs);
}
/**
* Get a fresh ECR authorization token from AWS
*/
private async fetchNewToken(): Promise<ECRTokenCache | null> {
const [error, response] = await tryCatch(
this.ecrClient.send(new GetAuthorizationTokenCommand({}))
);
if (error) {
this.logger.error("Failed to get ECR authorization token", { error });
return null;
}
const authData = response.authorizationData?.[0];
if (!authData?.authorizationToken || !authData.proxyEndpoint) {
this.logger.error("Invalid ECR authorization response", { authData });
return null;
}
// Decode the base64 token to get username:password
const decoded = Buffer.from(authData.authorizationToken, "base64").toString("utf-8");
const [username, password] = decoded.split(":", 2);
if (!username || !password) {
this.logger.error("Failed to parse ECR authorization token");
return null;
}
const expiresAt = authData.expiresAt || new Date(Date.now() + 12 * 60 * 60 * 1000); // Default 12 hours
const tokenCache: ECRTokenCache = {
token: password,
username,
serverAddress: authData.proxyEndpoint,
expiresAt,
};
this.logger.info("🔐 Successfully fetched ECR token", {
username,
serverAddress: authData.proxyEndpoint,
expiresAt: expiresAt.toISOString(),
});
return tokenCache;
}
/**
* Get ECR auth config for Docker operations
* Returns cached token if valid, otherwise fetches a new one
*/
async getAuthConfig(): Promise<Docker.AuthConfig | null> {
// Check if cached token is still valid
if (this.isTokenValid()) {
this.logger.debug("Using cached ECR token");
return {
username: this.tokenCache!.username,
password: this.tokenCache!.token,
serveraddress: this.tokenCache!.serverAddress,
};
}
// Fetch new token
this.logger.info("Fetching new ECR authorization token");
const newToken = await this.fetchNewToken();
if (!newToken) {
return null;
}
// Cache the new token
this.tokenCache = newToken;
return {
username: newToken.username,
password: newToken.token,
serveraddress: newToken.serverAddress,
};
}
/**
* Clear the cached token (useful for testing or forcing refresh)
*/
clearCache(): void {
this.tokenCache = null;
this.logger.debug("ECR token cache cleared");
}
}
+202 -11
View File
@@ -4,7 +4,13 @@ import {
type WorkloadManagerCreateOptions,
type WorkloadManagerOptions,
} from "./types.js";
import type { EnvironmentType, MachinePreset } from "@trigger.dev/core/v3";
import type {
EnvironmentType,
MachinePreset,
MachinePresetName,
PlacementTag,
} from "@trigger.dev/core/v3";
import { PlacementTagProcessor } from "@trigger.dev/core/v3/serverOnly";
import { env } from "../env.js";
import { type K8sApi, createK8sApi, type k8s } from "../clients/kubernetes.js";
import { getRunnerId } from "../util.js";
@@ -13,13 +19,45 @@ type ResourceQuantities = {
[K in "cpu" | "memory" | "ephemeral-storage"]?: string;
};
const cpuRequestRatioByMachinePreset: Record<MachinePresetName, number | undefined> = {
micro: env.KUBERNETES_CPU_REQUEST_RATIO_MICRO,
"small-1x": env.KUBERNETES_CPU_REQUEST_RATIO_SMALL_1X,
"small-2x": env.KUBERNETES_CPU_REQUEST_RATIO_SMALL_2X,
"medium-1x": env.KUBERNETES_CPU_REQUEST_RATIO_MEDIUM_1X,
"medium-2x": env.KUBERNETES_CPU_REQUEST_RATIO_MEDIUM_2X,
"large-1x": env.KUBERNETES_CPU_REQUEST_RATIO_LARGE_1X,
"large-2x": env.KUBERNETES_CPU_REQUEST_RATIO_LARGE_2X,
};
const memoryRequestRatioByMachinePreset: Record<MachinePresetName, number | undefined> = {
micro: env.KUBERNETES_MEMORY_REQUEST_RATIO_MICRO,
"small-1x": env.KUBERNETES_MEMORY_REQUEST_RATIO_SMALL_1X,
"small-2x": env.KUBERNETES_MEMORY_REQUEST_RATIO_SMALL_2X,
"medium-1x": env.KUBERNETES_MEMORY_REQUEST_RATIO_MEDIUM_1X,
"medium-2x": env.KUBERNETES_MEMORY_REQUEST_RATIO_MEDIUM_2X,
"large-1x": env.KUBERNETES_MEMORY_REQUEST_RATIO_LARGE_1X,
"large-2x": env.KUBERNETES_MEMORY_REQUEST_RATIO_LARGE_2X,
};
export class KubernetesWorkloadManager implements WorkloadManager {
private readonly logger = new SimpleStructuredLogger("kubernetes-workload-provider");
private k8s: K8sApi;
private namespace = env.KUBERNETES_NAMESPACE;
private placementTagProcessor: PlacementTagProcessor;
// Resource settings
private readonly cpuRequestMinCores = env.KUBERNETES_CPU_REQUEST_MIN_CORES;
private readonly cpuRequestRatio = env.KUBERNETES_CPU_REQUEST_RATIO;
private readonly memoryRequestMinGb = env.KUBERNETES_MEMORY_REQUEST_MIN_GB;
private readonly memoryRequestRatio = env.KUBERNETES_MEMORY_REQUEST_RATIO;
private readonly memoryOverheadGb = env.KUBERNETES_MEMORY_OVERHEAD_GB;
constructor(private opts: WorkloadManagerOptions) {
this.k8s = createK8sApi();
this.placementTagProcessor = new PlacementTagProcessor({
enabled: env.PLACEMENT_TAGS_ENABLED,
prefix: env.PLACEMENT_TAGS_PREFIX,
});
if (opts.workloadApiDomain) {
this.logger.warn("[KubernetesWorkloadManager] ⚠️ Custom workload API domain", {
@@ -28,6 +66,39 @@ export class KubernetesWorkloadManager implements WorkloadManager {
}
}
private addPlacementTags(
podSpec: Omit<k8s.V1PodSpec, "containers">,
placementTags?: PlacementTag[]
): Omit<k8s.V1PodSpec, "containers"> {
const nodeSelector = this.placementTagProcessor.convertToNodeSelector(
placementTags,
podSpec.nodeSelector
);
return {
...podSpec,
nodeSelector,
};
}
private stripImageDigest(imageRef: string): string {
if (!env.KUBERNETES_STRIP_IMAGE_DIGEST) {
return imageRef;
}
const atIndex = imageRef.lastIndexOf("@");
if (atIndex === -1) {
return imageRef;
}
return imageRef.substring(0, atIndex);
}
private clamp(value: number, min: number, max: number): number {
return Math.min(Math.max(value, min), max);
}
async create(opts: WorkloadManagerCreateOptions) {
this.logger.log("[KubernetesWorkloadManager] Creating container", { opts });
@@ -48,12 +119,13 @@ export class KubernetesWorkloadManager implements WorkloadManager {
},
},
spec: {
...this.#defaultPodSpec,
...this.addPlacementTags(this.#defaultPodSpec, opts.placementTags),
affinity: this.#getAffinity(opts.machine, opts.projectId),
terminationGracePeriodSeconds: 60 * 60,
containers: [
{
name: "run-controller",
image: opts.image,
image: this.stripImageDigest(opts.image),
ports: [
{
containerPort: 8000,
@@ -77,6 +149,14 @@ export class KubernetesWorkloadManager implements WorkloadManager {
name: "TRIGGER_ENV_ID",
value: opts.envId,
},
{
name: "TRIGGER_DEPLOYMENT_ID",
value: opts.deploymentFriendlyId,
},
{
name: "TRIGGER_DEPLOYMENT_VERSION",
value: opts.deploymentVersion,
},
{
name: "TRIGGER_SNAPSHOT_ID",
value: opts.snapshotFriendlyId,
@@ -228,21 +308,30 @@ export class KubernetesWorkloadManager implements WorkloadManager {
restartPolicy: "Never",
automountServiceAccountToken: false,
imagePullSecrets: this.getImagePullSecrets(),
nodeSelector: {
nodetype: env.KUBERNETES_WORKER_NODETYPE_LABEL,
},
...(env.KUBERNETES_SCHEDULER_NAME
? {
schedulerName: env.KUBERNETES_SCHEDULER_NAME,
}
: {}),
...(env.KUBERNETES_WORKER_NODETYPE_LABEL
? {
nodeSelector: {
nodetype: env.KUBERNETES_WORKER_NODETYPE_LABEL,
},
}
: {}),
};
}
get #defaultResourceRequests(): ResourceQuantities {
return {
"ephemeral-storage": env.EPHEMERAL_STORAGE_SIZE_REQUEST,
"ephemeral-storage": env.KUBERNETES_EPHEMERAL_STORAGE_SIZE_REQUEST,
};
}
get #defaultResourceLimits(): ResourceQuantities {
return {
"ephemeral-storage": env.EPHEMERAL_STORAGE_SIZE_LIMIT,
"ephemeral-storage": env.KUBERNETES_EPHEMERAL_STORAGE_SIZE_LIMIT,
};
}
@@ -252,20 +341,35 @@ export class KubernetesWorkloadManager implements WorkloadManager {
envtype: this.#envTypeToLabelValue(opts.envType),
org: opts.orgId,
project: opts.projectId,
machine: opts.machine.name,
};
}
#getResourceRequestsForMachine(preset: MachinePreset): ResourceQuantities {
const cpuRatio = cpuRequestRatioByMachinePreset[preset.name] ?? this.cpuRequestRatio;
const memoryRatio = memoryRequestRatioByMachinePreset[preset.name] ?? this.memoryRequestRatio;
const cpuRequest = preset.cpu * cpuRatio;
const memoryRequest = preset.memory * memoryRatio;
// Clamp between min and max
const clampedCpu = this.clamp(cpuRequest, this.cpuRequestMinCores, preset.cpu);
const clampedMemory = this.clamp(memoryRequest, this.memoryRequestMinGb, preset.memory);
return {
cpu: `${preset.cpu * 0.75}`,
memory: `${preset.memory}G`,
cpu: `${clampedCpu}`,
memory: `${clampedMemory}G`,
};
}
#getResourceLimitsForMachine(preset: MachinePreset): ResourceQuantities {
const memoryLimit = this.memoryOverheadGb
? preset.memory + this.memoryOverheadGb
: preset.memory;
return {
cpu: `${preset.cpu}`,
memory: `${preset.memory}G`,
memory: `${memoryLimit}G`,
};
}
@@ -281,4 +385,91 @@ export class KubernetesWorkloadManager implements WorkloadManager {
},
};
}
#isLargeMachine(preset: MachinePreset): boolean {
return preset.name.startsWith("large-");
}
#getAffinity(preset: MachinePreset, projectId: string): k8s.V1Affinity | undefined {
const nodeAffinity = this.#getNodeAffinityRules(preset);
const podAffinity = this.#getProjectPodAffinity(projectId);
if (!nodeAffinity && !podAffinity) {
return undefined;
}
return {
...(nodeAffinity && { nodeAffinity }),
...(podAffinity && { podAffinity }),
};
}
#getNodeAffinityRules(preset: MachinePreset): k8s.V1NodeAffinity | undefined {
if (!env.KUBERNETES_LARGE_MACHINE_POOL_LABEL) {
return undefined;
}
if (this.#isLargeMachine(preset)) {
// soft preference for the large-machine pool, falls back to standard if unavailable
return {
preferredDuringSchedulingIgnoredDuringExecution: [
{
weight: 100,
preference: {
matchExpressions: [
{
key: "node.cluster.x-k8s.io/machinepool",
operator: "In",
values: [env.KUBERNETES_LARGE_MACHINE_POOL_LABEL],
},
],
},
},
],
};
}
// not schedulable in the large-machine pool
return {
requiredDuringSchedulingIgnoredDuringExecution: {
nodeSelectorTerms: [
{
matchExpressions: [
{
key: "node.cluster.x-k8s.io/machinepool",
operator: "NotIn",
values: [env.KUBERNETES_LARGE_MACHINE_POOL_LABEL],
},
],
},
],
},
};
}
#getProjectPodAffinity(projectId: string): k8s.V1PodAffinity | undefined {
if (!env.KUBERNETES_PROJECT_AFFINITY_ENABLED) {
return undefined;
}
return {
preferredDuringSchedulingIgnoredDuringExecution: [
{
weight: env.KUBERNETES_PROJECT_AFFINITY_WEIGHT,
podAffinityTerm: {
labelSelector: {
matchExpressions: [
{
key: "project",
operator: "In",
values: [projectId],
},
],
},
topologyKey: env.KUBERNETES_PROJECT_AFFINITY_TOPOLOGY_KEY,
},
},
],
};
}
}
+4 -1
View File
@@ -1,4 +1,4 @@
import { type EnvironmentType, type MachinePreset } from "@trigger.dev/core/v3";
import type { EnvironmentType, MachinePreset, PlacementTag } from "@trigger.dev/core/v3";
export interface WorkloadManagerOptions {
workloadApiProtocol: "http" | "https";
@@ -23,11 +23,14 @@ export interface WorkloadManagerCreateOptions {
version: string;
nextAttemptNumber?: number;
dequeuedAt: Date;
placementTags?: PlacementTag[];
// identifiers
envId: string;
envType: EnvironmentType;
orgId: string;
projectId: string;
deploymentFriendlyId: string;
deploymentVersion: string;
runId: string;
runFriendlyId: string;
snapshotId: string;
+82 -66
View File
@@ -16,13 +16,14 @@ import {
type WorkloadRunAttemptCompleteResponseBody,
WorkloadRunAttemptStartRequestBody,
type WorkloadRunAttemptStartResponseBody,
type WorkloadRunLatestSnapshotResponseBody,
WorkloadRunSnapshotsSinceResponseBody,
type WorkloadServerToClientEvents,
type WorkloadSuspendRunResponseBody,
} from "@trigger.dev/core/v3/workers";
import { HttpServer, type CheckpointClient } from "@trigger.dev/core/v3/serverOnly";
import { type IncomingMessage } from "node:http";
import { register } from "../metrics.js";
import { env } from "../env.js";
// Use the official export when upgrading to socket.io@4.8.0
interface DefaultEventsMap {
@@ -62,6 +63,8 @@ type WorkloadServerOptions = {
export class WorkloadServer extends EventEmitter<WorkloadServerEvents> {
private checkpointClient?: CheckpointClient;
private readonly logger = new SimpleStructuredLogger("workload-server");
private readonly httpServer: HttpServer;
private readonly websocketServer: Namespace<
WorkloadClientToServerEvents,
@@ -122,7 +125,7 @@ export class WorkloadServer extends EventEmitter<WorkloadServerEvents> {
}
private createHttpServer({ host, port }: { host: string; port: number }) {
return new HttpServer({
const httpServer = new HttpServer({
port,
host,
metrics: {
@@ -150,7 +153,7 @@ export class WorkloadServer extends EventEmitter<WorkloadServerEvents> {
);
if (!startResponse.success) {
console.error("Failed to start run", {
this.logger.error("Failed to start run", {
params,
error: startResponse.error,
});
@@ -170,7 +173,6 @@ export class WorkloadServer extends EventEmitter<WorkloadServerEvents> {
paramsSchema: WorkloadActionParams,
bodySchema: WorkloadRunAttemptCompleteRequestBody,
handler: async ({ req, reply, params, body }) => {
console.log("headers", req.headers);
const completeResponse = await this.workerClient.completeRunAttempt(
params.runFriendlyId,
params.snapshotFriendlyId,
@@ -179,7 +181,7 @@ export class WorkloadServer extends EventEmitter<WorkloadServerEvents> {
);
if (!completeResponse.success) {
console.error("Failed to complete run", {
this.logger.error("Failed to complete run", {
params,
error: completeResponse.error,
});
@@ -207,7 +209,7 @@ export class WorkloadServer extends EventEmitter<WorkloadServerEvents> {
);
if (!heartbeatResponse.success) {
console.error("Failed to heartbeat run", {
this.logger.error("Failed to heartbeat run", {
params,
error: heartbeatResponse.error,
});
@@ -227,7 +229,7 @@ export class WorkloadServer extends EventEmitter<WorkloadServerEvents> {
{
paramsSchema: WorkloadActionParams,
handler: async ({ reply, params, req }) => {
console.debug("Suspend request", { params, headers: req.headers });
this.logger.debug("Suspend request", { params, headers: req.headers });
if (!this.checkpointClient) {
reply.json(
@@ -246,7 +248,7 @@ export class WorkloadServer extends EventEmitter<WorkloadServerEvents> {
const projectRef = this.projectRefFromRequest(req);
if (!runnerId || !deploymentVersion || !projectRef) {
console.error("Invalid headers for suspend request", {
this.logger.error("Invalid headers for suspend request", {
...params,
headers: req.headers,
});
@@ -282,7 +284,7 @@ export class WorkloadServer extends EventEmitter<WorkloadServerEvents> {
});
if (!suspendResult) {
console.error("Failed to suspend run", { params });
this.logger.error("Failed to suspend run", { params });
return;
}
},
@@ -294,7 +296,7 @@ export class WorkloadServer extends EventEmitter<WorkloadServerEvents> {
{
paramsSchema: WorkloadActionParams,
handler: async ({ req, reply, params }) => {
console.debug("Run continuation request", { params });
this.logger.debug("Run continuation request", { params });
const continuationResult = await this.workerClient.continueRunExecution(
params.runFriendlyId,
@@ -303,7 +305,7 @@ export class WorkloadServer extends EventEmitter<WorkloadServerEvents> {
);
if (!continuationResult.success) {
console.error("Failed to continue run execution", { params });
this.logger.error("Failed to continue run execution", { params });
reply.json(
{
ok: false,
@@ -319,41 +321,31 @@ export class WorkloadServer extends EventEmitter<WorkloadServerEvents> {
},
}
)
.route("/api/v1/workload-actions/runs/:runFriendlyId/snapshots/latest", "GET", {
paramsSchema: WorkloadActionParams.pick({ runFriendlyId: true }),
handler: async ({ req, reply, params }) => {
const latestSnapshotResponse = await this.workerClient.getLatestSnapshot(
params.runFriendlyId,
this.runnerIdFromRequest(req)
);
.route(
"/api/v1/workload-actions/runs/:runFriendlyId/snapshots/since/:snapshotFriendlyId",
"GET",
{
paramsSchema: WorkloadActionParams,
handler: async ({ req, reply, params }) => {
const sinceSnapshotResponse = await this.workerClient.getSnapshotsSince(
params.runFriendlyId,
params.snapshotFriendlyId,
this.runnerIdFromRequest(req)
);
if (!latestSnapshotResponse.success) {
console.error("Failed to get latest snapshot", {
runId: params.runFriendlyId,
error: latestSnapshotResponse.error,
});
reply.empty(500);
return;
}
if (!sinceSnapshotResponse.success) {
this.logger.error("Failed to get snapshots since", {
runId: params.runFriendlyId,
error: sinceSnapshotResponse.error,
});
reply.empty(500);
return;
}
reply.json({
execution: latestSnapshotResponse.data.execution,
} satisfies WorkloadRunLatestSnapshotResponseBody);
},
})
.route("/api/v1/workload-actions/runs/:runFriendlyId/logs/debug", "POST", {
paramsSchema: WorkloadActionParams.pick({ runFriendlyId: true }),
bodySchema: WorkloadDebugLogRequestBody,
handler: async ({ req, reply, params, body }) => {
reply.empty(204);
await this.workerClient.sendDebugLog(
params.runFriendlyId,
body,
this.runnerIdFromRequest(req)
);
},
})
reply.json(sinceSnapshotResponse.data satisfies WorkloadRunSnapshotsSinceResponseBody);
},
}
)
.route("/api/v1/workload-actions/deployments/:deploymentId/dequeue", "GET", {
paramsSchema: z.object({
deploymentId: z.string(),
@@ -367,7 +359,7 @@ export class WorkloadServer extends EventEmitter<WorkloadServerEvents> {
);
if (!dequeueResponse.success) {
console.error("Failed to get latest snapshot", {
this.logger.error("Failed to get latest snapshot", {
deploymentId: params.deploymentId,
error: dequeueResponse.error,
});
@@ -378,6 +370,31 @@ export class WorkloadServer extends EventEmitter<WorkloadServerEvents> {
reply.json(dequeueResponse.data satisfies WorkloadDequeueFromVersionResponseBody);
},
});
if (env.SEND_RUN_DEBUG_LOGS) {
httpServer.route("/api/v1/workload-actions/runs/:runFriendlyId/logs/debug", "POST", {
paramsSchema: WorkloadActionParams.pick({ runFriendlyId: true }),
bodySchema: WorkloadDebugLogRequestBody,
handler: async ({ req, reply, params, body }) => {
reply.empty(204);
await this.workerClient.sendDebugLog(
params.runFriendlyId,
body,
this.runnerIdFromRequest(req)
);
},
});
} else {
// Lightweight mock route without schemas
httpServer.route("/api/v1/workload-actions/runs/:runFriendlyId/logs/debug", "POST", {
handler: async ({ reply }) => {
reply.empty(204);
},
});
}
return httpServer;
}
private createWebsocketServer() {
@@ -391,14 +408,14 @@ export class WorkloadServer extends EventEmitter<WorkloadServerEvents> {
> = io.of("/workload");
websocketServer.on("disconnect", (socket) => {
console.log("[WorkloadSocket] disconnect", socket.id);
this.logger.log("[WS] disconnect", socket.id);
});
websocketServer.use(async (socket, next) => {
function setSocketDataFromHeader(
const setSocketDataFromHeader = (
dataKey: keyof typeof socket.data,
headerName: string,
required: boolean = true
) {
) => {
const value = socket.handshake.headers[headerName];
if (value) {
@@ -414,27 +431,26 @@ export class WorkloadServer extends EventEmitter<WorkloadServerEvents> {
}
if (required) {
console.error("[WorkloadSocket] missing required header", { headerName });
this.logger.error("[WS] missing required header", { headerName });
throw new Error("missing header");
}
}
};
try {
setSocketDataFromHeader("deploymentId", WORKLOAD_HEADERS.DEPLOYMENT_ID);
setSocketDataFromHeader("runnerId", WORKLOAD_HEADERS.RUNNER_ID);
} catch (error) {
console.error("[WorkloadSocket] setSocketDataFromHeader error", { error });
this.logger.error("[WS] setSocketDataFromHeader error", { error });
socket.disconnect(true);
return;
}
console.debug("[WorkloadSocket] auth success", socket.data);
this.logger.debug("[WS] auth success", socket.data);
next();
});
websocketServer.on("connection", (socket) => {
const logger = new SimpleStructuredLogger("workload-namespace", undefined, {
namespace: "workload",
const socketLogger = this.logger.child({
socketId: socket.id,
socketData: socket.data,
});
@@ -449,11 +465,11 @@ export class WorkloadServer extends EventEmitter<WorkloadServerEvents> {
};
const runConnected = (friendlyId: string) => {
logger.debug("runConnected", { ...getSocketMetadata() });
socketLogger.debug("runConnected", { ...getSocketMetadata() });
// If there's already a run ID set, we should "disconnect" it from this socket
if (socket.data.runFriendlyId && socket.data.runFriendlyId !== friendlyId) {
logger.debug("runConnected: disconnecting existing run", {
socketLogger.debug("runConnected: disconnecting existing run", {
...getSocketMetadata(),
newRunId: friendlyId,
oldRunId: socket.data.runFriendlyId,
@@ -467,14 +483,14 @@ export class WorkloadServer extends EventEmitter<WorkloadServerEvents> {
};
const runDisconnected = (friendlyId: string) => {
logger.debug("runDisconnected", { ...getSocketMetadata() });
socketLogger.debug("runDisconnected", { ...getSocketMetadata() });
this.runSockets.delete(friendlyId);
this.emit("runDisconnected", { run: { friendlyId } });
socket.data.runFriendlyId = undefined;
};
logger.log("wsServer socket connected", { ...getSocketMetadata() });
socketLogger.log("wsServer socket connected", { ...getSocketMetadata() });
// FIXME: where does this get set?
if (socket.data.runFriendlyId) {
@@ -482,7 +498,7 @@ export class WorkloadServer extends EventEmitter<WorkloadServerEvents> {
}
socket.on("disconnecting", (reason, description) => {
logger.log("Socket disconnecting", { ...getSocketMetadata(), reason, description });
socketLogger.log("Socket disconnecting", { ...getSocketMetadata(), reason, description });
if (socket.data.runFriendlyId) {
runDisconnected(socket.data.runFriendlyId);
@@ -490,11 +506,11 @@ export class WorkloadServer extends EventEmitter<WorkloadServerEvents> {
});
socket.on("disconnect", (reason, description) => {
logger.log("Socket disconnected", { ...getSocketMetadata(), reason, description });
socketLogger.log("Socket disconnected", { ...getSocketMetadata(), reason, description });
});
socket.on("error", (error) => {
logger.error("Socket error", {
socketLogger.error("Socket error", {
...getSocketMetadata(),
error: {
name: error.name,
@@ -505,7 +521,7 @@ export class WorkloadServer extends EventEmitter<WorkloadServerEvents> {
});
socket.on("run:start", async (message) => {
const log = logger.child({
const log = socketLogger.child({
eventName: "run:start",
...getSocketMetadata(),
...message,
@@ -521,7 +537,7 @@ export class WorkloadServer extends EventEmitter<WorkloadServerEvents> {
});
socket.on("run:stop", async (message) => {
const log = logger.child({
const log = socketLogger.child({
eventName: "run:stop",
...getSocketMetadata(),
...message,
@@ -545,7 +561,7 @@ export class WorkloadServer extends EventEmitter<WorkloadServerEvents> {
const runSocket = this.runSockets.get(run.friendlyId);
if (!runSocket) {
console.debug("[WorkloadServer] notifyRun: Run socket not found", { run });
this.logger.debug("notifyRun: Run socket not found", { run });
this.workerClient.sendDebugLog(run.friendlyId, {
time: new Date(),
@@ -556,14 +572,14 @@ export class WorkloadServer extends EventEmitter<WorkloadServerEvents> {
}
runSocket.emit("run:notify", { version: "1", run });
console.debug("[WorkloadServer] run:notify sent", { run });
this.logger.debug("run:notify sent", { run });
this.workerClient.sendDebugLog(run.friendlyId, {
time: new Date(),
message: "run:notify supervisor -> runner",
});
} catch (error) {
console.error("[WorkloadServer] Error in notifyRun", { run, error });
this.logger.error("Error in notifyRun", { run, error });
this.workerClient.sendDebugLog(run.friendlyId, {
time: new Date(),
-16
View File
@@ -1,16 +0,0 @@
{
"sourceType": "unambiguous",
"presets": [
[
"@babel/preset-env",
{
"targets": {
"chrome": 100
}
}
],
"@babel/preset-typescript",
"@babel/preset-react"
],
"plugins": []
}
+1
View File
@@ -0,0 +1 @@
../../.env
+4 -2
View File
@@ -9,7 +9,8 @@ node_modules
/app/styles/tailwind.css
# Ensure the .env symlink is not removed by accident
!.env
# Storybook build outputs
build-storybook.log
@@ -18,4 +19,5 @@ build-storybook.log
storybook-static
/prisma/seed.js
/prisma/populate.js
/prisma/populate.js
.memory-snapshots
+57
View File
@@ -0,0 +1,57 @@
import {
API_VERSION_HEADER_NAME,
API_VERSION as CORE_API_VERSION,
} from "@trigger.dev/core/v3/serverOnly";
import { z } from "zod";
export const CURRENT_API_VERSION = CORE_API_VERSION;
export const NON_SPECIFIC_API_VERSION = "none";
export type API_VERSIONS = typeof CURRENT_API_VERSION | typeof NON_SPECIFIC_API_VERSION;
export function getApiVersion(request: Request): API_VERSIONS {
const apiVersion = request.headers.get(API_VERSION_HEADER_NAME);
if (apiVersion === CURRENT_API_VERSION) {
return apiVersion;
}
return NON_SPECIFIC_API_VERSION;
}
// This has been copied from the core package to allow us to use these types in the webapp
export const RunStatusUnspecifiedApiVersion = z.enum([
/// Task is waiting for a version update because it cannot execute without additional information (task, queue, etc.). Replaces WAITING_FOR_DEPLOY
"PENDING_VERSION",
/// Task hasn't been deployed yet but is waiting to be executed
"WAITING_FOR_DEPLOY",
/// Task is waiting to be executed by a worker
"QUEUED",
/// Task is currently being executed by a worker
"EXECUTING",
/// Task has failed and is waiting to be retried
"REATTEMPTING",
/// Task has been paused by the system, and will be resumed by the system
"FROZEN",
/// Task has been completed successfully
"COMPLETED",
/// Task has been canceled by the user
"CANCELED",
/// Task has been completed with errors
"FAILED",
/// Task has crashed and won't be retried, most likely the worker ran out of resources, e.g. memory or storage
"CRASHED",
/// Task was interrupted during execution, mostly this happens in development environments
"INTERRUPTED",
/// Task has failed to complete, due to an error in the system
"SYSTEM_FAILURE",
/// Task has been scheduled to run at a specific time
"DELAYED",
/// Task has expired and won't be executed
"EXPIRED",
/// Task has reached it's maxDuration and has been stopped
"TIMED_OUT",
]);
export type RunStatusUnspecifiedApiVersion = z.infer<typeof RunStatusUnspecifiedApiVersion>;
+20 -42
View File
@@ -1,53 +1,31 @@
export function AISparkleIcon({ className }: { className?: string }) {
return (
<svg className={className} viewBox="0 0 18 18" fill="none" xmlns="http://www.w3.org/2000/svg">
<svg
className={className}
width="24"
height="24"
viewBox="0 0 24 24"
fill="none"
xmlns="http://www.w3.org/2000/svg"
>
<path
d="M14.9806 0.803884C14.8871 0.33646 14.4767 0 14 0C13.5233 0 13.1129 0.33646 13.0194 0.803884L12.7809 1.99644C12.7017 2.3923 12.3923 2.70174 11.9964 2.78091L10.8039 3.01942C10.3365 3.1129 10 3.52332 10 4C10 4.47668 10.3365 4.8871 10.8039 4.98058L11.9964 5.21909C12.3923 5.29826 12.7017 5.6077 12.7809 6.00356L13.0194 7.19612C13.1129 7.66354 13.5233 8 14 8C14.4767 8 14.8871 7.66354 14.9806 7.19612L15.2191 6.00356C15.2983 5.6077 15.6077 5.29826 16.0036 5.21909L17.1961 4.98058C17.6635 4.8871 18 4.47668 18 4C18 3.52332 17.6635 3.1129 17.1961 3.01942L16.0036 2.78091C15.6077 2.70174 15.2983 2.3923 15.2191 1.99644L14.9806 0.803884Z"
fill="url(#paint0_linear_11402_36656)"
fillRule="evenodd"
clipRule="evenodd"
d="M9.14286 4.85718C9.46177 4.85718 9.74205 5.06859 9.82966 5.37523L10.6041 8.08589C10.9431 9.27235 11.8705 10.1998 13.057 10.5388L15.7677 11.3132C16.0743 11.4008 16.2857 11.6811 16.2857 12C16.2857 12.319 16.0743 12.5992 15.7677 12.6868L13.057 13.4613C11.8705 13.8003 10.9431 14.7277 10.6041 15.9142L9.82966 18.6248C9.74205 18.9315 9.46177 19.1429 9.14286 19.1429C8.82394 19.1429 8.54367 18.9315 8.45605 18.6248L7.68158 15.9142C7.34259 14.7277 6.41517 13.8003 5.22871 13.4613L2.51806 12.6868C2.21141 12.5992 2 12.319 2 12C2 11.6811 2.21141 11.4008 2.51806 11.3132L5.22871 10.5388C6.41517 10.1998 7.34259 9.27235 7.68158 8.08589L8.45605 5.37523C8.54367 5.06859 8.82394 4.85718 9.14286 4.85718Z"
fill="#7655FD"
/>
<path
d="M5.94868 4.68377C5.81257 4.27543 5.43043 4 5 4C4.56957 4 4.18743 4.27543 4.05132 4.68377L3.36754 6.73509C3.26801 7.03369 3.03369 7.26801 2.73509 7.36754L0.683772 8.05132C0.27543 8.18743 0 8.56957 0 9C0 9.43043 0.27543 9.81257 0.683772 9.94868L2.73509 10.6325C3.03369 10.732 3.26801 10.9663 3.36754 11.2649L4.05132 13.3162C4.18743 13.7246 4.56957 14 5 14C5.43043 14 5.81257 13.7246 5.94868 13.3162L6.63246 11.2649C6.73199 10.9663 6.96631 10.732 7.26491 10.6325L9.31623 9.94868C9.72457 9.81257 10 9.43043 10 9C10 8.56957 9.72457 8.18743 9.31623 8.05132L7.26491 7.36754C6.96631 7.26801 6.73199 7.03369 6.63246 6.73509L5.94868 4.68377Z"
fill="url(#paint1_linear_11402_36656)"
fillRule="evenodd"
clipRule="evenodd"
d="M17.7143 2C18.0421 2 18.3278 2.22307 18.4072 2.54105L18.6538 3.5272C18.8777 4.42291 19.5771 5.12229 20.4728 5.34622L21.459 5.59276C21.7769 5.67225 22 5.95795 22 6.28571C22 6.61348 21.7769 6.89918 21.459 6.97867L20.4728 7.22521C19.5771 7.44914 18.8777 8.14851 18.6538 9.04423L18.4072 10.0304C18.3278 10.3484 18.0421 10.5714 17.7143 10.5714C17.3865 10.5714 17.1008 10.3484 17.0213 10.0304L16.7748 9.04423C16.5509 8.14852 15.8515 7.44914 14.9558 7.22521L13.9696 6.97867C13.6516 6.89918 13.4286 6.61348 13.4286 6.28571C13.4286 5.95795 13.6516 5.67225 13.9696 5.59276L14.9558 5.34622C15.8515 5.12229 16.5509 4.42291 16.7748 3.5272L17.0213 2.54105C17.1008 2.22307 17.3865 2 17.7143 2Z"
fill="#D946EF"
/>
<path
d="M12.9487 12.6838C12.8126 12.2754 12.4304 12 12 12C11.5696 12 11.1874 12.2754 11.0513 12.6838L10.8675 13.2351C10.768 13.5337 10.5337 13.768 10.2351 13.8675L9.68377 14.0513C9.27543 14.1874 9 14.5696 9 15C9 15.4304 9.27543 15.8126 9.68377 15.9487L10.2351 16.1325C10.5337 16.232 10.768 16.4663 10.8675 16.7649L11.0513 17.3162C11.1874 17.7246 11.5696 18 12 18C12.4304 18 12.8126 17.7246 12.9487 17.3162L13.1325 16.7649C13.232 16.4663 13.4663 16.232 13.7649 16.1325L14.3162 15.9487C14.7246 15.8126 15 15.4304 15 15C15 14.5696 14.7246 14.1874 14.3162 14.0513L13.7649 13.8675C13.4663 13.768 13.232 13.5337 13.1325 13.2351L12.9487 12.6838Z"
fill="url(#paint2_linear_11402_36656)"
fillRule="evenodd"
clipRule="evenodd"
d="M16.2857 14.8572C16.5932 14.8572 16.8661 15.0539 16.9633 15.3456L17.3388 16.472C17.481 16.8986 17.8157 17.2333 18.2423 17.3755L19.3687 17.751C19.6604 17.8482 19.8571 18.1212 19.8571 18.4286C19.8571 18.7361 19.6604 19.009 19.3687 19.1062L18.2423 19.4817C17.8157 19.6239 17.481 19.9586 17.3388 20.3852L16.9633 21.5116C16.8661 21.8033 16.5932 22 16.2857 22C15.9783 22 15.7053 21.8033 15.6081 21.5116L15.2326 20.3852C15.0904 19.9586 14.7557 19.6239 14.3291 19.4817L13.2027 19.1062C12.911 19.009 12.7143 18.7361 12.7143 18.4286C12.7143 18.1212 12.911 17.8482 13.2027 17.751L14.3291 17.3755C14.7557 17.2333 15.0904 16.8986 15.2326 16.472L15.6081 15.3456C15.7053 15.0539 15.9783 14.8572 16.2857 14.8572Z"
fill="#4F46E5"
/>
<defs>
<linearGradient
id="paint0_linear_11402_36656"
x1="9"
y1="0"
x2="9"
y2="18"
gradientUnits="userSpaceOnUse"
>
<stop stopColor="#E543FF" />
<stop offset="1" stopColor="#286399" />
</linearGradient>
<linearGradient
id="paint1_linear_11402_36656"
x1="9"
y1="0"
x2="9"
y2="18"
gradientUnits="userSpaceOnUse"
>
<stop stopColor="#E543FF" />
<stop offset="1" stopColor="#286399" />
</linearGradient>
<linearGradient
id="paint2_linear_11402_36656"
x1="9"
y1="0"
x2="9"
y2="18"
gradientUnits="userSpaceOnUse"
>
<stop stopColor="#E543FF" />
<stop offset="1" stopColor="#286399" />
</linearGradient>
</defs>
</svg>
);
}
@@ -0,0 +1,71 @@
export function AbacusIcon({ className }: { className?: string }) {
return (
<svg className={className} viewBox="0 0 24 24" fill="none" xmlns="http://www.w3.org/2000/svg">
<g clipPath="url(#clip0_16909_120578)">
<path
d="M4 3V21"
stroke="currentColor"
strokeWidth="2"
strokeLinecap="round"
strokeLinejoin="round"
/>
<path
d="M20 21V3"
stroke="currentColor"
strokeWidth="2"
strokeLinecap="round"
strokeLinejoin="round"
/>
<path
d="M8 5L8 6"
stroke="currentColor"
strokeWidth="2"
strokeLinecap="round"
strokeLinejoin="round"
/>
<path
d="M14 5L14 6"
stroke="currentColor"
strokeWidth="2"
strokeLinecap="round"
strokeLinejoin="round"
/>
<path
d="M15 10L15 11"
stroke="currentColor"
strokeWidth="2"
strokeLinecap="round"
strokeLinejoin="round"
/>
<path
d="M9 10L9 11"
stroke="currentColor"
strokeWidth="2"
strokeLinecap="round"
strokeLinejoin="round"
/>
<path
d="M12 15L12 16"
stroke="currentColor"
strokeWidth="2"
strokeLinecap="round"
strokeLinejoin="round"
/>
<path
d="M8 15L8 16"
stroke="currentColor"
strokeWidth="2"
strokeLinecap="round"
strokeLinejoin="round"
/>
<path
d="M3 21H21"
stroke="currentColor"
strokeWidth="2"
strokeLinecap="round"
strokeLinejoin="round"
/>
</g>
</svg>
);
}
@@ -0,0 +1,44 @@
export function ArchiveIcon({ className }: { className?: string }) {
return (
<svg className={className} viewBox="0 0 24 24" fill="none" xmlns="http://www.w3.org/2000/svg">
<g clipPath="url(#clip0_17398_782)">
<path
d="M19.2293 9C20.1562 9.00001 20.8612 9.83278 20.7088 10.7471L19.2781 19.3291C19.1173 20.2933 18.283 21 17.3055 21H6.69416C5.71662 21 4.88235 20.2933 4.7215 19.3291L3.29084 10.7471C3.13848 9.83289 3.8436 9.00021 4.77033 9H19.2293ZM9.95002 12.5C9.12179 12.5002 8.45002 13.1717 8.45002 14C8.45002 14.8283 9.12179 15.4998 9.95002 15.5H13.95L14.1033 15.4922C14.8597 15.4154 15.45 14.7767 15.45 14C15.45 13.2233 14.8597 12.5846 14.1033 12.5078L13.95 12.5H9.95002Z"
fill="currentColor"
/>
<rect x="2" y="3" width="20" height="4" rx="1" fill="currentColor" />
</g>
<defs>
<clipPath id="clip0_17398_782">
<rect width="24" height="24" fill="currentColor" />
</clipPath>
</defs>
</svg>
);
}
export function UnarchiveIcon({ className }: { className?: string }) {
return (
<svg className={className} viewBox="0 0 24 24" fill="none" xmlns="http://www.w3.org/2000/svg">
<g clipPath="url(#clip0_17398_66731)">
<path
d="M19.2027 10C20.1385 10 20.8456 10.8478 20.6782 11.7686L19.2984 19.3574C19.1254 20.3084 18.2972 21 17.3306 21H6.66945C5.70287 21 4.87458 20.3084 4.70167 19.3574L3.32179 11.7686C3.15438 10.8478 3.86152 10 4.79738 10H10.9995V16C10.9995 16.5521 11.4475 16.9997 11.9995 17C12.5518 17 12.9995 16.5523 12.9995 16V10H19.2027Z"
fill="currentColor"
/>
<rect x="11" y="4" width="2" height="6" fill="currentColor" />
<path
d="M15.5 6.5L12 3L8.5 6.5"
stroke="currentColor"
strokeWidth="2"
strokeLinecap="round"
strokeLinejoin="round"
/>
</g>
<defs>
<clipPath id="clip0_17398_66731">
<rect width="24" height="24" fill="currentColor" />
</clipPath>
</defs>
</svg>
);
}
@@ -0,0 +1,22 @@
export function ArrowTopRightBottomLeftIcon({ className }: { className?: string }) {
return (
<svg className={className} viewBox="0 0 24 24" fill="none" xmlns="http://www.w3.org/2000/svg">
<path
d="M14.8258 10.5L20.125 5.20083V8.5625C20.125 9.08027 20.5447 9.5 21.0625 9.5C21.5803 9.5 22 9.08027 22 8.5625V2.9375C22 2.41973 21.5803 2 21.0625 2H15.4375C14.9197 2 14.5 2.41973 14.5 2.9375C14.5 3.45527 14.9197 3.875 15.4375 3.875H18.7992L13.5 9.17417C13.1339 9.54029 13.1339 10.1339 13.5 10.5C13.8661 10.8661 14.4597 10.8661 14.8258 10.5Z"
fill="currentColor"
/>
<path
d="M2 21.0625V15.4375C2 14.9197 2.41973 14.5 2.9375 14.5C3.45527 14.5 3.875 14.9197 3.875 15.4375V18.7992L9.17417 13.5C9.54029 13.1339 10.1339 13.1339 10.5 13.5C10.8661 13.8661 10.8661 14.4597 10.5 14.8258L5.20083 20.125H8.5625C9.08027 20.125 9.5 20.5447 9.5 21.0625C9.5 21.5803 9.08027 22 8.5625 22H2.9375C2.69757 22 2.45765 21.9085 2.27459 21.7254C2.1847 21.6355 2.11689 21.5319 2.07114 21.4214C2.0253 21.3108 2 21.1896 2 21.0625Z"
fill="currentColor"
/>
<path
d="M14.8258 10.5L20.125 5.20083V10C20.125 10.5178 20.5447 10.9375 21.0625 10.9375C21.5803 10.9375 22 10.5178 22 10V2.9375C22 2.41973 21.5803 2 21.0625 2H14C13.4822 2 13.0625 2.41973 13.0625 2.9375C13.0625 3.45527 13.4822 3.875 14 3.875H18.7992L13.5 9.17417C13.1339 9.54029 13.1339 10.1339 13.5 10.5C13.8661 10.8661 14.4597 10.8661 14.8258 10.5Z"
fill="currentColor"
/>
<path
d="M2 21.0625V13.9375C2 13.4197 2.41973 13 2.9375 13C3.45527 13 3.875 13.4197 3.875 13.9375V18.7992L9.17417 13.5C9.54029 13.1339 10.1339 13.1339 10.5 13.5C10.8661 13.8661 10.8661 14.4597 10.5 14.8258L5.20083 20.125H10.0625C10.5803 20.125 11 20.5447 11 21.0625C11 21.5803 10.5803 22 10.0625 22H2.9375C2.69757 22 2.45765 21.9085 2.27459 21.7254C2.1847 21.6355 2.11689 21.5319 2.07114 21.4214C2.0253 21.3108 2 21.1896 2 21.0625Z"
fill="currentColor"
/>
</svg>
);
}
@@ -0,0 +1,94 @@
export function BunLogoIcon({ className }: { className?: string }) {
return (
<svg className={className} viewBox="0 0 80 70" xmlns="http://www.w3.org/2000/svg">
<path
id="Shadow"
d="M71.09,20.74c-.16-.17-.33-.34-.5-.5s-.33-.34-.5-.5-.33-.34-.5-.5-.33-.34-.5-.5-.33-.34-.5-.5-.33-.34-.5-.5-.33-.34-.5-.5A26.46,26.46,0,0,1,75.5,35.7c0,16.57-16.82,30.05-37.5,30.05-11.58,0-21.94-4.23-28.83-10.86l.5.5.5.5.5.5.5.5.5.5.5.5.5.5C19.55,65.3,30.14,69.75,42,69.75c20.68,0,37.5-13.48,37.5-30C79.5,32.69,76.46,26,71.09,20.74Z"
/>
<g id="Body">
<path
id="Background"
d="M73,35.7c0,15.21-15.67,27.54-35,27.54S3,50.91,3,35.7C3,26.27,9,17.94,18.22,13S33.18,3,38,3s8.94,4.13,19.78,10C67,17.94,73,26.27,73,35.7Z"
style={{ fill: "#fbf0df" }}
/>
<path
id="Bottom_Shadow"
data-name="Bottom Shadow"
d="M73,35.7a21.67,21.67,0,0,0-.8-5.78c-2.73,33.3-43.35,34.9-59.32,24.94A40,40,0,0,0,38,63.24C57.3,63.24,73,50.89,73,35.7Z"
style={{ fill: "#f6dece" }}
/>
<path
id="Light_Shine"
data-name="Light Shine"
d="M24.53,11.17C29,8.49,34.94,3.46,40.78,3.45A9.29,9.29,0,0,0,38,3c-2.42,0-5,1.25-8.25,3.13-1.13.66-2.3,1.39-3.54,2.15-2.33,1.44-5,3.07-8,4.7C8.69,18.13,3,26.62,3,35.7c0,.4,0,.8,0,1.19C9.06,15.48,20.07,13.85,24.53,11.17Z"
style={{ fill: "#fffefc" }}
/>
<path
id="Top"
d="M35.12,5.53A16.41,16.41,0,0,1,29.49,18c-.28.25-.06.73.3.59,3.37-1.31,7.92-5.23,6-13.14C35.71,5,35.12,5.12,35.12,5.53Zm2.27,0A16.24,16.24,0,0,1,39,19c-.12.35.31.65.55.36C41.74,16.56,43.65,11,37.93,5,37.64,4.74,37.19,5.14,37.39,5.49Zm2.76-.17A16.42,16.42,0,0,1,47,17.12a.33.33,0,0,0,.65.11c.92-3.49.4-9.44-7.17-12.53C40.08,4.54,39.82,5.08,40.15,5.32ZM21.69,15.76a16.94,16.94,0,0,0,10.47-9c.18-.36.75-.22.66.18-1.73,8-7.52,9.67-11.12,9.45C21.32,16.4,21.33,15.87,21.69,15.76Z"
style={{ fill: "#ccbea7", fillRule: "evenodd" }}
/>
<path
id="Outline"
d="M38,65.75C17.32,65.75.5,52.27.5,35.7c0-10,6.18-19.33,16.53-24.92,3-1.6,5.57-3.21,7.86-4.62,1.26-.78,2.45-1.51,3.6-2.19C32,1.89,35,.5,38,.5s5.62,1.2,8.9,3.14c1,.57,2,1.19,3.07,1.87,2.49,1.54,5.3,3.28,9,5.27C69.32,16.37,75.5,25.69,75.5,35.7,75.5,52.27,58.68,65.75,38,65.75ZM38,3c-2.42,0-5,1.25-8.25,3.13-1.13.66-2.3,1.39-3.54,2.15-2.33,1.44-5,3.07-8,4.7C8.69,18.13,3,26.62,3,35.7,3,50.89,18.7,63.25,38,63.25S73,50.89,73,35.7C73,26.62,67.31,18.13,57.78,13,54,11,51.05,9.12,48.66,7.64c-1.09-.67-2.09-1.29-3-1.84C42.63,4,40.42,3,38,3Z"
/>
</g>
<g id="Mouth">
<g id="Background-2" data-name="Background">
<path
d="M45.05,43a8.93,8.93,0,0,1-2.92,4.71,6.81,6.81,0,0,1-4,1.88A6.84,6.84,0,0,1,34,47.71,8.93,8.93,0,0,1,31.12,43a.72.72,0,0,1,.8-.81H44.26A.72.72,0,0,1,45.05,43Z"
style={{ fill: "#b71422" }}
/>
</g>
<g id="Tongue">
<path
id="Background-3"
data-name="Background"
d="M34,47.79a6.91,6.91,0,0,0,4.12,1.9,6.91,6.91,0,0,0,4.11-1.9,10.63,10.63,0,0,0,1-1.07,6.83,6.83,0,0,0-4.9-2.31,6.15,6.15,0,0,0-5,2.78C33.56,47.4,33.76,47.6,34,47.79Z"
style={{ fill: "#ff6164" }}
/>
<path
id="Outline-2"
data-name="Outline"
d="M34.16,47a5.36,5.36,0,0,1,4.19-2.08,6,6,0,0,1,4,1.69c.23-.25.45-.51.66-.77a7,7,0,0,0-4.71-1.93,6.36,6.36,0,0,0-4.89,2.36A9.53,9.53,0,0,0,34.16,47Z"
/>
</g>
<path
id="Outline-3"
data-name="Outline"
d="M38.09,50.19a7.42,7.42,0,0,1-4.45-2,9.52,9.52,0,0,1-3.11-5.05,1.2,1.2,0,0,1,.26-1,1.41,1.41,0,0,1,1.13-.51H44.26a1.44,1.44,0,0,1,1.13.51,1.19,1.19,0,0,1,.25,1h0a9.52,9.52,0,0,1-3.11,5.05A7.42,7.42,0,0,1,38.09,50.19Zm-6.17-7.4c-.16,0-.2.07-.21.09a8.29,8.29,0,0,0,2.73,4.37A6.23,6.23,0,0,0,38.09,49a6.28,6.28,0,0,0,3.65-1.73,8.3,8.3,0,0,0,2.72-4.37.21.21,0,0,0-.2-.09Z"
/>
</g>
<g id="Face">
<ellipse
id="Right_Blush"
data-name="Right Blush"
cx="53.22"
cy="40.18"
rx="5.85"
ry="3.44"
style={{ fill: "#febbd0" }}
/>
<ellipse
id="Left_Bluch"
data-name="Left Bluch"
cx="22.95"
cy="40.18"
rx="5.85"
ry="3.44"
style={{ fill: "#febbd0" }}
/>
<path
id="Eyes"
d="M25.7,38.8a5.51,5.51,0,1,0-5.5-5.51A5.51,5.51,0,0,0,25.7,38.8Zm24.77,0A5.51,5.51,0,1,0,45,33.29,5.5,5.5,0,0,0,50.47,38.8Z"
style={{ fillRule: "evenodd" }}
/>
<path
id="Iris"
d="M24,33.64a2.07,2.07,0,1,0-2.06-2.07A2.07,2.07,0,0,0,24,33.64Zm24.77,0a2.07,2.07,0,1,0-2.06-2.07A2.07,2.07,0,0,0,48.75,33.64Z"
style={{ fill: "#fff", fillRule: "evenodd" }}
/>
</g>
</svg>
);
}
@@ -0,0 +1,13 @@
export function ChevronExtraSmallDown({ className }: { className?: string }) {
return (
<svg className={className} viewBox="0 0 18 18" fill="none" xmlns="http://www.w3.org/2000/svg">
<path
d="M15 6L9.75926 12.1142C9.36016 12.5798 8.63984 12.5798 8.24074 12.1142L3 6"
stroke="currentColor"
strokeWidth="2"
strokeMiterlimit="1.00244"
strokeLinecap="round"
/>
</svg>
);
}
@@ -0,0 +1,13 @@
export function ChevronExtraSmallUp({ className }: { className?: string }) {
return (
<svg className={className} viewBox="0 0 18 18" fill="none" xmlns="http://www.w3.org/2000/svg">
<path
d="M3 12L8.24074 5.8858C8.63984 5.42019 9.36016 5.42019 9.75926 5.8858L15 12"
stroke="currentColor"
strokeWidth="2"
strokeMiterlimit="1.00244"
strokeLinecap="round"
/>
</svg>
);
}

Some files were not shown because too many files have changed in this diff Show More